AI in Healthcare | Building Patient Trust & Understanding

Session Focus

Coburn introduced the session by noting that while most AI discussion concerns power and impact, how the technology interfaces with individual patients is overridingly important, and that the panel deliberately mixed national policymakers, practicing clinicians and technologists.

Bohen framed the difference from other AI sessions: this one puts the patient at the center of the AI journey. He cited Mass General Brigham market research, consistent with findings elsewhere, showing patients become comfortable with AI in their care if they understand how it is being used and if a clinician is making the decisions.

How Patients Are Already Using AI

Pete Clardy MD described the view from Google, whose health team works across Search, YouTube and Gemini. Google Search handles roughly a billion health-intended searches a day.

Two trends stand out as people move from search toward generative tools. The questions are getting longer and more contextually complex, and the proportion of questions that are health or wellness focused is growing. He also sees people merging wellness data from wearables and geospatial sources with traditional clinical information.

His most striking observation concerned why patients share intimate details through openly available and not entirely safe mechanisms. They are getting value in the form of leveling historical information asymmetry. Quoting patient safety advocate Sue Sheridan, he said innovation for those people moves at the speed of necessity. Innovation moves at the speed of need, not at the speed of trust.

Allison Chang MD, PhD described what this looks like in clinic. It has become routine for a patient to hand her their phone showing a ChatGPT output listing questions the model said they should ask. The previous day she sat with a young man and went through every one.

Her position on public models was pragmatic. Most of the time the output is quite good, though models often miss clinical context and softer points. If a patient is trying to interpret a complex radiology read appearing in their patient gateway, she would rather they get some information from a sophisticated language model than sit with the anxiety or look somewhere less reliable.

What Clinicians Need Before They Trust a Model

Chang argued the burden of proof must be commensurate with the stakes of the clinical decision.

At Mass General Brigham clinicians can already use an AI scribe. Many colleagues restrict it to summarizing symptom discussion and are not ready to outsource the assessment and plan synthesis that contains the clinical reasoning.

For higher-stakes decisions she wants clinical trials. Her example was a lung nodule that could be early-stage cancer or entirely benign, where the choice is surgery, PET scan or invasive biopsy. Knowing there is an 18 percent risk of cancer is not the decision. The decision incorporates that risk alongside age, comorbidities, procedural risk and a shared conversation with the patient.

That conversation requires answers: what was the model trained on, was it validated in patients who look like you in a setting that looks like ours, and has it been monitored since deployment.

She reported a common view among computer scientists she works with, that once a model reaches an AUC of 0.9, nothing should stop its use with patients. Her response was that everyone’s heart is in the same place, but the oath taken is to do no harm, and an AUC does not tell you outcomes improved.

She also made a point about clinician psychology. The burden will be on clinicians not to get scared off by the term artificial intelligence, recognizing that many of these tools are simply other models and other inputs, much like the inputs already used. Whether patients react differently when the phrase is used warrants research.

Trust as a System, Not a Stamp

Eric Rosenthal MD gave the session’s most quotable formulation. For a health system, trust is not a stamp you put on the model, like SPF 55. It is a system you put around the model.

He laid out three sequential questions. Does the model work. Does it work here, meaning both this place and this moment in time. And does it still work.

His reasoning on the second is that it is effectively a rule that models do not transfer, because environments differ. Patients are on different medications, and a cancer regimen from three years ago may have changed.

He described building holdout data sets deliberately, not only an unseen fraction of data but unseen settings and a more recent time period, testing whether a model designed through 2023 still works in 2026.

On responsibility he used an autonomous vehicle analogy. Someone designed the algorithm, someone put it in a car, someone sold the system, someone got in. That is shared responsibility requiring shared governance. His recommendation was moving away from the checklist manifesto toward iterative, real-time dynamic telemetry, closer to pharmacosurveillance than to a committee sign-off.

Asked whether Mass General Brigham should prefer internally built models, he was not so sure. He works in an intensive care unit, and returning from an academic meeting he might change sedation practice. A prognostic model for coma recovery relying on signal related to sedation A stops working when the unit moves to sedation B. If clinicians are true to being agents of change, they will not be themselves soon enough.

The Data Bottleneck

Chang identified data as the true bottleneck. The perception is an embarrassment of riches, with every physician generating vast amounts of data per patient each year.

The reality is that most of it is not in a form usable for training and validation. The electronic health record is fundamentally optimized for billing, and billing codes, which are among the easiest data to extract rapidly, have been repeatedly shown not to correlate well with the clinical scenario.

Rosenthal extended the point about where validation should happen. Validating against a cleaned, filtered dataset with no missingness, then deploying into an environment with dirty data, spelling errors, incomplete documentation and time-warped waveforms, amounts to testing a model in the wild after training it in the zoo.

Clardy described two responses from the research side. Google and DeepMind platform open-weights models, distillates from frontier models like Gemini that are smaller and worse at coding or speaking French but enriched for medical language and image understanding, published with model cards explaining provenance so developer communities can adapt them locally.

Rosenthal added the complementary need to share open data, describing national infrastructure for secured clean room environments called trusted research environments, built with about 14 hospitals around the country in partnership with Mass General Brigham Digital. Data is available in a de-identified, safe form that cannot be taken home but can be used to evaluate a model, which he framed as inherently a fairness question: a researcher in California can ask whether their local data looks like the data in Boston.

The Federal View

Ken Callahan said HHS has heard the same questions and themes, having run a request for information on integrating AI into clinical care. Liability came up repeatedly.

He was explicit that the federal government has no interest in getting involved in actual clinical care or the relationship between patient and physician. What government can do is increase innovation by setting national standards, with the threshold of evidence scaling to the level of risk.

He named specific mechanisms already moving: the CMMI ACCESS model paying for outcomes and for AI in ways government has not previously done, and Rural Health Transformation Fund money going out to states for AI use.

On the tension between standards and pace, he pointed to administrative cost as an area FDA would not typically regulate but where promise is substantial, particularly given workforce shortages. He did not think the field is at, or will ever reach, a point of wanting full autonomy without the physician. HHS is leading a workgroup with CMS, FDA, ARPA-H and NIH working through these questions toward a national framework.

Five Years Out

Clardy pointed to TEFCA and related federal work unblocking information at the person level, and hoped people would come to control more of their own health information in secure, shareable ways, which he called an alignment question rather than a technology question.

He described an evolution toward triadic medicine, where patients use AI alongside human providers and providers use AI alongside clinical training. Looking further out, he noted that current understanding rests on language models adapted to other modalities, and that medical records are a poor proxy, comparing it to figuring out planes by looking at the exhaust. World models that understand what care actually looks like in an ICU or operating room are the next step.

Chang said five years is short in clinical terms, but expects within her career to have AI tools functioning almost as colleagues. What clinicians love about medicine is the conversation about difficult decisions and contradictory data, the text threads and tumor boards where they ask each other what they are missing. Her hope is handheld virtual colleagues that challenge preconceptions and surface patterns invisible to humans.

Her example was Sibyl, a model that takes an entire chest CT volume and predicts six-year lung cancer risk with high accuracy, without any visible abnormality or nodule present, which she described as transcending human perception.

She closed on the black box concern with a corrective. AI is not injecting uncertainty into medicine. Medicine is fundamentally uncertain, requiring constant decisions on imperfect information, and most clinicians are comfortable with uncertainty. The task is training doctors to weight these inputs appropriately.

Key Takeaways

1. Patients adopt AI at the speed of need, not the speed of trust, which explains willingness to share intimate details with unsafe tools.

2. Burden of proof should scale with clinical stakes. Scribing gets adopted; biopsy decisions need trials.

3. An AUC of 0.9 is not evidence of improved outcomes, which is where clinicians and computer scientists diverge.

4. Trust is a system around the model, not a certification of it, requiring surveillance rather than checklists.

5. Models rarely transfer, and locally built models expire too as clinical practice changes.

6. The EHR is optimized for billing, which is why healthcare’s apparent data abundance does not translate into training data.

7. Validation must match deployment conditions. Training in the zoo and testing in the wild is the recurring failure.