Home HealthYour AI Vendor’s Medical Exam Score Doesn’t Mean What You Think It Does

Your AI Vendor’s Medical Exam Score Doesn’t Mean What You Think It Does

by Staff Reporter
0 comments

Pitch decks for healthcare AI often lead with a familiar credential: the model passed a medical licensing exam. It sounds reassuring, but it tells buyers very little about whether the model can safely do real clinical work.

If health systems want to adopt AI at scale, they need evaluation that is open, reproducible, and continuously updated. That’s also much harder to build in healthcare than in general-purpose AI, for three reasons: First, patient data is locked down, secondly, public benchmarks go stale, and finally, AI agents are expanding what needs to be tested.

Why transparency matters and where it breaks down

A benchmark score is only as trustworthy as the method behind it. MedHELM, a clinician-validated framework, is a strong example. It covers 121 clinical tasks across five categories using a public methodology.

It’s also a good example of the core problem. Of the 37 datasets behind MedHELM, 14 are private and held inside the institutions that created them, and seven more are gated. Outside parties can only partly verify the published rankings, because full replication requires restricted patient records.

Even the best-case dataset has limits. MIMIC-IV underpins much of academic healthcare AI research, but access requires individual credentialing, ethics training, and a data use agreement that bars redistribution and sending the data to third-party commercial APIs. It also covers only 2008-2019, comes from one Boston academic medical center, and focuses on emergency and intensive care. Outpatient settings, community hospitals, and regional diversity are largely missing.

These restrictions exist to protect patient confidentiality under HIPAA, as they should. But they also keep clinical benchmarks from becoming turnkey, open-source test suites that companies can run internally before deploying a model. The main workaround is synthetic data, validated to match real records in difficulty, score distribution, and model rankings. And building and validating it takes sustained investment.

The contamination clock

Opening a benchmark creates a different problem. Frontier models are trained on huge amounts of public internet data, so once a test set is published, it can end up in training data. Newer models may then score higher because they have seen the questions, not because they reason better.

A study in Nature evaluating BiasMedQA – a benchmark of 1,273 clinical vignettes designed to test how leading prompts sway a model’s medical judgment, illustrates the gap. A frontier model scored 80% to 89% on publicly available prompts. On unpublished prompts built with the same logic, accuracy fell as low as 47%.

That leaves benchmark builders with a trade-off. Private test sets resist contamination but can’t be independently verified. Open test sets stay transparent but must be refreshed and funded indefinitely.

From answers to actions

As medical AI shifts from chatbots to agents that review charts, query electronic health records (EHRs), place orders and can work through payer portals, evaluation has to cover whole sequences of decisions, not just single answers. Did the agent form a sound clinical plan? Did it choose and run the right tools? Did it recover from errors and edge cases? Did it finish the task safely and legally?

New benchmarks are starting to address this. PhysicianBench, a Stanford-led benchmark published in May 2026, places agents in an EHR environment with real patient records and grades 100 long-horizon tasks adapted from real consultations between primary care physicians and specialists across 21 specialties. The best-performing model succeeded on only 46% of tasks on its first attempt, and open-source models reached at most 19%. 

HealthAdminBench, another from Stanford researchers, targets back-office work, which accounts for over $1 trillion in annual spending. It simulates an EHR, two payer portals and a fax system across 135 tasks, including prior authorization and appeals and denials management. Agents performed well on individual steps, with the top subtask success rate at 82.8%, but the best agent completed only 36.3% of tasks end to end. That gap shows reliability across a full workflow remains the hard part.

Interactive environments are much harder for a model to memorize, which gives agentic benchmarks some natural protection against contamination, but they’re far more costly to build and host. Additionally, evaluation shouldn’t be a one-time procurement checkbox. Buyers should be asking their vendors about: 

  • Task relevance: What data and scenarios make up the benchmark, and do they match our actual clinical and administrative workflows?
  • Freshness: When was the model last refreshed, which versions were tested, and how do you prevent data leakage?
  • Reproducibility: Can we, or an independent third party, run it and verify the results in a controlled setting?
  • Behavioral testing: Does it measure static recall or does it test the model as an agent completing end-to-end tasks in an interactive environment?

Ultimately, no single company or university can solve the privacy, contamination, and agent-testing problems alone. A trustworthy evaluation infrastructure will take health systems, research institutions, and vendors funding, building, and maintaining open, dynamic benchmarks together.

Photo: J Studios, Getty Images


David Talby, PhD, MBA, is the CTO of John Snow Labs. He has spent his career making AI, big data, and Data Science solve real-world problems in healthcare, life science, and related fields.

This post appears through the MedCity Influencers program. Anyone can publish their perspective on business and innovation in healthcare on MedCity News through MedCity Influencers. Click here to find out how.

You may also like

Leave a Comment

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More