Benchmark Performance ≠ Bedside Performance
An AI model diagnoses with 87% accuracy, and it can still be dangerous in the clinic. Why study numbers don’t tell us what we want to hear, and what counts instead.
87 percent. That is the diagnostic accuracy of an agentic AI system that colleagues from our institute published in Nature in late 2025. The human comparison group reached 78 percent. The agent beats the physician. On paper.
I consider this number important. And I consider it dangerous if you read it the wrong way.
The case I can’t shake
Picture the following case vignette: a patient, long-haul flight, then immobilization, then chest pain and shortness of breath. Any second-year medical student would name it immediately: pulmonary embolism, PE, an emergency.
An early ChatGPT system diagnosed it as jet lag. Recommendation: lie down, rest, see a doctor at some point.
This is not an edge case. It comes from a publication by our research group, published in early 2025. We had built case vignettes with classic emergency presentations and watched what the models did with them. The results were alarming: precisely with unambiguous emergencies (PE, sepsis, stroke), the models undertriaged dramatically. While they were overcautious on routine questions (overtriage), they systematically underestimated the critical cases.
Shortly afterward, Nature Medicine systematically confirmed the pattern: routine cases → overtriage. Genuine emergencies → undertriage. That is exactly the wrong behavior.
The problem with the benchmark
The 87 percent diagnostic accuracy comes from a GlossaryBenchmarkstandardized test for comparing models (e.g., SWE-bench, MMLU).Go to entry ↗ study. Benchmarks are controlled environments: cleanly structured case vignettes, complete symptom descriptions, clear questions. A patient in an outpatient clinic is the opposite of that.
The first problem: case vignettes are too good. No patient arrives with the complete symptom constellation a textbook describes. Pain is diffuse, the history has gaps, context is missing. The benchmarks don’t test what reality demands.
The second problem: multiple choice is not a proxy for a physician. Many benchmarks are based on tests like the USMLE, the American medical licensing exam. At some point the models became very good at them. The question is: is performance on multiple-choice questions a valid predictor of clinical judgment? I would say no. Recalling knowledge is not the same as delivering care.
The third problem is the technically trickiest: as soon as a benchmark is public, it can become training data. The model learns the right answers, not the medical logic behind them. The benchmark becomes worthless once it has circulated widely enough. Vendor-run benchmarks should therefore always be treated with caution: they are optimized for the vendors’ own products, not for independent comparability.
Bias on two levels
Poor benchmark performance is only half of the problem. The other half is GlossaryBiassystematic skew in data or outputs.Go to entry ↗, which is more complex because it operates on two levels.
Machine-side bias arises where training data doesn’t reflect reality. One well-known example: image datasets for dermatological diagnosis come disproportionately from American and East Asian clinics. The African continent is nearly absent. That doesn’t mean the model gets worse overall. It means it is systematically less reliable for certain populations, without that showing up in the aggregate accuracy figures. Aggregation bias: overall performance holds up, subgroup performance diverges considerably.
We saw this in our data. For underrepresented population groups, there is markedly more undertriage. Precisely where the systems were trained least, they make the most dangerous wrong decisions.
Human-side bias is what clinical psychology has been dealing with for decades: automation bias. People tend to trust a machine’s statements more than their own judgment, even when the machine is wrong. And this effect is amplified, not weakened, when the system explains its decision.
This is the so-called white-box paradox: GlossaryExplainabilitymaking it understandable why the AI decided as it did.Go to entry ↗ features (visual explanations of why the model arrives at a diagnosis) are actually meant to enable critical scrutiny. In practice, they increase trust, regardless of whether the explanation is correct. Transparency that produces a feeling of safety without delivering safety is not a gain.
What we need: a different evaluation cascade
Static benchmarks are not an end, but a beginning. What has to come next:
Retrospective validation: real patient data, historical, to check whether the model works in the actual patient population of the intended clinic.
Virtual EHR environments and dynamic simulations: in our working group, we described the concept of the digital twin for medical AI GlossaryEval (Evaluation)measuring how well a model solves a task.Go to entry ↗ in Nature Medicine in early 2025. The technology isn’t finished yet, but the direction is clear. Models must be tested in simulated clinical environments before they are used on real patients.
Prospective real-world studies: the most important, and the hardest to replace. How do the systems actually perform once they are embedded in the workflow? With real handoffs, real documentation gaps, real time pressure?
Post-implementation monitoring: with accompanying economic evaluation. An AI that is clinically valid but leads to more unnecessary referrals or doesn’t improve overall outcomes does not fulfill its purpose.
What models can do must be visible where they can’t
Another construction site we are working on right now: performance has to be communicated more granularly. A model that is well trained for internal medicine can be nearly useless for rare diseases. Aggregate accuracy figures don’t show that.
The Broad Model Card concept, which we are submitting right now, extends the familiar model card format (also recommended by the FDA) to systems that can theoretically cover thousands of diagnoses. Instead of a single overall accuracy, performance is presented across the entire disease atlas: ICD codes as a grid, granular across disease groups.
87 percent. And then?
A model with 87 percent benchmark accuracy is a promising candidate. Nothing more. Between that and bedside performance lies a road that includes retrospective validation, prospective studies, subgroup analysis, and post-market monitoring.
The numbers that appear in press reports rarely come from this cascade. They come from the first step. Knowing that changes how you read them.
AI has enormous potential in clinical care. But that potential isn’t realized by better models alone. It takes better evaluation, better communication of limits, and people who know when not to believe the machine.
Oscar is a postdoctoral researcher at the Else Kröner Fresenius Center for Digital Health (TU Dresden) and a co-founder of Denkmal.