How we test the AI models that review your charts

New AI models launch every few weeks, and every launch comes with impressive-sounding claims. None of those claims tell you whether a model is safe to put in front of a clinician finishing a chart. So before any model powers Cortex Lens, it runs the same benchmark: de-identified real primary-care charts, carrying the original human-made coding and documentation mistakes, scored against what a careful reviewer should have flagged.

Quick Answer

1. Every candidate model runs the full corpus of de-identified real charts — never demos, never cherry-picked examples.
2. We score precision first. A wrong flag interrupts a clinician and burns trust; a missed issue in an optional review tool does not.
3. Every published number carries a confidence interval. Small corpus, honest error bars.
4. Results are public at cortexcharts.com/benchmarks, including cost and speed — the numbers vendors usually leave out.

Why precision comes first

A chart-review overlay earns attention it doesn't own. Every flag pulls the clinician's eyes away from the patient chart they are trying to close. If the flag is right, that interruption paid for itself. If it is wrong, the tool just taught the clinician to ignore it.

That is why our headline metrics weight false positives as worse than false negatives. Alongside overall accuracy, we publish a trust-weighted score in which precision counts double. A model that flags less but is right more will beat a chattier model on that score — deliberately.

What this week's sweep found

Anthropic released Claude Sonnet 5 this week, and it went through the same benchmark as every other candidate — running the exact configuration Cortex uses in production. Two results stood out:

• When it flagged something, it was right far more often than the model reviewing charts today — its precision came in well above any previous model we have run through this pipeline.
• It also scored higher on overall accuracy, so the extra caution did not come at the cost of missing what matters.

It still is not in production. A promotion is a separate decision from a benchmark win: the candidate has to hold up across repeated runs, clear a precision floor, and make sense on speed and cost. That validation is underway. The full leaderboard, methodology, and caveats are on the benchmarks page.

Why publish any of this

Most clinical AI vendors ask you to trust a demo. We would rather show the eval: real charts, a named independent judge model, published error bars, and the weakest charts listed right next to the averages. If the numbers move — up or down — the page moves with them.