Cortex Engine Benchmarks
These benchmarks score Cortex against de-identified primary-care charts with real coding and documentation gaps. The useful question is not which model sounds smartest. It is which engine catches more clinically relevant issues without becoming slow, expensive, or noisy.
Claude Sonnet 5 — now running in Cortex — is right about 56% of what it flags, versus 37% for the previous model on the same review pipeline, at matching overall accuracy (46% vs 45%). Fewer wrong flags means fewer interruptions you learn to ignore — that is the trade we optimize for, and why it was promoted on 2026-07-01.
Every candidate — including this one — runs the same 10 de-identified real charts, is scored by an independent judge model, and repeats the run three times to check that it behaves consistently. It gets promoted into Cortex only after it beats the current model here, especially on avoiding false alarms.
10 charts is a small sample, so every score is published with its range. When two models' ranges overlap, we call it a tie instead of declaring a winner. Speed and cost are published too (1.1s per chart at the fast end, 0.11¢ at the cheapest) — the numbers vendors usually leave out.
GPT-OSS 120B (Cerebras)
OpenAI GPT-5.4 Nano
Validation runs · Claude Sonnet 5
All evaluated systems
Default sort is overall accuracy descending: the smartest row first. Click any column title to sort.
claude-sonnet-5
claude-opus-4-7
gpt-5.5
claude-sonnet-5
claude-sonnet-4-6
gemini-3.1-pro-preview
gpt-5.4
claude-haiku-4-5-20251001
gemini-3.5-flash
gpt-5.4-nano
gpt-oss-120b
Validation runs · Claude Sonnet 5 claude-sonnet-5scoped gold | Candidate run | 10 | 51% 32–70 | 59% | 63% | 66% | 2.5¢ | 14.5s | The pre-promotion validation runs of today's model |
Claude Opus 4.7 claude-opus-4-7 | Comparison | 10 | 48% 36–61 | 51% | 52% | 55% | 12.7¢ | 15.7s | Tested for comparison |
OpenAI GPT-5.5 gpt-5.5 | Comparison | 10 | 48% 30–65 | 58% | 42% | 91% | 5.6¢ | 27.2s | Accurate but slow and pricey |
Cortex today · Claude Sonnet 5 claude-sonnet-5scoped gold | In Cortex today | 10 | 46% 26–66 | 53% | 56% | 56% | 2.5¢ | 15.5s | The model reviewing charts now (promoted 2026-07-01) |
Previous model · Claude Sonnet 4.6 claude-sonnet-4-6scoped gold | Previous model | 10 | 45% 29–61 | 40% | 76% | 37% | 1.9¢ | 17.4s | Reviewed charts until 2026-07-01 |
Google Gemini 3.1 Pro gemini-3.1-pro-preview | Comparison | 10 | 44% 32–59 | 57% | 40% | 83% | 3.7¢ | 21.9s | Tested for comparison |
OpenAI GPT-5.4 gpt-5.4 | Comparison | 10 | 35% 20–53 | 40% | 34% | 46% | 1.4¢ | 9.4s | Tested for comparison |
Claude Haiku 4.5 claude-haiku-4-5-20251001 | Comparison | 10 | 35% 20–49 | 35% | 43% | 36% | 0.76¢ | 8.3s | Tested for comparison |
Google Gemini 3.5 Flash gemini-3.5-flash | Comparison | 10 | 33% 16–52 | 44% | 30% | 75% | 1.4¢ | 6.1s | Tested for comparison |
OpenAI GPT-5.4 Nano gpt-5.4-nano | Comparison | 10 | 29% 13–48 | 34% | 24% | 41% | 0.11¢ | 5.9s | Cheapest tested |
GPT-OSS 120B (Cerebras) gpt-oss-120b | Comparison | 10 | 21% 10–32 | 28% | 20% | 55% | 0.16¢ | 1.1s | Fastest tested, misses more |
Lowest overall-accuracy rows stay visible so a high mean cannot hide weak chart types.
| Chart | Gold | Flagged | Recall | Overall |
|---|---|---|---|---|
| chart-002 | 1 | 2 | 0% | 0% |
| chart-002 | 1 | 2 | 0% | 0% |
| chart-003 | 0 | 1 | 100% | 0% |
| chart-003 | 0 | 0 | 100% | 0% |
| chart-002 | 1 | 1 | 0% | 0% |
| chart-003 | 0 | 1 | 100% | 0% |
Lowest overall-accuracy rows stay visible so a high mean cannot hide weak chart types.
| Chart | Gold | Flagged | Recall | Overall |
|---|---|---|---|---|
| chart-002 | 1 | 2 | 0% | 0% |
| chart-002 | 1 | 2 | 0% | 0% |
| chart-003 | 0 | 1 | 100% | 0% |
| chart-003 | 0 | 0 | 100% | 0% |
| chart-002 | 1 | 1 | 0% | 0% |
| chart-003 | 0 | 1 | 100% | 0% |
Benchmark method
The published leaderboard uses the primary GPT-5.5 judge to create gold issues and match Cortex output. Code computes precision, recall, and overall accuracy (F1) from the match array — the benchmark does not trust judge arithmetic. A trust-weighted score (F0.5, precision counts double) is published alongside, because a wrong flag interrupts the clinician while a missed issue does not.
Every accuracy number carries a 95% confidence interval from a bootstrap over charts. On a corpus this size, two systems whose intervals overlap should be read as tied — we publish the intervals so you do not have to take a one-point ranking gap on faith.
We do not treat any single model as canonical truth. Consensus oracle v1 keeps GPT-5.5 as the published score for now, records Opus disagreement audits, and is designed to add newer frontier judges as they become clearly stronger.
Every published number comes from de-identified primary-care charts captured from real Practice Fusion workflows, carrying the original human-made coding and documentation mistakes. A synthetic smoke-test fixture exists in the repo but is excluded from all published metrics. Some charts have no encounter note, so note-dependent lanes are skipped for those rows.
Rows marked “scoped gold” run Cortex’s current product configuration, which deliberately leaves quality-measure gaps and prior-auth flags out of the product. Those rows are graded only on the issue types they are designed to catch — the same fairness rule applied to charts without an encounter note. Comparison models run a broader research configuration and are graded on the full issue list.
The public summary ignores single-chart smoke runs, archived prompt controls, and superseded variants; raw result files stay in the repo. Lane-level provider failures are tracked separately from total row failures.
This is still a small, primary-care-only corpus. Treat the ranking as a decision aid, not a clinical validation claim. Product-debatable and single-judge disagreements are flagged in the repo before they affect leaderboard scoring.
Judge: GPT-5.5 (Thinking, high effort) (high). Reproduce with pnpm eval --system <id>, then pnpm eval:summarize. Summary file: public/benchmarks/summary.json. Oracle policy: research/eval/consensus/v1/manifest.json.