Be Careful With 'Beats Jev': How to Read Decision Model Benchmarks

Image placeholder

Editorial, product-forward hero illustration — a clean, flat, geometric trophy on an off-white background, with a very large emerald-green asterisk floating above the trophy's upper right corner. Beneath the trophy, in a crisp monospace footnote line in pale slate text, a single receipt-style fine-print caveat reads: '*Vendor-reported. On vendor-selected dataset. At vendor-chosen threshold.' A thin pale-slate horizontal rule separates the trophy from the fine print. The trophy itself is navy-ink outline with emerald-green highlights; no color fill. One small navy-ink star sits on the trophy's base. Palette strictly off-white background, pale slate grays, emerald green accent, navy ink. Linear or Stripe editorial feel. No AI robots, no neural networks, no faces, no 3D rendering.

• Aspect ratio: 16:9
• Source size: 1600 × 900 px (sharp on retina at 760 px display width)
• Format: WEBP (preferred), PNG fallback
• Target file size: under 300 KB

Save to /public/images/blog/<slug>/, then set src and alt on this <MediaFigure /> to swap it in. Keep the ratio prop so the browser reserves the right space.

By Dr. Sanju Abraham, Fractional CTO — 180+ projects shipped, 34 funded companies. Published October 12, 2026.

Key takeaways

• Most head-to-head “beats Jev” claims are produced by the vendor itself, which means the benchmark was selected, run, and threshold-tuned by the team with the strongest incentive to look good on it.
• Benchmark contamination is a real and documented problem: some models score high on public benchmarks because those benchmarks ended up in the training data, not because the model generalises any better than its peers.
• Confidence and calibration are not the same thing, and vendor charts routinely conflate them — a 94% raw-accuracy score at a cherry-picked threshold tells you almost nothing about how the model behaves across the whole confidence range.
• The three-asterisk rule — a MAW framing — is the fastest filter: who ran the eval, which dataset was used, and at what threshold was the headline number computed.
• The only benchmark that actually matters is your own data, run through both models side-by-side in one afternoon, scored on the one or two metrics your downstream workflow genuinely cares about.

The short answer: Vendor benchmark charts in the decision model market are marketing artefacts first and measurement artefacts second. A chart that reads “Vendor X: 94% on CategoryCorrectness vs Jev: 88%” is almost always vendor-run, often on a dataset the vendor also curated, and nearly always reported at a single threshold that happens to make the rest of the field look weak. Before you let any such number change your roadmap, decode it with the three-asterisk rule and then run the own-data test on a weekend. This piece shows you how.

A founder I was advising last month forwarded me a vendor pitch deck. Slide seven: a single bar chart with two columns. The newcomer at 94%. Jev at 88%. No benchmark name. No sample size. No threshold label. “Should we switch?” she asked. The honest answer is that the chart contained roughly zero useful information, and the next twenty minutes of our call were spent teaching her how to read it. This piece is that twenty minutes, written down.

What does a typical “beats Jev” chart actually say?

A typical vendor benchmark chart says much less than it looks like it says. Here is the shape of the claim you will see repeatedly in 2026 pitch decks, landing pages, and X threads from the growing field of Jev-compatible decision APIs:

Illustrative only. “Vendor X scores 94% on CategoryCorrectness, compared to Jev at 88%. +6 points. Switch today.”

That sentence contains at least nine load-bearing words that are not defined anywhere on the page. What is CategoryCorrectness? Who defined it? Which dataset? How large? What was the threshold? Were the two models tested on the same questions in the same order? Did the vendor pick the subset Jev happens to be weakest on? Is 6 points inside the statistical noise of the sample size? None of these are hypothetical worries. They are the exact questions any serious ML evaluator asks before quoting the number out loud — and the exact questions a product leader must learn to ask before committing a quarter of engineering time to switching stacks.

The deeper issue is structural. Decision models are a new category — our market map of thirteen Jev-compatible decision APIs is roughly a year old, and the field has been grinding since OpenAI entered the market. In a new category, there is no agreed-upon evaluation harness yet. Every vendor rolls their own, which means every “beats Jev” chart is a self-report. The closest thing to a cross-vendor standard is the Stanford HELM framework, and even HELM cheerfully admits that benchmark reliability is an open research problem rather than a solved one.

The three-asterisk rule for reading vendor claims

Every vendor benchmark claim needs three asterisks checked before you believe it. Call it the three-asterisk rule — a MAW framing, meant to sit next to the three-band rule in your head as you read 2026 marketing material. The three asterisks are:

1. Asterisk 1 — Who ran the evaluation? Vendor-run or independent? Was the comparison model’s team present or notified?
2. Asterisk 2 — Which dataset, and could the model have seen it? Public benchmark that may be in training data, held-out set the vendor built themselves, or your own data?
3. Asterisk 3 — What threshold and metric were actually reported? Raw accuracy at a chosen cutoff, F1 across the whole confidence range, or something custom?

If any of the three asterisks is missing or vague, the number is marketing not measurement. That does not make it fraudulent — some vendor numbers are perfectly honest — it just means you cannot act on it alone.

Who ran the eval, and why does it matter so much?

Independence changes the number more than most buyers realise. When a vendor runs its own benchmark against a competitor, three structural biases show up — not through bad faith, usually, but through completely normal incentives.

• Prompt bias. The vendor’s engineers spent a year tuning prompts for their own API. They have never written a prompt for the competitor at that same level of care. Day-of comparisons under-represent the competitor by default.
• Dataset bias. The vendor chose the dataset. If they tried five datasets and reported the one where they looked best, you have no way of knowing.
• Threshold bias. Decision models return a confidence score with every answer. The vendor picked a single cutoff for the headline number. If that cutoff was chosen after seeing both models’ outputs, it is the textbook definition of a cherry-picked threshold.

None of these is fatal on its own. All three together, which is the normal state of a vendor-run chart, mean the headline number can swing five to ten points before anyone has done anything genuinely dishonest. For readers new to the vendor landscape, our earlier OpenAI, Microsoft, and Cloudflare entries in the decision API market piece covers who is incentivised to publish which kinds of charts.

What raises the credibility of a benchmark, in rough order:

• Vendor-run but with the comparison model’s team notified and given a chance to re-tune prompts.
• Third-party lab (HELM, academic group, independent evaluation firm).
• Peer-reviewed paper, pre-registered, with code and prompts public.
• Your own data, run by your own engineer, with no vendor involved.

The right-hand end of that list is where buying decisions should be made. The left-hand end is where pitch decks live.

Could the model have seen the test set already?

Benchmark contamination is the quiet scandal of the modern LLM leaderboard. When a model is trained on a very large corpus of web text and code, it is genuinely difficult to prove that the test set of a popular public benchmark was not somewhere in that corpus. The 2024 ICLR paper Proving Test Set Contamination in Black-Box Language Models by Oren and colleagues formalised this. They showed that you can detect contamination without access to the training data at all, using ordering statistics: a contaminated model finds the benchmark’s canonical answer order dramatically more likely than any shuffled version, which it should not if it has genuinely never seen the data.

The practical implication for decision model buyers is important and specific: when a vendor publishes a score on a public benchmark, you cannot assume the model has never seen the benchmark. The model may be pattern-matching to data it memorised during pre-training rather than generalising to your problem. That is not a problem on your own data, because your data was not on the public web. It is a problem the moment a pitch deck says “beats Jev on MMLU-Pro” or any other named public set.

Three heuristics help:

• Prefer private held-out sets. The best public benchmarks of 2026 use closed test splits specifically to defeat contamination. Microsoft Research’s MMLU-CF is one example.
• Favor new benchmarks over famous ones. If the benchmark predates the model’s training cutoff by more than six months, assume contamination risk is non-zero.
• Test on your own data. It is the one dataset you can be sure was not in anyone’s pre-training set.

Image placeholder

Infographic illustrating benchmark contamination risk — a horizontal timeline on an off-white background. Three stacked rows: top row shows a model's training cutoff date as a vertical navy line; middle row shows three public benchmarks as pale slate pills labeled 'Benchmark A (2022)', 'Benchmark B (2024)', 'Benchmark C (2026)' with the first two sitting left of the cutoff and the third sitting right; bottom row shades the area left of the cutoff in pale amber with the label 'contamination risk — may be in training data' and the area right of the cutoff in pale emerald with the label 'contamination-safe window'. A small call-out box in the top right reads 'Your own data = always safe'. Palette: off-white background, pale slate borders, amber + emerald accents, navy ink. Clean, flat, geometric. No AI robots.

• Aspect ratio: 16:9
• Source size: 1280 × 720 px (sharp on retina at 760 px display width)
• Format: PNG
• Target file size: under 500 KB

Save to /public/images/blog/<slug>/, then set src and alt on this <MediaFigure /> to swap it in. Keep the ratio prop so the browser reserves the right space.

Which thresholds and metrics were actually reported?

The third asterisk is the one most pitch decks hide hardest. Decision models do not return just a label; they return a label plus a confidence score, as we covered in detail in Why ‘70% Sure’ Is More Useful Than ‘Yes’. Every number you see on a benchmark chart is a slice through that distribution at a particular cutoff. Change the cutoff and the number moves — sometimes by double digits.

Four tricks you will see in the wild:

• Threshold cherry-picking. The vendor evaluated at the single confidence cutoff where their model happened to look best, then reported that number without naming it.
• Confidence–calibration conflation. The chart says “94% accurate” but what it means is “94% of items the model labeled ‘yes’ were correct — evaluated only on items where the model was more than 0.8 confident.” A genuinely useful chart reports accuracy across the whole confidence range, which is a calibration curve, not a bar.
• Subset selection. The eval covers 500 examples carefully chosen from a larger set. “Carefully” sometimes means “we dropped the ones we failed.”
• Metric choice. Raw accuracy, precision at a threshold, F1, Brier score, and expected calibration error are all legitimate numbers. They rank models differently. The vendor picked the one that ranked them first.

The companion piece to this one — The calibration feature nobody talks about — digs into why the calibration curve is often the more useful comparison than the headline accuracy bar. For now, the practical heuristic is: if the vendor chart does not show a calibration curve, assume the headline number is a point estimate at a flattering cutoff.

Honest vs misleading benchmark presentations, side by side

Here is the same hypothetical comparison, presented two ways. The first column is what a vendor pitch deck typically shows. The second is what the same underlying data would look like reported honestly.

What the chart claims	Vendor-flavored version	Honest version
Headline	”94% on CategoryCorrectness"	"F1 of 0.87 ± 0.03 on the public TriageQA v2 held-out set, n = 2,000”
Who ran it	Not stated	”Vendor-run; competitor team not notified; competitor prompts not tuned by competitor”
Dataset	”Internal benchmark"	"TriageQA v2, released March 2024; may overlap with pre-training”
Threshold	A single bar	A calibration curve from 0 to 1 confidence
Comparison	Just two bars	Both models, both prompts, same seeds, same examples, in same order
Takeaway offered	”Switch today"	"Pilot on your own data before switching”

The vendor-flavored row is not fraud. It is simply underspecified in ways that happen to flatter the vendor. The honest row is the shape of a chart you can act on without regret.

Run the own-data test in one afternoon

The only benchmark that actually matters is your own data, run through both models side-by-side on the exact problem you plan to solve. Here is the five-step protocol I use with every Fractional CTO client who is weighing a vendor switch. One afternoon, one engineer, no data science team required — a point echoed in our piece on your first decision model experiment.

1. Pull 200 real examples. From production logs, with the input and the known correct answer (or human-labeled best answer) for each. If you do not have 200, start with 50 and acknowledge the noise. Keep the exact same examples for both models.
2. Lock the prompt once. Write a single prompt that both models will see. Do not tune it per vendor. If the APIs are wire-compatible — many are, as our Jev-compatible APIs piece covers — this is a one-line change in your test harness.
3. Run both models, log everything. For each example, log the model’s label, its confidence, your known-correct label, and the latency. Nothing fancy. A CSV is sufficient.
4. Compute three numbers, not one. For each model: (a) accuracy at the same threshold, (b) accuracy on the slice where confidence is above 0.8, and (c) the volume of items the model is confident about at all. The first number is what the vendor shows you. The second two are where the real differences live.
5. Decide on cost per correct auto-decision. Multiply the per-call price by the number of items each model confidently handles correctly. The model that costs less per correct auto-decision wins, even if its headline accuracy is slightly lower. This is the number that determines your actual production bill.

That whole exercise fits inside four hours for a competent backend engineer and gives you a number that is actually about your business. For the broader selection framework around which this test sits, see our companion piece Choosing a Decision API in 2026.

Image placeholder

Infographic: a five-step horizontal workflow on an off-white background titled 'The own-data test'. Five circular numbered nodes connected by thin pale-slate arrows: 1) 'Pull 200 examples' with a document icon, 2) 'Lock one prompt' with a padlock icon, 3) 'Run both models' with a split-arrow icon, 4) 'Compute three numbers' with a bar-chart icon, 5) 'Pick lowest cost per correct auto-decision' with a dollar-sign icon. Each node is navy-outlined with an emerald-green number. Below the workflow, a thin pale-slate footnote line reads 'One afternoon. One engineer. No data science team.' Palette: off-white background, pale slate borders, emerald green numbers, navy ink. Clean, flat, geometric. No AI robots.

• Aspect ratio: 16:9
• Source size: 1280 × 720 px (sharp on retina at 760 px display width)
• Format: PNG
• Target file size: under 500 KB

Save to /public/images/blog/<slug>/, then set src and alt on this <MediaFigure /> to swap it in. Keep the ratio prop so the browser reserves the right space.

What this means for picking a decision API in 2026

The 2026 decision model market resembles the 2012 database market in one specific way. A cluster of vendors look roughly equivalent in their own slide decks and genuinely differ once you run them on your workload. We mapped the field in thirteen decision APIs: a market map and expanded on the frontier moves in OpenAI enters the decisions market. The practical takeaway is the same across all of them: never let a vendor chart be the thing that moves your roadmap.

Three rules of thumb for 2026 buyers:

• Ignore any benchmark that is missing one of the three asterisks. No independence, no dataset provenance, no reported threshold — no decision.
• Treat the own-data test as the actual benchmark. It is cheap. It is specific to your use case. It is not contaminated. Nothing else carries that combination.
• Switch only when the cost per correct auto-decision moves meaningfully. A six-point headline accuracy difference that disappears at your actual threshold is not a reason to re-wire a production system.

The field will mature. The benchmarks will get better. Public evaluation harnesses will emerge. Until they do, you are the evaluator, and the one afternoon it takes to prove or disprove a “beats Jev” claim is the cheapest insurance on an entire year of production decisions.

Frequently asked questions

Are vendor benchmark claims in the decision model market trustworthy?

Vendor-reported benchmarks are not fraudulent, but they are structurally biased. The vendor chose the dataset, tuned the prompt, picked the threshold, and ran the eval. Each of those choices can swing the headline number by several points before anyone has done anything genuinely dishonest. Treat a vendor chart as a lead, not a conclusion, and verify on your own data before switching anything.

What is benchmark contamination?

Benchmark contamination is the problem of a model having seen the test set during its own training. Because modern LLMs are trained on very large web corpora, popular public benchmarks are often inside the training data, which means a high score may reflect memorisation rather than generalisation. The 2024 ICLR paper “Proving Test Set Contamination in Black-Box Language Models” shows how to detect this without access to the training data itself.

What is the three-asterisk rule?

The three-asterisk rule is a MAW framing for reading any vendor benchmark claim. Before you act on a number, confirm the three asterisks: who ran the evaluation (independence), which dataset was used (contamination risk), and what threshold and metric were reported (cherry-picking risk). If any of the three is missing or vague, the number is marketing, not measurement. The rule pairs with the three-band rule for confidence scores.

How do I test a decision model on my own data?

Pull around 200 real examples from your production logs with known-correct labels. Lock a single prompt that both vendors will see. Run both models on exactly the same examples, logging label, confidence, and latency. Compute three numbers per model: accuracy at a shared threshold, accuracy above a high-confidence cutoff, and the volume of confident items. Decide on cost per correct auto-decision, not headline accuracy. The exercise fits in one afternoon.

Why isn’t raw accuracy the right metric?

Raw accuracy at a single threshold tells you how the model behaves at exactly one point on its confidence distribution. It hides calibration, the shape of the error curve, and the volume of confident decisions. The model that costs you less per correct auto-decision is almost never the model with the highest headline accuracy; it is the one with the right mix of accuracy, calibration, and confident volume for your specific workflow.

Should I ever trust a public benchmark score?

Yes, with caveats. Public benchmarks that use closed test splits, post-date the model’s training cutoff, and are run by independent labs such as Stanford HELM are legitimately informative. Public benchmarks that are famous, older than the model, and reported only by the vendor itself are the ones to view with the most skepticism. In both cases, your own data is a stronger signal.

Does the own-data test need a data scientist?

No. The own-data test is a product-and-engineering exercise, not a statistics project. A competent backend engineer can pull the examples, write the test harness, run both APIs, and compute the three numbers inside half a working day. The analysis is spreadsheet-level math. Where a data scientist helps is in refining the metrics over months of production data, but you do not need one to make the first decision.

---

If you are weighing a decision model switch right now and the vendor’s chart is the thing driving the conversation, that is exactly the kind of call a Fractional CTO is paid to make with you. One hour of the right reading of one chart — and one weekend of the own-data test — can save a quarter of engineering time and an entire production rollout. More pieces in this thread live in the AI Strategy archive.