Related reading: Evals Are Tests Wearing a Lab Coat called the eval-versus-test split a role artifact. The Visible Test Suite Is the Ceiling on What the Next Agent Can Deliver names the same ceiling at the team-codebase level. Tamper-Resistant Test Design is the structural companion on oracles the writer cannot have memorized.
The team spent a quarter porting its stack to the model that topped the leaderboard. The model never shipped.
The 2026 saturation review analyzed 60 widely used LLM benchmarks and flagged 29 as high or very high on a saturation index. Variance among frontier models at that saturation level approaches measurement noise. MMLU is routinely over 88% on frontier models. In parallel, a 2026 null-model paper documents a fixed-string output topping AlpacaEval 2.0 under automated judging. Multilingual contamination studies report leakage up to 91.8% on popular suites. Hugging Face retired its Open LLM Leaderboard in 2025 over the compounding of these effects. The public number is doing less work than its reader thinks.
The seductive conclusion is that benchmarks are useless. That conclusion is wrong, and off-voice. The sharper claim is narrower. Public benchmarks are weak signal on capability shape, usable as triage. They do not tell a team what the model will do on the team’s own tasks, in the team’s own vocabulary, against the team’s own invariants. The signal that moves the ship decision lives in a suite the team already half-owns.
The Public Leaderboards Measure Something Other Than Fit
The saturation numbers are measurement, not folklore. At a saturation index approaching 1.0 the discrimination between frontier models collapses into the noise band of the evaluation itself. A benchmark that cannot distinguish the top three models from each other is not a bad benchmark. It is a benchmark that has done its job and now reports a flat line. The reader who treats a flat line as “these models are all equally good” has read the chart backward.
The null-model result on AlpacaEval 2.0 is a different kind of signal. A fixed-string output, chosen to game the judge’s stylistic preferences, climbed the ranking against models that produced actual answers. The number the leaderboard reported was partly a measurement of capability and partly a measurement of alignment with the judge’s reward surface. The two components are hard to separate from outside.
Contamination compounds both. A benchmark whose items leaked into training corpora loses discrimination against models that happened to see the items. The 91.8% leakage figure on one multilingual study is the tail; the median leaks lower. The reader cannot know which model saw which items. The number is reported anyway.
The public number is doing a weaker job than the reader assumes it is doing. The industry response has been to build steeper hills: Humanity’s Last Exam, FrontierMath, GPQA Diamond were authored to resist the saturation that caught MMLU. They will saturate on the same curve, because the mechanism that saturated the easier benchmarks is the optimization pressure that produced the saturation on them. A fixed artifact optimized against by well-funded labs saturates. The adversarial design pushes the discrimination out a few quarters and does not change the long-run shape. Building a steeper hill does not change the climber’s incentive.
Capability Shape Is a Weak Signal. Ship Signal Lives Elsewhere.
Concede the triage role. Public benchmarks are useful for shortlist narrowing. A team deciding which three models to evaluate seriously spends an hour on the leaderboards and comes out of it with a reasonable shortlist. The alternative is a greenfield evaluation of every model on the market, which no team runs, so the leaderboards earn their keep as the triage layer.
The sharpening is at the next step. Once the shortlist is three models wide, the leaderboard numbers tell the team nothing about which of the three gets closest to the team’s tasks on the team’s vocabulary. The remaining variance is below the benchmark’s discrimination power. The ship decision is made somewhere else, or it is made on a coin flip wearing a dashboard.
The team that treats the leaderboard as the ship signal spends a quarter porting its stack to the model whose public number was largest and discovers at launch that the model’s domain fluency on the team’s actual workflows is a notch behind the one it did not pick. The leaderboard did not lie. It just answered a different question. The team asked which model was best, and the leaderboard answered which model scored highest on a public benchmark. Those are different questions when the benchmark is saturated and the shortlist is tight.
Your Private Eval Is Your Suite, Named Correctly
The eval wave built dashboards, golden datasets, judge models, specialized frameworks, and a job title. The private-eval argument is the operational consequence of the evals-wearing-a-lab-coat frame: the discipline the eval wave was reaching for is the test suite the team already half-owns, in the team’s own domain language, assembled from the same builders.
[Fact]
public void A_draft_to_a_churn_risk_customer_includes_the_retention_offer_and_stays_under_two_hundred_words()
{
var draft = emailAgent.Compose(
anEmailDraft()
.forA(churnRiskCustomer())
.about(aRecentFailedShipment()));
draft.Should().MentionThe(retentionOffer);
draft.wordCount.Should().BeLessOrEqualTo(200);
}That scenario is a test, a scenario, and an eval at once. The vocabulary is the team’s (churnRiskCustomer(), retentionOffer, aRecentFailedShipment()). The assertions are on named domain state and a concrete constraint, not on an exact string or an opaque golden output. A workflow that runs the production scenario tests against each shortlisted model and reports pass counts per domain concept is a private eval with no dashboard, no separate framework, and no job title. The report is a diff of two numbers.
The reader who was told evals are a new discipline will notice that the shape is familiar. That is the point. The industry split evals from tests to sell the framework. The team that collapses the split keeps the discipline and loses the overhead.
The Vocabulary Is the Barrier to Contamination
Public benchmarks leak because their vocabulary is public. The items are published, the protocols are published, the evaluation harnesses are published. Any training corpus assembled after the benchmark’s release is a candidate carrier.
A private eval written in the team’s own domain language is contamination-resistant for a structural reason. anEmailDraft().forA(churnRiskCustomer()) is a composition of builder calls the public corpus has never seen. The scenario names are the team’s; the invariants are the team’s; the oracle lives in a vocabulary a model’s training data could not have memorized. The oracle also survives model turnover: a new generation of models that have never seen the team’s suite scores on the same scenarios without re-tuning the oracle, because the oracle asserts on domain state rather than on a specific output phrasing.
The vocabulary-amplifies arc argued that the suite is leverage because agents recombine what they find. The private-eval argument extends it. The suite is also leverage because its oracles are contamination-resistant across model generations. The same investment pays twice.
The Suite You Already Maintain Is Half the Eval
A team that writes disciplined scenario tests already has most of a private eval. The missing piece is a workflow that runs the suite on each shortlisted model, collects the outputs, and asserts with model-tolerant oracles. One day of pipeline work on top of the existing suite replaces a quarter of eval-framework configuration.
The design that works is cheap. A CI stage per model-under-eval. A per-scenario pass/fail report. A per-concept rollup (how the model does on loyalty, on billing, on returns). A judge is model-generated only on the oracles that genuinely need natural-language scoring, with a human-authored rubric and a deterministic gate downstream of the judge; the rest of the oracles are direct assertions on domain state. The judge is a probabilistic evidence source inside a deterministic verdict, which is the house position on governed model judgment restated at the eval tier.
The result is a signal the leaderboard cannot produce. The team sees which model is strongest on billing scenarios, which model drafts churn-risk emails that keep the retention offer, which model gets the word-count constraint right under adversarial inputs. The ship decision rides on evidence the team authored, in a vocabulary the public leaderboards never saw, against invariants the team owns.
The Private Eval Is the Benchmark That Steers You
Public leaderboards shape shortlists. Private evals decide them.
The team that spent a quarter on the wrong model read the public number as the ship signal. The team that reads the public number as triage and the private eval as the verdict spends one day and ships the right model. The private eval’s cost is one day because the suite is already there. The public leaderboard’s cost in misread ship decisions is a quarter at a time.
The eval that matters lives in the same vocabulary as the production tests, assembled from the same builders, asserting on the same invariants. The split between “eval” and “test” is a role artifact; the private suite collapses it. The suite is the ceiling at the codebase level and the benchmark at the model-selection level, and both ceilings are set by the same discipline. The team that invests in the vocabulary pays once and collects at both tiers.
The leaderboard told the team which models to consider. The private eval told the team which model to ship. The team that mistook the first signal for the second ported the stack for nothing. The team that reads the signals in order spends a day and gets the ship decision right.