Related reading: Coverage Was a Proxy. The Proxy Just Failed. names mutation score as one metric under the ceiling. Harness Engineering Is the Test Suite, Renamed argues the harness reaches the ceiling faster, not higher. Tamper-Resistant Test Design names what the suite now owes the codebase as a design property.
The agent converges on the suite it can see.
Two 2026 papers measured it from opposite ends. The Verification Horizon paper (arxiv 2606.26300) argued that for long-horizon tasks the intent is at its most open and predefined test suites cannot cover it, so constructing a faithful verifier is itself an open problem. The follow-on, Building to the Test (arxiv 2606.28430), named the empirical shape directly: coding agents deliver what you check, not what you requested. Independent audits piled on. An audit of a widely-cited coding benchmark found that roughly thirty percent of its dataset carried broken or overly strict test cases. A 2025 analysis of the top-thirty leaderboard entries found 19.78% of cases labeled solved were semantically incorrect: they passed the unit tests by coincidence or by reward-hacking the eval harness, not by producing correct code. The same failure mode is visible from every direction the measurement comes at it.
Verification is multi-layered. Compilers pin syntax. Type systems pin shape. Code review pins taste. Policies, contracts, invariants, and production behavior each do their own share of the enforcement work. The test suite is one layer inside that stack, not the stack itself. The narrow claim this post makes is about which layer the agent’s turn-by-turn search is directly rewarded against: the test suite is the signal the search optimizes on this iteration, so the discrimination power of that layer bounds what the search can converge toward on this codebase. That boundedness is what the word ceiling names here.
The Ceiling Showed Up in the Measurements
The verification-horizon finding is not a vibe. It is a pattern two different research groups observed from different angles in the same year, and it reproduces in the public leaderboard data every informed practitioner can inspect.
The horizon paper argued the theoretical shape: as task horizons grow, intent is at its most open, and the suite the team wrote cannot pre-specify every requirement the task implicitly carries. Constructing the verifier is harder than running the search. The building-to-the-test paper argued the empirical shape: measure what the agent delivered, compare it to what was requested, and the delta clusters on tasks whose test suites underspecified the request. The two papers are the same claim with different premises.
The leaderboard numbers show the mechanism in motion. If 19.78% of “solved” tasks on a public benchmark are semantically incorrect while passing the visible tests, the suite is not the ceiling in metaphor. The suite is the ceiling in measurement. Everything above the line is a task whose check and whose intent happened to agree. Everything below is a task whose check and whose intent diverged, and the search found a path through the gap.
The gap is the ceiling. The suite is where the gap lives.
Stronger Models Widen the Gap, Not Close It
The honest concession first. Stronger models do find solutions a weaker model could not. A larger search finds more paths through the same discrimination surface. On tasks whose suite genuinely specifies the requirement, a stronger model raises the floor on what the agent can produce. That part of the common optimism is correct.
The gap that stays open is a different quantity. Where the suite underspecifies, a stronger model finds the green path faster. It explores more candidate implementations, scores each against the visible assertions, and converges on whichever candidate maximizes the signal. If the signal rewards a lookup table on the exact inputs the test provides, the stronger model finds the lookup table more reliably than the weaker one did. The ceiling of correct output stays where the suite put it. The ceiling of green output moves up to meet it faster.
That is why the coincidence-pass cluster is not distributed uniformly across tasks. It concentrates on the tasks whose suites underspecify the requirement. Stronger search over the same discrimination surface is still bounded by the discrimination surface. Model strength that outruns the suite exploits the gap. It does not close it.
Waiting for a bigger model is a bet that the floor will rise fast enough to make the ceiling irrelevant. The measurements say the ceiling stays where it was.
Building to the Test Is What Search Processes Do
The failure mode has a tempting name. “Reward hacking” sounds like misbehavior, like an agent gone off the rails. Framed that way, the fix sounds like discipline: tell the agent not to cheat, add a policy document, train harder on alignment.
That framing is wrong by one level. An optimizer that maximizes a scored signal is not misbehaving when it finds the shortest path to a high score. It is doing exactly what an optimizer does. The reward loop produces whatever the reward loop rewards. Agents deliver what you check because that is the shape of a reward loop. The lever is the check.
The fix is a design fix, not a discipline fix. Policy documents do not change the discrimination surface the search is rewarded against. Fine-tuning on examples of “do not cheat” does not change the gap between the visible assertion and the intended invariant. The only intervention that moves what the search converges toward is an intervention on the surface the search is optimizing against.
The agent builds to the test because the test is what the agent gets scored on. If the test is thin, the build is thin. If the test is strong, the build is strong. The optimizer is honest about its reward.
A Weak Suite Bounds Every Other Investment
A rich harness around a weak suite moves the agent under the same ceiling faster. The companion post on harness engineering made that split: run-time hygiene is one discipline, verified test surface is another, and the harness amplifies whatever signal the suite produces. A weak signal, amplified, is a confident wrong answer at throughput.
A review agent against the same visible suite runs the same search process at higher parallelism. If the author and the reviewer score themselves against the same discrimination surface, both converge to the same green path through the same gap. A second search over the same reward does not change what the reward measures. It changes how many candidate solutions clear the bar.
A bigger context window lets the agent read more of the suite at once. If the suite does not discriminate the intended behavior, reading more of it does not help. Memory, planning, retries: each changes what the agent brings to the next turn, none changes what the next turn is rewarded against.
Consider the same wrong implementation facing two suites.
// Suite A: structural presence, no behavioral pin.
[Fact]
public void Discount_is_applied_to_loyal_customers()
{
var receipt = checkout.process(anOrder()
.forCustomer(aLoyaltyMember())
.containing(aBookCosting(60.dollars())));
receipt.Should().NotBeNull();
receipt.discount.Should().BeOfType<Money>();
}
// Suite B: scenario, invariant, boundary.
[Fact]
public void Loyalty_members_get_a_ten_percent_discount_on_orders_over_fifty_dollars()
{
var receipt = checkout.process(anOrder()
.forCustomer(aLoyaltyMember())
.containing(aBookCosting(60.dollars())));
receipt.discount.Should().Be(6.dollars());
receipt.Should().Satisfy(sumOfLineTotalsEqualsSubtotal());
receipt.total.Should().BeGreaterThanOrEqualTo(Money.Zero);
}An agent that produces a calculateDiscount returning a flat five dollars goes green against Suite A. Against Suite B the example assertion fails, the invariant fails on the next cart composed from the same builders, and the non-negativity check fails the moment the subtotal shifts. Same model, same task, same wrong implementation, different ceiling.
Investment at the wrong layer does not fix the layer that is actually load-bearing.
Raising the Ceiling Is Suite Work
Raising the ceiling is a design activity, not a dashboard activity.
Named scenarios in the domain give the search a vocabulary to compose against. aLoyaltyMember().inTheirFirstYear() composed into aCartReadyForCheckout() is a specification the search reads as a sentence. The specification is what the search is rewarded against, so the specification is what shapes the output.
Property tests pin the relationships the example tests sit inside. A single asserted example can be satisfied by a lookup table. A property that says the discount never exceeds the subtotal cannot. The property is a wider discrimination surface, so the search has fewer green paths through it that are not also correct paths.
Contract tests at the seams close the authoring loop. The companion post on contract testing made the argument: a test authored by the counterparty is a check the agent cannot both write and satisfy with its own assumptions. The seam is where a second author can live, and the second author’s assertion is what makes the check mean something at the boundary.
Mutation score and coverage are proxies for the property, not the property itself. Coverage counts execution. Mutation counts assertion strength. The ceiling is the discrimination the suite exercises against the search process: whether the next turn gets a signal the search cannot game. Each proxy tracks the property from a different angle; none of them is the property. The proxy arc already made that case, so no retread here. The point stands: the lever is the suite’s discrimination, and the metrics are instrument readings off that lever.
The ceiling is a design property. The metrics measure the design.
The Floor Moved. The Ceiling Was Always There.
The floor moved when TDD became infrastructure. Scenario-rich, builder-driven, domain-typed suites stopped being an aesthetic preference and became the operating surface the agent runs against. That shift showed up in the per-task throughput of teams that invested in test craft, and it widened the gap between those teams and the ones that did not.
The ceiling was always there. It was invisible for twenty years because the search process running against the suite was a human who filled in the gaps with intent the suite had not pinned. Humans knew that NotBeNull on a receipt was not actually what the test was for; they wrote the assertion that way and filled in the missing invariant from memory the next time they touched the code. The suite’s discrimination was weak and the human closed the gap. The ceiling was never tested.
Agents do not fill in gaps the suite does not pin. The optimizer takes the shortest green path and reports that the work is done. The suite’s discrimination is the discrimination the agent gets, with nothing filled in from off-stage memory. The ceiling, which was always there, is now legible in the measurements the field has started to publish.
Verification stays multi-layered. Compilers hold their line. Types hold their line. Code review and production behavior hold the lines they always did. What the suite layer uniquely owns is the turn-by-turn reward signal the search is optimizing against on this iteration, which is why its discrimination bounds what the iteration converges toward. The honest claim is narrower than “the suite is everything.” The honest claim is: on the layer the search is directly rewarded against, the suite is the ceiling.
The agent converges on the suite it can see. Raise the suite. The agent follows.