Same model, same weights, same underlying capability. On the Terminal-Bench leaderboard, Claude Opus 4.6 scores 58.0 percent wrapped in Anthropic’s own Claude Code and 79.8 percent wrapped in a third-party harness called ForgeCode. That is a 21.8 percentage point swing from changing nothing but the scaffolding around the model, a gap wider than what usually separates one model generation from the next.
A position paper from Tessl, a London software startup, argues that gap is not a footnote. It is the reason coding agent benchmarks, the numbers that show up in every model launch post and every “best AI coding tool” comparison, measure something other than what people assume they measure. The paper, posted to arXiv in June by Maria Gorinova and five co-authors, is headed for the Agentic Software Engineering workshop at KDD 2026 in Jeju, South Korea this August. Its argument is narrow, technical, and, once you see the underlying data, hard to unsee in every leaderboard published since.
A Model Is Not an Agent
The paper’s central move is a distinction most benchmark coverage skips past. An LLM is a model. A coding agent like Claude Code, OpenAI’s Codex, or Cursor Agent is what the authors call an agent harness: a model wired into a tool-use loop, with a system prompt, file access, and some way of running tests. Above that sits a system harness, the outer orchestration layer that breaks a larger goal into tasks, hands each one to an agent harness, manages the repository and CI environment those tasks run against, and routes feedback, linters, test results, reviewer comments, back into the loop.
Benchmarks like SWE-Bench, HumanEval, and LiveCodeBench were built to score the model component, back when a coding benchmark meant asking an LLM to generate a working function in one pass. They still report a single end-to-end number. But a modern coding agent has at least five moving parts, tasks, the agent harness itself, the environment, the context fed into that environment, and the feedback signals used to grade and refine a solution, and the paper’s evidence says any one of those five can move the final score as much as swapping the underlying model.
The feedback layer alone splits into three tiers that current benchmarks collapse into one pass or fail. Inner-loop signals, tests, types, linting, arrive in seconds and catch narrow mistakes. Middle-loop signals, reviewer feedback, mutation testing, architecture checks, take minutes to hours and catch the kind of judgment calls a single unit test cannot see. Outer-loop signals, whether a pull request actually gets merged, whether it gets reverted later, take days and are the closest thing to ground truth, but they are exactly what a benchmark run in an afternoon cannot capture. A production system harness, the kind described in engineering writeups from both Anthropic and OpenAI and demonstrated in the paper’s own open-sourced NS2 reference implementation, uses all three. A benchmark that reports only a single pass rate uses none of that structure, similar to how MiniMax M2.7’s self-modifying training harness needed its own internal feedback loop just to know which of its hundred iterations actually helped.
The Terminal-Bench table is the sharpest illustration, but it is not the only data point. Morph Labs found Claude Opus 4.5 swinging 4 to 10 points between a standardized and a custom scaffold on the same benchmark. AI21, analyzing more than 200,000 SWE-Bench runs, found that orchestration choices, container allocation, and even evaluation seed selection materially move the pass rate with the model and harness both held fixed. Fan et al.’s SWE-Effi study goes further, showing that scaffold effectiveness is not additive with the base model. The same scaffold produces resolve rates that vary 2 to 3 times depending on which model sits underneath it, meaning the scaffold and model interact rather than simply stack.
The Second Problem: Grading Against One Answer
SWE-Bench, the dominant benchmark for agentic coding, works by taking a real GitHub issue and its eventual merged fix, then checking whether an agent’s patch makes the same hidden tests pass that the original pull request did. That is a reasonable way to check whether a bug got fixed. It is a much narrower way to check whether an agent wrote good code, because it grades closeness to one specific implementation rather than whether the underlying problem was actually solved well.
The paper cites three empirical studies that expose how much this costs in practice. SWE-Bench+ found solution leakage in 32.67 percent of issue text and passes under insufficient tests in 31.08 percent of cases. A differential-testing study found that 7.8 percent of patches marked as resolved actually fail tests written by the original developers, and 29.6 percent behave differently from the gold patch at runtime despite passing the benchmark’s own checks. A separate memorization study documented file-localization behavior in top models consistent with recalling training data rather than reasoning about the repository from scratch.
The gap between benchmark and reality shows up again in a real-world dataset called AIDev, which the paper cites as the clearest evidence of the disconnect: across 456,000 agent-authored pull requests in 61,000 live repositories, actual maintainer acceptance sits at 35 to 64 percent, well under the headline figures above 70 percent that models report on SWE-Bench Verified. A separate study from the AI safety research group METR reached a similar number from a different angle. Recruiting maintainers from scikit-learn, Sphinx, and pytest to review 296 AI-generated pull requests that had already passed the SWE-Bench automated grader, METR found maintainer merge decisions ran about 24 percentage points below the automated score, and, after adjusting for the ordinary noise in human review decisions, roughly half of test-passing patches would not actually be merged into the codebase.
Why Developers Should Care
None of this means coding agents are worse than advertised. It means the specific number attached to a model, “65 percent on SWE-Bench Verified,” is close to meaningless on its own, because it describes a system, not a model, and the system half of that pairing is invisible in the headline. Two vendors can quote the same base model at wildly different scores because one built a better harness, not a better model, and a reader has no way to tell which happened from the leaderboard alone.
That matters directly for anyone choosing between coding agents right now, a decision this site has covered before in a different modality, where the gap between benchmark scores and real capability shows the same pattern. The paper’s suggested fix is procedural rather than mathematical: benchmark submissions should report the model, harness version, and environment hash used, and should include at least one ablation swapping a non-model component while holding the rest fixed. Without that metadata, comparing two leaderboard entries is comparing two different systems while pretending to compare two models.
What the Paper Does Not Solve
The authors are upfront about the limits of their own argument, and it is worth taking that section seriously rather than skipping to the recommendations. Their proposed alternative, grading against broader behavioral invariants instead of a single reference patch, runs into what they call the operationalization gap: specifying what a “good” solution looks like in terms precise enough for automated grading, without accidentally encoding one particular implementation as the only acceptable answer. They admit this is the hardest open problem in the paper and do not claim to have solved it.
There is also a scope limit the paper does not dwell on. Every dataset it cites, SWE-Bench, Terminal-Bench, AIDev, skews toward well-known, actively maintained open-source repositories with conventional structure. Whether the same harness-driven variance holds inside a messy internal enterprise codebase with undocumented conventions, the environment where most professional coding agents actually operate, is not something any of the cited studies measured directly. The paper’s own NS2 system, built and open-sourced by the authors as a reference implementation, is a useful existence proof of a harness with component-level checks, but it is one team’s design, not a validated standard, and the authors present it that way rather than overselling it.
The paper also picks a specific, defensible fight rather than a sweeping one. In its own alternative-views section, it concedes that end-to-end scores do reflect real usage and are not worthless, only insufficient on their own, an admission that keeps the argument closer to other recent findings about the gap between benchmark performance and real-world reliability than to a wholesale dismissal of the benchmarking project.
What Happens Next
The industry response started before the paper even circulated widely. OpenAI has already stopped treating SWE-Bench Verified as its frontier reporting standard for exactly the reasons this paper documents, and Scale AI’s SWE-Bench Pro now recommends broader task horizons and multi-language coverage as a partial fix. Snorkel AI’s newly launched Senior SWE-Bench goes further in the direction the paper argues for, grading code quality through per-task rubrics written by the original repository maintainers rather than test-passing alone, explicitly citing the METR merge-rate gap as the problem it exists to close.
None of that resolves the operationalization gap the Tessl paper flags as the real bottleneck. Writing a grader that can tell good code from code that merely passes tests, without secretly grading against one hidden implementation, is a harder problem than swapping in a new dataset. Until someone solves it, the honest reading of any coding agent leaderboard is the one this paper pushes toward: treat the number as a description of a system you have not seen the components of, not a property of the model whose name sits next to it.
Leave a Reply