An audit of 254 SWE-bench submissions across four splits, run on published verdicts without executing any models, asking whether small leaderboard gaps really order the systems beneath them. On Verified the leading two entries each resolve 396 of 500 instances; the top ten share 285 successes and 51 failures, leaving 164 instances that distinguish them at all. Median nesting of frontier solution sets is 0.935 against a score-implied baseline of 0.774. Scores depend on the model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points, against an 8.8-point spread across the top thirty. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05, while the larger Test split separates 14 of 23. The authors release a partition and a five-step audit protocol, and recommend reporting comparison-set-specific resolution and model-scaffold provenance instead of reading aggregate gaps as rank.
Two entries at the top of Verified both resolve 396 of 500 instances. Under the usual reading that is a tie, and the leaderboard prints an ordering anyway because the numbers are not printed to the precision that would show it.
The audit's central move is to ask how much of the outcome set is actually shared. The top ten share 285 successes and 51 failures. Only 164 instances distinguish their outcomes at all. Median nesting of 0.935 says the solution sets sit almost entirely inside one another, where a score-implied baseline would predict 0.774.
This is the finding that should change how a team buys. Observed within-model scaffold ranges reach 29.8 percentage points. The entire spread across the top thirty entries is 8.8 points.
So "which model is best at coding" is, as measured, a smaller question than "which harness is wrapped around it". The authors are careful that the design is observational and does not identify causal scaffold effects, and six of nine cell-mean interaction tests survive Holm correction. The direction is still plain enough to act on: changing the scaffold is a bigger lever than changing the model, and it is the cheaper one.
Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs, and 14 of 23 on the larger Test split. The authors are explicit that this does not establish the systems are equivalent. It establishes that the benchmark at that size cannot tell them apart.
That distinction is the whole paper. A leaderboard that cannot resolve a difference is not evidence of no difference, and reading it as either a ranking or a tie are both errors.
Report the comparison set with the score. Record which scaffold produced it. Stop treating a two-point gap as a procurement decision. And when a benchmark saturates, keep it as a regression floor and go find something that still separates the systems you are choosing between.
@article{liu2026coding,
author = {Fengshuo Liu and Ying Liu and Ruize Sun and Lie Luo and Siyuan Guo},
title = {Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead},
journal = {ADMA 2026},
year = {2026},
eprint = {2609.17394},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.17394}
}Ejentic did not author this paper. Credit belongs to the authors named above; the note is ours.
2026
Read this before you let an agent call anything a customer will see the output of. The failure mode is not a crash, it is an API returning 200 with a body that means "I did not understand you". There is no exception to catch and no status code to branch on. The result we quote most often is the last one: the fix was one line of schema, not a better model.
2023
The benchmark we reach for when a client asks whether an agent can be trusted with a real repository. Passing tests is a much sharper signal than "the diff looks reasonable".
2026
Every RAG proposal we are handed assumes dense retrieval is the default and everything else is a downgrade. This is the study that says it depends on corpus size, and that above roughly 10 million tokens a lexical index built for nothing beats an agent that reasons its way through the corpus. It also costs 39 times fewer query tokens at the point where the agent is still winning.