The Benchmark Everyone Quoted for Coding Agents Just Got Disowned by OpenAI

The number every launch post quoted

For the last couple of years, if you wanted a one-number answer to "how good is this model at coding," you reached for SWE-bench Verified. It's a curated set of real GitHub issues paired with the pull request that actually fixed them, run through an agent harness and graded on whether the generated patch passes the project's own tests. Every frontier lab put the score in its launch announcement. Buyers put it in vendor comparison spreadsheets. It became the de facto standard the way a credit score becomes the de facto standard for creditworthiness — convenient, widely quoted, and not quite measuring what people assumed it measured.

In February 2026, OpenAI published a post titled "Why we no longer evaluate SWE-bench Verified," announcing it had stopped reporting the benchmark and was recommending other labs do the same. Its stated reason: the problems are pulled from open-source repositories that model providers train on, and the benchmark had become contaminated enough that gains on it no longer reliably tracked real coding skill. OpenAI pointed to SWE-bench Pro — a harder successor with held-out splits and private commercial repos — as the interim replacement, while it builds evaluations designed from the start to resist this kind of leakage.

This wasn't really a surprise

The contamination problem had been visible in the research literature for a while before a frontier lab said it out loud. An analysis by Aleithan et al. found that roughly 94% of SWE-bench's instances consist of issues that predate the training cutoffs of the LLMs commonly evaluated on it — meaning the model may have seen the issue, the discussion, and possibly the fix itself during training, not just learned a general pattern for fixing similar bugs. A separate study by Zhou et al. identified direct data leakage in a meaningful slice of the benchmark. On top of that, SWE-bench is exclusively Python, which rewards models (and the harnesses built around them) for getting good at one language's idioms and one style of bug-fixing task, not software engineering in general.

There's a second, quieter problem that matters more for buyers than contamination does: scores had bunched up near the ceiling, with frontier models converging within a fraction of a point of each other, while the actual gap in usefulness between those agents in production was obviously not that small. When a benchmark saturates, differences in reported score stop separating good agents from bad ones — they mostly separate who tuned their scaffolding against the public leaderboard most aggressively. One analysis of SWE-bench Pro reportedly found that swapping only the agent scaffold — the orchestration and tool-use logic wrapped around the same underlying model — produced a swing of over 20 percentage points. If that's representative, the harness is doing more work than the model, which is precisely the opposite of what a model-comparison benchmark is supposed to tell you.

What this means if you're the one buying, not building

None of this means SWE-bench was worthless, or that benchmarks in general are theater. It means a specific number, treated as a specific kind of proof, stopped being proof. A few practical conclusions follow for anyone evaluating a coding agent or an AI dev-tools vendor rather than building the model itself:

A leaderboard score is a screening signal, not a due-diligence result. It tells you a vendor cleared a bar other vendors also cleared. It does not tell you the agent will handle your codebase, your stack, or your team's actual bug reports — especially if your stack isn't Python-heavy open source, which is exactly the part of the world SWE-bench underrepresents.

Ask what the benchmark number is actually protecting against. SWE-bench Pro's pitch is held-out tasks and private repos specifically so the model can't have memorized the answer. Terminal-Bench, a separate 2026 benchmark from Stanford and the Laude Institute, takes a different angle — real command-line workflows instead of single-file Python patches — and frontier agents score well under two-thirds on it, which at least means there's real headroom left to measure. A benchmark with headroom left is more informative than one everyone has already maxed out.

Treat scaffold and model as separate purchasing decisions. If orchestration logic can swing results by double digits, then "which model powers this agent" is a smaller part of the buying decision than "how is this vendor's harness built" — how it retries, how it decides a task is done, what context it keeps between steps. Ask vendors to show their harness decisions, not just their model's name and a score.

Run your own held-out set before you sign anything. Pull 15-20 real, closed tickets from your own repo — the kind your team actually handles, not toy bugs — and have the vendor's agent attempt them without ever having seen your codebase before the trial. Judge each result the way a reviewer would judge a pull request: would a human on your team actually merge this, or does it pass the tests while missing the point of the ticket. That's a smaller, slower benchmark than SWE-bench, and it's the only one that can't be contaminated, because you wrote the tasks yourself.

The takeaway

The lesson from OpenAI retiring its own favorite coding benchmark isn't "benchmarks are fake." It's that any benchmark a vendor didn't design around your use case is measuring something adjacent to the question you're actually asking. Before you buy a coding agent on the strength of a leaderboard number, run it against twenty tickets from your own backlog and judge the output the way your own reviewers would — that's the only score that was never for sale.