The spread test:
why 62 points beats 6.4

DeepSeek's DeepSWE score moved from 12.8 to 62.7. That is a 4.9x jump in one release cycle. On the same benchmark family, V4-Flash went from 7.3 in preview to 54.4 official.

Damaging admission before anything else: I cannot confirm those two DeepSWE numbers came from an identical measurement. The 12.8 is listed for V4-Pro-Preview. The 62.7 is listed for V4-Pro Max, a maximum-reasoning configuration. Some of that "leap" may be a dial being turned, not a model being rebuilt.

Here is the part nobody disputes. On the old coding leaderboards, the top four models sit within a few points of each other. On the new agentic ones, they spread by more than 60 points. That gap is the whole story, and it should change how you pick a coding agent this month.

The Spread Test

A benchmark is only worth reading while it still spreads the field.

Run the numbers. A May 28, 2026 benchmark audit put the SWE-Bench Pro leaderboard at GPT-5.5 82.6%, Claude Opus 4.7 82%, Gemini 3.5 Flash 79.8%, and DeepSeek V4 Pro 76.2%. Top to bottom, a 6.4 point spread. Read that table by itself and every frontier model looks interchangeable.

Then a four-person team at Datacurve built DeepSWE from scratch and ran the same models. GPT-5.5 hit 70%, Opus 4.7 hit 54%, Gemini 3.5 Flash hit 28%, and DeepSeek V4 Pro hit 8%. That is a 62 point spread. Same field, resolved nearly 10x wider than SWE-Bench Pro manages, 62 divided by 6.4.

So sort every benchmark you read into two buckets. Saturated rulers compress the field and tell you nothing about your buying decision. Resolving rulers separate the field and predict something. The Spread Test is the only filter you need: look at the gap between first and last place before you look at anyone's score.

The 500 IQ Intern Who Cannot Find the Light Switch

Static coding benchmarks measure whether a model writes a good patch when you hand it the problem. Agentic benchmarks measure whether it can find the problem first. Different job, different failure modes.

DeepSWE, Terminal-Bench 2.1, DSBench, and NL2Repo all test the second thing. The model gets a repo, a shell, some tools, and a goal. Then it has to navigate files, run commands, read errors, and recover when step four blows up. DeepSeek's own agent materials list V4-Flash official at 82.7 on Terminal-Bench 2.1, and release coverage puts Pro-tier Terminal-Bench numbers up near 87.9, though the figure shifts depending on which variant and tracker you read.

An agent is a 500 IQ intern. It can write a beautiful function. It also cannot find the light switch in your build system without twenty minutes of flailing. Static benchmarks grade the function. Agentic benchmarks grade the flailing.

Here is why the old rulers broke. In February, OpenAI published that it no longer evaluates SWE-bench Verified, because GPT-5.2 could reproduce a gold patch verbatim from a single sentence of task description. Claude Opus 4.5 could recall the exact inline comment text from a specific pull request diff line. When the answers are on the public internet, the test measures memory, not engineering.

Which is how one benchmark page ends up listing three different "best" SWE-bench Pro scores at once: 59.1% for GPT-5.4 xHigh on Scale's standardized public set, 80.0% for Claude Fable 5 in a vendor aggregate, and 47.1% for Claude Opus 4.6 on Scale's private commercial set. All three are real. Most articles quoting a score never say which set they mean. An ounce of care in picking your ruler saves a pound of pain in production.

Now translate it into money, because that is where this bites. V4-Pro output tokens run $3.48 per million against $25 for Claude Opus 4.7 and $30 for GPT-5.5, roughly a 7.2x and 8.6x gap. The benchlm profile for DeepSeek V4 Pro (High), released August 13, 2026, lists $0.43 input and $0.87 output with a 1M context window. On paper that is not close.

Back-of-napkin, using DeepSWE's own success rates. Say an agentic task burns 1M output tokens per attempt. At 8% success you need about 12.5 attempts, so 12.5 times $3.48 is $43.50. At 70% success you need about 1.43 attempts, so 1.43 times $25 is $35.75. The cheap model costs more per finished job.

That is the number to build your evaluation around. Not score. Not price per million. Cost per resolved issue, measured on your repo. Reviewers already report high token burn and slow responses on V4 in real workflows, which is exactly the tax that napkin math exposes.

Leaderboards are Ferraris. Beautiful, fast, and they do not haul anything. Your own twenty-task eval on your own codebase is a tractor. Ugly, boring, and it does the work.

Three signals inside the same shift

2031

Zoom out and the benchmark race matters less than what it reveals about positioning.

In April 2026, NIST's Center for AI Standards and Innovation evaluated DeepSeek V4 Pro and concluded its capabilities lag the frontier by about eight months. That is the whole open-weight thesis in one sentence. Eight months behind, MIT licensed, roughly 7x cheaper on output tokens, 1.6 trillion total parameters with 49 billion active. Closed labs sell certainty. Open weights sell optionality.

Over a five-year arc, that trade compounds in a specific direction. If your workload tolerates an eight-month capability lag, you get frontier-adjacent performance at commodity prices and you keep the weights. If it does not, no discount saves you. The asymmetry is not about which model wins. It is about which workloads you can afford to route to the cheap tier without your customers noticing.

The second-order shift is who holds the pen. A four-person team shipped a benchmark and moved the procurement conversation for the entire industry. Benchmark authors are becoming kingmakers, and I think that power is under-priced right now. Whether DeepSWE itself survives contact with optimization pressure is an open question, because every ruler eventually saturates. That impermanence is the pattern, not the exception.

My read on this: the 12.8 to 62.7 story is real evidence of fast iteration, and weak evidence of supremacy. NIST also flagged serious safety gaps in earlier DeepSeek models, including jailbreak susceptibility and agent hijacking risk. Those risks scale with autonomy. A model you hand a shell and a repo is a very different threat surface than a chatbot.

What to Build This Weekend

Build your own resolving ruler. It takes one afternoon.

First, pull 20 closed issues from a repo you actually maintain. Pick ones where the fix is already merged, so you have a known-good answer. Write a simple pass or fail check for each, usually just the test suite that shipped with the fix. That check is called a verifier, and it is the difference between an eval and a vibe.

Second, run three models against those 20 tasks with the same scaffolding. Same prompt, same tools, same retry limit. Log tokens in, tokens out, wall-clock time, and pass or fail.

Third, compute cost per resolved issue. Total spend divided by tasks passed. That single number will contradict at least one leaderboard you currently trust, and I would bet on the contradiction being large.

Fourth, before you hand any agent write access to your repository, read the tool page. Directory entries for autonomous dev agents like Auto-Dev list the execution surface and the permissions the thing requests. Read that first. An agent with shell access and no sandbox is not a productivity tool, it is a liability with good marketing.

Things will break. Half your tasks will fail for stupid reasons like a missing environment variable or a stale lockfile. That is normal and it is the point. Fix the harness, rerun, and get your reps in. When you need a break from the terminal, Eyedropper gives you one random color a day and sends you outside to photograph it, which is a better reset than refreshing a leaderboard.

Start with 20 tasks. Not 200. Simple always defeats complex, and a small ruler you trust beats a big one you cannot reproduce.