We are building a pipeline in which a bug report filed from inside NAOMS is picked up and worked by an AI coding session on its own. A person describes what went wrong (a report you get to read before it leaves your machine), and a session investigates the code and proposes a fix, with no one having to route it by hand.

Work that runs on every report is metered work: every job has a price. So before settling on which model does it, we asked a narrow question and measured the answer. Can a model cheaper than Claude's Sonnet 5 or Opus 5.5 do this job, and what does one job cost, in tokens and in money, on bugs from our own history?

How to test a bug-fixer when you already know the answers

The trick is to use bugs that are already fixed. For each one, version control keeps the code as it stood just before the fix landed. Put a model in that older copy, hand it the original complaint, and see whether it finds and repairs what the real fix repaired. The real fix is the answer key.

Two precautions make this honest. We deleted the project's internal notes from the test copy, because bug write-ups there often name the fix outright. And we checked each prompt so that no file or function name from the real fix leaked into it — though in 4 of the 13 cause-finding cases the symptom text itself is found only in the file at fault, so a text search can point to it.

Every model ran inside the same harness — the program that wraps a model, gives it tools such as "read this file" or "search for this word", and loops until it is done. Here that was Claude Code run non-interactively, with only the model swapped. Each round of that loop — the model thinks, asks for a tool, reads the result — is one turn. We capped each job at 60 turns. In the fixing rounds the models could read, search and edit files, but could not run anything: no shell, no tests of their own. In the cause-finding round they could only read and search.

We ran three rounds:

1. Find the cause. Thirteen fixed bugs, in Rust, TypeScript and JavaScript. The model had only the symptom and had to name, in ranked order, the file and function at fault. We counted how often the right file came first.
2. Fix it, with the tests handed over. Five of those bugs, all TypeScript or JavaScript. Along with the report, the model got the regression tests that shipped with the real fix — tests written to fail while the bug exists and pass once it is gone — and was told not to edit them.
3. Fix it from the report alone. The same five bugs, no tests. This is what the real pipeline faces.

Rounds 2 and 3, run this way, turned out to test the wrong thing, and we later redid the fixing question properly — see "The wrong test, and the right one" below.

flowchart LR
  R["an already-fixed bug:
the original report"] --> T["the code as it was
just before the fix"]
  T --> M["model reads, searches, edits
(up to 60 turns, no shell)"]
  M --> P["the model's patch"]
  P --> G1["run the real fix's tests"]
  P --> G2["two blinded AI reviewers"]
  P --> G3["does the code still load?"]

The six models were Opus 5.5 and Sonnet 5 from Anthropic, DeepSeek V4 Pro and DeepSeek V4.1 Flash, and Xiaomi's MiMo V2.6 Pro and MiMo V2.6 Flash. "Pro" and "Flash" are each vendor's larger and smaller tiers.

What came back, finding the cause

How to read it:

• "Cause found" counts the right file ranked first. The two Flash models gave no wrong answers in that round; their misses were jobs that ran out of turns without answering (three for DeepSeek Flash, four or five for MiMo Flash, which also gave one correct answer in prose instead of the required format).
• Every figure is one run per model, thirteen bugs. A one-case difference in this table is not a ranking.

The fixing rounds are a separate story, and the one worth trusting is the one further down this page — the numbers for those first two fixing rounds are superseded and are not repeated here.

Price per token is not price per bug

The cause-finding round already shows this on its own: DeepSeek V4.1 Flash answered 5 of 13 cases within 30 turns and 10 of 13 within 60, all at a few cents a job, and it never gave a wrong answer — its misses were turns running out, not wrong guesses. A cheap model that needs more turns is not necessarily a worse deal; what matters is the price times the turns a model needs to finish, not the list price per token in isolation.

The fixing round, redone properly (see below), sharpens the same point. DeepSeek V4 Flash needed up to 80 tool-use steps and 6.5 to 12 minutes of wall time per job under our real bug-fixing procedure, and still finished at $0.042. MiMo V2.6 Flash took longer still — 26 to 30 minutes per job — and cost more, $0.053, while also fixing fewer of the five bugs correctly. Slower and cheaper is not the same claim as slower and better; on this set, DeepSeek beat MiMo on both cost and correctness.

Each turn resends the whole conversation so far, so a model that needs many turns pays to re-read its own accumulating context every time, discounted by caching. On an earlier run of our triage job with Sonnet, a typical report involved about 747,000 tokens re-read from the conversation cache — a discounted store of text the model has already seen — against 5,600 tokens of actual output. Most of what you pay for in an agentic job is the model re-reading its own context, not producing new text. (See "Cutting the fixed prompt" below for one way we cut that re-read cost.)

What this job costs on a Claude subscription instead

Everything above prices Claude at the API's pay-as-you-go rate. Plenty of teams instead pay Anthropic a flat monthly fee, so we measured what that changes.

Anthropic's Claude Max 20x plan lists at $200 a month plus tax. It buys a five-hour session limit and a weekly limit shared across every model and across the Claude app and Claude Code, with no published token or dollar figure attached to that weekly limit — so we measured it ourselves, the same way we measured everything else here: we priced the work done in a week at API list price and divided by how much of the weekly limit that work used. By that measure a job on Max costs about 1.75% of the same job's API list price (our measurements ranged 1.4%–3.1%). Applied to a Sonnet-sized job like our own report-triage job — $0.53 at API list with the one-hour cache — that is roughly $0.009 per job, cheaper than the cheapest candidate models measured here (MiMo Flash and DeepSeek Flash, estimated at $0.015 to $0.036).

That number has real limits. It does not carry over to the bug-fixing job above without re-measuring it, which we have not done. It assumes Max weighs every model's usage in proportion to its API price, which Anthropic has not published; its help center says only that Opus costs several times more per turn than Sonnet. It is one week of measurements, not a guarantee for any other week. And a subscription is a different product from pay-as-you-go: its five-hour and weekly caps put a hard ceiling on how much work one seat can push through, however cheap each job looks by this arithmetic. Anthropic's own usage terms also describe the advertised Pro and Max limits as assuming ordinary, individual use of Claude Code, and say that teams building an automated product or service on it should use metered API access instead.

The wrong test, and the right one

We first measured fix quality by giving each model only the report, with no shell and no way to run its own patch or any test — not even the real fix's own regression tests, in the round where those were withheld. Under that setup, hidden tests mostly graded whether a model happened to choose the same variable and field names as the original author, and two blinded AI reviewers who agreed on every one of thirty grades still both missed the same two patches crashing the moment the code loaded. That taught us something about grading, but nothing trustworthy about which model fixes bugs best.

The owner ruled that round invalid. NAOMS's own procedure for fixing a bug, PROC-BUG, requires reproducing the reported failure as a failing test before touching production code, then fixing it, then proving the fix green. A model with no way to run a test cannot do that step, so it cannot be said to have followed our own procedure — it can only be said to have guessed and gotten lucky or unlucky. We redid the fixing question with that gap closed.

Fixing it under PROC-BUG

This time, DeepSeek V4 Flash and MiMo V2.6 Flash each got a tool to run tests alongside the usual read/search/edit tools, and a prompt that mirrors NAOMS's own seeded PROC-BUG steps: reproduce the reported failure as a red test, fix the minimal root cause, then prove the reproducer and the affected tests green. We did not run Claude under this protocol — we stopped the Claude runs to save the shared Claude account's usage — so nothing in this table compares directly to Opus or Sonnet. (The model names are the providers' own: the earlier rounds called DeepSeek through its deepseek-flash alias, recorded as V4.1 Flash; this round used the model id deepseek-v4-flash. We have not confirmed whether those are the same model, so read the two DeepSeek Flash results as separate.)

Giving DeepSeek a real test tool cost about 60% more per job than letting it guess without one — and it removed every failure the same model had produced without one, on this small set. That is the owner's point, measured: without the ability to run a test, "fixed" is just a guess dressed up as a patch.

On one MiMo case, the model changed a key's name in its output, and the reviewer failed it because a downstream test consumer expects a different name. We overruled that grade on the historical record: the test consumer the reviewer relied on was added to the codebase a month after the real fix shipped, and the model's patch actually matched an existing precedent elsewhere in the project. We keep that grade marked with an asterisk in our own notes — a reviewer relying on evidence from the wrong point in the project's history is itself worth remembering.

A few honest limits. This is five bugs, one run each, per model. The reviewer this time was a Sonnet model that had already seen the real fix before grading, so it was not a blind judge — it is a check against a known answer key. MiMo's dollar figure is computed from its reported token counts times its published rates, not read from a bill; DeepSeek's uses NAOMS's own pricing table.

A local model, on our own hardware

We also tried a bug fix on a model running entirely on our own GPU, instead of any cloud API: Qwen3.8-27B on a single RTX 5090. There is no full test set here — this is one bug, one run, and the model ran without a test tool. It got the fix right (the real fix's hidden tests passed 5/5), but it took 57 minutes, against 6.5 to 12 minutes for the cloud jobs — which were also running tests, so the local model was slower while doing less. It ran at a GPU clock capped to keep the card stable — about 67 tokens a second. The owner stopped further local runs as too slow to pursue right now, and we never put it through PROC-BUG. Running the model locally removes the per-token bill (electricity remains), but on this one data point, the wall-clock cost was far too high to call it a candidate yet.

Cutting the fixed prompt

Separately, we measured what Claude Code's own harness costs before a model does any work at all. Its fixed per-call prompt — the tool schemas, instructions and project files sent on every single turn — measured 24,759 tokens by default. Restricting which tools are advertised to the model and dropping the auto-loaded CLAUDE.md/AGENTS.md context cut that to 3,543 tokens, with no change in which tools the model could actually use to fix a bug. Separately, prompt caching was already serving 96 to 97% of a job's input tokens from a steep, discounted cache rate rather than full price — a leaner fixed prompt shrinks what that cache has to serve on every turn, starting from the first one. This applies to work run through Claude Code's own harness specifically; the PROC-BUG numbers above use NAOMS's own harness and are unaffected by it.

A failed experiment

We also tried a second job: routing. Given a report, which of the project's roughly 1,600 work items does it belong to? For each of 68 cases with known answers, a model saw the report and a list of every item's short name, with no tools, and picked one.

All six models scored at or below 8.8% — the score you get by always answering the most common item. Random guessing across the 36 possible answers gives 2.8%, and the models hovered around that. We stopped the runs early.

The cause was the test set, not the models. When the set was assembled, every report containing a word from its own item's name had been removed, so that a model could not win by simple word-matching. The set's own notes said so. A list made of nothing but those names therefore carried almost no information about the right answer, by construction. Six very different models all near zero meant the experiment was measuring itself. Routing well likely needs what the fixing job had: the ability to search the code.

What we can and cannot claim

On the cause-finding round, the two Pro models matched or beat Sonnet and Opus at naming the right file, at a fraction of the cost order Claude has shown elsewhere in this project. On the fixing round, redone under PROC-BUG, DeepSeek V4 Flash fixed every one of five bugs correctly and cheaply; MiMo V2.6 Flash fixed three outright and still cost more per job, despite also being a "Flash" tier model. We have not run Opus or Sonnet under PROC-BUG, so we cannot say where Claude would land on this exact protocol — only that, between these two, DeepSeek was both more accurate and cheaper; and for DeepSeek itself, adding the test run turned three or four correct fixes into five, and raised the price by under two cents a job.

That is five bugs and thirteen bugs, mostly one run each, fixes in TypeScript and JavaScript, and — for the PROC-BUG round — two models out of six, with a reviewer that had already seen the answer. It does not show that any of these models is as good as Claude in general, and none of it extends past this small sample. It also leaves out questions a price table cannot answer, such as where each provider processes your code and what it may keep. (NAOMS's own rules for which model is allowed to receive data are a separate system, described in there is no ungoverned route to a model.)

The method, though, carries over to any team picking a model for agentic work:

• Measure the cost of a finished job, because token counts vary several-fold between models.
• Grade with an answer key from your own history, and test the grader with a patch that should pass and one that should fail.
• Require the model to run its own tests before calling anything fixed. A model with no way to check its own work will occasionally call a broken patch done; ours did, until we gave it a way to prove otherwise, and that requirement more than paid for itself in removed failures.
• Read a uniform score as a warning about the test, not a verdict on the models.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.