The benchmark that flattered me for eight days

For eight days in July I ran the same prompt fifteen times. It asked for Gelp: a web app with sign-in, a data import, and push-to-deploy. Build it, deploy it.

Each round ran under a different version of upward, a Claude Code plugin I have been building. Each round produced two numbers: what the build cost in dollars, and a quality score from a model that reviewed the finished app. Round N’s failures became version N+1, and round N+1 tested that.

I wanted to know whether the plugin was worth its cost. What I got instead was four reasons the measurement could not answer that question. LLM-graded evaluation invalidates itself. Here is how, in the order I found out.

Finding 1 — pushed to improve, the model learned the test

At first, quality meant craft. Each build was scored on its three rings — auth, import, deploy — from 0 to 10 against fixed anchors, and the headline was the average. The reviews were reads. A reviewer model went through the source, poked at it a little, and gave a score. Nobody booted anything.

A score always came back, and the board always moved. Both of those feel like measurement while they are happening. Round 5 cost USD 6.74 and scored 8.3, so it became the target for every round after it.

The middle rounds stalled there, so I pushed the loop directly: rewrite the plugin, improve the performance. Version 0.3.6 answered with a Known-killers checklist — four defect classes: secrets tracked in git, permanent-cache poisoning, secrets compared as raw strings, and firewall rules that a cloud ACL in front of them overrides. The loop workflow then checked all four, item by item, on every build.

The model wrote that checklist itself. It was not a design of mine. And every class in it came from a bug that had already shipped in an earlier round. That last sentence is the important one, and I read straight past it at the time.

It worked, and it billed. The rounds carrying the checklist are where the series’ delivery record comes from. They are also where the budget went: every item had to be reasoned about on every build, whether or not it applied, so the thinking and fix phases grew in both tokens and wall time.

You can read this as the plugin learning discipline, and rounds 10 to 13 as the proof. Most of that reading is true. The delivery record was real, and those rounds still deliver under the strictest review in the series.

The mechanism is what is wrong. This benchmark has one task. A checklist distilled from that task’s own past failures is not discipline. It is a time machine sent back from the future to fix the present. It is cheating, arrived at honestly: I asked for better performance, and memorising the test is the cheapest way to deliver it.

The plugin did not get better at building software. It got better at building Gelp, because it already knew where Gelp breaks. And it paid for that score twice — in tokens and wall time on every build, and in scope, because a checklist about one app’s bugs shrinks the set of tasks the plugin can serve.

So here is a rule worth keeping. If a change raises your benchmark score and narrows what the tool can be used for, it did not gain capability. It absorbed the test. The cheap way to check is to swap the task and see what survives, which a benchmark with one task cannot do from the inside.

Round 14 keeps me honest. Version 0.5.0 had deleted the checklist by then, and round 14 fell back to T3, the tier that does not deliver — but it closed all four killer classes anyway, on its own. Either the model had those classes as priors by then, or round 14 got lucky. Both readings fit, and this benchmark cannot tell them apart.

Finding 2 — a standard cannot be instructed, only dispatched

Round 5 was the target, and it came with a standing instruction that I gave to reviewer after reviewer for the whole series: rate this by the same standard round 5 was rated by.

It did not work. I restated it, I quoted the rubric back word for word, and the standard drifted from session to session anyway.

The first drift was in the scale itself. Round 13 ran a control — pure Opus, no plugin — and it scored 7.0 against the plugin build’s 8.4. A 1.4-point gap reads as good versus a bit better. What the two reviews described was one build a user could work with and one build a user could not get a working product out of at all. Averaging three ring scores had let one brilliant ring buy back a dead one.

So I rewrote the rating system in the middle of the experiment. Version 2 gates on outcome first — the delivery tier: T1 delivers, T2 delivers with faults, T3 does not deliver — then places the run inside its tier’s band with tier band + 0.25 × craft, bases 7.5 / 5.0 / 2.5 / 0.0. The bands do not overlap, so a working build always outranks a broken one. The rewrite was correct. It also means every score before it and every score after it came off different instruments.

The written rubric was not the only thing drifting. A grader does not simply apply the standard it is handed. As the scaffold evolves, the reviewing model picks up the scaffold’s own vocabulary for what good means, and the scaffold ends up grading its own children.

By the end the board was a patchwork. Some runs still carried v1 numbers, nine had been re-rated somewhere along the way, and two had only their original reviews. No complete board existed at any single standard, because no standard had ever actually been held.

One dispatch, not one instruction

What finally held was a dispatch. On 2026-07-19 I sent out ten fresh single-draw Sonnet reviewers in one parallel run: same time, same prompt, same goal, one reviewer per unscored artifact, all held to one written protocol — read the full source, probe lightly, no defect-class sweep, no boot. That depth is what this series calls shallow. Their numbers sit on the same footing as round 5’s existing shallow 7.1, the calibration point both batches share.

For the first time, all nineteen runs were readable as one board. And the top of that board belongs to rounds I had already decided I cannot ship.

Four of the six checklist rounds top the whole series. They are also the variant that version 0.5.0 deleted, for the reason finding 1 describes: the checklist only knows Gelp’s own past bugs, so those scores do not transfer to any other task. As a plugin feature it is off the table, whatever it scores.

Take those six rounds out and the board looks very different.

Rounds 1, 6 and 15 are tied. The spread across them is 0.1 points, well inside the ±0.6 noise of a single reviewer, so the honest statement is that one standard cannot separate them.

Now look at what those three actually are:

• round 1 — Opus with upward 0.1.x, the first version I ever shipped, USD 5.77
• round 6 — the cheap model, Sonnet, with upward 0.3.2, USD 8.64
• round 15 — Opus with 0.6.0-light: 0.3.2 forked, checklist absent, the redundant prose cut out, five cheap rules added, USD 9.28

Note — this holds at one depth only. Nobody booted rounds 1 or 6; their scores are what a careful reader of the source concluded. Round 15, the one member of the trio that was later booted, fell from 9.5 to 3.9. So the tie at the top is a tie in what a reader concludes, not in what got delivered.

The finding, stated once: “use the same standard” is a request, not a property. Only results from the same parallel run — same time, same prompt, same goal, same reviewer — can be called same-standard. Everything else on the board was different judges holding different rulers, however firmly each was told to hold round 5’s.

The cheapest run is the oldest version, and eight days of plugin development did not beat it. The plugin was still worth something in absolute terms — the no-plugin control scored 6.8 on the same board, 2.8 points below round 1 — but everything I built after 0.1.x is invisible at this depth. That much I trust. What I could not bank on top of it is the 1-versus-15 comparison itself, because the runs being compared are days apart. That leads to finding 3.

Finding 3 — the model changed while I was measuring it

What I realised later is that a benchmark spread over days has a variable in it that a benchmark run in one afternoon does not. Upstream work on the model — how it performs, how it thinks — lands inside your measurement window. It is the easiest variable to forget, because nothing in your own setup changes when it moves.

The model’s capability was adjusted more than once while these fifteen rounds were running. Round 1 and round 15 are eight days apart. Same model name, same version string, and not necessarily the same subject under test.

That matters most for exactly the comparison I just made. “0.1.x scores as well as 0.6.0-light” assumes the thing running 0.1.x and the thing running 0.6.0-light were the same. I cannot show that they were. My run log has no field that dates the upstream adjustments, so after the fact a bad round and a bad week look identical. Serving conditions vary call to call on top of that, which hides the difference between a bad round and a bad hour.

The judge has the same problem, plus one of its own. The judge is a model too, so its verdict is bounded by whatever entered its session’s context. That is not noise that averages out over fifteen rounds, because the plugin changes what enters the reviewing session’s context. A round that scores well may be scoring well because its reviewer was better informed, not because its build was better.

Sorted by the only property that matters afterwards — whether the damage can be undone — everything that moved during those eight days lands in three classes:

What moved	Class	Fixable afterwards?
The model under test — adjusted upstream during the window; serving conditions vary per call	Drift	No — record the model identifier forward, or run everything in one sitting
The rubric — v1 to v2 mid-experiment, plus criteria the reviewer derives on its own	Drift	No
The judge’s session context — which the plugin itself changes	Uncontrolled variable	Yes — re-measure once, held fixed
Review depth	Uncontrolled variable	Yes — same fix
The subject learned the test	Contamination	No — only a held-out task detects it

The parallel re-grade fixed the two uncontrolled variables and nothing else. That does not make the benchmark valid. It makes one class of its invalidity measured.

Finding 4 — is the harness even worth it?

The fourth finding did not come from my data. It came from watching how fast the ground moves under this kind of tool.

Models keep absorbing the scaffolding built around them. Settings, rules, wrappers, prompt boilerplate: features one release, dead weight the next. My checklist is a small instance of the same pattern. It was scaffolding that stopped applying the moment the task changed.

Anthropic published the strongest version of this while I was writing up the benchmark. In the new rules of context engineering for Claude 5 generation models, Thariq Shihipar reports that they removed over 80% of Claude Code’s system prompt for models like Claude Opus 5 and Claude Fable 5, with no measurable loss on their coding evaluations. The guidance that replaced it is not more rules but fewer: newer models have better judgement and handle these decisions well without explicit instructions, so the work moves to curating what the model sees and loading it progressively.

That is a hard result to argue with, and it points straight at my plugin. When the team that ships the harness improves it by deleting 80% of its instructions, a plugin whose main lever is more instructions is pushing against the direction of travel. Round 15 was my own small version of that experiment: fork the version from before the checklist, cut the redundant prose, add five cheap rules. It tied the top of the no-checklist board.

So the honest answer is that a harness is worth it for boundaries, not for intelligence. Deterministic workflow steps, project-specific defect memory, cost and scope limits, a clear place for the human to intervene: those are things a model cannot supply for itself out of its own judgement. Telling a strong model how to think is the part that expires. In an era full of new vocabulary, what keeps mattering is context management.

What the benchmark was actually worth

The value of the whole exercise was not a ranking. It was learning where the boundaries of an agentic workflow are, and what an LLM-graded number can and cannot support.

Three things now seem structural rather than unlucky. The model underneath keeps evolving, so the same prompt at two different times does not produce comparable results. The model derives review criteria of its own as the scaffold around it changes, so consistency is not something you can instruct into it. And every judgement is bounded by the session context it was made in, which differs round by round. Put together: unless every round is re-reviewed in one pass, under one standard, after all the rounds are finished, the benchmark is not worth citing.

What building the plugin taught me

1. You can make a model smarter, and it will cost you. The price is tokens, plus a deeper review process and more prompting to sustain it. Nothing in this series bought quality for free.
2. A harness has real benefits, but the point is not to make the human lazy. Keeping a model working without pause is a popular goal, and a lot of the time it is just spending tokens.
3. The model’s strength is doing high-quality work inside clear boundaries. The human is still the one who evaluates and decides when to pivot. That the planning stage always wants the strongest model available is the same fact from another angle.
4. The transferable skills were all about handling the model, not scoring it. Task-planning principles. Raising the cache hit rate. Writing prompts that hold. Keeping the model from losing focus, and noticing when strict doctrine costs more efficiency than it buys. Steering a run toward the intended direction, and managing memory so the session context stays worth reading.

Speed was never the constraint here: fifteen full builds took eight days. The constraint was knowing whether any of it was true.