What ralphctl found when it finally measured itself

Where this picks up

The first time ralphctl reported on itself, it said 84.3% of tasks passed on the first attempt and that not one acceptance criterion had failed. Both numbers were true. Both were also narrower than they looked. They describe 36 of the 105 tasks in the report and say nothing about the other 69, because those ran before the harness could write a verdict down.

That is this post. Eleven releases, v0.14.0 released on 30 June through v0.21.0 on 24 August, plus six commits sitting on top of the tag at ac3110ef.¹ 294 commits. The TypeScript under src/ went from 742 files to 914 and from 82,541 lines to 113,390, which is the honest way to say it: the repo's tracked file count went 1,380 to 1,769, so an unqualified "742 to 914 files" would be off by nearly half. Production dependencies over that whole stretch: still exactly seven, and I can list them, which is the only version of that claim worth anything. Commander, cross-spawn, ink, proper-lockfile, react, typescript-result, zod. Breaking changes: zero. Every BREAKING marker in the changelog predates v0.14.0, and the newest migration note says "Nothing to do by hand." The last post was the month I stopped breaking your data. This one is what came after, and it is additive the whole way down.

The last post asked whether I could leave the harness running overnight. This one asks whether the overnight run was any good. For four articles' worth of releases I could not answer that. I tuned prompts, flipped defaults and rearranged escalation rungs by feel, shipped them, and felt better afterwards. Feeling better is not a measurement. So I built the report and pointed it at my own data root.

The line the first report drew

Start with the thing you should not have to wonder about, then look closely at where its proof lives. The checks themselves, typecheck plus lint plus the full test suite, came back green on both sides of the line this section is about. Of the 69 tasks that predate the instrument, 68 were attempted and 67 of those have a green run of the suite on record; the exception is a blocked task whose only attempt aborted before the checks could run, and the 69th is the todo 1 in the block below, never attempted at all. But only three of those tasks carry harness-recorded exit codes, from the two May sprints where a verifyScript was configured. For the rest, the green survives as command transcripts inside the loop's round artifacts. Keep that distinction in hand; it is this post's subject wearing a different hat.

ralphctl runs stats shipped in v0.20.0. It is a read-only fold over data the harness had been persisting all along: every attempt, every verdict, every settled task, with no new writes and no new schema. It answers the questions I had been guessing at, including the attempts-to-done distribution, which escalation rung resolved each stall versus fell through, and per-criterion pass rates.

Here is what it said the first time I ran it, across 21 sprints and 105 real tasks of my own usage. This block is pasted from the commit that shipped the feature, where I recorded it as the pre-flip baseline:

Harness outcomes — 21 sprints · 105 tasks

Summary
  first pass     86/102 done on attempt 1 (84.3%)
  criteria       126/400 passed (31.5%) · 0 failed · 274 unknown · 36 tasks graded

Outcome mix
  done           102  (clean 102 · with warning 0)
  blocked        2
  todo           1

Plateau
  tasks          1 task (1.0%)

Escalation rungs
  rung              granted  resolved  fell-through  unsettled
  model                   1         0             1          0

So what are the 274? I went into the data root and correlated every task's recorded criteria verdicts against its sprint date. The answer is a step, not a slope:

BEFORE v0.14.0 (released 2026-06-30):    0 / 69 tasks graded
FROM   v0.14.0 onward:                  36 / 36 tasks graded

Not a single one of the 69 before the cut. Every one of the 36 after it, no exceptions. v0.14.0 is the release that made the evaluator emit a structured verdict per criterion, and it lands exactly on the seam.

That changes what 0 failed means. It is not a suspiciously green number covering an absence. It is a real result over the 126 criteria that were actually graded, at full coverage every day since grading existed.

The denominators are worth a second look too, because they are not the same number twice. First pass is 86 of 102, not of 105: it is a rate over tasks that actually reached done, and the two blocked tasks and one still-open task are not in it. Criteria pass rate is 126 of 400, which counts every criterion ever written, recorded or not. Two rates, two populations, and the report labels each one with the denominator it is quoting because I got that wrong on paper before I got it right in code. A first-pass rate is a claim about the tasks that finished. It says nothing at all about the ones that did not.

Here is the part that is still uncomfortable, and it is smaller and more interesting than "I wasn't testing." The first thing a new instrument tells you is how much of your own history it can never reach. The 68 tasks that ran came back green wherever their checks ran, and for all but three the proof is a transcript in a round artifact, not an exit code the harness wrote down. What I will never have is a per-criterion record of them, because you cannot grade the past retroactively. You can only start recording, and from that day forward the record is complete. Anyone who has bolted observability onto a running system has hit this same wall: the dashboards are flawless from the moment you deploy them, and blind for every incident before.

"Unknown" is not "passed"

An acceptance criterion in ralphctl is a line the planner writes into the task before any code exists, phrased so a second model can check it: what has to be true for this task to count as done. The evaluator grades each one, and it always did. It is worth being precise about what was actually missing, because it is narrower than "the instrument didn't exist". The inputs existed and were machine-checkable: a task from before the boundary carries a populated verificationCriteria array, entries like {"id": "C1", "check": "auto", "command": "pnpm typecheck && pnpm lint && pnpm test"}, and its round artifacts record that command coming back green. The v0.14.0 changelog says the same thing from the code's side, noting that structured verdicts supersede "the earlier free-text <criterion_checkpoint> reasoning protocol, which is removed." The criterion was written. The command ran. The answer came back as prose, and prose is not a record.

So the missing artifact was one thing: criteriaVerdicts, a place to write the answer down. Four pieces of work got it there. Only one of them changed what the evaluator checks. The other three changed whether its answer survived.

The verdict was already written. Nothing could read it. Every evaluator round in this dataset, back to the earliest persisted sprint in late May, wrote an operator-readable evaluation.md beside the round's artifacts, with each dimension's pass, fail or n/a plus the finding behind it. Until v0.20.0, nothing in the TUI or the CLI ever opened that file. A failed attempt showed you eval failed · attempt 2 and stopped there. The evidence existed on disk and the product had no door to it. v0.20.0 added the door: press v on a focused task for a scrollable overlay, or run ralphctl task evaluation <taskId> to print the same file for a pipe.

That is a humbling class of bug. Not a wrong answer, a right answer nobody could reach. If you can only see the aggregate, you will trust the aggregate.

Verdicts stopped being prose. In v0.14.0 the evaluator started emitting a structural verdict per criterion, pass or fail plus a one-line evidence citation, and the harness folds those into a per-task map from the signal itself rather than parsing them back out of the model's write-up. A schema refinement rejects an overall "passed" that lists a failing criterion, which is the kind of contradiction free-text grading lets through without complaint.

The rubric grew a fifth floor and a real N/A. Robustness joined correctness, completeness, safety and consistency, and a Zod refinement now rejects any terminal verdict missing a floor dimension. The rule I care about more is the one on N/A. An explicit not-applicable has to carry a non-empty finding, same as a fail does. "Not applicable" can no longer be silence. Before that, N/A had a habit of leaking into critiques as if it were a failure, which I fixed in a quality sweep at the start of July.

And the real fix starts before generation. v0.20.0 added a deterministic plan critic: a zero-token pass that runs between the model's proposed task list and my approval prompt. It catches missing verification criteria, duplicate criterion ids, and auto commands that are placeholders, prose, or multi-line shell nobody can run. It is advisory. It never blocks the chain, it just prints its findings above "Approve plan?" and lets me decide.

Notice the shape of that. The cheapest place to fix an ungradeable criterion is before it exists. Every other fix in this section makes a verdict survive. The plan critic stops an ungradeable criterion from being written at all, and it costs nothing to run because it never calls a model.

None of these four made the harness stricter. The gate was always there and it always ran. What they did was turn the evaluator's answer from something I read once in a terminal into something the project keeps. A check you cannot count is a check you cannot improve, and for 69 tasks that was every check I ran.

What you do once you can measure

Here is the second act, and it is the part I did not plan for. Once the harness could report on itself, "this feels better" stopped being an acceptable reason to change it. Something had to fill the gap, and what filled it was other people's papers.

At ac3110ef, thirteen distinct arXiv papers are cited at 42 sites across 23 source files. That is not a bibliography in a README, it is inline in the code that acts on them:

grep -rahoE "arXiv[ :]?[0-9]{4}\.[0-9]{4,5}" src/ | sort | uniq -c | sort -rn

The -a matters: one of the 23 files carries a literal NUL byte inside a template string, so without the flag BSD grep declares the file binary and silently drops two citations. A census undercounting itself is exactly the kind of bug this post is about.

The escalation ladder carries most of it. Four remedies fire in priority order when a task stalls, and each one has an argument behind it.

Take the first rung. When ralphctl bumps the generator to a stronger model, it raises the evaluator's reasoning effort in lockstep on every rung. That is not symmetry for its own sake. It comes from The Verification Horizon, whose claim is that no fixed reward function stays effective as the policy it grades keeps improving, so verification has to co-evolve with the generator.² A grader you never strengthen becomes a rubber stamp the moment the thing it grades gets better than it. The evaluator's model still never escalates, only its effort, which is a cost decision I am watching rather than defending.

The best-of-N rung is where the citation habit gets interesting. When a task has already been nudged at the top of the ladder and plateaued again, ralphctl samples n candidates on the same model and picks between them by verification first, then judging. The rung draws on Scaling Test-Time Compute for Agentic Coding³ for the mechanism and on Trae Agent⁴ for the selection order. And then the source comment says this, at escalation-policy.ts:93-99:

"selects among them by verification then judging (arXiv 2604.16529 (RTV): harness-level N-candidate selection lifted SWE-bench Verified 67.4 → 73.6 at N=16 — RTV measured N=16, not the 2-4 cap used here; arXiv 2601.22129 (SWE-Replay) motivates the tight cap via a descriptive read of its concave scaling curve — a small N captures a disproportionate share of the gain; […])"

Two hedges are doing work in there, and they hedge different things. The first is a departure. RTV measured its lift at sixteen candidates and ralphctl caps at two to four, because sixteen is not a bill I want, so the comment says so rather than letting "grounded in RTV" paper over the gap. The second is an interpretation. SWE-Replay is not cited for a claim it makes. The comment says ralphctl read a concave shape out of its curve and calls that a descriptive read, so nobody mistakes my reading for the authors' finding.

That pair is the argument for the whole habit. A citation that only ever agrees with you is decoration. Anyone reading that ladder deserves to know which rungs stand on a result and which stand on my reading of one.

The habit cuts against features too. Prior-learnings injection stays deliberately thin because Evaluating AGENTS.md found that context files do not generally improve task success rates while raising inference cost by over 20% on average.⁵ The prompt engine tail-compresses long prior-context sections rather than growing them, because Lost in the Middle documents the U-shaped attention curve that buries whatever sits in the middle of a long context.⁶ Both of those are papers arguing me out of shipping more context, and both are cited at the line that does the trimming.

The rollup and the first decision it justified shipped in the same pull request. Best-of-N had been built, tested, and unreachable since the release before, sitting at a default of 0. That commit flipped it to 2. My own message says why the two halves are one change: "the rollup is what makes the flip measurable (its effect on the done-with-warning rate is the first before/after)." Turning on a feature and being able to tell whether it helped are the same piece of work, and for four articles' worth of releases I had been doing only the first half.

The paper that argues against me

Everything above is a paper the codebase cites. This one is not. I went looking for the strongest published argument against the design I had just spent eleven releases hardening, and I found one. It is not in the census. The code does not reference it anywhere.

Is Escalation Worth It? A Decision-Theoretic Characterization of LLM Cascades, Dylan Bouchard, May 2026.⁷ A cascade sends every query to a cheap model first and escalates on some signal. A router decides before generation which model gets the query at all. The abstract's finding: a lightweight pre-generation router exceeds the best cascade policy on four of five datasets.

The reason matters more than the score. The router does not win because it has a better signal about which query is hard. It wins because "cascades pay the cheap model before any escalation decision." The cheap model's generation cost is spent on every single query, including the ones that were always going to end up at the larger model. The cascade's structure is the cost, and no amount of tuning the escalation signal touches it.

ralphctl's ladder is a cascade. Not a cousin of one, the thing itself. Every task starts on the tier my preset picked and only climbs after the loop has already burned attempts finding out it should have started higher. The economic presets, the whole "let the task earn the expensive model" posture I was pleased with in the last post, are exactly the structure the paper measures as second-best.

I have not fixed this and I am not going to pretend I have a rebuttal. The paper's setting is single-turn queries and mine is a multi-turn generator-evaluator loop, which is a real difference and also exactly the sort of difference people reach for when they would rather not update. The honest position is that I now have a report that could settle it. Route-first versus climb-later, same task population, runs stats --json on both. I have not run it.

Meanwhile: one more backend, one contract

Three providers became four over this stretch. OpenCode landed in v0.19.0: it is vendor-neutral, fronts 75-plus model providers plus local runtimes, and its free tier needs no credentials at all, so a fresh install can drive a real session without signing in to anything.

The provider count is the boring half. What makes adding the next one cheap is a contract test. In v0.20.0 I added table-driven headless and interactive port conformance suites that drive every adapter over a scripted in-process fake child process against one shared contract table. A dropped working directory, a prompt inlined into argv where it should be piped, a missing kill rung: all of those now fail at pnpm test instead of in a live session forty minutes into a sprint. Adding a provider is mostly the new adapter plus one-line edits to about fifteen shared Record<AiProvider, …> tables. The compiler tells you every place a new provider needs an answer, and the conformance table tells you whether the answer is right.

One loop from the last post closes here. Anthropic's Fable 5, which got export-controlled server-side in mid-June and forced me to build a suspended-models list, came back in v0.17.0. The suspension lifted, the model is pickable again, opt-in only, with no preset row and no default escalation rung. The list I built for it stays, because that was never going to be the last time.

What to steal

Build the instrument earlier than you think you need it. Not because the work before it was bad, mine ran green, but because the record starts the day you build the thing that writes it. The 68 tasks I ran before 30 June sit permanently outside the report. I can tell you all but one ran their checks green, mostly on the word of a transcript. I cannot tell you which criteria they passed on, and I never will be able to. Instrumenting late buys you a complete record from today and a permanent hole behind you, and the hole is the harder half: it never shows up as a failure, only as an absence, and absences do not page anyone.

Then make the report say what it does not know. 274 unknown in its own column beside 0 failed is why I read the shape of my own data in about ten seconds. Fold those 274 into a percentage, or drop them from the denominator, and the same report says "126 of 126 passed" and teaches nothing. A report that hides its own coverage is worse than none, because it is confidently wrong instead of merely missing.

The measurement did not make ralphctl better on its own. It made the next argument possible. That is also why a paper I disagree with is now in this post: once you can measure, "I prefer my design" is a claim you have to go and check. The router-versus-ladder question is now answerable, which is a much better place than where it was two months ago, filed under taste. Answering it properly means building the router first, because ralphctl has no pre-generation routing today, but the report that would settle it already exists.

ralphctl is at v0.21.0, on npm, four backends, MIT. The concepts underneath, without the release numbers, are in the field guide to agent harnesses, and what a harness has to earn before you trust what it produces is its own post.

Source on GitHub. npm package. MIT license.