Nightly improver: eval-gated learning over the run ledger (item 5 of 6)

[catomean]

Item 5 of the "Ledger and the Loop" sequence. Items 1–4 and 6 shipped 2026-08-02 (#133, #134, #135 here; orangecat#547, orangecat#556). This one was deliberately held back: it learns from the run ledger, and on the day the ledger shipped it was empty. Building an improver against zero rows produces a plausible-looking job that has never once been exercised on real data.

Why it waits (measured, not guessed)

As of 2026-08-02 15:20 UTC, on prod:

signal	count
runs created, last 7d	34
runs with cost_usd (metered)	0
run_escalations rows	0
non-success closes, last 7d	31

The zeros are not a bug: usage reporting and the ladder both deployed after the last nightly autopilot (04:00 UTC). The first real data lands on the next nightly run.

Note the 31/34 non-success rate — mostly partial and timeout (reaper closes at +1h). That is precisely the signal the improver is supposed to metabolize, and it means there is a real problem here worth learning from, not a synthetic one.

Start condition

Build when all hold (one query, below):

• >= 25 runs with non-null cost_usd
• >= 5 rows in run_escalations
• >= 7 distinct days covered by metered runs

select
  (select count(*) from orchestration_runs where cost_usd is not null)                        as metered_runs,
  (select count(*) from run_escalations)                                                      as escalations,
  (select count(distinct date_trunc('day', created_at))
     from orchestration_runs where cost_usd is not null)                                      as metered_days;

What to build

A nightly job (after the autopilot window, not inside it) that reads the last 7 days of orchestration_runs + run_escalations + OrangeCat outcomes and emits:

1. A brief — spend by project, cost per successful outcome, failure clusters by error signature, which escalation rungs were reached and whether escalating actually helped (success rate at rung N vs N-1).
2. Exactly one proposed change — a prompt/config/gate edit, as a draft PR. Not a list; one, with the evidence that motivated it.

Non-negotiables

• Propose-only. It opens a PR; it never self-applies. An agent that edits its own prompt on its own evidence has no outside check.
• Eval-gated. A proposal needs a measurable before/after on replayed history, or it doesn't ship. Without a scoring function, "improvement" is indistinguishable from drift. The eval harness is the real prerequisite and can be built before the threshold is met.
• One change per night, max. Compounding unverified edits is how you lose the ability to attribute a regression.

Order of work

1. Eval harness + replay over historical runs (buildable now — does not need the threshold).
2. The brief (read-only, no proposals) — run it for a few nights and check it says true things.
3. Proposal generation + draft PR, gated on the eval.

[catomean]

Blocked on #145: the start gate checks for ≥25 metered runs, but token accounting has never written a row (0 of 15 runs), so this trigger cannot currently fire. #145 must land before the gate becomes real.

[catomean]

The start condition is met. Re-ran the gate query from the body today (2026-08-26):

signal	required	measured
runs with cost_usd	≥ 25	81	✅
run_escalations rows	≥ 5	34	✅
distinct metered days	≥ 7	20	✅

Metered window is 2026-08-03 → 2026-08-25, $162.31 total. The zeros this issue was parked on are gone; it is unblocked.

I did not start building. Instead I ran the read-only half by hand — the "brief" this issue specifies as step 2 — because if a hand-run brief says nothing useful, an automated one won't either. It says quite a lot, including one thing that should change the design.

Cost per successful outcome varies 8×

project	metered runs	spend	done	errored	$/done
orangecat	10	$50.60	7	3	$7.23
printcraft	10	$33.26	6	4	$5.54
datacat	15	$26.53	14	1	$1.89
surf-your-life	14	$15.99	12	2	$1.33
truthseeker	13	$11.36	13	0	$0.87

orangecat costs 8.3× per successful run what truthseeker does, and one third of its runs error. Two projects account for 52% of all spend. This is exactly the "spend by project, cost per successful outcome" output item 1 asks for, and it is actionable today without any automation.

The escalation ladder has never once been resolved by a human

 level  | opened | resolved_success | resolved_manual | still_open | avg_fail_streak
--------+--------+------------------+-----------------+------------+-----------------
 retry  |     22 |               11 |               0 |         11 |             1.0
 patch  |      6 |                3 |               0 |          3 |             2.0
 human  |      5 |                3 |               0 |          2 |             4.8
 replan |      1 |                0 |               0 |          1 |             3.0

resolved_manual is zero at every rung, including the rung named human. Escalations either get closed by a later success or stay open forever — 17 of 34 are still open. Whatever the ladder is doing, being acted on by a person is not part of it. That belongs in the brief's "whether escalating actually helped" section, and it is a finding about the ladder rather than about any agent.

⚠️ The hazard: success is a bar, not a measurement

Outcomes over the last 14 days:

 partial | 52
 timeout | 33
 success | 24
 error   |   5
 (null)  |   3

Success is 21% of closed runs; partial + timeout is 73%. The body predicts this ("31/34 non-success... precisely the signal the improver is supposed to metabolize") and reads it as a real problem to learn from.

I think that reading is a trap, and it matters because it would be encoded into the scoring function. timeout is assigned by the reaper closing runs at +1h — it describes the window, not the work. partial is assigned by the definition-of-done gate. Both are properties of where we set the bar, and this has bitten before: a previous investigation concluded the non-success rate was the bar, not agent quality.

So an improver scored on outcome = success would be optimizing a threshold it does not control and cannot see. The available moves that raise that number — loosen the DoD gate, extend the reaper window, narrow task scope — all look like improvements and none of them are. That is drift wearing an eval's clothing, and the non-negotiable "a proposal needs a measurable before/after" does not protect against it: the measurement would faithfully show the number going up.

Recommendation for whoever builds step 1: the eval harness should score against outcomes the agent actually controls, and hold the bar fixed as an input rather than treating it as part of the score. Concretely, before writing the scoring function, separate timeout runs that were making progress when reaped from those that were stuck — the first is a window problem and the second is an agent problem, and today they are the same string in the same column. Until they can be told apart, cost-per-successful-outcome (which the table above shows is already discriminating, and which no threshold change can inflate) is the sounder objective.

Leaving open, unblocked, with step 1 the next thing to build. The numbers above can be regenerated at any time; nothing here needed the automation to exist.

🤖 Generated with Claude Code

[catomean]

Correction to my comment above, and it strengthens rather than weakens the case for waiting.

I reported that resolved_by = 'manual' was zero at every rung and read it as a finding about the ladder — "being acted on by a person is not part of it." That reading was wrong. There was no code path that could write 'manual'. The value was declared in the type and produced by nothing, so the zero was not a measurement of human behaviour; it was a dead enum branch. Nothing could have been concluded from it either way.

Digging into why turned up a real defect, now fixed in #381:

The ladder advanced on isFailingOutcome but resolved only on outcome = 'success', so partial — the most common outcome — could neither advance a ladder nor clear one. Eighteen ladders were open at once, none with a single success since opening. Repaired in production: 18 → 6, with the six that remain genuinely lacking a qualifying run.

Why this matters for the improver

It is a worked example of the hazard I flagged above, and it lands harder than I put it.

The escalation ladder is a system that already consumed these outcomes to make decisions. It was silently broken for months precisely because success/partial/timeout do not mean what their names suggest: partial is not "a bit of a failure", it is work landed, bar not cleared, and timeout describes the reaper's one-hour window rather than the run. Two reasonable engineers reading those names composed a state machine with no exit.

An improver trained on the same vocabulary would inherit the same misreading, with no state machine to make it visible — it would just emit confident proposals. And the historical data is now doubly unsafe to replay: the escalation rungs recorded before #381 were produced by the broken rule, so "was this project escalated at the time?" is not a trustworthy feature.

Reinforces the order in my previous comment: fix the outcome taxonomy first — specifically, split timeout into reaped-while-progressing vs genuinely-stuck — then build the eval harness. Cost-per-successful-outcome remains the sounder objective in the meantime, since no threshold change can inflate it.

The start condition is still met. The data is still worth learning from. It just should not be learned from through this vocabulary.

🤖 Generated with Claude Code

[catomean]

Taxonomy prerequisite: first half done, in #383 — live and verified.

I proposed splitting timeout into reaped-while-progressing vs genuinely-stuck. Measuring first turned up a different and larger split hiding in the same column, so that is what got fixed:

timeout was also covering "no agent ever saw this prompt" — 29 of 157.

Not an inference: the runner itself acked those commands verified: false. The reaper already computed that fact (runNeverStarted, for its staleness predicate) and discarded it when choosing the label, so runs no agent received were recorded as the agent failing. They advanced escalation ladders and darkened outcome streaks. surf-your-life reached the human rung with ten of them behind it.

Production is relabelled (undelivered, 29 rows, zero mislabelled remaining), and going forward the reaper stamps it directly.

For the improver specifically

The historical record is now honest about this distinction, so a replay can separate "the project's agent failed" from "the fleet failed to deliver". That was previously impossible from the outcome column, and it is a third of what looked like project failure.

It is also a concrete demonstration of the hazard, not just an argument for it: a system that consumed these outcomes to make decisions — the escalation ladder — was wrong for months, and wrong in a way that looked exactly like a project doing badly.

What is still conflated

The original split I proposed remains partly open:

• partial still covers both agent finished with follow-up work and reaped mid-progress, corrected upward by repo evidence. They are distinguishable today only by payload.evidence (26 of 144 over 60 days), not by the outcome itself.
• timeout — now 129 rows — still means "reaped, no evidence of work, and the runner never said it failed to start". Some of those were genuinely stuck; some were making progress and got cut at the one-hour mark with the evidence check either bounded out (MAX_CHECKS_PER_SWEEP = 5), missing a repo URL, or lacking a GitHub token.

So before building the eval harness, partial is the remaining ambiguity worth closing — and unlike the delivery case there is no equivalent hard signal, so it needs an actual decision about what evidence counts rather than a relabel.

Start condition still met. Data now materially more honest than when this issue was written.

🤖 Generated with Claude Code