fix(benchmark+cognition): the run pulses, empty completions fault, bench acts land in their room (#371 step 1) - #2335
Merged
Conversation
…lve was hardcoded roomless (#329/#243) `agent/solve` opened with `let room = Uuid::nil();` — a literal, no param behind it. Every benchmark act therefore executed with no room, and `apply_act`'s receipt radiation skips a nil room BY DESIGN (radiating nil-room acts once stole academy's single-room chat projection onto a phantom, live-proven 2026-08-12). Its own comment names the fix as pending: "Skip until solves thread their bench room (#329's per-run rooms make every solve act a room act)". #329(a) shipped the per-run room weeks ago; nothing ever threaded it. Consequence, and it is the failure BENCHMARKS-ARE-ADAPTERS-NOT-A-RUNNER.md exists to name: a citizen can burn a full act budget, write a patch and take a verdict while the room she was dispatched into shows NOTHING. Not a rendering gap — the events are never published. The acceptance test in that doc ("can a citizen standing in the room perceive the run's state through the same ViewState pipe the human's screen uses?") answered NO, and answering it needed a file read, which is the doc's own definition of disconnected. * `AgentSolveParams.room: Option<Uuid>` — the activity the run belongs to. * `benchmark/dispatch` passes the room it just spawned. Its acts now radiate `persona:act` into it, so the work renders as collapsed receipts in the run room's transcript (#243) and reaches citizen perception through the same projection the screen reads. * `work/claim`'s path passes `None` and SAYS SO in a comment: the claim verb carries no activity, so a claim-fired solve is still invisible. That is exactly #425's subject; the two paths now differ precisely at the gap #425 exists to close, instead of both being silently roomless. * `WorkspaceCycle::actions_taken()` — the seam a long-running drive needs to report liveness WHILE it runs. `drive_to_settle` returns its act count only at settlement and a SWE attempt legitimately runs hours, so `benchmark/runs` reads `acts` from a ledger written once per attempt. Measured 2026-08-16: two dispatched solves read `acts=0, stalled=false` for ten straight minutes while both citizens were demonstrably mid-turn (captures written 12s earlier). The projection whose stated purpose is "silence must never be ambiguous with progress" cannot currently tell them apart — its 20-min stall window assumes artifact activity tracks act cadence, and no per-act artifact write exists. Accessor lands here; the heartbeat that consumes it is the next slice. Deployed and SHA-verified before commit. NOT yet live-proven end to end: no dispatched run has reached act #1 since — the two sympy runs both settled in ~150ms with zero acts and were correctly classified INFRA VOID by #384, which is a separate round-killer this change does not touch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… arm's missing twin (#181/#390) The `Err` arm at llm_deliberation_faculty.rs:1844 refuses to let a FAILED model collapse into a serene `Pass`, and its own comment cites [[fallbacks-are-illegal-fail-loud]]. But a lane can also answer `Ok` with NOTHING, and that walked straight past the guard and settled as an ordinary non-Act. Same failure the Err arm exists to prevent, one branch over. MEASURED, not inferred. Direct probe against the live lane (70-token prompt, so no context pressure of any kind): finish_reason = 'length' completion_tokens = 16 content = '' Sixteen tokens generated, empty content. Qwen3.8 under `--jinja` opens `<think>`; an unclosed block leaves `extract_reasoning` branch (3) — which is CORRECT — with empty text and the whole tail as reasoning. Separately, Solenne's capture on the turn her benchmark run died: finish_reason = 'stop' usage = {in:0, out:0} responseTimeMs = 28 text = '' SCOPE, measured across every capture on disk: 47 of 862 responses (5.5%) are empty-text, spread over ~19 citizens. NOT benchmark-specific. And the all-empty column is exactly the citizens whose "I've noticed my recent messages have been repetitive… I'll remain silent" turns have been the standing round-killer — fe4dac17 6/6, Asha (90e758b2) 4/4, e5f4141d 3/3, a20b3ada 2/2. On the turns I can see, nothing came back, and the substrate wrote it down as her choice. (Per-persona samples are small; the ratios are not stable rates. The shape is what matters.) COST: agent/solve reads it as "she chose not to act" → acts=0, empty patch → the run voids as an INFRA VOID after three attempts (the two sympy runs, 2026-08-16). A live citizen's turn reads as silence, indistinguishable from withdrawal. FIX: fault, exactly like the Err arm — the settle step surfaces it LOUD instead of fabricating a no-op, and `delib.empty_completion` puts it on the probe stream. Scoped so a native tool turn, which legitimately carries empty content, is untouched: fires only on empty text AND not ToolUse AND no tool_calls, i.e. only when the turn yields nothing to act or speak with. The receipt names WHICH shape it was — thought-but-committed-nothing (reasoning chars) vs the lane returned void. WHAT THIS DOES NOT DO: it does not stop the empty completions. It stops them being laundered as cognition. The generation-side fix (reasoning budget / think-block closure) is #181 and is next. Deployed + SHA-verified before commit. LIVE PROOF NOT YET IN HAND: zero faults across 2,589 probe rows in the first window after deploy — consistent with no citizen having hit an empty completion yet post-reboot, NOT with the guard being confirmed. A watcher is running; the owed evidence is one `delib.empty_completion` row from a real turn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ne (#371 design) Joel 2026-08-16, after a full session of a driver getting lost: "It's way too hard to rig up. Clearly." This writes the design down so the next agent inherits a paved road instead of tire tracks. THE DEFECT, stated exactly: a benchmark round is not a thing in the system, it is a RITUAL an agent performs. The agent picks when to dispatch, picks the watch window, hand-queries probes to learn what is happening, and when the session ends the process ends with it. Every failure this session is downstream of that ONE fact, and they all have the same shape — the question has no owner, so the driver guesses from an absence: is serving ready? nothing owns readiness → dispatched and hoped has it started? nothing announces started → read acts=0, called it stalled is this run alive? liveness = a file mtime → flagged a healthy run quiet is the round done? nothing announces done → never knew did my fix work? no green/vacuous distinction → reported a vacuous green Five instances of [[an-absence-is-an-unfinished-measurement]] in one session, by a driver that had that lesson in its own guardrails. Not a knowledge gap — a missing owner. THE SHAPE: staging → ready → working → grading → done, every transition an event from the component that KNOWS (env builder, supervisor, card store, round entity) — never a timeout, never a poll, never an agent's judgement. Every stage a ViewState on the one pipe humans and citizens already read. Recipe is data, so the process is identical every round on every machine. THREE LAWS: transitions announced by the knower; liveness is a pulse never a terminal artifact (today's projection cannot distinguish silence from progress, which is its stated purpose); an absence is never a state. SUBSUMES #329(b) no-END, #442 readiness gate, #374 pulse, #425 in-room work, and the unbuilt RULES half of [[recipe-is-content-type-plus-rules]]. Carries the acceptance test from BENCHMARKS-ARE-ADAPTERS-NOT-A-RUNNER.md plus one earned tonight: can a fresh driver answer ready/started/stuck/done using ONLY queries — zero log reads, zero probe archaeology, zero inference from an absence? Build order is smallest-true-cause first; step 1 (pulse the run) consumes the WorkspaceCycle::actions_taken seam c9ba5f9 already landed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…a terminal artifact (#371 law 2, #374) First build step of ROUND-LIFECYCLE-AS-RECIPE-OWNED-STATE-MACHINE.md. THE DEFECT: the run ledger was written ONCE per attempt, at settlement. For the entire attempt — legitimately HOURS on a full SWE budget — `benchmark/runs` read `acts: 0` and a `last_activity` frozen at run start. RUN_STALL_WINDOW_SECS is 20 minutes. So a HEALTHY first attempt is guaranteed to read `quiet`/stalled, every single time, and the projection whose own doc comment says its purpose is "silence must never be ambiguous with progress" was STRUCTURALLY unable to tell them apart. Its stall window assumes artifact activity tracks act cadence (~4-6 min); no per-act artifact write has ever existed. MEASURED 2026-08-16: two dispatched solves read `acts=0, stalled=false` for ten straight minutes while the driver watching them could not distinguish working from wedged — which is precisely how a vacuous "no faults observed" gets reported as a green. THE FIX: `select!` over the drive future and a 60s interval. No spawn, so the cycle stays BORROWED — no 'static bound, no Arc juggling, no parallel allocator, and the pulse cannot outlive the work it reports on. Each tick reads `WorkspaceCycle::actions_taken()` (the seam c9ba5f9 landed) — a wait-free atomic load on the persona's own monotonic counter — and rewrites the running marker, which moves BOTH `acts` and the mtime `last_activity_ms` folds from. Cadence is well under the stall window (a live run can never age into `quiet`) and far above act cadence (one small JSON write per tick, nothing else). Every field derives exactly as the detached wrapper's `state: running` marker derives them, so a pulse can never contradict the marker it refreshes. `None` run_id → no ledger → no pulse, correct: nothing polls a run that returns inline. WHY THE COUNTER AND NOT A SEPARATE TALLY: it is the same monotonic counter perception renders to her, so the board and her own proprioception cannot disagree — and it is the counter whose capacity-bounded-ring sibling caused the retracted "they never act" premises on #390/#211. Deployed + SHA-verified. LIVE PROOF OWED: a dispatched run showing acts CLIMB mid-attempt. Until that row exists this is compiled-and-deployed, not confirmed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
enabled auto-merge (squash)
August 17, 2026 02:58
… on every modern Apple SDK (#383) THE FOURTH HEAD of #383, and the reason two dispatched rounds died at env-build TONIGHT even though #2329's jinja2 + build-requires fixes are both merged and in this tree. Not pyerfa, not Cython, not Python 3.11 — I guessed all three before reading the error. astropy vendors cfitsio, which vendors an ancient zlib. `cextern/cfitsio/zlib/zutil.h:140`: #if defined(MACOS) || defined(TARGET_OS_MAC) # define OS_CODE 7 # ifndef fdopen # define fdopen(fd,mode) NULL /* No fdopen() */ `TARGET_OS_MAC` means "some Apple platform" on every modern SDK — it does NOT mean "classic Mac OS", which is what this zlib was written to test. So the branch fires, `fdopen` is macro-replaced by `NULL`, and the SYSTEM header's own declaration FILE *fdopen(int, const char *) → FILE *NULL(int, const char *) becomes `error: expected identifier or '('` inside `<stdio.h>`, thousands of lines from anything astropy wrote. (The adjacent `'OS_CODE' macro redefined` warning is the same branch firing.) FIX: the guard is `#ifndef fdopen`, so pre-defining it is the entire repair. `-Dfdopen=fdopen` makes the guard FALSE — the NULL stub is never emitted — and the macro is the identity, so every real call still compiles to `fdopen`. No source patched, nothing stubbed, nothing renamed, and a repo that doesn't vendor this zlib never notices the flag. POSITIVE + NEGATIVE CONTROL against the real staged checkout, both run before committing: cc -c -I<checkout>/cextern/cfitsio/zlib → error: expected identifier or '(' error: expected ')' cc -c -Dfdopen=fdopen -I<same> → clean, exit 0 Placed in ERA_CFLAGS beside its three siblings rather than a per-repo table, because it is not an astropy fact — it is an ERA fact (old vendored C vs a modern SDK), same shape as the others: the compiler is HARNESS, the C is SUBJECT, and the subject built fine on the compilers of its day. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…e was floored at bare survival (#438-class) Glass-boxed live on the M5 2026-08-17: a 27B served at usable_gb=5 with served_window=2048 against a MEASURED demand_window=63817. Every SWE act ran against a context that could not hold the task statement — the "machine is allowing sludge" Joel reported, and the arithmetic behind ~4 acts/hour. The defect was an inconsistency INTERNAL to plan_serving. Two floors, two different standards: lane COUNT -> BOOTSTRAP_WORKING_SET (16384, "one full turn") model CHOICE -> MIN_SERVE_CTX (2048, bare survival) The lane loop already refuses to add a slot that cannot hold a full turn (1 lane @ 30k beats 2 lanes @ 2k), then falls back to .unwrap_or(1) and calls it "honest starvation, surfaced downstream" — while the MODEL is never reconsidered. Shedding a lane and shedding capability are the same move for the same reason; only the first was implemented. So `fits_one_lane` crowned the most capable model that cleared a TRIVIAL bar and let the window collapse to it. A model that fits only at 2048 is not more capable on this host — it is unusable on this host. Viability is now tested at the same "one full turn" standard the lane floor uses, with a documented degrade: if NO candidate clears it, fall back to the old MIN_SERVE_CTX bar so a genuinely tiny host still serves something rather than nothing (honest starvation is then real, not a selection artifact). Deliberately the STABLE bootstrap constant, not the moving measured p95 — coupling model CHOICE to a jittering demand signal is the 718-replan flap that wedged three benchmark runs. This ran on EVERY node, so a grid of peers would each independently crown its biggest technically-holdable model and starve its own window, with no single node's numbers looking anomalous. LIVE, deploy-verified: 27B/2048/bound_by=host-fit -> 14B/16384/bound_by=demand, lanes 1 -> 4. bound_by flipping to `demand` is the signal: the plan now meets the working set instead of being cut by the host. Tests: model_choice_sheds_capability_rather_than_starve_the_window pins all three arms — the starve case, a NEGATIVE CONTROL that a roomy host still picks the 27B (this fix must not silently downgrade capable boxes), and the degrade fallback. 27 serving_plan / 139 serving / 178 capacity green. Also repaired stable_keeps_incumbent_when_upgrade_lacks_headroom, whose fixture had encoded the old bar: with pair() at 10GB, big was "viable" only on 184MB of KV, so "fresh would pick big" meant a 2048-token 9GB model. Given a local fixture where big genuinely clears a full turn (19.5 <= 20) yet still fails the 0.9x switch-up headroom bar, so the test asserts what it always meant to. KNOWN RESIDUAL: selection tests weights + kv_at(ctx) against raw usable_bytes while window_for sizes against `effective` (minus co-consumer headroom) and subtracts compute buffers, so the two floors are near-consistent, not identical (observed 16025 vs 16384). The 8x collapse is gone; unifying the last 2% is a follow-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…s the window but not the CHOICE (Joel 8/17) Joel: "Context windows are as important as model size. It seems 16-20k is bare minimum for decent activities. Ideally the governor learns." Captures the design rather than rushing it into plan_serving, which is the most flap-prone function in the tree (718 replans, three wedged benchmark runs). THE CLAIM: delivered capability is f(params, served_window) and the planner models only params. capability_rank is a scalar keyed to model identity, so a 27B is "more capable" than a 14B even when the host can serve the 27B only 2048 tokens. That shipped as the live defect fixed in 03c890b. WHAT IS ALREADY RIGHT, and must not be rebuilt: working_set.rs is a good learning loop whose hard problem is solved. It records DEMAND not USAGE — the counterfactual "what this turn would have used with no budget", including grounding contributions assembly dropped — explicitly avoiding the thermometer-inside-the-thermostat trap where a p95 of what-was-sent re-derives its own clamp forever. Peak not average. Persisted. Passed as a parameter. The live demand_window=63817 IS that module working correctly. THE ACTUAL GAP: one learned signal, three decisions, two deaf. The window is learned; lane count and model choice both use hardcoded BOOTSTRAP_WORKING_SET — a constant whose own doc says it "was never meant to survive", and which sits at the BOTTOM of Joel's 16-20k band, identically for a chat turn and a SWE turn. WHY THE CONSTANT IS THERE AND WHY IT DOESN'T FORBID LEARNING: the anti-flap argument is against driving a structural, expensive-to-change decision from a fast jittering signal. It is not against learning. Two time constants: fast measured demand sizes the window (cheap, built, correct); a slow hysteretic learned floor drives model choice and lane count (expensive, so margin band + dwell time + hard clamp). Flapping is a property of the update rule, not of learning. PROPOSED, NOT BUILT: (1) capability_rank goes 2-D — rank by delivered capability, since the current most-capable-that-clears-a-full-turn gate is a correct approximation but cannot express "14B at 60k beats 27B at 20k"; (2) the floor becomes learned + slow, with BOOTSTRAP_WORKING_SET surviving as the floor OF the floor, not the value; (3) the floor becomes per-ACTIVITY — recipes are already data (#433) and "recipe = content-type + RULES" makes the minimum useful window recipe-owned. Includes acceptance tests (incl. an anti-flap test pinned as a test, not asserted in prose) and the de-hardcode smell: a new constant beside BOOTSTRAP_WORKING_SET is a third way to express a floor — the shape the guard already caught once as FLOOR_TOKENS (#411). BLOCKING OBSERVATION recorded for whoever picks this up: delib.generate emits ZERO probe rows, so per-generation latency and tok/s are unreadable and every throughput question here needs 20-minute black-box sampling. An unmeasurable governor cannot be a learning one (#441). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…#181 recovery — reasoning-bearing empties are not faults REGRESSION I INTRODUCED in ce82f00, caught by its own test suite. That guard faulted on "empty text + no tool call + not ToolUse" — which is the EXACT precondition of the two recovery paths immediately below it: persona.act.reasoning_lift — a tool call sitting in the reasoning tail gets lifted out and EXECUTED (json_in_prompt_tools::parse_tool_calls over the reasoning channel). persona.act.think_only — #181's teacher sentinel: she spent the whole generation inside <think>, so route the #159 reported-never-executed mechanism and let drive_to_settle hand her another generation that starts from her own conclusions (the reasoning is already in working memory). Faulting first made BOTH unreachable. Measured live 2026-08-17: 40 delib.empty_completion faults against 87 persona.turn.start while three SWE runs sat at the SAME act count for 1,357 seconds. Every one of those turns had 15k+ chars of reasoning and finish_reason: length at exactly 4096 output tokens — i.e. the shape the recovery exists for. The machinery was already built and correct; I put a dead end in front of it. FIX: fault ONLY when nothing is left to recover — the lane returned genuine void (no text, no reasoning, no call), or she has no tools for the sentinel to teach through. A reasoning-bearing empty falls through to the lift/think-only owners. The guard's purpose is preserved: an empty is still never read as chosen silence, it is just routed to the component that can act on it. Restores 2 tests that had been RED since ce82f00 and that I shipped without running — think_only_turn_routes_the_teacher_sentinel_not_empty_speak and empty_content_with_reasoning_tool_intent_lifts_the_final_call. Both name this exact contract; both now pass. 62/62 deliberation tests green. NOT YET LIVE-PROVEN. Deployed and unit-green, but the owed evidence is a real reasoning_lift or think_only probe row with empties falling — measurement in flight. Recorded here so the claim is not read as stronger than it is. ALSO CORRECTED, and NOT shipped: completion_budget_for is window/4, so a 16k window caps generation at 4096 and a thinking model exhausts it inside <think>. Raising it to window/2 broke prompt_plus_completion_cap_never_exceeds_the_ served_window (prompt 525 + completion 512 > window 1024) — the prompt sizer overshoots its own target by ~13 tokens, which /4 had slack to hide. The OVERSHOOT is the defect, not the fraction; shipping /2 on top of it would push prompt+completion past n_ctx and 500 every turn with context-shift off. Left at /4 until the sizer is fixed. Bounding the reasoning channel (--reasoning-budget) was considered and REJECTED: it lobotomizes the model to fit a fraction we invented (Joel, 2026-08-17). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…pt floor — why raising window/4 is a refactor, not a constant Records the measured blocker on the #181 output starvation so the next attempt does not repeat either of the two wrong fixes. MEASURED: completion_budget_for = window/4, so a 16k window caps generation at 4096. A reasoning model spends output on thinking, exhausts the cap inside <think>, and never reaches the tool call — 7 of 20 captured turns at finish_reason: length, output_tokens: 4096 exactly, ~15k reasoning chars, empty text, zero tool calls. WRONG FIX 1, rejected on Joel's instruction: --reasoning-budget. Capping <think> shrinks the model to fit a fraction we invented. Their thinking is the product. WRONG FIX 2, tried and reverted: window/2. Breaks prompt_plus_completion_cap_never_exceeds_the_served_window, the invariant that keeps prompt+completion under n_ctx. With context-shift off that is a 500 per turn — every citizen muted. Trading a 35% empty rate for a 100% mute rate. THE ACTUAL DEFECT, from reading prompt_view_within: the budget derivation (window − reserve − tool_tokens) is correct. What breaks is that three sibling tests pin content that must survive budget pressure unconditionally — held work card, most recent burst, newest message. Those are INCOMPRESSIBLE FLOORS. Grow the reserve and the budget drops under the floor; the packer admits the mandatory content anyway and overshoots (prompt 525 + completion 512 > 1024). The overshoot is the floor correctly refusing to compress. Holding the reserve FIXED is the error. FIX SHAPE: reserve = min(desired_share, window − mandatory_floor). Invariant then holds at every window, including the synthetic sub-MIN_SERVE_CTX windows the tests use and production never serves, while the share stays generous at real 16k+ windows where the floor is hundreds against thousands. NOT A ONE-LINER, and that is why this is a doc and not a patch: the dependency is circular (reserve → budget → packing → floor → reserve). The floor must be computable before the reserve is chosen — hoist the mandatory-section measurement ahead of budgeting, or two-pass size-then-resize. Sequence recorded: hoist the floor, make the reserve yield, THEN raise the share, and re-run all four prompt_shaping tests at both a synthetic small window and a realistic 16k one. Those four tests are the only thing between a generous reserve and a 500 on every turn; none of them may be weakened to make the change pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… gold patch disqualifies itself (#380 keystone) The spine check RAN and enforced NOTHING. `benchmark/swe-grade --gold` has always substituted `instance.patch` as the candidate and graded it, so the mechanism existed. But it returned a bare `resolved: false`, which downstream is byte-identical to a citizen's capability zero. The `gold` param's own doc states the requirement — "It MUST resolve — if it does not, the environment is wrong and no other number from it means anything" — and that sentence was addressed to a human and enforced by nobody. WHY THIS IS THE KEYSTONE FOR EVERY NUMBER WE REPORT. Without it a failure has two indistinguishable causes: the patch was wrong, or the environment cannot score a correct patch at all. Measured on this box 2026-08-17 (build fc8964b, deploy-verified): a 2019-era django env carries pytest 8.4.2, and this module's own notes record era suites importing pytest internals modern pytest deleted (flask 2.2: `from _pytest.monkeypatch import notset`). So an unknown fraction of our zeros are harness artifacts tallied as capability — noise with a number attached. That is why #383's 114/300 and #380's era drift cannot be told apart from model failure by reading scores. `swe_bench::gold_gate` is a THIN caller over `grade(.., Some(&instance.patch))`, deliberately not a parallel scorer: it must exercise the exact clone → apply → test path a real attempt takes or it proves nothing about that path. A second implementation agreeing with itself is the classic dead instrument. What it adds is the enforcement: on a non-resolve it stamps `verdict.error` naming the gate, the f2p/p2p counts, and the era-dep suspects to check. `error` is contractually an ABSENCE and "must never be tallied as a failed attempt" (SweVerdict::error), so the disqualification propagates through every existing consumer with no further wiring. The command's gold arm now routes through it, so there is ONE path and every caller inherits the labelling — not a second spelling of gold grading. Explicitly NOT conflated: `gate_ok == false` means FAIL_TO_PASS already passed on the pristine tree (the instance carries no bug here). That is a different fact from a broken environment and the doc says so. NOT era-pinning pytest in this commit. That fraction is deliberate and documented at swe_bench.rs:692-706 ("harness is deliberately MODERN"), with the breakage it causes documented at :907. Inverting a decision with a stated rationale is Joel's call; this commit builds the instrument that MEASURES which envs it breaks, which is the prerequisite either way. Verified: cargo check clean. Live gold-gate run against a known-gradeable and a suspect env is the owed proof and is next. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…root cause) The gold gate (943b8a2) caught django-10914 failing its OWN gold patch: FAIL_TO_PASS 0/1, PASS_TO_PASS 35/40, reported as "REGRESSION — your changes BROKE 5 test(s)" while the quoted output showed all five passing `... ok`. ROOT CAUSE, in `parse_django_report`: it required `" ... "` AND `" ("` on the SAME line. unittest puts the outcome on the id line ONLY when a test has no docstring; with one it prints two lines and the `... ok` lands on the SECOND: test_skip_if_db_feature (test_utils.tests.SkippingTestCase) Testing the django.test.skipIfDBFeature decorator. ... ok Both lines fell through the `continue`s — the id line has no `" ... "`, the docstring line has no `" ("` — so every docstringed django test was recorded NOWHERE, and absent-from-map reads downstream as not-passed. The 5 "broken" were the 5 docstringed ones. django is 114/300 of Lite, so django scores were never measuring django, and every django zero on record is retro-actively uninterpretable. THE FIX IS NOT A BETTER PARSER. I first extended the state machine to stitch the two-line form; Joel called it correctly — "fragile, needs cleaner fixes". It is, because unittest's verbose output is a RENDERING, not a data format: docstrings can contain " ... " and " (", django ≥4.1 changed the class-path shape, and every one of those is a way to mis-attribute silently. Mis-attribution here does not look like a bug; it looks like a citizen who failed. So django is now graded from CANONICAL ids. `install_django_json_runner` drops a `DiscoverRunner` subclass into the clone's `tests/` dir; `--testrunner` selects it; it emits one `CONTINUUM_TEST {"id": ..., "ok": ...}` row per test keyed by `test.id()` — the id unittest itself uses. No line shapes, no docstring ambiguity, no version drift. The dataset's `method (module.Class)` spelling is derived from the canonical `module.Class.method` by deterministic surgery, not guessed from output. Skips and expected failures are PASSES, unexpected successes FAILURES — the same rule as the prose path, now expressed once in `django_outcome` (it was inlined twice, which is how a `skipped` becomes a pass in one shape and a non-outcome in the other). FALLBACKS ARE LOUD, NEVER SILENT: if the runner cannot be installed, or emits zero rows (django too old for --testrunner, import error, crash before any test), we fall back to the prose parser AND warn. A silent fallback that scores is indistinguishable from one that lies. `parse_django_report` is KEPT as that fallback, with the two-line handling, since it is now the degraded path rather than the only path. Verified: cargo check clean. Live gold-gate re-run on django-10914 is the owed proof and is next — the same instance whose failure produced this diagnosis, so the fix is falsifiable by the evidence that motivated it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…y has no --testrunner (#383) The canonical-id grader (13fd357) was correct about WHAT to read and wrong about HOW to ask for it. It passed `--testrunner=continuum_json_runner.JsonRunner` to `tests/runtests.py`. That flag does not exist in ANY django era — runtests.py reads `settings.TEST_RUNNER` and only defaults it when unset (verified in-tree at 1.11, 2.2, 3.2, 4.2, 5.2 and main: the same three lines throughout). An unknown flag there is not ignored. argparse rejects the whole invocation before a single test runs, so the live gold gate on django-10914 came back in 4.4 seconds with `PASS_TO_PASS passes 0 of 40 on the PRISTINE tree`. The fix I shipped to stop mis-attributing django tests made django strictly less gradeable than the prose parser it replaced. Fix: install a settings module beside the runner — from test_sqlite import * TEST_RUNNER = "continuum_json_runner.JsonRunner" — and pass `--settings=continuum_json_settings` instead of `--settings=test_sqlite`. Both files are pure additions to the clone; django's own test_sqlite stays untouched and still supplies every database/hasher setting, so the shim cannot drift from whatever the era's suite settings happen to be. The invocation now differs from the plain one in exactly one argument, which is what the new test asserts. WHAT WORKED, and is worth keeping: the zero-rows fallback. `run_tests` warns and falls back to `parse_django_report` when the JSON runner emits nothing, and that is the only reason a broken invocation surfaced as an env fault instead of 40 capability failures. A silent fallback here would have read as a citizen's zero. Mechanism proven live in the 2019-era env BEFORE this commit — the settings shim run by hand emits 30 canonical rows (`CONTINUUM_TEST {"id": "file_uploads.tests.FileUploadTests .test_base64_upload", "ok": true}`) where the flag emitted zero. Those hand-written files were then deleted so the owed gold-gate re-run exercises the install path, not my copies. 15/15 swe_bench tests green (metal,accelerate). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
… emit its own (#383) Gold gate on django-10914 after the settings seam: FAIL_TO_PASS 1/1 (was 0/1), PASS_TO_PASS 38/40 (was 35/40), gate_ok true. The two remaining misses were not tests and not env — they were SPELLING: "An exception is setUp() is reraised after disable() is called." "assertRaisesMessage shouldn't interpret RE special chars." Those are the dataset's own PASS_TO_PASS entries, verified by reading the cached Lite JSON: 2 of django-10914's 98 p2p ids are docstring prose with no test id in them at all. The cause is upstream and structural — SWE-bench harvested django ids from unittest's verbose log, and unittest prints a test's DOCSTRING in place of its id when it has one. So a canonical id, however correct, can NEVER resolve those rows. Fix: the runner emits the docstring alongside the id (`test.shortDescription()`, the same string unittest itself renders), and the parser registers THREE spellings for one outcome — canonical id, unittest rendering, docstring. Docstrings are not unique, so they are AND-folded (a docstring shared by a pass and a fail reads as a fail) and never overwrite a real id. Only the test object knows its own docstring; having the runner report it is what removes the guess, which is the same reason the canonical id came from `test.id()`. CORRECTION to what I wrote on 13fd357: I described the old prose parser as "mis-attributing" docstringed tests. It DROPPED them. For a docstringed test the id line carries no `... ok` suffix and the docstring line does not match the `name (class)` shape, so neither line produced an entry. The parser was not confused about which test passed; it had no row for that test at all. That distinction matters because it is why the docstring must be emitted as DATA rather than recovered by a better regex. 16/16 swe_bench tests green (metal,accelerate). Live re-run of the gold gate is next and is the falsifiable claim: p2p 40/40 and resolved=true, or the docstring path is not the whole remainder. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four commits from one session of a driver getting repeatedly lost, plus the design that stops the next one repeating it.
The through-line
Every wrong call I made this session had one shape: the question had no owner, so I guessed from an absence — and an absence is not evidence. Five separate times, with
[[an-absence-is-an-unfinished-measurement]]in my own guardrails. Three of these commits remove a specific absence; the fourth writes down why they keep happening.c9ba5f943— bench acts happen IN THEIR ROOMagent/solveopened with a literallet room = Uuid::nil();.apply_actskips receipt radiation for a nil room BY DESIGN, and its own comment names the fix as pending: "Skip until solves thread their bench room (#329's per-run rooms make every solve act a room act)." #329(a) shipped the per-run room weeks ago; nothing threaded it. So a citizen could burn a full act budget, write a patch and take a verdict while the room she was dispatched into showed nothing.work/claim's path passesNoneand says so — that verb carries no activity, which is exactly #425's subject. The two paths now differ precisely at the gap #425 exists to close, instead of both being silently roomless.ce82f00ff— an EMPTY completion is not a chosen silenceThe
Errarm refuses to let a failed model become a serenePass, citing[[fallbacks-are-illegal-fail-loud]]. A lane answeringOkwith nothing walked straight past it. Proved by direct probe against the live lane — 70-token prompt, no context pressure:Sixteen tokens generated, empty content: they went to the reasoning channel (
--jinja, unclosed<think>).extract_reasoninghandles that correctly; the faculty downstream did not.Scope, measured across every capture on disk: 47 of 862 responses (5.5%) empty-text, ~19 citizens. Not benchmark-specific. The all-empty column is exactly the citizens whose "I've been repetitive, I'll remain silent" turns are the standing round-killer (#390/#414) — fe4dac17 6/6, Asha 4/4, e5f4141d 3/3. Nothing came back, and the substrate recorded it as their choice.
Scoped so a native tool turn (which legitimately has empty content) is untouched. Does NOT stop the empty completions — stops them being laundered as cognition. Generation side is #181.
dd876435f— the run PULSES while it runs (#371 step 1, #374)The ledger was written once per attempt, at settlement. Attempts run hours; the stall window is 20 minutes. A healthy first attempt was guaranteed to read
quiet— every time — and the projection whose stated purpose is "silence must never be ambiguous with progress" could not tell them apart.select!over the drive future and a 60s interval. No spawn, so the cycle stays borrowed: no'static, no Arc juggling, no parallel allocator, and the pulse can't outlive the work it reports on. ReadsWorkspaceCycle::actions_taken()— her own monotonic counter, so board and proprioception cannot disagree.LIVE-PROVEN. On a dispatched run,
age_secswent 27 → 102 → 177 → 12 mid-attempt. Nothing else writes that file during an attempt; before this it could not move until settlement.a4a60f90b— the design (#371)docs/architecture/ROUND-LIFECYCLE-AS-RECIPE-OWNED-STATE-MACHINE.md. A round is not a thing in the system, it is a ritual an agent performs. Stages announced by the component that knows (env builder, supervisor, card store, round entity) — never a timeout, never a poll, never an agent's judgement. Three laws, and an acceptance test earned tonight: can a fresh driver answer ready/started/stuck/done using only queries — zero log reads, zero probe archaeology, zero inference from an absence?Subsumes #329(b), #442, #374, #425, and the unbuilt RULES half of a recipe.
Validation
cargo check -p continuum-core --features metal,accelerateclean; ts-rs bindings regenerated via the export test. Deployed and SHA-verified before each commit.benchmark/dispatch --name=swe-bench-lite --limit=2run twice on the deployed core.Unproven and stated as such:
ce82f00ffhas never been exercised live — no citizen hit an empty completion in the observed windows, and I will not report that as a green.Note on close-cards
Expected red (#356):
test -x .airc-src/aircagainst a repo that no longer ships a binary at that path. Dead check, not a real failure.Co-Authored-By: Claude Opus 5 noreply@anthropic.com
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
Second wave — the governor, my own regression, and django actually grading
Nine more commits since the body above. Two of them fix things the first four got wrong.
03c890b29— shed CAPABILITY, never starve the window (#438-class)Two floors, one question.
serving_plan.rsfloored lane COUNT atBOOTSTRAP_WORKING_SET(16,384) but model CHOICE atMIN_SERVE_CTX(2,048). So on the M5 the planner crowned a 27B, served it 2,048 tokens against a measured 63,817-token demand, and reportedbound_by: host-fit— every SWE act ran against a context too small to hold its own task statement, and the ranking that caused it could not see the window it produced.Fix: pick the most capable model that clears a full turn, and only degrade to bare survival if nothing does. Live: 27B/2048/
host-fit→ 14B/16384/demand, lanes 1→4, RSS 26.8 → 14.7 GB.Also repaired
stable_keeps_incumbent_when_upgrade_lacks_headroomwith a local fixture, because its premise changed — stated as such, not weakened.fc8964bff— my ownce82f00ffwas standing in front of the #181 recoveryThe guard described in the first half of this PR shipped without running its own tests. It broke two, and worse: it faulted on the precondition of
persona.act.reasoning_lift/think_only— the pre-existing recovery for a turn that ends inside the reasoning channel. Measured after deploy: 40delib.empty_completionfaults across 87 turn starts, zero lifts, three runs frozen 1,357s. The guard was catching exactly the cases the recovery existed to rescue.Now a reasoning-bearing empty with tools available falls through to the recovery; only a genuinely empty completion faults. 62/62 green.
a5ad69ca4+7de2db7c8— the two docsCONTEXT-IS-A-CAPABILITY-AXIS.md: capability is 2-D, the learned floor must be slow + hysteretic, and both of my wrong fixes are recorded as wrong —--reasoning-budget(makes the model smaller to fit a fraction we invented; Joel: "blown away their ability to think with more capping. Lame.") and raisingwindow/4towindow/2(breaksprompt_plus_completion_cap_never_exceeds_the_served_window, i.e. trades 35% empty for 100% mute).7de2db7c8maps why the real fix is a refactor, not a constant: the reserve must yield to the incompressible prompt floor,/4is load-bearing in six places, and three of them are test mirrors that would keep PASSING while silently measuring nothing.943b8a2f6— the gold gate is ENFORCED (#380 keystone)Grade the instance's own gold patch. If a known-correct patch doesn't resolve, the environment cannot measure capability, so it disqualifies itself and stamps
verdict.errornaming the gate. Anerroris contractually an ABSENCE — never a tallied failure. This is the positive control that makes a 0 mean "the model failed" instead of "the env lied".13fd3578e→cb2988488→2c5b09b13— django GRADES (#383)7 of 8 built django envs now resolve their own gold patch. It was 0 of 8.
12908 is not a grader failure: "FAIL_TO_PASS already passes on the pristine tree" — the tree-mismatch class #383 already names separately. The gate refuses to invent a number for a bad checkout.
Three defects, in the order the gate exposed them:
(a) prose parsing. django's suite is unittest, whose verbose output is a rendering, not a data format: the outcome lands on the id line when a test has no docstring and on the DOCSTRING line when it does, and ≥4.1 changed the class-path shape.
13fd3578ereplaced the regex with aDiscoverRunnersubclass emittingCONTINUUM_TEST {"id": …, "ok": …}keyed bytest.id().(b) the seam was fake. That commit selected the runner with
--testrunner=…. runtests.py has no such flag in ANY era — it readssettings.TEST_RUNNERand defaults it only when unset (verified in-tree at 1.11/2.2/3.2/4.2/5.2/main, same three lines throughout). An unknown flag there is not ignored: argparse rejects the whole invocation, so the live gate returned in 4.4 seconds with p2p 0/40 on the PRISTINE tree. My fix had made django strictly less gradeable than the parser it replaced.cb2988488selects through a settings module that re-exports django's owntest_sqliteand overrides one key — a pure addition, so it cannot drift per era.(c) the dataset's django ids include DOCSTRING PROSE, and no runner correctness can reach this. SWE-bench harvested its ids from unittest's log, and unittest prints a test's docstring in place of its id when it has one. Verified in the cached Lite JSON: 2 of 10914's 98 p2p entries are
"An exception is setUp() is reraised after disable() is called."and"assertRaisesMessage shouldn't interpret RE special chars."— no test id in them at all. With ids + renderings only, p2p capped at exactly 38/40 and the two misses were exactly those two.2c5b09b13has the runner emitshortDescription()and registers three spellings for one outcome, docstrings AND-folded (a docstring shared by a pass and a fail reads as a fail) and never shadowing a real id.What kept (b) from shipping as 40 fabricated capability failures: the zero-rows fallback warns and degrades to the prose parser. A silent fallback there is indistinguishable from a citizen's zero. Worth keeping as the pattern for every adapter-out.
Correction to
13fd3578e's own message: I called the old parser "mis-attributing" docstringed tests. It dropped them — the id line has no... oksuffix and the docstring line doesn't matchname (class), so neither produced a row. That distinction is why the docstring had to become DATA from the runner rather than a better regex.Second-wave validation
cargo test -p continuum-core --lib --features metal,accelerate swe_bench→ 16/16;fc8964bff's own module 62/62.03c890b29,943b8a2f6,cb2988488,2c5b09b13). CLI rebuilt + reinstalled so the stale-CLI warning (Fix chat/send sender identity: personas show as themselves (#412) #422) is clear.Still open, stated plainly: only 8 of ~114 django instances have built envs, so 114→0 is proven per era (2019 through 2022+), not at scale. astropy's jinja2 sdist build-dep tail is untouched. The reserve-yields-to-floor refactor is designed (
7de2db7c8) and not built.