feat(bbh): resilient sweep — stream results, abort-on-exhaustion, auto-resume#5
Open
nickmeinhold wants to merge 8 commits into
Open
feat(bbh): resilient sweep — stream results, abort-on-exhaustion, auto-resume#5nickmeinhold wants to merge 8 commits into
nickmeinhold wants to merge 8 commits into
Conversation
The BBH sweep wrote results only once, at the end of the loop, from an in-memory list — so any process death mid-run (Max usage exhausted, kill, laptop sleep) lost the entire run. Worse, a usage-limit error was caught per-call and recorded as passed=False, so an exhausted run limped to completion writing hundreds of failure rows that summarize() averaged into pass rates as if they were wrong answers (silent corruption). Three fixes: - Stream each TaskResult to disk with a flush; partial progress now survives process death (OS page cache outlives the process). - Detect genuine Max-usage exhaustion (usage limit / credit / 429) and abort loudly with the exact resume command, instead of poisoning the dataset. Transient signals (timeout / 529) still record-and-continue. - Add --resume <jsonl>: skip already-completed (task_id, arm) pairs and append, so an interrupted sweep continues where it stopped. Output file format is byte-identical (meta line + one row per result); existing analyze_sweep.py / RUN_FOR_NICK.md commands are unaffected. Verified: unit tests for the classifier + resume parse, and a live n=1 end-to-end (fresh write lands on disk, resume skips the done pair). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
run_bbh_resumable.sh wraps run_bbh_pilot.py: on a usage-exhaustion abort (exit 2) it extracts the partial-results path, waits BACKOFF_SECONDS for the Max window to recover, and re-invokes with --resume — looping up to MAX_RETRIES until the sweep completes or a hard error occurs. Lets a long n=100+ run survive hitting the 5-hour usage window with no human in the loop. Verified: syntax check, path-extraction unit, live n=1 run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gen_progress.py renders the streaming sweep JSONL into a self-contained auto-refreshing HTML status page (overall + per-arm progress, pass rates, cost units, ETA). publish_progress.sh force-pushes it to the gh-pages branch every 5min (under GitHub Pages' ~10 builds/hr soft limit) from an isolated worktree until the run PID exits. Live at enspyrco.github.io/echo. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The harness shells out to `claude --print` from inside the Echo project
dir, where `claude -p` loads the project CLAUDE.md, runs SessionStart
hooks, and ingests the task-restore injection. Every BBH model call was
therefore answering AS the Echo agent ("All 8 tasks restored. Now to
your question...") rather than the benchmark question.
This — not a parser bug or "Sonnet burying the format line" — is the
real cause of the n=30/n=99 Sonnet unparseables and the 98-125s wall
times (loading a huge CLAUDE.md + hooks per call). Accuracy was
confounded too.
Fix: pass `--setting-sources ""` so none of {user,project,local}
settings load (no hooks/CLAUDE.md/task-restore) while keeping the Max
OAuth auth (--bare drops auth, so it's unusable here).
Verified via scripts/repro_sonnet_unparse.py on the 3 known failing
tasks: all now parse cleanly, wall times drop ~6-10x (99s->6s), and
date_understanding/5 goes unparseable->correct. 25 parser/scoring unit
tests still pass.
IMPLICATION: all prior BBH sweeps (n=15/30/99) are contaminated and
must be re-run with the fixed harness before #336/#619.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First BBH sweep run with the --setting-sources "" fix (commit 23030a2), so model calls no longer inherit the project CLAUDE.md/hooks/task-restore. Clean n=30 (3 subtasks × 10): haiku-only 0.867 10.4s 0 unparse sonnet-only 0.900 10.3s 0 unparse echo-judge 0.867 17.4s 0 unparse echo-oracle 0.933 14.0s 0 unparse vs contaminated 20260625T013951Z_bbh_n30.jsonl: sonnet-only was 0.833 @ 40.7s with 3 unparseables — the contamination cost Sonnet ~7 pts and 4x latency, and had pushed it BELOW Haiku. Clean, Sonnet (0.900) edges Haiku (0.867) — direction restored — and zero unparseables across all arms. Separation is +1 task at n=30, so still needs a clean n=99 to be decisive; committing the datapoint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full n=99 (3 subtasks × 33) on the --setting-sources "" fixed harness. Clean n=99: haiku-only 0.848 11.3s 0 unparse sonnet-only 0.848 10.2s 0 unparse echo-judge 0.869 16.8s 0 unparse echo-oracle 0.899 14.3s 0 unparse vs contaminated 20260625T014218Z_bbh_n99.jsonl (sonnet-only 0.657 @ 53s, 20 unparseables): the contamination cost Sonnet ~19 pts and 5x latency. Findings: - Sonnet-vs-Haiku separation: NONE on this slice. Both 0.848, and identical per-subtask (causal 22/33, date 29/33, logical 33/33). These subtasks are near-ceiling and don't discriminate the two models. A decisive separation needs harder subtasks (logical_deduction_seven, etc). - Complementarity IS real despite the aggregate tie: echo-oracle 0.899 > both baselines because Haiku and Sonnet miss DIFFERENT items (oracle recovers causal_judgement 27/33 vs baselines' 22/33). That gap is the headroom Echo routing targets. - echo-judge 0.869 edges above both baselines but leaves ~3 pts of the oracle headroom uncaptured — the real judge isn't yet as good as the perfect router. This is where cross-family judges (#619) come in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Experiment A: tested whether HARDER BBH subtasks reveal a Sonnet-vs-Haiku
separation that the near-ceiling easy slice couldn't (n=33 x 3, clean
--setting-sources "" harness).
Subtasks chosen as the reasoning-heavy "hard" siblings:
logical_deduction_seven_objects, tracking_shuffled_objects_seven_objects,
temporal_sequences
Result — every subtask is a CEILING for BOTH model tiers:
haiku-only 99/99 1.000
sonnet-only 99/99 1.000
echo-judge 98/99 0.990 (one routing miss)
echo-oracle 99/99 1.000
Per-subtask: haiku & sonnet BOTH 33/33 on all three.
KEY FINDING: current Haiku already aces the MCQ BBH subtasks that used to
discriminate model tiers (incl. 7-object state-tracking). So:
- No Sonnet-vs-Haiku accuracy separation exists to route on.
- No oracle->echo-judge headroom (1-task noise gap) for a better judge to
close, so the OpenAI cross-family judge run (Experiment B) was correctly
NOT launched — nothing to capture.
Implication: accuracy-based routing value cannot be demonstrated on MCQ BBH
with 2026 Claude models. Forward paths: (1) extend harness to score
free-form subtasks (word_sorting/dyck_languages/multistep_arithmetic — the
genuinely hard ones, currently unscoreable), (2) move to a non-saturated
benchmark (MMLU-Pro #620, GPQA), or (3) reframe Echo's value from accuracy
to cost/latency-neutral routing ("route to cheap, lose nothing").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… blog Rigorous check that the HumanEval headline predates/escaped the project-context contamination. Re-ran haiku-only over HumanEval 100-163 on the fixed harness (--setting-sources ""): 63/64 = 0.984, EXACTLY the blog's published baseline. An exact reproduction is only possible if the contamination never affected the HumanEval runs (the task-restore SessionStart hook postdated them; coherent high pass rates were incompatible with the contamination signature anyway). So the blog's HumanEval headline foundation is sound. (sonnet-only / echo-judge / echo-small-judge not yet re-run this pass — baseline match is the decisive contamination test.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The BBH sweep wrote results only once, at the end of the loop, from an in-memory list. Any process death mid-run (Max usage exhausted, kill, laptop sleep) lost the entire run. Worse, a usage-limit error was caught per-call and recorded as
passed=False, so an exhausted run limped to completion writing hundreds of failure rows thatsummarize()averaged into pass rates as if they were wrong answers — silent corruption of the headline metric.This matters now: we're scaling to n=100 (~720 Claude calls) on the Max 20× plan, where brushing the 5-hour usage window is plausible.
What
analyze_sweep.py/RUN_FOR_NICK.mdunaffected.usage limit/credit/429) with the exact resume command, instead of poisoning the dataset. Transient signals (timeout/529) still record-and-continue — one blip doesn't kill a run.--resume <jsonl>— skip already-completed(task_id, arm)pairs and append. Resume is clean because the task list is deterministic (load_bbh(subtasks, n_per_subtask, start)), so "what's done" is recoverable from the output file alone.run_bbh_resumable.sh— unattended wrapper: catches the exit-2 abort, backs offBACKOFF_SECONDS, re-invokes with--resume, up toMAX_RETRIES. A long sweep survives the usage window with no human in the loop.Verification (by running, not just reasoning)
🤖 Generated with Claude Code