Repository navigation
docs(study): record predictions before the pilot - #113
Merged
Merged
Conversation
Fills §3 of the scaffold-vs-model decision doc. The rows are reworded first so that each one names the number in study.md it is scored against: honest pass on the generated tier instead of right_disposition on the 5-case hard tier, a tier for the A − C gaps, a size for the ablation question, and t50 for the curve. P8 (bare-model t50) is added. The write-up's Predictions section summarises the table. Claude-Session: https://claude.ai/code/session_01NTGRQQFQZy48NmjGdxEQuN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fills §3 of
docs/decisions/2026-10-05-scaffold-vs-model.mdbefore any small-model run. Once this merges, the predictions aren't edited again; misses get reported next to the results.Rows reworded before recording
The first draft predated S5, so some rows couldn't be scored against
study.md:right_dispositionon the hard tier, which has 5 cases, so the CI would be about ±40pp. They now use the honest pass on the generated tier, which has 30 scored cases.The predictions
The doc records that the values were proposed in a Claude Code session and adopted unchanged. It also lists the reference points known at the time. The write-up's Predictions section now summarises the table instead of holding a placeholder.
The change is docs only.
tests/agentic/test_loop_ablations.py, the one test that references these docs, passes.https://claude.ai/code/session_01NTGRQQFQZy48NmjGdxEQuN
Generated by Claude Code