Repository navigation
heldout-v2: freeze 37 task specs (authored by Hange) - #144
Conversation
howlcipher
left a comment
There was a problem hiding this comment.
Verdict: merge-ready (COMMENT-only, head 99e102f0ce74f503790308280ad6babfce9b2825, base be5fc0e2).
1. Spec-only. The diff is 4 files, all under experiments/token-efficiency-heldout-v2/: tasks/README.md, tasks/build_specs.py, tasks/manifest.json and spec_freeze_test.go. There are no solutions/, no reference implementations, no results and no language or runtime changes. The manifest schema (id, category, description, inputs, output, kind, expected) and the in-freeze build_specs.py oracle match the heldout-v0 and v1 freezes on main, so expected is no leak beyond the established format. python3 build_specs.py regenerates manifest.json byte-identically (git status clean afterwards).
2. Pins and counts. I recomputed sha256 for all three pinned files and each matches: README c2d9173d…be262, build_specs.py a2ba3abc…3727, manifest.json 8607a8d2…67d7. TestHeldoutV2SpecsFrozen passes. The test pins every file in tasks/. There are 37 tasks with ids w01–w37, contiguous. Per category: text, table, numbers, lists, strings and records have 4 each, and logic has 13.
3. Overlap. I compared against heldout-v0 (h01–h32), heldout-v1 (v01–v34), the falsifiable-v0 manifest (30 tasks) and the token-efficiency-v0 a/t solution sources (60). The v0 harness has no prose specs, so I compared against code. Results: 0 ID clashes, 0 normalized-text duplicates and 0 identical input fixtures. The highest fuzzy similarity was 0.71 (difflib ratio), and the highest Jaccard was ≤0.52.
The flagged pairs are all distinct:
- w33/v29: dense-rank a leaderboard with a two-key sort, versus the argmin of order totals. Different task.
- w36/h22: an ordered first-failing-rule cascade over three predicates, versus a single range check. w36 needs early exit; h22 doesn't.
- w02/h20: collapse consecutive duplicate lines (uniq), versus distinct tags in first-seen order. They give different results on
a,b,a, and the data and shape differ.
Other near pairs I checked, all distinct: - w12/h20 (0.71): only the JSON-array output wording is shared.
- w06/h10 (0.70): grouped sum by a month slice of the date, versus a count by dept.
- w21/h28 (0.69): manager→reports with the top excluded, versus team→members.
- w24/v26 and w24/v10: a nested-array sum across carts, versus a flat sum and a straight mapping.
Within v2, the highest pair is w10/w31 at 0.71: H:MM:SS formatting versus base-2 conversion. Distinct.
4. CI, gofmt, tests. The CI build check is completed/success. gofmt -l . is empty. go vet on the new package is clean. A full local go test ./... exited 1: apps/status_api hit the 10-minute timeout and apps/task_api/lifecycle failed with an unexpected id 3. Both are unrelated to this diff, which is experiments-only. Each passes in isolation on base and on head (exit 0), so this looks like a flake from shared state or ports when the full suite runs in parallel. Not a blocker for this PR, but worth an issue.
5. Hard nos. All intact: no -compile-bc/HFIR change, TestProdFlipCriteriaLock untouched, no 4d74dbcf, and no live-LLM or H1 PASS/KILL claim (the only rg hits are fixture strings in w36).
Nits (non-blocking):
- The README overlap claim is stated against the 60 harness tasks, which have no prose specs. That's fine, but say so.
- The
apps/*_apitest flakiness above.
This is the spec-only freeze for held-out v2. Hange Zoë (bot-rnd-0001) authored it; I'm committing it verbatim. It has 37 tasks, w01–w37: text 4, table 4, numbers 4, lists 4, strings 4, records 4, logic 13. It contains no solutions and no language changes.
SHA-256 hashes:
TestHeldoutV2SpecsFrozen pins these hashes. Control-flow implementation will start only after this merges.