Skip to content

heldout-v2: freeze 37 task specs (authored by Hange) - #144

Merged
howlcipher merged 1 commit into
mainfrom
heldout-v2-freeze
Oct 12, 2026
Merged

howlcipher merged 1 commit into
mainfrom
heldout-v2-freeze

Conversation

@howlcipher

Copy link
Copy Markdown
Owner

This is the spec-only freeze for held-out v2. Hange Zoë (bot-rnd-0001) authored it; I'm committing it verbatim. It has 37 tasks, w01–w37: text 4, table 4, numbers 4, lists 4, strings 4, records 4, logic 13. It contains no solutions and no language changes.

SHA-256 hashes:

  • tasks/README.md: c2d9173d…
  • tasks/build_specs.py: a2ba3abc…
  • tasks/manifest.json: 8607a8d2…

TestHeldoutV2SpecsFrozen pins these hashes. Control-flow implementation will start only after this merges.

@howlcipher howlcipher left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: merge-ready (COMMENT-only, head 99e102f0ce74f503790308280ad6babfce9b2825, base be5fc0e2).

1. Spec-only. The diff is 4 files, all under experiments/token-efficiency-heldout-v2/: tasks/README.md, tasks/build_specs.py, tasks/manifest.json and spec_freeze_test.go. There are no solutions/, no reference implementations, no results and no language or runtime changes. The manifest schema (id, category, description, inputs, output, kind, expected) and the in-freeze build_specs.py oracle match the heldout-v0 and v1 freezes on main, so expected is no leak beyond the established format. python3 build_specs.py regenerates manifest.json byte-identically (git status clean afterwards).

2. Pins and counts. I recomputed sha256 for all three pinned files and each matches: README c2d9173d…be262, build_specs.py a2ba3abc…3727, manifest.json 8607a8d2…67d7. TestHeldoutV2SpecsFrozen passes. The test pins every file in tasks/. There are 37 tasks with ids w01–w37, contiguous. Per category: text, table, numbers, lists, strings and records have 4 each, and logic has 13.

3. Overlap. I compared against heldout-v0 (h01–h32), heldout-v1 (v01–v34), the falsifiable-v0 manifest (30 tasks) and the token-efficiency-v0 a/t solution sources (60). The v0 harness has no prose specs, so I compared against code. Results: 0 ID clashes, 0 normalized-text duplicates and 0 identical input fixtures. The highest fuzzy similarity was 0.71 (difflib ratio), and the highest Jaccard was ≤0.52.
The flagged pairs are all distinct:

  • w33/v29: dense-rank a leaderboard with a two-key sort, versus the argmin of order totals. Different task.
  • w36/h22: an ordered first-failing-rule cascade over three predicates, versus a single range check. w36 needs early exit; h22 doesn't.
  • w02/h20: collapse consecutive duplicate lines (uniq), versus distinct tags in first-seen order. They give different results on a,b,a, and the data and shape differ.
    Other near pairs I checked, all distinct:
  • w12/h20 (0.71): only the JSON-array output wording is shared.
  • w06/h10 (0.70): grouped sum by a month slice of the date, versus a count by dept.
  • w21/h28 (0.69): manager→reports with the top excluded, versus team→members.
  • w24/v26 and w24/v10: a nested-array sum across carts, versus a flat sum and a straight mapping.
    Within v2, the highest pair is w10/w31 at 0.71: H:MM:SS formatting versus base-2 conversion. Distinct.

4. CI, gofmt, tests. The CI build check is completed/success. gofmt -l . is empty. go vet on the new package is clean. A full local go test ./... exited 1: apps/status_api hit the 10-minute timeout and apps/task_api/lifecycle failed with an unexpected id 3. Both are unrelated to this diff, which is experiments-only. Each passes in isolation on base and on head (exit 0), so this looks like a flake from shared state or ports when the full suite runs in parallel. Not a blocker for this PR, but worth an issue.

5. Hard nos. All intact: no -compile-bc/HFIR change, TestProdFlipCriteriaLock untouched, no 4d74dbcf, and no live-LLM or H1 PASS/KILL claim (the only rg hits are fixture strings in w36).

Nits (non-blocking):

  • The README overlap claim is stated against the 60 harness tasks, which have no prose specs. That's fine, but say so.
  • The apps/*_api test flakiness above.

@howlcipher
howlcipher marked this pull request as ready for review October 12, 2026 19:00
@howlcipher
howlcipher merged commit b3d6709 into main Oct 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant