Held-out token benchmark (heldout-v0): HowlFrame 1.31x Python on unseen tasks (measure only)

[howlcipher]

This adds a held-out token benchmark. It measures only: no built-in, syntax, compiler or VM change.

Spec freeze: f510cd01 contains only the 32 task specs and the oracle, committed before any solution existed. The hashes are in the journal and enforced by TestHeldoutSpecsFrozen.

Result: HowlFrame loses to Python on held-out tasks

	HowlFrame / Python, median per-task ratio	Wins / ties / losses	vs JS	vs Go
cl100k_base	1.309x	0 / 1 / 29	1.087x (6/1/23)	0.499x (30/0/0)
o200k_base	1.299x	0 / 1 / 29	1.016x (13/0/17)	0.493x (30/0/0)

The v0 corpus gave 0.983x. That result came from built-ins shaped for those tasks and does not generalize.

By category, HowlFrame / Python ranges from CSV at 1.84x down to reshape at 1.08x. The two file-walking tasks are HowlFrame gaps: there is no directory listing construct.

Hot spots:

1. No index or slice primitives (CSV column lookup, skipping the header).
2. Counting into a map takes about 25 tokens, against Python's Counter.
3. No map/filter/sum/min/max/reverse, so every accumulation is an imperative loop.

Findings (recorded, not fixed):

• Sugar inside write_file/to_lower arguments is not expanded.
• or is not short-circuit.
• map_get on a missing key returns "".
• Lists taken out of a map have value semantics.

Contents

• experiments/token-efficiency-heldout-v0/: frozen tasks/, cmd/heldoutverify, count_tokens.py (reuses v0's stripping and offline guard), run.sh, gaps.json and verification_test.go.
  • The test enforces coverage, all-pass, matching hashes and the gap rules.
• References: 32 each in Python, JS and Go, and 30 in HowlFrame.
• Regenerated results, REPORT.md, and the journal docs/journals/2026-10-12_heldout_token_benchmark.md.

Validation (local, unsandboxed)

• go build, go vet and go test ./...: exit 0.
• go test ./experiments/...: passes.
• Held-out run.sh: 122/122 pass.
• v0 run.sh: unchanged.

No production -compile-bc or HFIR change, #90 stays Partial, and there were no live LLM or H1 runs.

[howlcipher]

Dev-lead review (COMMENT only), head 6ec043254fb5a82642d04979a69fcf8836e7435a, base 24f6a10f.

Freeze integrity: pass. f510cd01 adds only tasks/{README.md,manifest.json,build_specs.py}, with no solutions. git diff f510cd0..HEAD -- tasks/ is empty, so no spec or oracle changed after the freeze. TestHeldoutSpecsFrozen pins the frozen files by sha256. The commit order is correct: the freeze commit comes first, and refs, verifier and report come after. The 7 categories (text/csv/aggregate/parse/validate/reshape/walk) are generic, and I didn't see any spec shaped to favor or punish one language.

Verifier: pass. go test ./experiments/token-efficiency-heldout-v0/... passes locally. TestHeldoutVerificationCoverage checks the sha256 of every solution, requires exact coverage (tasks×4 − gaps), and requires every gap to have a reason and no solution file. There are 122 passing refs. The 2 gaps (h31/h32, walk) are listed explicitly in gaps.json, not dropped.

Report honesty: pass. I recomputed from results/tokens.csv over the 30 tasks HowlFrame solves:

• vs Python: median per-task ratio is 1.309 (cl100k) / 1.299 (o200k), ratio of medians is 1.151, and wins/ties/losses are 0/1/29 for both encodings.
• Medians: HowlFrame 72.5, Python 63, JS 69/73, Go 162.5/164.5.
• vs JS: 1.087 / 1.016. vs Go: 0.499 / 0.493.

All of these match REPORT.md and the PR body. The report opens with "does not generalize", gives the 0/1/31 figure with gaps counted as losses, and states the limits: one author, tokenizer proxy, no live LLM. I spot-checked h09 (csv→records). The HowlFrame ref is idiomatic for current main. Its length comes from the missing zip/comprehension and header-skip idioms, not from padding.

Non-blocking: one sentence saying that leaving out the 2 gaps flatters HowlFrame's median would help, since the 0/1/31 line implies it but doesn't say it. HowlFrame does worst on csv (1.84x), aggregate (1.48x) and parse (1.40x), and walk can't be expressed. Those are good candidates for the next slice.

Scope: the changes are experiments/token-efficiency-heldout-v0/** plus a journal. Nothing touches internal/, syntax, the VM, built-ins or the original 60-task benchmark.

Hard nos: intact. No -compile-bc/HFIR flip, #90 stays Partial, no live LLM, no H1 PASS/KILL claim.

CI: build was in progress at review time.

Verdict: merge-ready on content. Okabe may undraft and squash-merge once CI is green.