Token-efficiency benchmark: HowlFrame vs Python/JS/Go (measure only)

[howlcipher]

Measure-only slice. Tokens per task on the 60 falsifiable-v0 tasks, with references in HowlFrame, Python, JS and Go. Each reference is oracle-verified: 240/240 pass. Counted offline with tiktoken==0.8.0 (cl100k_base, o200k_base).

Result: HowlFrame uses more tokens than Python and JS on all 60 tasks and fewer than Go on all 60.
Median tokens (cl100k): Howl 160 / Py 119 / JS 122 / Go 250. Median per-task ratios: 1.42x Py, 1.37x JS, 0.64x Go (o200k: 1.42 / 1.27 / 0.64).

Report: experiments/token-efficiency-v0/REPORT.md. Journal: docs/journals/2026-10-12_token_efficiency_benchmark.md. Run: TOKBENCH_PYTHON= experiments/token-efficiency-v0/run.sh

Hard nos: no syntax/compiler/VM changes, HFIR default and -compile-bc untouched, #90 Partial, no live LLM or H1 PASS/KILL. The only harness change is a small exported materialize helper in experiments/falsifiable-v0/harness.

[howlcipher]

Dev-lead review (COMMENT only), head 4645d0b821d4fcf8e0c42f395e26c0a9b31faa3c.

Verdict: merge-ready on content. CI build is green. Okabe may undraft and squash-merge.

Scope. 253 files against merge-base 8793d6ed. 251 are under experiments/token-efficiency-v0/. The other two are docs/journals/2026-10-12_token_efficiency_benchmark.md (journal, fine) and a new experiments/falsifiable-v0/harness/materialize.go (11 lines). That file only wraps the existing validateTask and materialize in an exported MaterializeReferenceSandbox. It changes no scoring, oracle or Diff behavior. Nothing touches internal/, the compiler, the VM, syntax or CI. It merges cleanly onto current main dc46a4ec.

Hard nos: all intact. HFIR default and -compile-bc are not flipped. #90 stays Partial. There are no live LLM calls: count_tokens.py disables tiktoken's uncached loader so it fails offline, and run.sh does no pip install. There is no H1 claim.

Methodology

• tiktoken==0.8.0 is pinned and its version is asserted at runtime. Both cl100k_base and o200k_base are counted.
• The raw count includes all source text (comments, imports, Go boilerplate and build tag). A secondary comments- and blank-lines-stripped variant is also reported.
• Oracle pass is checked by tokverify (hfexp arm-a for HowlFrame; shared materializer, Diff, UnauthorizedDiff and CompareOracle for the other languages). verification.json has 240/240 passing. The counter matches each source file's SHA-256 against verification and rejects any coverage that isn't exactly 60×4.
• I recomputed the ratios from results/tokens.json, which has 480 rows. Median per-task ratios for cl100k are 1.422 vs Python, 1.368 vs JS and 0.640 vs Go. For o200k they are 1.417, 1.270 and 0.636. Medians are Howl 160, Py 119, JS 122, Go 250. These match REPORT.md and the PR body. A loose "1.3x JS" undersells cl100k slightly (1.37); quote 1.37 for cl100k and 1.27 for o200k.
• Threats to validity are stated honestly: single-author/template bias, oracle visible during authoring, 30 pairs with identical source, tokenizer as a proxy rather than a model, Go boilerplate, capability declarations excluded.

Non-blockers

1. Baseline staleness. Main has moved to dc46a4ec with #128–#130, which add dogfood token-density sugar (desugar, trim/lower/join, etc.). The HowlFrame references and numbers here reflect 8793d6ed syntax. Label them as the pre-sugar baseline, and rerun or add a v0.1 on current main before using them to motivate compact-surface work.
2. The last section of REPORT.md is stale. It says "Changes are left uncommitted…" and "cannot claim green repository CI", but the work is committed and CI is green. Please update that wording in a follow-up.
3. The Dependencies sentence in REPORT.md is garbled ("the preinstalled a Python with (set TOKBENCH_PYTHON…)").
4. The JS/Go comment stripper does not handle regex or template literals. That is documented, and it doesn't affect the headline raw counts.
5. The native verifier only diffs trees, with no host isolation. That is documented as reference-only.