heldout-v2: freeze 37 task specs (authored by Hange)

experiments/token-efficiency-heldout-v2/spec_freeze_test.go

@@ -0,0 +1,26 @@
+package heldoutv2
+
+import (
+	"crypto/sha256"
+	"fmt"
+	"os"
+	"testing"
+)
+
+// The frozen heldout-v2 spec files must match the hashes recorded at freeze time.
+func TestHeldoutV2SpecsFrozen(t *testing.T) {
+	frozen := map[string]string{
+		"tasks/README.md":      "c2d9173ddf1ad4586d6bb02ed72b9197ce752f0fcd2e0ec0a3641ae8112be262",
+		"tasks/build_specs.py": "a2ba3abc8c3151c6c27dd35200a5c79bd6a551de0b4e59cd66b5a35f5a773727",
+		"tasks/manifest.json":  "8607a8d255115a225311208636e6563073aeace0e839a63b0e9f71aa17a467d7",
+	}
+	for path, want := range frozen {
+		data, err := os.ReadFile(path)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if got := fmt.Sprintf("%x", sha256.Sum256(data)); got != want {
+			t.Errorf("%s changed after freeze: %s", path, got)
+		}
+	}
+}

experiments/token-efficiency-heldout-v2/tasks/README.md

@@ -0,0 +1,46 @@
+# Held-out token-efficiency tasks, third set (heldout-v2): frozen specs
+
+This file and the others in `tasks/` (`manifest.json`, `build_specs.py`) are
+**frozen**. They were written before any control-flow/logic built-in or
+reference solution, and before anything was measured.
+
+- Author: R&D seat bot-rnd-0001 (not the implementer of the measured slice).
+- Base: main `be5fc0e2`.
+- `manifest.json` is generated by `python3 build_specs.py` and is byte-identical
+  on every run.
+
+## Corpus
+
+There are 37 tasks in 7 categories. None of them repeats or rewords a task from
+token-efficiency-v0 (release-gating evidence tasks a01–a30, t01–t30),
+token-efficiency-heldout-v0 (h01–h32) or token-efficiency-heldout-v1 (v01–v34).
+About a third of the set (13 tasks) is logic/control-flow work: branching,
+early exit, modular arithmetic, searching, ranking and small state machines.
+The tasks were written as general programs, not around any planned built-in.
+There are no directory-walk tasks.
+
+| Category | Tasks |
+|---|---|
+| text | w01–w04 |
+| table | w05–w08 |
+| numbers | w09–w12 |
+| lists | w13–w16 |
+| strings | w17–w20 |
+| records | w21–w24 |
+| logic | w25–w37 |
+
+## Running a reference
+
+The rules are the same as heldout-v0 and heldout-v1:
+
+- Each reference runs with its working directory set to a fresh sandbox
+  containing only the task's `inputs`.
+- It may read under `in/` and must write exactly the one `output` path.
+- `text` outputs are compared exactly, after trailing newlines are removed
+  from both sides. `json` outputs are compared as decoded values.
+- `expected` values come from the spec-time oracle in `build_specs.py`, which
+  is part of this freeze. The oracle is not a counted or verified reference.
+- HowlFrame references are built with `build --profile governed` and run with
+  `-run-bc` and only `filesystem:read=in`, `filesystem:write=out`.
+- A task that a language cannot express with the features on the measured
+  commit is recorded as a gap with a reason, not as a solution.