heldout-v3: freeze 35 task specs (authored by Lain)

experiments/token-efficiency-heldout-v3/spec_freeze_test.go

@@ -0,0 +1,26 @@
+package heldoutv3
+
+import (
+	"crypto/sha256"
+	"fmt"
+	"os"
+	"testing"
+)
+
+// The frozen heldout-v3 spec files must match the hashes recorded at freeze time.
+func TestHeldoutV3SpecsFrozen(t *testing.T) {
+	frozen := map[string]string{
+		"tasks/README.md":      "309081e4062d432c95d649ca4642d477d0724f482acd15c6889b6532b4f8928c",
+		"tasks/build_specs.py": "199d297eefc2286fbe05a30cfbf574f461cbbd24680290b669732ccb6372e2a1",
+		"tasks/manifest.json":  "85bcde28428d27ce835efaf86d153f30094c3a70a43ae2be1f2f4eae6adcaa0c",
+	}
+	for path, want := range frozen {
+		data, err := os.ReadFile(path)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if got := fmt.Sprintf("%x", sha256.Sum256(data)); got != want {
+			t.Errorf("%s changed after freeze: %s", path, got)
+		}
+	}
+}

experiments/token-efficiency-heldout-v3/tasks/README.md

@@ -0,0 +1,61 @@
+# Held-out token-efficiency tasks, fourth set (heldout-v3): frozen specs
+
+This file and the others in `tasks/` (`manifest.json`, `build_specs.py`) are
+**frozen**. They were written before any records/JSON slice or reference
+solution, and before anything was measured.
+
+- Author: Assurance seat bot-assurance-0001 (not the implementer of the measured slice).
+- Base: main `d887b96b`.
+- Freeze date: 2026-10-12.
+- `manifest.json` is generated by `python3 build_specs.py` and is byte-identical
+  on every run.
+
+## Corpus
+
+There are 35 tasks in 7 categories, weighted toward records and JSON
+(17 tasks): nested get-in paths, deep merge, flatten/unflatten, null versus
+missing keys, grouping into records, key renaming, canonical JSON out with
+sorted keys, object diff, nested predicates, defaults for missing paths,
+composite keys, JSON Lines in, value transforms, depth, entries, and set/del
+patches. The rest are spread across text, table, numbers, lists, strings and
+logic. The tasks were written as general programs, not around any planned
+built-in. There are no directory-walk tasks.
+
+| Category | Tasks |
+|---|---|
+| records | x01–x17 |
+| text | x18–x20 |
+| table | x21–x23 |
+| numbers | x24–x26 |
+| lists | x27–x29 |
+| strings | x30–x32 |
+| logic | x33–x35 |
+
+## Overlap check
+
+None of the tasks repeats or rewords a task from token-efficiency-v0
+(a01–a30, t01–t30), falsifiable-v0 (the experiment harness, t01–t30),
+token-efficiency-heldout-v0 (h01–h32), token-efficiency-heldout-v1 (v01–v34)
+or token-efficiency-heldout-v2 (w01–w37). Method: every description in those
+manifests (and the v0 author solutions) was listed on main `d887b96b` and each
+v3 task was compared against all of them by hand for the same operation,
+the same input shape or a near paraphrase. Ideas that were close were dropped
+or changed (for example, no id-to-name maps, inverting objects, grouping
+members into arrays, sku totals, uniq/RLE, interleaving or rotation). Input
+file names and data values are new.
+
+## Running a reference
+
+The rules are the same as heldout-v0, heldout-v1 and heldout-v2:
+
+- Each reference runs with its working directory set to a fresh sandbox
+  containing only the task's `inputs`.
+- It may read under `in/` and must write exactly the one `output` path.
+- `text` outputs are compared exactly, after trailing newlines are removed
+  from both sides. `json` outputs are compared as decoded values.
+- `expected` values come from the spec-time oracle in `build_specs.py`, which
+  is part of this freeze. The oracle is not a counted or verified reference.
+- HowlFrame references are built with `build --profile governed` and run with
+  `-run-bc` and only `filesystem:read=in`, `filesystem:write=out`.
+- A task that a language cannot express with the features on the measured
+  commit is recorded as a gap with a reason, not as a solution.