Records/JSON sugar (pure) + heldout-v3 harness and before/after results

change_log.md

@@ -3,6 +3,7 @@
 ## Unreleased
 
 ### Added
+* Records and JSON sugar (heldout-v3 records slice): `(rec k v …)` record literals with name keys, `(get m k …)` / `(put m k … v)` nested path read/write, `(read_json p)` / `(write_json p v)` one-call file JSON under the existing filesystem grant, `(pluck lst k)`, `(select lst k …)`, `(group_by x lst key)`, `(merge a b …)`. Desugar-only: no opcode or capability; arity entries; Go/JS reject with `UNSUPPORTED_ON_TARGET`; governed scan fails closed through every form. `parse_json` now runs under `-run`. `encode_json` already writes sorted keys, so output is stable.
 * Dogfood round 4: `apps/fetch_cli`, `apps/template_cli`, `apps/stamp_api`; `(map_has)`, `(when)`/`(unless)`, n-ary `and`/`or` desugar. Suite total HowlFrame 2331 vs Python 3025 / Go 3295 / Node 2927. Journal: `docs/journals/2026-10-10_dogfood_apps_round4.md`.
 * Dogfood round 3: `apps/wc_cli`, `apps/json_get`, `apps/counter_api`; density sugar `(str …)` / `(nz x d)`; `list_len` accepts string/bytes; `http_server` startup `store_open`/`store_put`/`do`/`let`. Real-app harness extended; totals HowlFrame 1907 vs Python 2395 / Go 2642 / Node 2297. Journal: `docs/journals/2026-10-10_dogfood_apps_round3.md`.
 * Dogfood round 2: `apps/notes_cli`, `apps/auth_api`, `apps/grep_cli` built on standalone bytecode; `store_open` accepts a runtime URI expression; AST density sugar `(switch …)` and `(inc x)`. Real-app token harness extended with Python/Go/Node ports and parity. Journal: `docs/journals/2026-10-10_dogfood_apps_round2.md`.

docs/direct_execution_design.md

@@ -33,7 +33,9 @@ Wired into `main()` via a new `-run` boolean flag:
 
 ### Explicitly out of scope for Phase 1
 
-`http_server`/`route`/`middleware`, `struct`/`parse_json`/`res`/`res_json`, `db_connect`/`sql_query`, `spawn`, `fetch`, `llm_generate`/`fuzzy_cast`/`assert_semantic`, `try_let`, `patch`, `with_context`, `test`, `match`, `exec`, `read_file`/`write_file`/`mkdir`, `web_app`/JS target. Attempting any of these under `-run` produces a clear "not supported under -run (Phase 1)" JSON error naming the specific node, not a silent no-op or crash. `read_file`/`write_file`/`mkdir` were deliberately deferred rather than included: the Go backend's usage pattern for these depends on `try_let` to catch the `(value, error)` tuple they return, and `try_let` itself is Phase 2 scope — including the I/O primitives without an error-handling primitive to pair them with would mean inventing an ad hoc error convention with no precedent, better decided alongside `try_let` itself.
+`http_server`/`route`/`middleware`, `struct`/`res`/`res_json`, `db_connect`/`sql_query`, `spawn`, `fetch`, `llm_generate`/`fuzzy_cast`/`assert_semantic`, `try_let`, `patch`, `with_context`, `test`, `match`, `exec`, `read_file`/`write_file`/`mkdir`, `web_app`/JS target. Attempting any of these under `-run` produces a clear "not supported under -run (Phase 1)" JSON error naming the specific node, not a silent no-op or crash. `read_file`/`write_file`/`mkdir` were deliberately deferred rather than included: the Go backend's usage pattern for these depends on `try_let` to catch the `(value, error)` tuple they return, and `try_let` itself is Phase 2 scope — including the I/O primitives without an error-handling primitive to pair them with would mean inventing an ad hoc error convention with no precedent, better decided alongside `try_let` itself.
+
+`parse_json` has since been added to `-run` (heldout-v3 records slice): `(parse_json T text)` decodes an in-memory string, bytes or byte-number list with the same `decodeJSONValue` the bytecode `PARSE_JSON` uses, and needs no capability.
 
 ### Value representation
 

docs/reference/construct_coverage.md

@@ -50,7 +50,9 @@
 | `filter` | No | None | Yes | No | - | AUTHORITATIVE |
 | `for` | Yes | None | No | No | - | AUTHORITATIVE |
 | `fuzzy_cast` | No | None | No | Yes | - | AUTHORITATIVE |
+| `get` | No | None | Yes | No | - | AUTHORITATIVE |
 | `go_import` | No | None | No | Yes | - | AUTHORITATIVE |
+| `group_by` | No | None | Yes | No | - | AUTHORITATIVE |
 | `html_escape` | Yes | None | No | No | - | AUTHORITATIVE |
 | `http_body` | No | None | Yes | No | - | AUTHORITATIVE |
 | `http_request` | Yes | network | No | No | - | AUTHORITATIVE |
@@ -82,6 +84,7 @@
 | `match` | No | None | No | Yes | - | AUTHORITATIVE |
 | `max` | No | None | Yes | No | - | AUTHORITATIVE |
 | `max_by` | No | None | Yes | No | - | AUTHORITATIVE |
+| `merge` | No | None | Yes | No | - | AUTHORITATIVE |
 | `middleware` | No | None | No | Yes | - | AUTHORITATIVE |
 | `min` | No | None | Yes | No | - | AUTHORITATIVE |
 | `min_by` | No | None | Yes | No | - | AUTHORITATIVE |
@@ -97,11 +100,15 @@
 | `pad_right` | No | None | Yes | No | - | AUTHORITATIVE |
 | `parse_json` | Yes | None | No | No | - | AUTHORITATIVE |
 | `patch` | No | None | Yes | No | - | AUTHORITATIVE |
+| `pluck` | No | None | Yes | No | - | AUTHORITATIVE |
 | `print` | Yes | None | No | No | - | AUTHORITATIVE |
+| `put` | No | None | Yes | No | - | AUTHORITATIVE |
 | `range` | No | None | Yes | No | - | AUTHORITATIVE |
 | `read_file` | Yes | filesystem | No | No | - | AUTHORITATIVE |
+| `read_json` | No | None | Yes | No | - | AUTHORITATIVE |
 | `read_line` | Yes | None | No | No | - | AUTHORITATIVE |
 | `read_text` | No | None | Yes | No | - | AUTHORITATIVE |
+| `rec` | No | None | Yes | No | - | AUTHORITATIVE |
 | `reduce` | No | None | Yes | No | - | AUTHORITATIVE |
 | `regex_match` | Yes | None | No | No | - | AUTHORITATIVE |
 | `req_header` | Yes | None | No | No | - | AUTHORITATIVE |
@@ -116,6 +123,7 @@
 | `rtrim` | No | None | Yes | No | - | AUTHORITATIVE |
 | `schema` | No | None | No | Yes | - | AUTHORITATIVE |
 | `schema_bridge` | Yes | None | No | No | - | AUTHORITATIVE |
+| `select` | No | None | Yes | No | - | AUTHORITATIVE |
 | `semantic_match` | No | None | No | Yes | - | AUTHORITATIVE |
 | `set` | Yes | None | No | No | - | AUTHORITATIVE |
 | `sget` | No | None | Yes | No | - | AUTHORITATIVE |
@@ -171,3 +179,4 @@
 | `while` | Yes | None | No | No | - | AUTHORITATIVE |
 | `with_context` | No | None | Yes | No | - | AUTHORITATIVE |
 | `write_file` | Yes | filesystem | No | No | - | AUTHORITATIVE |
+| `write_json` | No | None | Yes | No | - | AUTHORITATIVE |

experiments/token-efficiency-heldout-v3/REPORT.md

@@ -0,0 +1,82 @@
+# Held-out v3: records/JSON slice, measured blind
+
+Specs in `tasks/` are frozen (authored by bot-assurance-0001, base `d887b96b`,
+pinned by `spec_freeze_test.go`) and were not edited. This directory adds the
+harness, references and results only.
+
+## Protocol
+
+1. **Phase 1, blind.** The records/JSON forms were designed and committed
+   (`e18ad5d`) before anything in this directory was opened, guided only by
+   heldout-v2 evidence (records flat at 1.19x Python; cost in s-expression
+   structure and JSON I/O).
+2. **Phase 2.** After that commit the specs were read and the references were
+   written: Python, JS and Go once (`solutions/`), HowlFrame twice:
+   - `solutions/howl/` baseline: only features on main `a6bb53b6`. Verified
+     35/35 with a binary built from `a6bb53b6` and with the slice binary.
+   - `postrec/solutions/howl/` the same programs rewritten to use the slice
+     where it applies (`run_postrec.sh`). They do not build on `a6bb53b6`.
+3. `run.sh` / `run_postrec.sh` replay every reference in a fresh sandbox
+   (`cmd/heldoutv3verify`, same rules as v2: governed build, `-run-bc`,
+   `filesystem:read=in`, `filesystem:write=out`), then count tokens with
+   tiktoken 0.8.0 (cl100k_base, o200k_base; `count_tokens.py` reuses the v0
+   comment stripping). `TOKBENCH_PYTHON=/workspace/tokvenv/bin/python`.
+4. Results are pinned by `results_freeze_test.go` and `postrec_test.go`;
+   `verification_test.go` checks coverage and source hashes. No gaps.
+
+## Headline (cl100k_base; o200k_base in `postrec/results/compare.md`)
+
+| HowlFrame vs | Median per-task ratio before → after | Total tokens before → after | W/T/L before → after |
+|---|---|---|---|
+| Python | 1.282 → **1.186** | 3783 → 3547 / 2840 (1.332 → **1.249**) | 3/0/32 → **8/0/27** |
+| JS | 1.103 → **1.024** | 3783 → 3547 / 3304 (1.145 → **1.074**) | 13/0/22 → **16/0/19** |
+| Go | 0.460 → **0.434** | 3783 → 3547 / 8095 (0.467 → **0.438**) | 35/0/0 → **35/0/0** |
+
+Median tokens per task: HowlFrame 105 → 99, Python 77, JS 94, Go 221.
+
+| Category | n | Median howl/py before → after | Total howl/py before → after |
+|---|---|---|---|
+| records | 17 | 1.195 → 1.113 | 1.244 → 1.118 |
+| text | 3 | 1.282 → 1.282 | 1.377 → 1.377 |
+| table | 3 | 1.411 → 1.400 | 1.431 → 1.427 |
+| numbers | 3 | 1.260 → 1.260 | 1.411 → 1.370 |
+| lists | 3 | 1.247 → 1.137 | 1.184 → 1.047 |
+| strings | 3 | 1.458 → 1.458 | 1.436 → 1.436 |
+| logic | 3 | 1.766 → 1.766 | 1.684 → 1.654 |
+
+## Reading it honestly
+
+- HowlFrame still **loses to Python on 27 of 35** tasks and on total tokens
+  (1.25x). The slice moved records from 1.24x to 1.12x of Python (total), not
+  to parity. Against JS it is now roughly even; against Go it wins every task.
+- What paid: `read_json`/`write_json` (every JSON task), `get` (1 token
+  shorter per read, multi-step paths collapse), `rec` for literal records.
+- What did not: `put`/`get` take a static path, so the three tasks with
+  runtime dotted paths (x01, x04, x17) still loop by hand and stay at
+  1.1-1.5x. `merge` is shallow, so deep merge (x02) is still a recursive
+  defun. `pluck`/`select`/`group_by`/`merge` were not used by any v3
+  reference: the blind guess at those shapes did not match this corpus.
+- Remaining structural costs seen in the baseline: no type predicate (object
+  vs array is tested with `(starts_with (encode_json v) "{")`), recursion
+  needs a typed `defun`, `(call f …)` and explicit `return`, `append` is a
+  statement (no value), and list equality compares references.
+- Non-record categories were untouched by design; their ratios moved only
+  where a JSON read/write appears.
+
+## Findings for follow-up (not fixed here)
+
+- HFIR bytecode: in `(print (encode_json (parse_json Any s)) (let (t (parse_json Any s)) …))`
+  the first argument prints the raw text `s` instead of the re-encoded value
+  (reproduces on `a6bb53b6`; AST bytecode and the interpreter are correct).
+  The parity test in `records_cli_test.go` avoids that shape.
+- `encode_json` converts int64 to float64 (`normalizeForJSON`), so integers
+  above 2^53 lose precision on output. No v3 task hits it.
+
+## Notes on the frozen specs (from review of #152; specs not edited)
+
+- `tasks/README.md` says the input file names are new; five reuse earlier
+  names with new contents (`config.json`, `people.json`, `prices.json`,
+  `nums.txt`, `words.txt`). Contents and operations are new.
+- x01 says "compact JSON" while the oracle uses Python's default separators;
+  no x01 output contains a separator, so it does not affect scoring. Tighten
+  the wording in v4.

experiments/token-efficiency-heldout-v3/cmd/heldoutv3verify/main.go

@@ -0,0 +1,229 @@
+// Command heldoutv3verify replays every held-out reference in a fresh sandbox
+// and compares its single output with the frozen expected value.
+package main
+
+import (
+	"bytes"
+	"context"
+	"crypto/sha256"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io/fs"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"reflect"
+	"sort"
+	"strings"
+	"time"
+)
+
+type Task struct {
+	ID       string            `json:"id"`
+	Category string            `json:"category"`
+	Inputs   map[string]string `json:"inputs"`
+	Output   string            `json:"output"`
+	Kind     string            `json:"kind"`
+	Expected json.RawMessage   `json:"expected"`
+}
+
+type Gap struct {
+	Task   string `json:"task"`
+	Lang   string `json:"lang"`
+	Reason string `json:"reason"`
+}
+
+type Result struct {
+	Task   string `json:"task"`
+	Lang   string `json:"lang"`
+	Pass   bool   `json:"pass"`
+	SHA256 string `json:"sha256"`
+	Detail string `json:"detail,omitempty"`
+}
+
+var exts = map[string]string{"howl": "howl", "python": "py", "js": "js", "go": "go"}
+
+func main() {
+	root := flag.String("root", "experiments/token-efficiency-heldout-v3", "benchmark root")
+	howl := flag.String("howlframe", "/tmp/howlframe", "HowlFrame binary")
+	howlDir := flag.String("howl-dir", "", "HowlFrame solutions directory (default <root>/solutions/howl)")
+	resultsDir := flag.String("results", "", "results directory (default <root>/results)")
+	langs := flag.String("langs", "howl,python,js,go", "comma-separated languages to replay (results are only written for the full set)")
+	flag.Parse()
+	var manifest struct{ Tasks []Task }
+	must(readJSON(filepath.Join(*root, "tasks/manifest.json"), &manifest))
+	var gaps []Gap
+	must(readJSON(filepath.Join(*root, "gaps.json"), &gaps))
+	gapped := map[string]bool{}
+	for _, g := range gaps {
+		gapped[g.Task+"/"+g.Lang] = true
+	}
+	work, err := os.MkdirTemp("", "hf-heldout-v3-verify-")
+	must(err)
+	fmt.Println("Replay artifacts:", work)
+	var results []Result
+	failed := false
+	for _, lang := range strings.Split(*langs, ",") {
+		passed, total := 0, 0
+		for _, t := range manifest.Tasks {
+			if gapped[t.ID+"/"+lang] {
+				continue
+			}
+			total++
+			src, _ := filepath.Abs(filepath.Join(*root, "solutions", lang, t.ID+"."+exts[lang]))
+			if lang == "howl" && *howlDir != "" {
+				src, _ = filepath.Abs(filepath.Join(*howlDir, t.ID+"."+exts[lang]))
+			}
+			data, readErr := os.ReadFile(src)
+			r := Result{Task: t.ID, Lang: lang, SHA256: fmt.Sprintf("%x", sha256.Sum256(data))}
+			if readErr != nil {
+				r.Detail = readErr.Error()
+			} else {
+				r.Detail = verify(filepath.Join(work, lang+"-"+t.ID), src, lang, *howl, t)
+				r.Pass = r.Detail == ""
+			}
+			if r.Pass {
+				passed++
+			} else {
+				failed = true
+			}
+			fmt.Printf("%s %s pass=%v %s\n", lang, t.ID, r.Pass, r.Detail)
+			results = append(results, r)
+		}
+		fmt.Printf("%s: %d/%d passed\n", lang, passed, total)
+	}
+	out, _ := json.MarshalIndent(results, "", "  ")
+	res := filepath.Join(*root, "results")
+	if *resultsDir != "" {
+		res = *resultsDir
+	}
+	if *langs != "howl,python,js,go" {
+		if failed {
+			os.Exit(1)
+		}
+		return
+	}
+	must(os.MkdirAll(res, 0o755))
+	must(os.WriteFile(filepath.Join(res, "verification.json"), append(out, '\n'), 0o644))
+	if failed {
+		os.Exit(1)
+	}
+}
+
+func verify(dir, src, lang, howl string, t Task) string {
+	if err := os.MkdirAll(filepath.Join(dir, "out"), 0o755); err != nil {
+		return err.Error()
+	}
+	for p, content := range t.Inputs {
+		full := filepath.Join(dir, p)
+		os.MkdirAll(filepath.Dir(full), 0o755)
+		if err := os.WriteFile(full, []byte(content), 0o644); err != nil {
+			return err.Error()
+		}
+	}
+	absDir, _ := filepath.Abs(dir)
+	var cmd []string
+	switch lang {
+	case "howl":
+		art := filepath.Join(filepath.Dir(absDir), filepath.Base(absDir)+".hfbc")
+		if o, err := run(absDir, 30*time.Second, howl, "build", "--profile", "governed", src, "-o", art); err != nil {
+			return "build: " + o
+		}
+		cmd = []string{howl, "-run-bc", "-allow-caps", "filesystem:read=" + filepath.Join(absDir, "in") + ",filesystem:write=" + filepath.Join(absDir, "out"), art}
+	case "python":
+		cmd = []string{"python3", src}
+	case "js":
+		cmd = []string{"node", src}
+	case "go":
+		bin := absDir + ".bin"
+		if o, err := run(filepath.Dir(absDir), 60*time.Second, "go", "build", "-o", bin, src); err != nil {
+			return "go build: " + o
+		}
+		cmd = []string{bin}
+	}
+	if o, err := run(absDir, 10*time.Second, cmd[0], cmd[1:]...); err != nil {
+		return "run: " + o
+	}
+	// Exactly the inputs (unchanged) plus the one output may exist.
+	var files []string
+	filepath.WalkDir(absDir, func(p string, d fs.DirEntry, err error) error {
+		if err == nil && !d.IsDir() {
+			rel, _ := filepath.Rel(absDir, p)
+			files = append(files, filepath.ToSlash(rel))
+		}
+		return nil
+	})
+	want := []string{t.Output}
+	for p := range t.Inputs {
+		want = append(want, p)
+	}
+	sort.Strings(files)
+	sort.Strings(want)
+	if !reflect.DeepEqual(files, want) {
+		return fmt.Sprintf("file set %v, want %v", files, want)
+	}
+	for p, content := range t.Inputs {
+		got, _ := os.ReadFile(filepath.Join(absDir, p))
+		if string(got) != content {
+			return "input modified: " + p
+		}
+	}
+	got, err := os.ReadFile(filepath.Join(absDir, t.Output))
+	if err != nil {
+		return err.Error()
+	}
+	return compare(got, t)
+}
+
+func compare(got []byte, t Task) string {
+	if t.Kind == "text" {
+		var want string
+		json.Unmarshal(t.Expected, &want)
+		g := strings.TrimRight(string(got), "\n")
+		if g != strings.TrimRight(want, "\n") {
+			return fmt.Sprintf("text mismatch: got %q want %q", g, want)
+		}
+		return ""
+	}
+	var g, w any
+	if err := json.Unmarshal(got, &g); err != nil {
+		return "invalid JSON output: " + err.Error()
+	}
+	json.Unmarshal(t.Expected, &w)
+	if !reflect.DeepEqual(g, w) {
+		return fmt.Sprintf("json mismatch: got %s want %s", bytes.TrimSpace(got), t.Expected)
+	}
+	return ""
+}
+
+func run(dir string, timeout time.Duration, name string, args ...string) (string, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+	c := exec.CommandContext(ctx, name, args...)
+	c.Dir = dir
+	out, err := c.CombinedOutput()
+	if ctx.Err() != nil {
+		return "timeout", ctx.Err()
+	}
+	s := string(out)
+	if len(s) > 400 {
+		s = s[:400]
+	}
+	return s, err
+}
+
+func readJSON(path string, v any) error {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return err
+	}
+	return json.Unmarshal(data, v)
+}
+
+func must(err error) {
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(2)
+	}
+}