HFIR model adapter: full -compile-hfir-bc language coverage (#88)

[howlcipher]

Brings the HFIR model adapter (hfir-model-adapter/v1, #88) up to full coverage of the language that -compile-hfir-bc lowers. This PR adds no new opcode, capability, grant, lowering or language form.

What changed

• Transport kinds: the transport accepts every kind LowerToBytecode lowers from source (hfir.AdapterNodeKinds, 49 kinds). New ones: defun/param/call/return, while, for, try/catch, read_file, write_file, mkdir, exec, fetch, parse_json, is_nil, map_keys, html_escape, attr_escape, regex_match, time_now, sleep (transport role duration maps to the unnamed graph edge), cli_args, and req_query/req_header/req_path. (map_get <expr> key) now has a target edge.
• Role validator: roles are matched in order and support ? (optional) and * (repeated) suffixes. Kinds with more than one valid shape list each alternative.
• Control edges: for/while control edges come from the validated roles, the same way if already worked. The schema still has no control-edge field.
• Type check fixes: BOOL consts are typed bool, to_int/to_float return numbers, str_split returns a list, and string + string is accepted. literal_kind is never carried on non-const nodes.
• Bounds: raised to 4096 nodes, 256 inputs per node and 1 MiB per transport so that real programs fit.
• New flag: experimental -hfir-adapter-roundtrip (used with -compile-hfir-bc) encodes the source graph as adapter JSON, strictly decodes it and runs CompileCandidate.

Coverage test

TestModelAdapterCoversArityTable:

• All 386 corpus programs that -compile-hfir-bc lowers (experiments, tests, examples, apps) round-trip through the adapter to a byte-identical artifact.
• 124 of 159 arity-table forms are covered, either by the corpus or by explicit samples.
• 35 forms are exempt, each with a reason. The test checks that every exempt form stays outside -compile-hfir-bc: it must not appear in a compiled corpus program, and where a sample exists, both paths must reject it. If a form becomes lowerable later, the test fails until the adapter covers it.
• Exempt forms: spawn, spawn_agent, task, grants, llm_generate, confidence, lazy_synthesize, the circuit forms, the optimize forms, schema_bridge, achieve, db/store/s* forms, http_server/res/res_json/res_header/http_res, web_app/wasm_app, module/export, plus read_line, http_request and req_method. Those last three are on the promote-blocker fence, and lowering them would change the locked prod-flip criteria, so they stay out of scope here.
• A mutation check confirmed it: dropping a kind from the adapter fails the test.

Other tests

• Unit tests for kinds that must stay outside the transport (agent, process, model, store, route and server forms all return HFIR_TRANSPORT_KIND) and for the ordered role shapes.
• A loops-plus-functions candidate runs correctly.
• exec, fetch, read_file, write_file, mkdir and env reached through the adapter fail closed with no grant.

Invariants

• TestHFIRBenchmarkSolutionParity: 0 of 268 mismatches.
• The sha256 pins and freeze tests are unchanged and pass.
• Go/JS UNSUPPORTED_ON_TARGET behaviour is unchanged.
• The promote-blocker fence is unchanged.
• TestProdFlipCriteriaLock was updated only for the model-adapter row. It now asserts that the transport carries the source-lowered kinds and still rejects match, spawn_agent, grants, llm_generate, http_server, res, store_put, read_line and http_request. The criteria doc row and the docs/reference "still rejects" sentences were updated to match. Historical journals were not edited.
• Hard nos: no production -compile-bc/HFIR flip, Write flagged http_server bytecode and stop #90 stays Partial, no tip-lock, no DOM.
• go test ./... exits 0 locally.
• Token benchmark (TOKBENCH_PYTHON=/workspace/tokvenv/bin/python): the v2 postcf and v3 postrec reruns show no change in the results files. v3 postrec median is still 1.186 with 8 wins.

The breaking changes for adapter clients (bounds and type check) are documented in change_log.md.

[howlcipher]

Review of #167 at 5c59030c8dc13f2d3f782bd3db3b097ffc7419c1 (base 5009e035)

Verdict: not merge-ready yet. The code is sound, and coverage is real and well tested. But (a) the lock change loosens a criterion and needs Owner sign-off, and (b) one cheap crash fix should land here, because this PR turns the adapter into the full-language intake for model proposals.

I used a fresh clone in /workspace/hf167-* with a base worktree. Port 8080 was free, and I started no servers.

Gates (all real exit codes)

• gofmt -l . is empty. go vet ./... exit 0. go test ./... exit 0.
• Focused -v run exit 0: TestModelAdapterCoversArityTable (386 round-trip byte-identical; 124/159 covered, 35 exempt), TestProdFlipCriteriaLock, TestHFIRBenchmarkSolutionParity, all 10 TestAuthority* (#163), and TestIssue141* arity/drift/fuzz.
• Mutation: deleting the regex_match case from nodeRoleAlternatives makes the coverage test fail (exit 1, 6 corpus programs: adapter rejected ... HFIR_TRANSPORT_KIND). Restored afterwards.
• The diff touches no files under benchmarks/ or experiments/, so pins, freeze and token-benchmark inputs are untouched.
• No tip-lock, no live LLM, no H1 claim.

1. TestProdFlipCriteriaLock and the criteria row, line by line

Change	Class
Old loop asserted DecodeCandidate rejects defun, call, return, while, for, read_file, write_file, mkdir, exec, fetch, regex_match, html_escape, attr_escape, time_now, sleep, try. The new code asserts these 16 are members of AdapterNodeKinds().	Loosening (the criterion is reversed)
match moves from the old list to the new reject list	Neutral
New reject asserts for spawn_agent, spawn, grants, llm_generate, http_server, http_server_start, res, store_put, read_line, http_request	Strengthening
Doc row: "leaves the transport allow-list on the Phase-1 adapter subset ... #88 stays Pending" becomes "widened nodeRoles to every kind LowerToBytecode lowers"	Loosening
"What this spike does": drops "still rejects html_escape, attr_escape, regex_match, time_now, and sleep" and rewords the rest to "accepts exactly ... still rejects kinds the source lowerer does not lower"	Loosening (plus a wording nit: the test checks membership, not "exactly")

The promote-blocker fence is byte-unchanged: only 2 doc hunks and 1 test hunk changed, and neither touches the fence. read_line, http_request and req_method stay exempt. req_method is rejected by kind in my probe, but it's missing from the lock's reject list (nit: add it). The flip is gated behind the new flag in -compile-hfir-bc only, *compileBc is untouched, and "#90 stays Partial" is intact. Nothing makes HFIR or the adapter the prod default.

A criterion was loosened, so Owner sign-off is needed under the "additive or strengthening only" rule. I agree with the change itself. It's the point of #88.

2. Size limits

Corpus needs, measured over 422 programs through -hfir-adapter-roundtrip:

• Max nodes: 3046 (heldout-v2/postcf/.../w10.howl). 82 programs exceed the old 128.
• Max inputs per node: 48 (examples/repo_analyst).
• Max bytes: 416,017 (tests/test_let_chain.howl).

Boundary probes:

• 4097 nodes, 257 inputs and >1 MiB are all rejected with HFIR_TRANSPORT_BOUNDS.
• 256 inputs is accepted and fast (3 ms).
• A 4095-node linear binary chain decodes in 90 ms. Compile takes 1.5 to 4.4 s and allocates about 4.3 GB (quadratic in lowering). There's no stack overflow, since Go stacks grow.
• The ordered ?/* validator is greedy and single-pass, so there's no backtracking. 255 alternating param/body roles is rejected in about 2 ms.

Two pre-existing DoS bugs, both reproduced on 5009e035. Neither depends on the limits, but this PR widens the surface:

• [blocker, fix here] A 4-node cyclic proposal kills the process. Take a = binary(b, b) and b = binary(a, a). validateTransportTypes.infer has no in-progress mark and runs before transportCycle, so the result is fatal error: stack overflow (not recoverable). Fix: run transportCycle before the type check, or mark in-progress in infer.
• [follow-up, blocks any live run] Exponential lowering on shared pure subexpressions. If each binary level references the previous node twice, the cost doubles per level. At depth 16 compile takes 0.13 s and 196 MB, at 20 it takes 2.2 s and 3.1 GB, and at 22 (a 3.7 KB proposal) it hits OOM. validateNoSharedEffects only blocks effectful sharing. The fix is to reject pure sharing at intake, or memoize/emit locals in lowering, or cap the tree-expanded size.

Recommendation:

• Keep 4096 nodes and 1 MiB (corpus max is 3046 nodes and 416 KB).
• Lower inputs to 64 (corpus max is 48).
• Add a nesting-depth or expanded-size cap so lowering cost is bounded by input size.

3. sleep duration

• The role mapping round-trips losslessly (duration ↔ unnamed edge).
• Units are milliseconds, through the same ToInt(x) * time.Millisecond as the interpreter, so there's parity.
• Negative values return immediately. Floats truncate (0.01 becomes 0).
• string and bool fail with runtime TYPE_ERROR. The transport type check doesn't catch them (nit).
• MaxInt64 overflows time.Duration and returns immediately. This is pre-existing and the same on all paths.
• 1e15 ms sleeps unbounded under DefaultExecutionPolicy(), which has no deadline. The instruction limit doesn't bound sleeps. With Deadline set, the VM stops at the deadline: LIMIT_EXCEEDED ... wall-clock deadline exceeded at 3.00 s, and a while true (sleep 1) loop stops the same way.
• So yes, a model-proposed sleep can hang the run unless the runner sets a deadline. Any adapter harness must set one, or clamp sleep. The interpreter's time.Sleep ignores the deadline (pre-existing).

4. Adversarial input and fuzz

Kind and role rejections:

• These are all rejected with HFIR_TRANSPORT_KIND: spawn_agent, spawn, grants, llm_generate, http_server(_start), res, store_put/open, match, read_line, http_request, req_method, use, module, export, EXEC, exec and Exec.
• Wrong role (exec with argv) and a while without body both give HFIR_TRANSPORT_ROLE.
• for whose body is itself gives HFIR_TRANSPORT_EFFECT. while whose body is program gives HFIR_TRANSPORT_CYCLE. Calling an undefined function gives HFIR_BYTECODE_INVALID.

Effects never run without a grant:

• exec with a dynamically built cmd ("/bin/" + "sh") is denied with CAPABILITY_DENIED ... process.
• try/catch around a denied exec does catch CAPABILITY_DENIED (exit 0, error JSON printed), but the command never runs. That matches source try_let.
• req_query outside a server gives UNDEFINED_VAR req.
• return outside a defun compiles and exits 1 with no output (nit: better rejected at intake).

Governed path: the adapter output is byte-identical to -compile-hfir-bc on 386 programs and runs on the same VM checks, so the #163 rules apply unchanged.

Fuzz: 600 mutated proposals (kind, role, edge retarget, add/drop edge, value, literal_kind), seeded from a defun+for+try/exec+while+read_file+sleep graph:

• 0 panics, 0 effect leaks.
• 88 ran (all exit 0). The rest were rejected: ROLE 306, LITERAL 113, UNREACHABLE 40, EFFECT 17, VALUE 6, BYTECODE_INVALID 4, ENTRY 1.
• 25 mutants formed cycles and were skipped, because they would trigger the stack-overflow bug above.

5. Coverage test and type checks

The 386/386 round-trip, the 124/35 split and the mutation result are confirmed above.

Only 11 of 35 exemptions have a sample that is checked against -compile-hfir-bc. The other 24 have an empty sample and are only checked to be absent from compiled corpus programs. I spot-checked store_open, sget, http_server, res, db_connect, optimize_block, lazy_synthesize and module, and -compile-hfir-bc rejects all of them (exit 1). Nit: add samples so the claim holds for all 35.

Type fixes behave as intended:

• to_int("5") < 7 is accepted and prints true.
• These are still rejected: string + number, to_string(x) < 7, string + list, bool - number.

Asks

1. Owner sign-off on the lock and criteria-row loosening.
2. In this PR: run the cycle check before the type check (it's a one-line move), plus a regression test for the cyclic binary crash.
3. File as a follow-up that blocks any live-model run: the exponential/quadratic lowering cost and a deadline/sleep clamp for adapter runs.
4. Nits: add req_method to the lock's reject list, lower inputs to 64, reword "accepts exactly", add exemption samples, and type-check the sleep operand at intake.

Once 1 and 2 are done, I'd call it merge-ready.

[howlcipher]

Re-review of #167 at 0d505666c7461daff6371225d05863508d403cbc

Verdict: code merge-ready; merge gated on Owner sign-off for the lock change. Every item from my last review is fixed, and I found no new blockers.

I used a fresh worktree, which I removed afterwards. Port 8080 was free and I started no servers.

Gates (real exit codes)

• gofmt -l . is empty. go vet ./... exit 0. go test ./... exit 0 (75 packages ok).
• Focused -v run exit 0: TestModelAdapterCoversArityTable (386 programs round-trip; 126/159 covered, 33 exempt), TestProdFlipCriteriaLock, TestHFIRBenchmarkSolutionParity, all 10 TestAuthority* and all 5 TestIssue141*.
• The new hfir unit tests pass: BinaryCycleRejectedBeforeTypes, ReturnOutsideDefunRejected, SleepDurationTypeChecked and InputsPerNodeBound.
• The diff vs 5c59030 touches only 6 files and nothing under benchmarks/ or experiments/, so pins and freeze are unchanged.

1. Cycles

All of these now return HFIR_TRANSPORT_CYCLE with no crash:

• My a↔b repro.
• binary rings of 3, 4 and 50 nodes.
• Self-loops on binary and program.
• A while whose body or condition (through is_nil) points back at the loop, a for whose iterable (a list) contains the loop, and an if whose else branch is itself.
• A let whose value is itself.
• A defun whose body calls with the defun as an argument.
• A cycle reachable only from a dead if false branch, and a fully unreachable cycle.

Fuzz: 600 mutations, cycle-forming cases included (32 of them), with a new seed and return in the base graph:

• 0 panics and 0 leaked effects (no pwned, no passwd contents).
• 88 ran. The rest were rejected: ROLE 317, LITERAL 121, CYCLE 6, EFFECT 6, RETURN 2, and so on. The other cycle-forming mutants were rejected by earlier checks first.

2. Inputs

• 64 inputs is accepted, 65 gives HFIR_TRANSPORT_BOUNDS.
• The corpus still round-trips 386/386 byte-identically.

3. Return rule

The path restriction matches the description exactly.

Accepted:

• Directly under program, or partway through it (the statements after it are skipped).
• Under sequence, a let body, or an if then-branch.
• Inside a while within a defun.

Rejected with HFIR_TRANSPORT_RETURN:

• In a while or for body.
• In a sequence inside a while, an if inside a while, or a let inside a for.
• As a try expression, in a try success body, or in a catch.
• As a print value, a let value, an if condition, a call argument, or a return value.

Runtime, on 3 paths: -run, -compile-bc then -run-bc, and -compile-hfir-bc then -run-bc. The adapter round-trip is byte-identical to the last of these.

• (do (print "before") (return 7) (print "after")) prints before and exits 7 on all 3 paths.
• An if that returns 3 halts before a following write_file, so no file is written. Exit is 3 on all paths.
• An exec that is denied before a return 0 fails with CAPABILITY_DENIED and exit 1 on all paths, so the return does not mask the denial (the denial fires first).
• Receipts (-receipt) are written on both bytecode paths when main returns early.
• There is no deferred-cleanup construct in the lowered subset that a return could skip.

The rule is sound. I recommend confirming it as is. A main-level return behaves like exit with that code, and exit is already allowed anywhere. Restricting return to the final statement would add complexity without closing any authority gap.

4. Exemption samples

• Blanking store_open's sample locally fails the test with exempt form store_open has no sample (exit 1). I reverted it afterwards.
• All 33 exemptions now carry samples, and each sample is checked to use its form.
• module and export round-trip byte-identically through -hfir-adapter-roundtrip and run (prints 1, exit 0). Nit: they count as covered because ResolveModules erases them before HFIR, which is worth a comment in the criteria doc.

5. sleep type check

• Rejected at transport (HFIR_TRANSPORT_TYPE): string, bool, list and dict operands, string + string, and env.
• Accepted: int, float, 1+1 and to_int.
• An unknown symbol is still accepted and fails at runtime. That's expected for a dynamic value.

6. Lock diff vs 5c59030

• Test: one line, which adds "req_method" to the reject list. That's strengthening.
• Doc: only the "What this spike does" sentence changed. "accepts exactly" became "carries each ... (a membership check, not an exact-set check)". That's neutral and more accurate.
• Nothing further was loosened, and the promote-blocker fence is byte-identical.
• *compileBc is untouched, there's no flip, and #90 stays Partial.
• The loosening from the first commit (reject becomes carry for the 16 kinds, and the criteria row) is still present, so Owner sign-off is still required.

Remaining, not blocking this PR

• Exponential lowering on shared pure subexpressions (depth 22 runs out of memory) and the quadratic cost on deep chains. Both are pre-existing and should block any live-model run until they're fixed or capped.
• Adapter runs need a wall-clock deadline or a sleep clamp. DefaultExecutionPolicy has no deadline.

I don't see either filed on this PR yet. Please file them as issues.