Skip to content

SM-12: a semantic module nests to max_scope_depth, not 128 levels - #529

Merged
afflom merged 6 commits into
release/v0.4from
fix/snapshot-depth
Nov 15, 2026
Merged

afflom merged 6 commits into
release/v0.4from
fix/snapshot-depth

Conversation

@humuhumu33

@humuhumu33 humuhumu33 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #528.

What changes

  • The semantic-module JSON reader takes a bound (SemanticModule::parse_bounded); the linker passes the project's max_scope_depth as the compile stack hosts it. The depth is measured without recursing (json_depth) before anything recurses on it. Past the bound: LLT4001 naming max_scope_depth (the prim append advice stays).
  • The notation reader takes the same bound (parse_module_bounded, Reader::with_max_depth). Past it: LLP2001, now naming max_scope_depth.
  • Every later reading of what the linker admitted reads to the same depth: the document JSON, declaration IDs, definition digests, the snapshot's values, and the backend's declaration spans read without the 128-level limit; render_build runs on a thread of the compile stack's size; SemanticSnapshot::from_checked, to_json, and canonical_bytes run on a thread of the largest compile stack's size (1 GiB reserved, not committed), so an SDK caller on any stack gets its bytes.
  • The authoring API (lexlean::sdk: read_module, write_module, to_data_form, to_notation, declaration_spans, is_notation) reads, converts, and writes a semantic module to 1024 levels, the default max_scope_depth, on a stack of its own; it read the notation to 256 and the JSON to 128 on its caller's stack. AU-08, which reads every committed module through it, failed on the 1000-cell fixture in the first gate run; it passes now (34 min locally).
  • Unchanged: SemanticModule::parse with no project (127 levels), tooling modules (128, LLT4013), core-module notation, production program depth (512).

Byte identity

No published artifact moves: cargo xtask check-golden passes for every project, and the diff touches no identity input (language/, schemas/, embedded goldens), so no compiler-semantics ID moves. A module within the old limits reads, renders, and hashes as before. No regeneration is needed.

Evidence

  • conformance_sm_12 (4 s): the 1000-cell fixture checks and builds; its snapshot is written on a 2 MiB thread, the snapshot ID equals the digest of the bytes, and the bytes nest past 1000 levels; 1100 cells are LLP2001; under max_scope_depth = 150, 145 cells check, 146 and 147 are LLT4001, 148 is LLP2001.
  • conformance_sm_12 also reads, converts both ways, and writes the 1000-cell fixture through the authoring API on a 2 MiB thread, and refuses the 1100-cell one.
  • Planted defects, each restored, each failing conformance_sm_12: the linker reading with parse (127 levels); to_json re-reading with the bounded Json::parse (panic); canonical_bytes on its caller's stack (STATUS_STACK_OVERFLOW); the authoring API reading the notation to 256 levels; the authoring API on its caller's stack (STATUS_STACK_OVERFLOW).
  • lexlean verify with pinned Lean 4.32.1 (Windows host, debug compiler): 256 cells verified in 45 s, 1000 cells in 29 s.
  • SP-02 asserted the 130-level let-chain refusal; it now asserts the chain checks, and that assertion passes locally. Its later Lean-backed part ran out of memory (std::bad_alloc) compiling a production certificate on the Windows host; this change does not touch production lowering or certificates, and the Linux gate is the judge.
  • AU-01, AU-03 (a 50,000-level notation is still LLP2001), AU-06 pass; fmt, clippy -D warnings, validate-spec-links pass locally. validate-model and the full gate run in CI.

Decisions for the owners

  • This reverses Language 1.2: list literals are limited to about 122 elements by the semantic-module JSON reader #375 (keep the 128-level limit, advise prim append). SM-12's statement changes; the register count does not.
  • Delivery to Prism: PrismPM vendors v0.4.0, and release/v0.4 has moved the language-1.2 identity since. Taking this fix through the next release from this line rebuilds every Prism app (because of those other moves, not this fix). A patch on v0.4.0 carrying only this fix would keep existing holos byte-identical, but departs from the release-line model.

🤖 Generated with Claude Code

A list nests one JSON level per element, and the front end read a module's
data with a fixed 128-level limit and its notation with 256, so a list of
more than about 122 elements was refused, the snapshot panicked writing its
own bytes past about 115, and a debug build overflowed the caller's stack
rendering 100. Both readers now take the project's max_scope_depth as the
compile thread's stack hosts it, and every later reading of what the linker
admitted (rendering, declaration IDs, definition digests, the snapshot and
its canonical bytes) reads to the same depth on a stack sized for it. A
module within the old limits reads, renders, and hashes as before; no
identity input changes. Reverses #375's decision to keep the 128-level limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@afflom afflom left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two bounded source reviews credit the admission path: the linker uses the effective compile depth, iterative JSON depth preflight precedes unbounded recursive reads, canonical/unknown-member checks remain, and notation refuses before exceeding its bound. No additional parser/linker blocker or concrete byte-recipe change for formerly accepted values was found. Reported Windows tests/plants/Lean timings remain author evidence; no checks were repeated.

P2 — Remove the caller-stack fallback in lib.rs:240–264. If spawn_scoped cannot create/reserve the sized worker, Err(_) => run() executes the same recursive snapshot/render work on the unknown caller stack. This defeats the stated SPEC§25.5 never-on-caller guarantee precisely on a resource failure. Propagate a registered resource failure through the public fallible boundary, and explicitly resolve the infallible snapshot methods; do not silently revert to the unsafe execution path. Reuse the existing closed error model.

Also disposition returned snapshot lifecycle in the any/default-stack claim. from_checked returns recursive owned snapshot data with ordinary destruction; sizing construction/serialization alone does not cover that returned value's later drop/clone. to_json explicitly documents caller care for its returned recursive Json. The new1000-cell test does exercise snapshot destruction on its2MiB caller, so this is not a claim that destruction is untested or an observed crash at1000. Account for all admitted depths/caller constraints honestly, without inventing a numeric safety guarantee.

Keep the small-stack R5 repair distinct from the new language-limit contract; historical #375/#403 remain delivered under their original scope. Existing owner retains the focused correction, lifecycle disposition and final-head qualification. No identity writer is indicated by the inspected27-path source delta. Independent receipts pr529-274-depth-admission-review/receipt.json and pr529-downstream-source-review/receipt.json; no runtime, producer, logs or CI controls performed.

@afflom afflom mentioned this pull request Nov 14, 2026

afflom commented Nov 14, 2026

Copy link
Copy Markdown
Member

@humuhumu33 Bounded reproduction confirms the snapshot panic, and there is a useful distinction in the PrismPM contract.

  • With the frozen native compiler from ebcc8731 (binary SHA-256 65c79632f37bf8489fdb3e029cb5f10def664c29c4242f6d550611b2beb319ca), a 120-cell list checks and builds but snapshot serialization exits 101 with snapshot DTO uses the canonical JSON domain: recursion limit exceeded. SDK canonical_bytes and snapshot_id also panic at 120 after construction succeeds. At 122, construction itself panics. At 123/130/131, admission returns LLT4001. The snapshot source blob matches current main, v0.4.0, release/v0.4 and Patch the single Lean pin to 4.32.2 for kernel soundness #518, but those heads were compared, not executed.
  • Linux debug builds at 100/120 passed with both 8 MiB and 2 MiB caller-stack bounds. This does not reproduce or contradict the reported Windows stack overflow.
  • PrismPM's published v0.4.0 tag resolves to b48ee2ffa4d2097685054dc350b57aa204fc7080. Its acceptance-vector projector already resolves closed zero-argument aliases at every list tail. Its tests cover owner preservation, cycles and the shared alias budget. List append remains a different, unsupported application-list form.
  • A bounded LexLean fixture with 65 cons cells ending in (chunkB) and a 66-cell chunkB passed lock/check/build/snapshot, without append. That establishes the 131-element source/snapshot construction alternative, not PrismPM runtime or installed-SDK acceptance.

Please confirm the exact PrismPM/SDK revision behind the reported PP2001 and whether that consumer still rejects closed-tail chunks. This is dependency-contract coordination only, not a request for a new PrismPM implementation, SDK build or release. The real panic still needs the existing #529 owner repair, including the outstanding sized-thread failure fallback and snapshot lifecycle review; no parallel fix writer or identity regeneration has been started.

…ng fix

Only CHANGELOG.md conflicted: the nesting entry stays first, #498's follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SDK read a module's notation to 256 levels and its JSON with serde's
128-level limit on its caller's stack, so it could not read the 1000-cell
fixture, and AU-08, which reads every committed module through it, failed
in the gate. Consumers read and write application sources through it too.
It now reads, converts, and writes a semantic module to 1024 levels, the
max_scope_depth a project is created with, on a stack of its own sized
like a compile thread's for that depth. Tooling and core modules keep
their readers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@afflom afflom left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bounded current-head review credits the SDK admission repair: semantic notation uses bound1024, JSON depth is measured iteratively before recursive reading, and authoring operations use the sized worker when its creation succeeds. The complete18-path delta comprises incoming11f plus8 authored SDK/contract/test/doc paths; no new language/schema/embedded-golden or expected-product change was found. Existing source credit is retained; reported local tests and timings were not repeated.

P2 remains and now affects the new SDK entry points: sdk.rs:33–35 deep() calls unchanged on_stack, whose Err(_) => run() fallback executes recursive work on the unknown caller stack when the sized worker cannot be created. Repair the shared failure path through the existing closed error model and explicitly disposition infallible methods. Adding the SDK wrapper does not satisfy the earlier5900904759 finding.

The returned-value lifecycle also remains separate from worker sizing. snapshot.rs is unchanged; SDK read_module additionally returns recursive serde_json::Value to its caller, where later clone/traversal/drop are outside the worker. The1000-cell2MiB source fixture includes destruction, so this is not a claim of an observed1000-cell crash. Explicitly cover the supported lifecycle/caller contract or narrow stronger claims; do not infer a universal stack guarantee from that fixture.

Existing owner retains this repair, inherited#498 findings, exact consumer/SDK disposition and final-head acceptance. Receipt pr529-a3c-source-delta/receipt.json. No producer, test, CI control or #431/oldTC03 validation was run or admitted; this review does not qualify the separate patch release.

github-actions Bot and others added 2 commits November 15, 2026 01:00
Keep #431's Json::from_serde for the snapshot value inside this PR's
deep-stack wrapper, place from_str_unbounded ahead of the duplicate-key
check's (#520) doc comment, and keep both CHANGELOG entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3M8ULnzEQwLLngyBBX
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3M8ULnzEQwLLngyBBX
Keep both MAX_COMPILE_STACK_BYTES definitions from #503 (native and
WASI) at the crate visibility this PR needs for on_stack callers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3M8ULnzEQwLLngyBBX
@afflom
afflom merged commit ab91576 into release/v0.4 Nov 15, 2026
68 checks passed
@afflom
afflom deleted the fix/snapshot-depth branch November 15, 2026 09:52
@afflom afflom mentioned this pull request Nov 15, 2026

afflom commented Nov 15, 2026

Copy link
Copy Markdown
Member

Post-landing acceptance audit: finalbca7a7cb proposal CI43686923596 completed64 successful jobs/764 successful steps plus3 skipped Intel obligations; exact proposal repro passed. Actual ab91576 has a different tree from final proposal; actual CI43701355779 is pending with zero jobs, while actual repro passed.

This establishes the merge's presence for release ancestry, not full premerge or actual-tree acceptance. #528 remains open pending full-body/source disposition and qualification; prior fallback/lifecycle findings and #431/oldTC03 STOP are not cleared by external landing. No stopped source/runtime review or adoption, skipped/canceled aggregation, retrospective approval, or new producer/retry occurred. Preserve original evidence and use the existing owner lane. Complete inventory receipt cbf6af24c464f1d8df9902664a5331bc4f6f9b7e81246895425f7def029f9241.

humuhumu33 added a commit that referenced this pull request Nov 15, 2026
…#535, #536) into the arrays branch

The generated outputs that both sides rewrote take release/v0.4's copy, or
are removed where the line no longer has them; regenerate.yml rewrites the
whole set from this tree.

Lean moves to 4.32.2 with #518, so the array authority follows the other Lean
rows: LEAN-ARRAY-4-32-1 becomes LEAN-ARRAY-4-32-2, cited at f3b06c705e6c.
The three files its statement rests on (src/include/lean/lean.h,
src/Init/Prelude.lean, src/Init/Data/Array/Set.lean) are byte-identical at
f054605aea4b and f3b06c705e6c, so the statement carries over unchanged.

runtime-1.2/library.toml keeps the line's toolchain and the branch's Arrays
module. LANGUAGE_1_2_SEMANTICS_ID is the merged tree's, as RP-10 recomputes it.
CHANGELOG and VERIFICATION keep both sides' entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants