Skip to content

uorc: ncm context-mixing level (--best) and the standard kernel basis - #115

Merged
afflom merged 10 commits into
claude/project-thread-e7b16tfrom
claude/project-thread-84bxra
Nov 9, 2026
Merged

afflom merged 10 commits into
claude/project-thread-e7b16tfrom
claude/project-thread-84bxra

Conversation

@afflom

@afflom afflom commented Nov 9, 2026

Copy link
Copy Markdown
Member

Requested by Alex · project thread

Closes #62. Closes #107. Stacked on #112 (base claude/project-thread-e7b16t); it retargets to main once #112 merges.

Result (Canterbury, release binary, scripts/bench.mjs --steps 0)

file bytes uorc uorc --best gzip -9 bzip2 -9 xz -9e
alice29.txt 152089 53011 38462 54179 43202 48528
asyoulik.txt 125179 48041 35491 48816 39569 44592
cp.html 24603 7965 6580 7973 7624 7652
fields.c 11150 3159 2552 3127 3039 3032
grammar.lsp 3721 1346 1114 1234 1283 1292
kennedy.xls 1029744 61984 17165 209721 130280 51868
lcet10.txt 426754 134996 94469 144418 107706 119488
plrabn12.txt 481861 187573 132637 194264 145577 165456
ptt5 513216 53531 30605 52377 49759 39860
sum 38240 11068 9243 12768 12909 9500
xargs.1 4227 1872 1560 1748 1762 1812
total 2810784 564546 369878 730625 542710 493080

What changes

ncm, a context-mixing coder (client/gen/ncm_model.py), extrapolated from the lexlean-r4 neuromorphic model with bounded plasticity:

  • 17 hashed nibble contexts with adaptive-rate counters, a match model, a record model and word hashes.
  • Two gated mixers with clamped Hebbian weight updates and an integer stretch/squash.
  • A 32-bit binary arithmetic coder.
  • Integer-only throughout.

The model is written once as an IR (client/gen/ir.py) and compiled three ways:

  • to LexLean (irlex.py → Codec/NcmEncode, Codec/NcmDecode);
  • to a UORC kernel graph (irgraph.py, ncmgraph.py);
  • by hand to native Rust (crates/uorc/src/ncm.rs).

All of these were checked byte for byte against a C spec on Canterbury. The graph also decodes correctly on the reference-semantics mirror.

Standard kernel basis (client/gen/gen_basis.py):

  • The ulz and ncm decoder libraries (121 blocks, 8.5 KB) form root 0 of a UORC-BASIS-V1 basis, with a fixed identity.
  • Archives are now linked: a conditional archive (scope 1) names that identity, and its body is one entry block that applies a library decoder to the payload. Linked ulz framing is about 95 bytes, down from a 2.6 KB embedded kernel.
  • Under the core semantics basisGet returns data, not code. So decoding links the basis: Archive.materialize prepends the library blocks, and the result is the self-contained archive the reference path already runs.
  • A mismatched identity is rejected: the native path and materialize both compare it.

Window frames (from the speed lane): roots 1.. of the same basis are previous frames. The model is folded in here so the client is exported only once; the native side follows in the speed lane.

Levels:

  • Model: Client.compress(input, window, steps, level).
  • Native: Level::Fast (default) and Level::Best, the CLI's --best, a best flag on the wasm exports, and a checkbox on the Pages page.
  • Decompress needs no level, because the archive names its decoder.

Native fast path:

  • archive::linked_archive/linked_length, and fast_decode for linked archives with any object lengths.
  • ncm::decode_canonical checks encode(decode(p)) == p in one pass, by comparing the bytes the encoder would emit as the decoder shifts.

Certificate-friendly LexLean (no semantic change): without these changes the rust-std preservation certificate for the ncm encoder ran out of 13 GB.

  • Scalars live in a small table instead of a 36-field record.
  • Long statement lists are split into functions.
  • Literals go first in + as well as *.
  • Literal loop counts reference the state, so the kernel can't unroll the model's setup at check time.

generated/rust is re-exported (node scripts/client-export.mjs, pinned LexLean 4c41d395).

scripts/bench.mjs adds a uorc --best row, and uorc inspect names a linked archive's representation (ulz or ncm).

Tests

  • native_archives_are_the_model_archives now covers both levels: native archives must equal the exported model's byte for byte.
  • linked_archives_decode_by_linking_the_basis: a non-canonical linked ulz archive is declined natively and decoded through linking plus the reference semantics. The same decline is checked for ncm.
  • archive.rs tests check that the basis identity binds basis-library.hex, that a resealed archive with another identity is declined, and that linked round trips match the model's archive sizes.
  • ncm tests: round trips, and the one-pass canonical check agrees with a real re-encode on every one-bit and one-byte mutation of a payload.
  • best_level_beats_fast_on_text (including multi-object archives).
  • The CLI test checks that inspect reports ulz for a default archive and ncm for a --best one.
  • cargo test --release -p uorc, cargo clippy -p uorc -p uorc-wasm --all-targets -D warnings and cargo fmt --check pass locally.

UC-BAS-01 can cite the linked-basis tests above (left to the conformance lane, #108).



Generated by Claude Code

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…P, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs
Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs
A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs
@afflom afflom self-assigned this Nov 9, 2026
@afflom
afflom merged commit daa639b into claude/project-thread-e7b16t Nov 9, 2026
14 checks passed
afflom added a commit that referenced this pull request Nov 9, 2026
… (re-land #115 on main) (#116)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: optimize the client crate in test builds, lighter debug differential tests, ulz history window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: rustfmt the ulz history test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* build: optimize the exported reference crates in test builds

The differential tests run the exported decoder graph; unoptimized, the uorc
unit tests took 30 min of the 60 min acceptance leg (main is over budget
too). At opt-level 1 they take 4.5 min for about 6 min more compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

---------

Co-authored-by: Claude <noreply@anthropic.com>
afflom added a commit that referenced this pull request Nov 9, 2026
* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: optimize the client crate in test builds, lighter debug differential tests, ulz history window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: rustfmt the ulz history test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* build: optimize the exported reference crates in test builds

The differential tests run the exported decoder graph; unoptimized, the uorc
unit tests took 30 min of the 60 min acceptance leg (main is over budget
too). At opt-level 1 they take 4.5 min for about 6 min more compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing level (--best) and the standard kernel basis (#115)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

---------

Co-authored-by: Claude <noreply@anthropic.com>

* uorc: native window frames (each frame matches into the two before it)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

---------

Co-authored-by: Claude <noreply@anthropic.com>
afflom added a commit that referenced this pull request Nov 10, 2026
…ball (#120)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: optimize the client crate in test builds, lighter debug differential tests, ulz history window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: rustfmt the ulz history test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* build: optimize the exported reference crates in test builds

The differential tests run the exported decoder graph; unoptimized, the uorc
unit tests took 30 min of the 60 min acceptance leg (main is over budget
too). At opt-level 1 they take 4.5 min for about 6 min more compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level (--best) and the standard kernel basis (#115)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

---------

Co-authored-by: Claude <noreply@anthropic.com>

* uorc: native window frames (each frame matches into the two before it)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm v2, 3.9% smaller on Canterbury

The context-mixing coder gains:
- two adaptive probability maps (selected by the partial byte and by an
  order-3 hash) that refine the mixed probability;
- a third mixer, selected by the byte before the previous one and the
  bit position;
- a plasticity that decays from 128/65536 to 24/65536 per unit of
  x * err as the input is seen;
- a check byte in each 16-slot bucket's unused slot 0, with a neighbour
  pair and recycling by use, so colliding contexts stop sharing counts.

Model IR, LexLean, kernel graph and the native port are byte-identical
to the C spec on Canterbury. The ncm library grows, so the basis
identity and the linked templates are regenerated, and generated/rust
is re-exported. No released archive uses the previous identity.

Canterbury, uorc --best: 355,559 B (was 369,878; xz -9e 493,080).

bench.mjs passes --force, since the client refuses to overwrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* ulz: hash4 chain finder with hash3 head (model, mirror, native)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* Re-export generated Rust for the hash4 finder; rustfmt ulz.rs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* Restore the finder export in generated/rust (fix the bad merge resolution)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

---------

Co-authored-by: Claude <noreply@anthropic.com>
afflom added a commit that referenced this pull request Nov 10, 2026
…k decode (#125)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: optimize the client crate in test builds, lighter debug differential tests, ulz history window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: rustfmt the ulz history test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* build: optimize the exported reference crates in test builds

The differential tests run the exported decoder graph; unoptimized, the uorc
unit tests took 30 min of the 60 min acceptance leg (main is over budget
too). At opt-level 1 they take 4.5 min for about 6 min more compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level (--best) and the standard kernel basis (#115)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

---------

Co-authored-by: Claude <noreply@anthropic.com>

* uorc: native window frames (each frame matches into the two before it)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm v2, 3.9% smaller on Canterbury

The context-mixing coder gains:
- two adaptive probability maps (selected by the partial byte and by an
  order-3 hash) that refine the mixed probability;
- a third mixer, selected by the byte before the previous one and the
  bit position;
- a plasticity that decays from 128/65536 to 24/65536 per unit of
  x * err as the input is seen;
- a check byte in each 16-slot bucket's unused slot 0, with a neighbour
  pair and recycling by use, so colliding contexts stop sharing counts.

Model IR, LexLean, kernel graph and the native port are byte-identical
to the C spec on Canterbury. The ncm library grows, so the basis
identity and the linked templates are regenerated, and generated/rust
is re-exported. No released archive uses the previous identity.

Canterbury, uorc --best: 355,559 B (was 369,878; xz -9e 493,080).

bench.mjs passes --force, since the client refuses to overwrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* ulz: hash4 chain finder with hash3 head (model, mirror, native)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* Re-export generated Rust for the hash4 finder; rustfmt ulz.rs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: --best skips the per-frame ncm self-decode on compress

The native ncm encoder is byte-identical to the model, and every encoder
output decodes to its input, so the self-decode only repeated the
encode's cost. Same bytes; docs 9.1 MB --best 22.8 s -> 16.3 s, code
18.6 MB 11.4 s -> 8.9 s on 4 cores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NpWuTWFrL9kYU72Rv

* uorc: ncm-only compress, x86 frames, parallel block decode

Compress writes one coder, ncm, with the raw archive as fallback. The
Level enum and the ulz candidate are gone; --best is accepted and changes
nothing. The ulz decoder stays so v0.1 and v0.2 archives decode.

- Window frames read their history through ncm (ncmRunWith): the model's
  contexts and match table see the frames before, not only ulz matches.
- A per-frame x86 branch filter (E8/E9 rel32) runs under ncm when a frame
  is dense in calls (hits > 0 and n <= 256 hits).
- The basis library is the v0.2 library byte for byte with blocks
  appended, because graph block references are relative. The client
  accepts the v0.2 identity with its frozen templates, so v0.2 archives
  (default and --best, single and window frames) decode unchanged.
- Blocks: the encoder starts a block every 4 frames (BLOCK, a policy, not
  a wire rule). A frame that reads no window starts a block, so the
  decoder finds blocks from the frame headers and decodes them on
  separate cores; decompress, verify and decompress_stream all do.
  Old streams are one block.

Regenerated the certified export (generated/rust).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NpWuTWFrL9kYU72Rv

---------

Co-authored-by: Claude <noreply@anthropic.com>
afflom added a commit that referenced this pull request Nov 10, 2026
* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: optimize the client crate in test builds, lighter debug differential tests, ulz history window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: rustfmt the ulz history test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* build: optimize the exported reference crates in test builds

The differential tests run the exported decoder graph; unoptimized, the uorc
unit tests took 30 min of the 60 min acceptance leg (main is over budget
too). At opt-level 1 they take 4.5 min for about 6 min more compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level (--best) and the standard kernel basis (#115)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

---------

Co-authored-by: Claude <noreply@anthropic.com>

* uorc: native window frames (each frame matches into the two before it)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm v2, 3.9% smaller on Canterbury

The context-mixing coder gains:
- two adaptive probability maps (selected by the partial byte and by an
  order-3 hash) that refine the mixed probability;
- a third mixer, selected by the byte before the previous one and the
  bit position;
- a plasticity that decays from 128/65536 to 24/65536 per unit of
  x * err as the input is seen;
- a check byte in each 16-slot bucket's unused slot 0, with a neighbour
  pair and recycling by use, so colliding contexts stop sharing counts.

Model IR, LexLean, kernel graph and the native port are byte-identical
to the C spec on Canterbury. The ncm library grows, so the basis
identity and the linked templates are regenerated, and generated/rust
is re-exported. No released archive uses the previous identity.

Canterbury, uorc --best: 355,559 B (was 369,878; xz -9e 493,080).

bench.mjs passes --force, since the client refuses to overwrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* ulz: hash4 chain finder with hash3 head (model, mirror, native)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* Re-export generated Rust for the hash4 finder; rustfmt ulz.rs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: --best skips the per-frame ncm self-decode on compress

The native ncm encoder is byte-identical to the model, and every encoder
output decodes to its input, so the self-decode only repeated the
encode's cost. Same bytes; docs 9.1 MB --best 22.8 s -> 16.3 s, code
18.6 MB 11.4 s -> 8.9 s on 4 cores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NpWuTWFrL9kYU72Rv

* uorc: store identical frames once, as κ-labelled reference frames

A frame identical to an earlier one becomes a reference frame: a conditional
archive over the basis [library, F] (F's κ-label, the same identity a window
frame computes) whose 10-byte body returns basis root 1. No library block, no
basis identity change and no export: the exported decoder verifies it given
window [F]. The encoder labels each frame (SHA-256 chain) and skips coding
repeats; decoders read headers first and keep only frames a later reference
names, in memory up to 256 MiB then in an unnamed temp file. A stream without
repeats is byte-identical to before. decompress_stream now takes Read + Seek;
the CLI spools a piped stdin to a temp file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3M5Q4GpN89jhJUkexu

* uorc: ncm-only compress, x86 frames, parallel block decode

Compress writes one coder, ncm, with the raw archive as fallback. The
Level enum and the ulz candidate are gone; --best is accepted and changes
nothing. The ulz decoder stays so v0.1 and v0.2 archives decode.

- Window frames read their history through ncm (ncmRunWith): the model's
  contexts and match table see the frames before, not only ulz matches.
- A per-frame x86 branch filter (E8/E9 rel32) runs under ncm when a frame
  is dense in calls (hits > 0 and n <= 256 hits).
- The basis library is the v0.2 library byte for byte with blocks
  appended, because graph block references are relative. The client
  accepts the v0.2 identity with its frozen templates, so v0.2 archives
  (default and --best, single and window frames) decode unchanged.
- Blocks: the encoder starts a block every 4 frames (BLOCK, a policy, not
  a wire rule). A frame that reads no window starts a block, so the
  decoder finds blocks from the frame headers and decodes them on
  separate cores; decompress, verify and decompress_stream all do.
  Old streams are one block.

Regenerated the certified export (generated/rust).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NpWuTWFrL9kYU72Rv

* uorc: store identical frames once, as κ-labelled reference frames

A frame identical to an earlier one becomes a reference frame: a conditional
archive over the basis [library, F] (F's κ-label, the same identity a window
frame computes) whose 10-byte body returns basis root 1. No library block, no
basis identity change and no export: the exported decoder verifies it given
window [F]. The encoder labels each frame (SHA-256 chain) and skips coding
repeats; decoders read headers first and keep only frames a later reference
names, in memory up to 256 MiB then in an unnamed temp file. A stream without
repeats is byte-identical to before. decompress_stream now takes Read + Seek;
the CLI spools a piped stdin to a temp file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3M5Q4GpN89jhJUkexu

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants