Skip to content

uorc: window frames, so matches cross 8 MiB frame boundaries - #117

Merged
afflom merged 10 commits into
mainfrom
claude/project-thread-e7b16t
Nov 9, 2026
Merged

afflom merged 10 commits into
mainfrom
claude/project-thread-e7b16t

Conversation

@afflom

@afflom afflom commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Requested by Alex · project thread

With 8 MiB frames, matches could not reach across a frame boundary, which cost up to 4x in ratio on large inputs. This PR implements natively the window-frame model that #116 exported.

What changes

  • Window frames: every frame after the first is a conditional archive over the basis [standard library, the outputs of the previous ≤2 frames], named by its UORC-BASIS-V1 identity (Archive.windowIdentity). Its entry decodes the ulz payload with those frames as history. A one-frame stream is unchanged. There is no wire-format change and no re-export.
  • archive.rs:
    • window_identity, window_archive, window_decode and basis_of.
    • chain_parts hashes the 16 MiB of roots without joining them.
    • window_decode checks the seal and the basis identity on a second core while the payload decodes.
  • lib.rs:
    • New compress_frame_with and decompress_frame_with.
    • Compress stays parallel, because every frame's window is known input.
    • Decompress, verify and decompress_objects now run frames in order and hold only the last two outputs.
    • Archives that are not window frames (raw, the linked ulz frame 0, and ncm under --best) ignore the window, as the model's decompress does.
  • Wasm and page: a new window_reset(). compress_frame and decompress_frame carry the window across calls, so compressBlob and decompressBlob stream windows too.

Results

Release binary on 4 cores, against xz -9 -T1. Sizes are in bytes.

input without windows this PR xz -9
4 × 10 MiB generated Rust (40 MiB) 1,602,959 430,605 404,604
10 MiB generated Rust (2 frames) not measured 421,036 399,976
repo tarball (128 MB) not measured 35,653,338 33,100,520

On the 40 MiB input, compression takes 0.41 s (xz: 8.9 s). Decompression takes 0.21 s (xz: 0.05 s) and is now sequential.

Known gap, next PR: high-entropy frames, such as the compressed blobs inside the tarball, take up to 20 s per 8 MiB to compress. That makes the tarball take 44 s, against 36 s for xz. The cause is the ulz match finder: a 3-byte hash into 2^16 buckets with depth-48 chains is cache-miss bound when nothing matches, and it also misses long-distance matches in such data. Fixing it changes the encoder, so the model's LzEncode has to change with it.

Tests

cargo test --release -p uorc passes 39 unit tests and 7 CLI tests. The new tests are:

  • window_frames_are_the_model_archives: the native window archive is byte-identical to the exported compress input window 0 0 for windows of 1 and 2, and a wrong window is refused.
  • window_frames_decode_by_the_reference_semantics: the exported decoder reconstructs a window frame against its window.
  • windows_carry_matches_across_frames: a 3-frame repeat costs less than 1.07 frames' archive, and stream, verify and in-memory decoding all round-trip.
  • chain_parts_is_chain_of_the_concatenation.

cargo clippy -p uorc -p uorc-wasm --all-targets -D warnings and cargo fmt --check are clean.


🤖 Generated with Claude Code

https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

claude and others added 10 commits November 9, 2026 18:43
Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…ntial tests, ulz history window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
The differential tests run the exported decoder graph; unoptimized, the uorc
unit tests took 30 min of the 60 min acceptance leg (main is over budget
too). At opt-level 1 they take 4.5 min for about 6 min more compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…#115)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

---------

Co-authored-by: Claude <noreply@anthropic.com>
…-e7b16t

# Conflicts:
#	crates/uorc-wasm/src/lib.rs
#	crates/uorc/src/archive.rs
#	crates/uorc/src/bin/uorc.rs
#	crates/uorc/src/lib.rs
#	web/app.js
#	web/index.html
#	web/uorc.js
…to claude/project-thread-e7b16t

# Conflicts:
#	crates/uorc-wasm/src/lib.rs
#	crates/uorc/src/archive.rs
#	crates/uorc/src/bin/uorc.rs
#	crates/uorc/src/lib.rs
@afflom
afflom merged commit bb7a636 into main Nov 9, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants