Skip to content

uorc: native codec fast path (faster than xz) and framed streaming of any size - #112

Merged
afflom merged 6 commits into
mainfrom
claude/project-thread-e7b16t
Nov 9, 2026
Merged

afflom merged 6 commits into
mainfrom
claude/project-thread-e7b16t

Conversation

@afflom

@afflom afflom commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Requested by Alex · project thread

Closes #105. Closes #106. Rebased onto main after #110.

What changes

  • crates/uorc/src/ulz.rs: a native Rust port of the ulz codec (Uorc.Codec.LzEncode / LzDecode), operation for operation.
  • crates/uorc/src/archive.rs: the archive layout, the kernel prefix and suffix, and the UORC-SHA256-CHAIN-v1 seal (sha2, which uses SHA-NI where the CPU has it). These mirror Uorc.Codec.Archive. A test checks the kernel bytes against Archive.lex.tex.
  • One header parser: uorc client: inspect, verify, safe writes, multi-file archives #110's frame.rs now builds on archive.rs for the seal, the ULEB writer and the kernel prefix. It also provides the header-only frame_length that streaming uses. There is no second parser or second chain implementation.
  • Byte identity: compress writes exactly the archive the exported model writes. The test native_archives_are_the_model_archives compares the two directly.
  • No re-encode check on decode: the native decoder accepts only payloads on which every reading agrees. It declines everything else, and those archives go to the exported reference decoder. native_decoder_agrees_with_the_reference_where_it_accepts checks this on damaged, resealed payloads.
  • Frames: a uorc stream is standard self-contained archives laid end to end, one per 8 MiB of input. A one-frame stream is byte for byte the model's archive. There is no wire-format change and no export regeneration.
  • Objects and trees across frames: each frame's objectLengths lists the pieces of objects it holds.
    • The first piece of every frame after the first continues the last object of the frame before. An object that ends exactly on a boundary is continued by an empty piece.
    • A plain stream is the one-object case, so compress_objects(&[x]) == compress(x).
    • Native archives are written with their object lengths directly, with no relabel pass. Archives the reference decoder handles are still relabelled to one object, as in uorc client: inspect, verify, safe writes, multi-file archives #110.
  • Directory-only trees: these were one object, which read as a plain stream. They now carry an empty end object.
  • Streaming API: compress_stream and decompress_stream (any Read to any Write) keep memory bounded by the frames in flight. Frames run in parallel (CLI --threads N, default every core). decompress, verify and decompress_objects also run frames in parallel.
  • CLI: keeps uorc client: inspect, verify, safe writes, multi-file archives #110's commands, exit codes and safe writes.
    • Plain compress and decompress stream through the Pending temporary file.
    • Decompress reads the first header to tell a tree from a plain stream.
    • inspect and verify walk multi-frame streams.
  • Stack: the 1 GiB virtual reservation is used only for calls into the exported reference code.
  • Wasm and Pages:
    • The module gains compress_frame, decompress_frame, frame_length and frame_size.
    • inspect returns one description per archive.
    • The page streams single files of any size frame by frame (compressBlob/decompressBlob).

Throughput

Release binary on a 4-core container. Times are wall clock and include process start. xz -9 -T1 is the comparator.

file uorc bytes xz -9 bytes uorc compress xz -9 compress uorc decompress xz -d
Rust source, 1 MiB 71,891 59,680 0.035 s 0.298 s 0.009 s 0.006 s
ELF binary, 1 MiB 596,860 469,264 0.25 s 0.33 s 0.039 s 0.038 s
generated Rust, 10 MiB 403,129 388,872 0.21 s 2.39 s 0.025 s 0.033 s

Ratio cost of framing: it costs nothing below 8 MiB. Above that, matches can't cross frame boundaries. On four repeats of a 10 MiB file the result is 1.60 MB, against 0.39 MB for xz.

The agreed fix is window frames: each frame is a conditional archive over the previous frames' outputs. The model side is with the ratio lane's export, and the native side follows here. With a window of two frames the same input measures 0.41 MB.

Tests

  • cargo test --release -p uorc: 27 unit tests and 7 CLI tests pass. That includes multi-frame object streams, frame plans, and verify across frames.
  • cargo clippy -p uorc -p uorc-wasm --all-targets -D warnings and cargo fmt --check -p uorc -p uorc-wasm are clean.
  • [profile.dev.package.uorc] opt-level = 3, and the heavier exported-model comparisons are release-only, so the debug acceptance leg fits its budget.

🤖 Generated with Claude Code

https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

@afflom afflom self-assigned this Nov 9, 2026
@afflom
afflom force-pushed the claude/project-thread-e7b16t branch from 20539db to 1cd3d59 Compare November 9, 2026 16:13
@afflom
afflom changed the base branch from claude/project-thread-jl6avw to main November 9, 2026 16:14
Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…ntial tests, ulz history window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
@afflom
afflom force-pushed the claude/project-thread-e7b16t branch from 1cd3d59 to d8c07cf Compare November 9, 2026 18:43
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
The differential tests run the exported decoder graph; unoptimized, the uorc
unit tests took 30 min of the 60 min acceptance leg (main is over budget
too). At opt-level 1 they take 4.5 min for about 6 min more compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
@afflom
afflom merged commit b9f48c1 into main Nov 9, 2026
14 checks passed
afflom pushed a commit that referenced this pull request Nov 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV
afflom added a commit that referenced this pull request Nov 10, 2026
* uorc client: LexLean-exported Rust compress/decompress, CLI, wasm, Pages site, release and bench

Uorc.Client composes Search.Engine (whole-buffer compress under a step budget)
and the ArchiveTrial decode phases (decompress). scripts/client-export.mjs
stages the closure, marks it executable, works around LexLean 0.4.0 LLV7015
with a typed let wrap, and runs the certified production build into
generated/rust. Work in progress: the exported Rust is far slower than the
Lean-native build and overflows the stack on deep recursion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* Export the client with shared-box LexLean; pin it; tighten wasm stack

The client crates are re-exported with LexLean at 7da569d (rust-std keeps
datatype fields behind Rc), pinned in scripts/client-export.mjs until the
lowering is released. Compress runs 18x and decompress 5x faster than the
inline-record export, and the WebAssembly stack drops from 256 MiB to 64 MiB.
The generated crates leave the workspace members so the workspace fmt and
clippy gates do not re-lint or reformat LexLean's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* Keep the client roots non-executable in the authority graph

The authority graph links Client against a closure that is not marked
executable, so lexlean check refused Client's executable definitions
(LLT4001). The export now adds executability and the two production
annotations on its staged copy; the staged data form equals the one the
committed packages were exported from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* release: check out complete history

The template trust root requires every checkout to fetch full history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* deny: license the exported client crates under the repository licence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* client: pin LexLean to claude/uorc-pin (shared-box lowering #441 + in-place byte update #443)

The export pin moves to the branch that carries both LexLean patches; the
re-exported client gains the std runtime's Bytes::update and is otherwise
unchanged. The pin moves to a LexLean release once both ship.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* client: ulz kernel archives (LZ + range coder as a UORC program graph)

compress now writes a standard UORC program archive whose graph is the ulz
decoder kernel (45 blocks, ~3 KB on the wire) and whose payload is the ulz
encoder's output, or the raw archive when that is smaller. The program search
runs only when --steps asks for it. decompress recognises a kernel archive by
rebuilding it byte for byte, runs the certified LexLean decoder, and keeps the
result only if re-encoding reproduces the payload; every other archive takes the
reference path. The current reference client decodes kernel archives by the
graph semantics, and a test decodes one that way.

1 MB inputs: rust.rs 54,366 B (gzip -9 61,786, xz -9 46,924), elf.bin 202,696 B
(gzip 238,109, xz 183,444); compress 0.7-1.9 s, decompress 0.7-1.9 s.

The codec and the client live in client/src, outside the authority project,
because the codec's tables use the in-place byte update that only the pinned
LexLean has; client-export stages client/src over src. client/gen holds the
writing aids (regenerate with client/gen/build.sh; check_graph.py runs the
kernel graph by a Python mirror of the reference semantics).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* client: raw archives decode by a rebuild check instead of the reference path

decompress recognises a raw archive by rebuilding it from its body and
comparing bytes, so a stored input decodes in milliseconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* client: smaller ulz kernel graph (2988 -> 2598 bytes)

Scalars move to low store keys with per-width accessors, and the payload
is copied into the store once instead of being threaded through every
block. Archives shrink by about 390 bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM

* Settle the open conformance rows and benchmark Silesia on release

Registry (#108):
- Bind UC-REAL-01, UC-API-01, UC-WIRE-01, UC-INT-01, UC-MACH-01,
  UC-MACH-02, UC-SYN-01 and UC-OPT-01 to the shipped client, narrowed to
  what it does, with tests in crates/uorc/tests/conformance.rs and the
  uorc-wasm exports test.
- UC-OBJ-01 and UC-BAS-01 stay open and say why in their own text.
- Retire UC-PROD-01, UC-CTL-01, UC-VIW-01, UC-STG-01, UC-EXE-01,
  UC-CHK-01 and UC-INT-06 (old Hologram/View product model), with the
  xtask tests and runtime scripts that only served them.

Benchmark (#51):
- bench.mjs takes several corpora and writes a per-corpus size, ratio
  and throughput summary to bench.{json,md,html}.
- release.yml runs it in its own release-only job on Canterbury and
  Silesia; the table goes into the release notes and the Pages site.
- UC-PERF-01 states this measure-only disposition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV

* registry: canonical lexlean fmt after retiring rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV

* uorc-wasm: inspect returns a JSON array since #112

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV

* Bind UC-BAS-01 to the standard basis and follow #116's level API

- UC-BAS-01 is build evidence: archives at both levels name the
  UORC-BASIS-V1 identity, and a resealed archive naming another basis
  fails to decode. Only UC-PERF-01 (measure only) and UC-REC-01
  (external record) remain open, so open_claims.rs is gone.
- Row tests and the wasm test take compress's level argument; OPT-01
  also checks --best is never larger than the default; WIRE-01 and
  OPT-01 describe linked archives and the ncm candidate.
- bench.mjs measures the default level and --best on every corpus.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream input of any size in bounded frames Client throughput: compress and decompress at least as fast as xz

2 participants