Repository navigation
uorc: native codec fast path (faster than xz) and framed streaming of any size - #112
Merged
Merged
Conversation
afflom
force-pushed
the
claude/project-thread-e7b16t
branch
from
November 9, 2026 16:13
20539db to
1cd3d59
Compare
Throughput (#105): crates/uorc now carries native Rust ports of the model's hot paths, byte-identical to it: - ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode. - archive.rs: the archive layout, kernel prefix/suffix and the UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive. compress writes exactly the archive the exported model writes (tested against it). The native decoder only accepts payloads on which every reading agrees (code < range throughout, match sources inside the output, no overrun, payload consumed exactly) and so needs no canonical re-encode check; anything else goes to the exported reference decoder, and a differential test checks agreement on damaged payloads. Streaming (#106): a uorc stream is standard self-contained archives laid end to end, one per 8 MiB frame; a one-frame stream is the model's archive byte for byte. compress_stream/decompress_stream keep memory bounded by frames in flight, frames run in parallel (--threads), and the native paths need no large stack (the 1 GiB reservation now applies only to the exported reference code). The wasm page streams files through frames too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…ntial tests, ulz history window Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
afflom
force-pushed
the
claude/project-thread-e7b16t
branch
from
November 9, 2026 18:43
1cd3d59 to
d8c07cf
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
The differential tests run the exported decoder graph; unoptimized, the uorc unit tests took 30 min of the 60 min acceptance leg (main is over budget too). At opt-level 1 they take 4.5 min for about 6 min more compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
afflom
pushed a commit
that referenced
this pull request
Nov 9, 2026
afflom
pushed a commit
that referenced
this pull request
Nov 9, 2026
afflom
pushed a commit
that referenced
this pull request
Nov 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV
afflom
added a commit
that referenced
this pull request
Nov 10, 2026
* uorc client: LexLean-exported Rust compress/decompress, CLI, wasm, Pages site, release and bench Uorc.Client composes Search.Engine (whole-buffer compress under a step budget) and the ArchiveTrial decode phases (decompress). scripts/client-export.mjs stages the closure, marks it executable, works around LexLean 0.4.0 LLV7015 with a typed let wrap, and runs the certified production build into generated/rust. Work in progress: the exported Rust is far slower than the Lean-native build and overflows the stack on deep recursion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * Export the client with shared-box LexLean; pin it; tighten wasm stack The client crates are re-exported with LexLean at 7da569d (rust-std keeps datatype fields behind Rc), pinned in scripts/client-export.mjs until the lowering is released. Compress runs 18x and decompress 5x faster than the inline-record export, and the WebAssembly stack drops from 256 MiB to 64 MiB. The generated crates leave the workspace members so the workspace fmt and clippy gates do not re-lint or reformat LexLean's output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * Keep the client roots non-executable in the authority graph The authority graph links Client against a closure that is not marked executable, so lexlean check refused Client's executable definitions (LLT4001). The export now adds executability and the two production annotations on its staged copy; the staged data form equals the one the committed packages were exported from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * release: check out complete history The template trust root requires every checkout to fetch full history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * deny: license the exported client crates under the repository licence Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * client: pin LexLean to claude/uorc-pin (shared-box lowering #441 + in-place byte update #443) The export pin moves to the branch that carries both LexLean patches; the re-exported client gains the std runtime's Bytes::update and is otherwise unchanged. The pin moves to a LexLean release once both ship. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * client: ulz kernel archives (LZ + range coder as a UORC program graph) compress now writes a standard UORC program archive whose graph is the ulz decoder kernel (45 blocks, ~3 KB on the wire) and whose payload is the ulz encoder's output, or the raw archive when that is smaller. The program search runs only when --steps asks for it. decompress recognises a kernel archive by rebuilding it byte for byte, runs the certified LexLean decoder, and keeps the result only if re-encoding reproduces the payload; every other archive takes the reference path. The current reference client decodes kernel archives by the graph semantics, and a test decodes one that way. 1 MB inputs: rust.rs 54,366 B (gzip -9 61,786, xz -9 46,924), elf.bin 202,696 B (gzip 238,109, xz 183,444); compress 0.7-1.9 s, decompress 0.7-1.9 s. The codec and the client live in client/src, outside the authority project, because the codec's tables use the in-place byte update that only the pinned LexLean has; client-export stages client/src over src. client/gen holds the writing aids (regenerate with client/gen/build.sh; check_graph.py runs the kernel graph by a Python mirror of the reference semantics). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * client: raw archives decode by a rebuild check instead of the reference path decompress recognises a raw archive by rebuilding it from its body and comparing bytes, so a stored input decodes in milliseconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * client: smaller ulz kernel graph (2988 -> 2598 bytes) Scalars move to low store keys with per-width accessors, and the payload is copied into the store once instead of being threaded through every block. Archives shrink by about 390 bytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Nk2H9geAZ2GKFCNvM * Settle the open conformance rows and benchmark Silesia on release Registry (#108): - Bind UC-REAL-01, UC-API-01, UC-WIRE-01, UC-INT-01, UC-MACH-01, UC-MACH-02, UC-SYN-01 and UC-OPT-01 to the shipped client, narrowed to what it does, with tests in crates/uorc/tests/conformance.rs and the uorc-wasm exports test. - UC-OBJ-01 and UC-BAS-01 stay open and say why in their own text. - Retire UC-PROD-01, UC-CTL-01, UC-VIW-01, UC-STG-01, UC-EXE-01, UC-CHK-01 and UC-INT-06 (old Hologram/View product model), with the xtask tests and runtime scripts that only served them. Benchmark (#51): - bench.mjs takes several corpora and writes a per-corpus size, ratio and throughput summary to bench.{json,md,html}. - release.yml runs it in its own release-only job on Canterbury and Silesia; the table goes into the release notes and the Pages site. - UC-PERF-01 states this measure-only disposition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV * registry: canonical lexlean fmt after retiring rows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV * uorc-wasm: inspect returns a JSON array since #112 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV * Bind UC-BAS-01 to the standard basis and follow #116's level API - UC-BAS-01 is build evidence: archives at both levels name the UORC-BASIS-V1 identity, and a resealed archive naming another basis fails to decode. Only UC-PERF-01 (measure only) and UC-REC-01 (external record) remain open, so open_claims.rs is gone. - Row tests and the wasm test take compress's level argument; OPT-01 also checks --best is never larger than the default; WIRE-01 and OPT-01 describe linked archives and the ncm candidate. - bench.mjs measures the default level and --best on every corpus. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Pb2pARmowTZwiuFLV --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Alex · project thread
Closes #105. Closes #106. Rebased onto
mainafter #110.What changes
crates/uorc/src/ulz.rs: a native Rust port of the ulz codec (Uorc.Codec.LzEncode/LzDecode), operation for operation.crates/uorc/src/archive.rs: the archive layout, the kernel prefix and suffix, and the UORC-SHA256-CHAIN-v1 seal (sha2, which uses SHA-NI where the CPU has it). These mirrorUorc.Codec.Archive. A test checks the kernel bytes againstArchive.lex.tex.frame.rsnow builds onarchive.rsfor the seal, the ULEB writer and the kernel prefix. It also provides the header-onlyframe_lengththat streaming uses. There is no second parser or second chain implementation.compresswrites exactly the archive the exported model writes. The testnative_archives_are_the_model_archivescompares the two directly.native_decoder_agrees_with_the_reference_where_it_acceptschecks this on damaged, resealed payloads.objectLengthslists the pieces of objects it holds.compress_objects(&[x]) == compress(x).compress_streamanddecompress_stream(anyReadto anyWrite) keep memory bounded by the frames in flight. Frames run in parallel (CLI--threads N, default every core).decompress,verifyanddecompress_objectsalso run frames in parallel.Pendingtemporary file.inspectandverifywalk multi-frame streams.compress_frame,decompress_frame,frame_lengthandframe_size.inspectreturns one description per archive.compressBlob/decompressBlob).Throughput
Release binary on a 4-core container. Times are wall clock and include process start.
xz -9 -T1is the comparator.Ratio cost of framing: it costs nothing below 8 MiB. Above that, matches can't cross frame boundaries. On four repeats of a 10 MiB file the result is 1.60 MB, against 0.39 MB for xz.
The agreed fix is window frames: each frame is a conditional archive over the previous frames' outputs. The model side is with the ratio lane's export, and the native side follows here. With a window of two frames the same input measures 0.41 MB.
Tests
cargo test --release -p uorc: 27 unit tests and 7 CLI tests pass. That includes multi-frame object streams, frame plans, and verify across frames.cargo clippy -p uorc -p uorc-wasm --all-targets -D warningsandcargo fmt --check -p uorc -p uorc-wasmare clean.[profile.dev.package.uorc] opt-level = 3, and the heavier exported-model comparisons are release-only, so the debug acceptance leg fits its budget.🤖 Generated with Claude Code
https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk