Repository navigation
uorc: window frames, so matches cross 8 MiB frame boundaries - #117
Merged
Merged
Conversation
Throughput (#105): crates/uorc now carries native Rust ports of the model's hot paths, byte-identical to it: - ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode. - archive.rs: the archive layout, kernel prefix/suffix and the UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive. compress writes exactly the archive the exported model writes (tested against it). The native decoder only accepts payloads on which every reading agrees (code < range throughout, match sources inside the output, no overrun, payload consumed exactly) and so needs no canonical re-encode check; anything else goes to the exported reference decoder, and a differential test checks agreement on damaged payloads. Streaming (#106): a uorc stream is standard self-contained archives laid end to end, one per 8 MiB frame; a one-frame stream is the model's archive byte for byte. compress_stream/decompress_stream keep memory bounded by frames in flight, frames run in parallel (--threads), and the native paths need no large stack (the 1 GiB reservation now applies only to the exported reference code). The wasm page streams files through frames too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…ntial tests, ulz history window Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
The differential tests run the exported decoder graph; unoptimized, the uorc unit tests took 30 min of the 60 min acceptance leg (main is over budget too). At opt-level 1 they take 4.5 min for about 6 min more compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…#115) * uorc: native codec fast path and framed streaming (#105, #106) Throughput (#105): crates/uorc now carries native Rust ports of the model's hot paths, byte-identical to it: - ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode. - archive.rs: the archive layout, kernel prefix/suffix and the UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive. compress writes exactly the archive the exported model writes (tested against it). The native decoder only accepts payloads on which every reading agrees (code < range throughout, match sources inside the output, no overrun, payload consumed exactly) and so needs no canonical re-encode check; anything else goes to the exported reference decoder, and a differential test checks agreement on damaged payloads. Streaming (#106): a uorc stream is standard self-contained archives laid end to end, one per 8 MiB frame; a one-frame stream is the model's archive byte for byte. compress_stream/decompress_stream keep memory bounded by frames in flight, frames run in parallel (--threads), and the native paths need no large stack (the 1 GiB reservation now applies only to the exported reference code). The wasm page streams files through frames too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk * uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending) The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the UORC kernel graph and ported natively) and the standard kernel basis: the ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by linked (scope 1) archives so small files carry no kernel. Compress gains a level (--best tries ncm). The exported client is not regenerated yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs * client: ncm LexLean sized for the rust-std certificate Scalars live in a small table instead of a rebuilt record, long statement lists are split into functions, literals go first in additions, and literal loop counts name the state, so the preservation certificate does not unroll the model's setup. Output is byte-identical to before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs * uorc model: window frames over the standard basis A frame can name the outputs of the K preceding frames (K = 1 or 2) as basis roots 1..K after the kernel library (root 0), so its ulz payload is coded against that window and matches reach across frame boundaries. - LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history window (decode/encode are the empty-history case, byte for byte as before). - ulz library block runWith(payload, history); window entries apply it to basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check decodes K=1 and K=2 window frames on the mirror. - Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]), windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith, windowMaterialize (keeps scope 1 and the identity; inlines the library). - Client: compress(input, window, steps, level); decompress(archive, window, work) passes scope conditional + basis [library, window..] to the reference trial for window frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk * client: fix window archive model syntax; refresh exact axiom lists Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs * generated: re-export the client (ncm, linked basis, window frames) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs * uorc: ncm context-mixing level and the standard kernel basis ncm, a context-mixing coder extrapolated from the neuromorphic model, is written once as an IR and compiled to LexLean, to a UORC kernel graph and natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1 basis; archives are linked to it by identity, so small files no longer carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the smaller archive; on Canterbury it is smaller than xz -9e on every file. inspect names linked archives' representation (ulz or ncm). Closes #62. Closes #107. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs --------- Co-authored-by: Claude <noreply@anthropic.com>
…-e7b16t # Conflicts: # crates/uorc-wasm/src/lib.rs # crates/uorc/src/archive.rs # crates/uorc/src/bin/uorc.rs # crates/uorc/src/lib.rs # web/app.js # web/index.html # web/uorc.js
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…to claude/project-thread-e7b16t # Conflicts: # crates/uorc-wasm/src/lib.rs # crates/uorc/src/archive.rs # crates/uorc/src/bin/uorc.rs # crates/uorc/src/lib.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Alex · project thread
With 8 MiB frames, matches could not reach across a frame boundary, which cost up to 4x in ratio on large inputs. This PR implements natively the window-frame model that #116 exported.
What changes
[standard library, the outputs of the previous ≤2 frames], named by its UORC-BASIS-V1 identity (Archive.windowIdentity). Its entry decodes the ulz payload with those frames as history. A one-frame stream is unchanged. There is no wire-format change and no re-export.archive.rs:window_identity,window_archive,window_decodeandbasis_of.chain_partshashes the 16 MiB of roots without joining them.window_decodechecks the seal and the basis identity on a second core while the payload decodes.lib.rs:compress_frame_withanddecompress_frame_with.decompress_objectsnow run frames in order and hold only the last two outputs.--best) ignore the window, as the model'sdecompressdoes.window_reset().compress_frameanddecompress_framecarry the window across calls, socompressBlobanddecompressBlobstream windows too.Results
Release binary on 4 cores, against
xz -9 -T1. Sizes are in bytes.On the 40 MiB input, compression takes 0.41 s (xz: 8.9 s). Decompression takes 0.21 s (xz: 0.05 s) and is now sequential.
Known gap, next PR: high-entropy frames, such as the compressed blobs inside the tarball, take up to 20 s per 8 MiB to compress. That makes the tarball take 44 s, against 36 s for xz. The cause is the ulz match finder: a 3-byte hash into 2^16 buckets with depth-48 chains is cache-miss bound when nothing matches, and it also misses long-distance matches in such data. Fixing it changes the encoder, so the model's
LzEncodehas to change with it.Tests
cargo test --release -p uorcpasses 39 unit tests and 7 CLI tests. The new tests are:window_frames_are_the_model_archives: the native window archive is byte-identical to the exportedcompress input window 0 0for windows of 1 and 2, and a wrong window is refused.window_frames_decode_by_the_reference_semantics: the exported decoder reconstructs a window frame against its window.windows_carry_matches_across_frames: a 3-frame repeat costs less than 1.07 frames' archive, and stream, verify and in-memory decoding all round-trip.chain_parts_is_chain_of_the_concatenation.cargo clippy -p uorc -p uorc-wasm --all-targets -D warningsandcargo fmt --checkare clean.🤖 Generated with Claude Code
https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk