Skip to content

ulz: hash4 chain finder, 3.7x faster and 2.1% smaller on a 128 MB tarball - #120

Merged
afflom merged 30 commits into
mainfrom
claude/project-thread-e7b16t
Nov 10, 2026
Merged

afflom merged 30 commits into
mainfrom
claude/project-thread-e7b16t

Conversation

@afflom

@afflom afflom commented Nov 10, 2026

Copy link
Copy Markdown
Member

Requested by Alex · project thread

Closes #118. This is stacked on #119 (ncm v2), so the diff includes #119 until it merges. The new commits are ecb0597 and the re-export after it.

Result (release binary, 4 cores, default level)

input before: bytes / compress s now: bytes / compress s xz -9 -T1
128 MB mixed tarball 35,653,338 / 44 s 34,899,652 / 11.9 s 33,100,520 / 36 s
40 MiB repeated text 430,605 425,184 404,604

The tarball decompresses in 2.7 s. On one high-entropy 8 MiB frame the encoder goes from 14–19 s to 1.4–1.8 s.

What changes

The ulz match finder in the encoder model (client/src/Uorc/Codec/LzEncode.lex.tex, generated by client/gen/gen_enc.py), the Python mirror (client/gen/ulz.py) and the native port (crates/uorc/src/ulz.rs). The bitstream and the decoder do not change, so every existing archive still decodes.

  • Old finder: 3-byte hash, 2^16 buckets, chain depth 48. On high-entropy data every bucket chain is long and mostly wrong, so it was cache-miss bound.
  • New finder: 4-byte hash chains in 2^hb buckets, where hb = bitlen(n+1) clamped to 16..22. A 3-byte latest-position table supplies length-2 and length-3 candidates. The head3 candidate is checked first, then the hash4 chain is walked to depth 48 (stopping at length 64).
  • Model memory: head3 at word 4096, head4 at word 69632, and the chain after it. EncS gains h4div, h4size and chainAt. The table sizes depend on the input, as the certificate needs.

Checks

  • The native encoder and the Python mirror are byte-identical on 9 fixtures (empty, short, runs, text, random, mixed), both plain and with a history window.
  • lexlean check passes. generated/rust is re-exported (node scripts/client-export.mjs, LexLean 4c41d39), and the rust-std certificates pass.
  • cargo test --release -p uorc passes: 39 unit tests (including native_archives_are_the_model_archives and window_frames_are_the_model_archives), 7 CLI tests and 10 conformance tests. cargo clippy -p uorc -p uorc-wasm --all-targets -D warnings and cargo fmt -p uorc --check are clean.

Generated by Claude Code

claude and others added 30 commits November 9, 2026 16:13
Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…P, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs
Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
…ntial tests, ulz history window

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs
A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
The differential tests run the exported decoder graph; unoptimized, the uorc
unit tests took 30 min of the 60 min acceptance leg (main is over budget
too). At opt-level 1 they take 4.5 min for about 6 min more compile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs
…#115)

* uorc: native codec fast path and framed streaming (#105, #106)

Throughput (#105): crates/uorc now carries native Rust ports of the
model's hot paths, byte-identical to it:
- ulz.rs: the LZ + range coder of Uorc.Codec.LzEncode/LzDecode.
- archive.rs: the archive layout, kernel prefix/suffix and the
  UORC-SHA256-CHAIN-v1 seal (sha2), mirroring Uorc.Codec.Archive.
compress writes exactly the archive the exported model writes (tested
against it). The native decoder only accepts payloads on which every
reading agrees (code < range throughout, match sources inside the
output, no overrun, payload consumed exactly) and so needs no canonical
re-encode check; anything else goes to the exported reference decoder,
and a differential test checks agreement on damaged payloads.

Streaming (#106): a uorc stream is standard self-contained archives laid
end to end, one per 8 MiB frame; a one-frame stream is the model's
archive byte for byte. compress_stream/decompress_stream keep memory
bounded by frames in flight, frames run in parallel (--threads), and the
native paths need no large stack (the 1 GiB reservation now applies only
to the exported reference code). The wasm page streams files through
frames too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* uorc: ncm context-mixing coder and a linked standard kernel basis (WIP, export pending)

The ncm coder (client/gen/ncm_model.py, one IR compiled to LexLean, to the
UORC kernel graph and ported natively) and the standard kernel basis: the
ulz and ncm decoder libraries bound by one UORC-BASIS-V1 identity, named by
linked (scope 1) archives so small files carry no kernel. Compress gains a
level (--best tries ncm). The exported client is not regenerated yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* client: ncm LexLean sized for the rust-std certificate

Scalars live in a small table instead of a rebuilt record, long statement
lists are split into functions, literals go first in additions, and literal
loop counts name the state, so the preservation certificate does not unroll
the model's setup. Output is byte-identical to before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc model: window frames over the standard basis

A frame can name the outputs of the K preceding frames (K = 1 or 2) as
basis roots 1..K after the kernel library (root 0), so its ulz payload is
coded against that window and matches reach across frame boundaries.

- LzDecode.decodeWith / LzEncode.encodeWith: the codec against a history
  window (decode/encode are the empty-history case, byte for byte as before).
- ulz library block runWith(payload, history); window entries apply it to
  basisGet 1 (++ basisGet 2). Graph mirror gains basisGet; gen_basis --check
  decodes K=1 and K=2 window frames on the mirror.
- Archive: windowIdentity (UORC-BASIS-V1 chain over [library, window..]),
  windowArchive, ulzWindowArchive, windowDecode, fastDecodeWith,
  windowMaterialize (keeps scope 1 and the identity; inlines the library).
- Client: compress(input, window, steps, level); decompress(archive, window,
  work) passes scope conditional + basis [library, window..] to the
  reference trial for window frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk

* client: fix window archive model syntax; refresh exact axiom lists

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* generated: re-export the client (ncm, linked basis, window frames)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

* uorc: ncm context-mixing level and the standard kernel basis

ncm, a context-mixing coder extrapolated from the neuromorphic model, is
written once as an IR and compiled to LexLean, to a UORC kernel graph and
natively. The ulz and ncm decoder libraries form root 0 of a UORC-BASIS-V1
basis; archives are linked to it by identity, so small files no longer
carry a 2.6 KB kernel. Level::Best (--best) also tries ncm and keeps the
smaller archive; on Canterbury it is smaller than xz -9e on every file.

inspect names linked archives' representation (ulz or ncm).

Closes #62. Closes #107.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs

---------

Co-authored-by: Claude <noreply@anthropic.com>
…-e7b16t

# Conflicts:
#	crates/uorc-wasm/src/lib.rs
#	crates/uorc/src/archive.rs
#	crates/uorc/src/bin/uorc.rs
#	crates/uorc/src/lib.rs
#	web/app.js
#	web/index.html
#	web/uorc.js
…to claude/project-thread-e7b16t

# Conflicts:
#	crates/uorc-wasm/src/lib.rs
#	crates/uorc/src/archive.rs
#	crates/uorc/src/bin/uorc.rs
#	crates/uorc/src/lib.rs
…-84bxra

# Conflicts:
#	crates/uorc-wasm/src/lib.rs
#	crates/uorc/src/archive.rs
#	crates/uorc/src/bin/uorc.rs
#	crates/uorc/src/lib.rs
The context-mixing coder gains:
- two adaptive probability maps (selected by the partial byte and by an
  order-3 hash) that refine the mixed probability;
- a third mixer, selected by the byte before the previous one and the
  bit position;
- a plasticity that decays from 128/65536 to 24/65536 per unit of
  x * err as the input is seen;
- a check byte in each 16-slot bucket's unused slot 0, with a neighbour
  pair and recycling by use, so colliding contexts stop sharing counts.

Model IR, LexLean, kernel graph and the native port are byte-identical
to the C spec on Canterbury. The ncm library grows, so the basis
identity and the linked templates are regenerated, and generated/rust
is re-exported. No released archive uses the previous identity.

Canterbury, uorc --best: 355,559 B (was 369,878; xz -9e 493,080).

bench.mjs passes --force, since the client refuses to overwrite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NtwZjtFZ8oFtv7gMs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NkSq1MUwK7keT67Uk
@afflom
afflom merged commit 3ba0dc4 into main Nov 10, 2026
14 checks passed
afflom pushed a commit that referenced this pull request Nov 10, 2026
… arrived through the branch history)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NpWuTWFrL9kYU72Rv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ulz encoder: the match finder stalls on high-entropy data and misses long matches

2 participants