Repository navigation
uorc v0.4: choice rule tests x86 before size, keeps the v0.3 entry in small schedules - #136
Merged
Merged
Conversation
…xt buckets, squash LUT, reused next-byte hashes A profile is a set of the 17 contexts; a left-out context is never looked up or adapted and would feed every mixer 0, so the payload is exactly the model with that input held at 0. The full profile is byte-identical to v0.3. Mixer rows are 16 wide (one cache line) when the profile has at most 13 contexts. Byte-identical for every profile: a context bucket's 16 slots pack probability and count into one 64-byte line, squash is a 4096-entry integer table, and the context hashes prefetch_next computes for the expected byte are reused when that byte comes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
…-gated Profile bit 17 adds a second match model keyed on the 4 bytes before a position (found from 4 agreeing bytes). Profile bit 18 adds a repeat model: the byte at the distance of the last match found keeps being predicted after a mismatch, with its run length as confidence, a hashed context of its expected byte, run and the previous byte (context 13 under this bit), and the match state that selects mixer 1 and the first APM. Profile 0x1ffff is the v0.3 coder byte for byte; both bits are 6.2% smaller on eight test files (3 MB of .rlib: 744,349 -> 674,978 B; xz -9e 685,552). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…rofiled coder Profile word: bits 0-16 keep contexts, F_SHORT (17) and F_REP (18) the match features; V03 = 0x1ffff is the v0.3 coder byte for byte, FULL = 0x7ffff. Native equals lane 1's native for every feature combination and equals the IR (client/gen/ncm_model.py) for masked profiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
…s start at weight 0 Bit 19 (F_SEL) now carries the selection of mixer 1 and the first APM by the two match models' states: it pays on long inputs (+0.6% without it on 9.5 MB) and costs on short ones (-1.0% without it on 150 files under 32 KiB). The short match and repeat inputs start at weight 0 so they only count once they predict. Profile 0x7ffff (bits 17, 18) is now 0.4% smaller than v0.3 on the short files; FULL is 0xfffff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…ta family A strategy is a transform (none or the x86 filter) and an ncm profile (FULL, MID = FULL without match-state selection, V03). A frame's family (tiny, small, x86 code, text, other) orders every strategy by a bounded 1..16 weight; the encoder codes the first two of that schedule and keeps the smallest archive. The weights are learned by the rule30 rule (+2 winner, -1 others) over measured outcomes of 328 training files, committed with the test that re-learns them. On those files the budget-2 schedule is within 0.01% of trying all six strategies and 3.7% smaller than the v0.3 rule. Client.lex.tex states the same rule (portfolio/schedule/family) over Archive.profileCandidate; archive.rs carries interim profile forms until the profile templates land. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…ep their meaning
The standard basis gains ncmRunProfile (the coder of client/gen/ncm_model.py
under a profile word) after every v0.3 block, which stay byte for byte, as
the v0.2 ones do. A profiled entry carries its profile word as a natural
constant after the payload. v0.3 archives name the v0.3 identity and decode
with their frozen suffixes; the native client, Archive.lex.tex and
NcmDecode/NcmEncode (runProfile) read every era.
Archive.profileCandidate and archive::{linked,window}_archive_profile are
what the strategy portfolio calls. inspect reports ncm_profile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
…h profileCandidate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…to claude/project-thread-8f7s6w
A lone frame now compresses in about the time of one strategy (3 MB .rlib: 11.4 s -> 5.8 s on 4 cores; v0.3 5.4 s). The choice does not depend on the order, so the archives are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
… for LexLean Profile 0x1ffff is the v0.3 coder byte for byte, so profileCandidate (and the native linked/window profile archives) write the old-shape v0.3 entry over the v0.3 identity: byte-identical to v0.3 and without a profile word. Client.weight and its callers bind fam, since a binder spelled family would capture the declaration (LLT4001). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
A v0.3 stream's reference frame names its frame's κ-label over the v0.3 library. The decoder's store now offers each decoded frame under both labels (archive::frame_labels), so v0.3 streams with reference frames keep decoding under the appended library. Checked end to end: a 16 MiB A+A stream compressed by v0.3.0 decodes byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
The first scheduled strategy codes with learning on: every coded bit charges each kept context the cost (an integer log LUT, 256 log2(4096/p)) of the bit with that context's term left out of the three mixers, less the cost with it. The contexts that saved nothing are dropped, the learned profile is coded too, and that frame is kept when it is no larger: fewer contexts, faster decode, never a larger archive. Model: ncm_model encodeLearn (GN gains, COST table), NcmEncode.learnProfile, Archive.learnedCandidate, Client.portfolioLearned; native ncm::encode_learn, byte-identical to the model on the checked inputs. The decoder and the library are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
examples/portfolio_outcomes.rs regenerates src/portfolio-outcomes.tsv (one line per training file, sorted by name), from which the weight table is learned and checked. portfolio_streams_do_not_depend_on_threads codes frames of all five families with 1-4 threads and requires identical streams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
…to claude/project-thread-8f7s6w
ncm::encode_below abandons a payload as soon as it reaches a caller's bound; a payload it finishes is the unbounded one. The portfolio bounds each strategy by the raw archive and, when strategies run one after another (--threads 1, a full batch, wasm), the second by the first's archive too; the two run side by side only when the batch leaves a core free, so --threads 1 uses one core. Archives are unchanged: all 332 training files give identical bytes. Single-thread CPU on 8 test files 80.9 -> 78.9 s, 2 MB random 15.6 -> 13.1 s; incompressible input cannot stop early without a guess. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
… small schedules family() now tests x86 density before size, so a small executable reaches the x86 filter. The weight table is retrained on archive lengths (capped at raw) instead of payload lengths. The V03 strategy writes the v0.3-shaped entry, 7-8 B shorter than a profiled one, and with archive lengths that gain is no longer taken as a loss: tiny and small frames now schedule V03 first. Client.lex.tex carries the same rule and table. The re-export changes only uorc_compress; the library identity 9272ab03 and the generated decoder are unchanged. Training set (332 files): 11,197,900 B, against 11,198,281 on v0.4 main and 11,484,742 on v0.3.0. One file is larger than under v0.3.0: 079_Pinyin.pm, +11 B (text family; V03 is not in that family's top two). The UC-BAS-01 test now checks both eras: a small text binds the v0.3 basis, a larger one the current basis. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…nt landed in #133) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
afflom
pushed a commit
that referenced
this pull request
Nov 11, 2026
#136's choice rule (x86 before size) moves into portfolio::family; its retrained weights are reseeded in model/engine.toml and weights.rs is regenerated from them. The outcomes table and its example stay removed: the engine decides the weights over a κ-named corpus. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3LeJgXkUvvAfeAGt9A
afflom
pushed a commit
that referenced
this pull request
Nov 11, 2026
Takes lane 3's resolution of #136 against the portfolio table (its weights reseeded in weights.rs) and main's factory sets with the frontier entries; re-decides the offline corpus at main's export identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QshCSRwiddS64zU3j
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two choice-rule regressions found after #133, before v0.4.0 is tagged.
Changes
family()(nativeportfolio.rsandClient.lex.tex) now testsx86Densebefore the size classes, so a small executable reaches the x86 filter.WEIGHTS/weightTableare now learned from archive lengths, capped at raw, instead of payload lengths. The V03 strategy writes the v0.3-shaped entry, which is 7-8 B shorter than a profiled one. With archive lengths, that gain is no longer counted as a loss, so tiny and small frames now schedule V03 first. Schedules (top two): tiny [V03, FULL], small [V03, MID], x86 [MID+x86, FULL], text [FULL, MID], other [FULL, MID].portfolio-outcomes.tsvis regenerated byexamples/portfolio_outcomes.rs, which now records archive lengths.uorc_compress. The library identity9272ab03is unchanged. The generated decoder code is byte-identical; only the hash of its source bundle moved.factory.identityis re-recorded.Measured (release build, 4 vCPU)
Compress time:
--threads 1: 38.2 s on main, 35.4 s with this PR.--threads 4, alternated runs: 65.9 / 64.7 s on main, 69.2 / 67.8 s with this PR.Trade on record
One training file is larger than under v0.3.0. It is
079_Pinyin.pm, at 38,561 → 38,572 B (+11). It is in the text family, where V03 wins by a few bytes but is not in that family's top two strategies.Compress time is about 4% higher on large multi-frame input. In the "other" family, FULL and MID tie at weight 12, and portfolio order runs FULL first, with the learned-mask encode.
A tie variant was measured natively and not adopted. In that variant, equal weights go to the cheaper strategy: no filter before x86, then V03, MID, FULL. Its results:
This PR keeps the ratio-first choice.
Checks
cargo test --release -p uorcpasses, includingnative_archives_are_the_model_archives, the portfolio tests andportfolio_streams_do_not_depend_on_threads.cargo clippy -p uorc --all-targets -D warningsis clean.🤖 Generated with Claude Code
https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH