Skip to content

uorc v0.4: choice rule tests x86 before size, keeps the v0.3 entry in small schedules - #136

Merged
afflom merged 25 commits into
mainfrom
claude/project-thread-2twygm
Nov 11, 2026
Merged

afflom merged 25 commits into
mainfrom
claude/project-thread-2twygm

Conversation

@afflom

@afflom afflom commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Fixes the two choice-rule regressions found after #133, before v0.4.0 is tagged.

Changes

  • x86 before size. family() (native portfolio.rs and Client.lex.tex) now tests x86Dense before the size classes, so a small executable reaches the x86 filter.
  • Retrained weights. WEIGHTS / weightTable are now learned from archive lengths, capped at raw, instead of payload lengths. The V03 strategy writes the v0.3-shaped entry, which is 7-8 B shorter than a profiled one. With archive lengths, that gain is no longer counted as a loss, so tiny and small frames now schedule V03 first. Schedules (top two): tiny [V03, FULL], small [V03, MID], x86 [MID+x86, FULL], text [FULL, MID], other [FULL, MID].
  • Training data. portfolio-outcomes.tsv is regenerated by examples/portfolio_outcomes.rs, which now records archive lengths.
  • Re-export. The export changes only uorc_compress. The library identity 9272ab03 is unchanged. The generated decoder code is byte-identical; only the hash of its source bundle moved. factory.identity is re-recorded.
  • UC-BAS-01 test. The test now covers both eras: a small text binds the v0.3 basis, and a larger one binds the current basis.
  • Merged main 7452a14 (Pages client: Prism product model of the page, generated from it, with browser acceptance #132). That change touches decode paths and the web/model crates only.

Measured (release build, 4 vCPU)

set v0.3.0 v0.4 main this PR
332-file training set 11,484,742 11,198,281 11,197,900
8 reference files 2,725,985 2,513,499 2,511,464
40 MiB mix 11,472,111 11,471,978

Compress time:

  • 8 MiB at --threads 1: 38.2 s on main, 35.4 s with this PR.
  • 40 MiB at --threads 4, alternated runs: 65.9 / 64.7 s on main, 69.2 / 67.8 s with this PR.

Trade on record

One training file is larger than under v0.3.0. It is 079_Pinyin.pm, at 38,561 → 38,572 B (+11). It is in the text family, where V03 wins by a few bytes but is not in that family's top two strategies.

Compress time is about 4% higher on large multi-frame input. In the "other" family, FULL and MID tie at weight 12, and portfolio order runs FULL first, with the learned-mask encode.

A tie variant was measured natively and not adopted. In that variant, equal weights go to the cheaper strategy: no filter before x86, then V03, MID, FULL. Its results:

  • 40 MiB time equal to main: 64.8 / 60.8 s, against 61.8 s.
  • Zero files larger than v0.3.0.
  • The training set totals 11,199,661 B. That is 1,380 B (0.012%) above v0.4 main, and the "other" family accounts for +1,816 B of it.

This PR keeps the ratio-first choice.

Checks

  • cargo test --release -p uorc passes, including native_archives_are_the_model_archives, the portfolio tests and portfolio_streams_do_not_depend_on_threads.
  • cargo clippy -p uorc --all-targets -D warnings is clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH

…xt buckets, squash LUT, reused next-byte hashes

A profile is a set of the 17 contexts; a left-out context is never looked
up or adapted and would feed every mixer 0, so the payload is exactly the
model with that input held at 0. The full profile is byte-identical to
v0.3. Mixer rows are 16 wide (one cache line) when the profile has at
most 13 contexts.

Byte-identical for every profile: a context bucket's 16 slots pack
probability and count into one 64-byte line, squash is a 4096-entry
integer table, and the context hashes prefetch_next computes for the
expected byte are reused when that byte comes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
…-gated

Profile bit 17 adds a second match model keyed on the 4 bytes before a
position (found from 4 agreeing bytes). Profile bit 18 adds a repeat model:
the byte at the distance of the last match found keeps being predicted after
a mismatch, with its run length as confidence, a hashed context of its
expected byte, run and the previous byte (context 13 under this bit), and the
match state that selects mixer 1 and the first APM. Profile 0x1ffff is the
v0.3 coder byte for byte; both bits are 6.2% smaller on eight test files
(3 MB of .rlib: 744,349 -> 674,978 B; xz -9e 685,552).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…rofiled coder

Profile word: bits 0-16 keep contexts, F_SHORT (17) and F_REP (18) the
match features; V03 = 0x1ffff is the v0.3 coder byte for byte, FULL =
0x7ffff. Native equals lane 1's native for every feature combination and
equals the IR (client/gen/ncm_model.py) for masked profiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
…s start at weight 0

Bit 19 (F_SEL) now carries the selection of mixer 1 and the first APM by the
two match models' states: it pays on long inputs (+0.6% without it on 9.5 MB)
and costs on short ones (-1.0% without it on 150 files under 32 KiB). The
short match and repeat inputs start at weight 0 so they only count once they
predict. Profile 0x7ffff (bits 17, 18) is now 0.4% smaller than v0.3 on the
short files; FULL is 0xfffff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…ta family

A strategy is a transform (none or the x86 filter) and an ncm profile
(FULL, MID = FULL without match-state selection, V03). A frame's family
(tiny, small, x86 code, text, other) orders every strategy by a bounded 1..16
weight; the encoder codes the first two of that schedule and keeps the
smallest archive. The weights are learned by the rule30 rule (+2 winner,
-1 others) over measured outcomes of 328 training files, committed with the
test that re-learns them. On those files the budget-2 schedule is within
0.01% of trying all six strategies and 3.7% smaller than the v0.3 rule.
Client.lex.tex states the same rule (portfolio/schedule/family) over
Archive.profileCandidate; archive.rs carries interim profile forms until the
profile templates land.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…ep their meaning

The standard basis gains ncmRunProfile (the coder of client/gen/ncm_model.py
under a profile word) after every v0.3 block, which stay byte for byte, as
the v0.2 ones do. A profiled entry carries its profile word as a natural
constant after the payload. v0.3 archives name the v0.3 identity and decode
with their frozen suffixes; the native client, Archive.lex.tex and
NcmDecode/NcmEncode (runProfile) read every era.

Archive.profileCandidate and archive::{linked,window}_archive_profile are
what the strategy portfolio calls. inspect reports ncm_profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
…h profileCandidate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
A lone frame now compresses in about the time of one strategy (3 MB .rlib:
11.4 s -> 5.8 s on 4 cores; v0.3 5.4 s). The choice does not depend on the
order, so the archives are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
… for LexLean

Profile 0x1ffff is the v0.3 coder byte for byte, so profileCandidate (and the
native linked/window profile archives) write the old-shape v0.3 entry over the
v0.3 identity: byte-identical to v0.3 and without a profile word. Client.weight
and its callers bind fam, since a binder spelled family would capture the
declaration (LLT4001).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
A v0.3 stream's reference frame names its frame's κ-label over the v0.3
library. The decoder's store now offers each decoded frame under both
labels (archive::frame_labels), so v0.3 streams with reference frames keep
decoding under the appended library. Checked end to end: a 16 MiB A+A stream
compressed by v0.3.0 decodes byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
The first scheduled strategy codes with learning on: every coded bit charges
each kept context the cost (an integer log LUT, 256 log2(4096/p)) of the bit
with that context's term left out of the three mixers, less the cost with it.
The contexts that saved nothing are dropped, the learned profile is coded too,
and that frame is kept when it is no larger: fewer contexts, faster decode,
never a larger archive. Model: ncm_model encodeLearn (GN gains, COST table),
NcmEncode.learnProfile, Archive.learnedCandidate, Client.portfolioLearned;
native ncm::encode_learn, byte-identical to the model on the checked inputs.
The decoder and the library are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3Mtk8bNt1Y8AGfBfPY
examples/portfolio_outcomes.rs regenerates src/portfolio-outcomes.tsv (one
line per training file, sorted by name), from which the weight table is
learned and checked. portfolio_streams_do_not_depend_on_threads codes frames
of all five families with 1-4 threads and requires identical streams.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
ncm::encode_below abandons a payload as soon as it reaches a caller's bound;
a payload it finishes is the unbounded one. The portfolio bounds each
strategy by the raw archive and, when strategies run one after another
(--threads 1, a full batch, wasm), the second by the first's archive too; the
two run side by side only when the batch leaves a core free, so --threads 1
uses one core. Archives are unchanged: all 332 training files give identical
bytes. Single-thread CPU on 8 test files 80.9 -> 78.9 s, 2 MB random
15.6 -> 13.1 s; incompressible input cannot stop early without a guess.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
… small schedules

family() now tests x86 density before size, so a small executable
reaches the x86 filter. The weight table is retrained on archive lengths
(capped at raw) instead of payload lengths. The V03 strategy writes the
v0.3-shaped entry, 7-8 B shorter than a profiled one, and with archive
lengths that gain is no longer taken as a loss: tiny and small frames
now schedule V03 first.

Client.lex.tex carries the same rule and table. The re-export changes
only uorc_compress; the library identity 9272ab03 and the generated
decoder are unchanged.

Training set (332 files): 11,197,900 B, against 11,198,281 on v0.4 main
and 11,484,742 on v0.3.0. One file is larger than under v0.3.0:
079_Pinyin.pm, +11 B (text family; V03 is not in that family's top two).

The UC-BAS-01 test now checks both eras: a small text binds the v0.3
basis, a larger one the current basis.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
…nt landed in #133)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PhocgkyQBi9JbhCuH
@afflom
afflom merged commit efd3d17 into main Nov 11, 2026
16 checks passed
afflom pushed a commit that referenced this pull request Nov 11, 2026
#136's choice rule (x86 before size) moves into portfolio::family; its
retrained weights are reseeded in model/engine.toml and weights.rs is
regenerated from them. The outcomes table and its example stay removed:
the engine decides the weights over a κ-named corpus.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3LeJgXkUvvAfeAGt9A
afflom pushed a commit that referenced this pull request Nov 11, 2026
Takes lane 3's resolution of #136 against the portfolio table (its weights
reseeded in weights.rs) and main's factory sets with the frontier entries;
re-decides the offline corpus at main's export identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QshCSRwiddS64zU3j
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants