Skip to content

Pages: worker pool on the visitor's cores, progress within a frame, bounded decode memory - #140

Merged
afflom merged 25 commits into
mainfrom
claude/project-thread-1dvr1c
Nov 11, 2026
Merged

afflom merged 25 commits into
mainfrom
claude/project-thread-1dvr1c

Conversation

@afflom

@afflom afflom commented Nov 11, 2026

Copy link
Copy Markdown
Member

Requested by Alex · project thread

The v0.4 follow-up to #132. Based on main efd3d17. Bytes are unchanged: the page's archives still equal uorc::compress, and the uorc tests pass.

Worker pool on the visitor's cores

  • The page's worker plans each frame with the library (FrameEncoder::plan). A pool of workers then codes the frame's two portfolio strategies side by side, and the library's own rule (uorc::choose) keeps one.
  • The pool holds one worker per core, up to 16, and at most two per GiB of device memory. All the workers share one compiled module.
  • The page's worker codes alone until two pool workers have loaded, so the first file never waits for the pool to start.
  • The wasm test planned_frames_coded_by_strategy_are_the_library_stream checks the planned path against uorc::compress, including reference frames.

Same 10 MiB of prose, 4 vCPU, headless Chromium:

Compress Decompress
One worker (before) 168 KB/s 563 KB/s
Worker pool 369 KB/s 561 KB/s

Decompression of this input stays on one worker, as it does natively. Every frame is a window frame, and its model reads the two frames before it.

Progress within a frame

  • ncm calls a progress hook every 16 KiB it codes. That hook is the wasm import uorc.progress.
  • A frame's share of the bar keeps moving through any extra pass, such as a learned re-encode or the v0.3 entry in small schedules.
  • A short last frame, which is coded when the stream ends, now reports its progress too.
  • The acceptance found two stalls of more than 2 s before these fixes. The longest stall is now 552 ms.

Decode memory

Measured in Node on streams of distinct 8 MiB frames:

Frames Before After
3 241 MiB 226 MiB
6 257 MiB 250 MiB
12 257 MiB 250 MiB

Two changes made this:

  • ncm's model reads the window's frames in place, not from a joined copy.
  • In WebAssembly, memory never shrinks, so a model's tables (about 100 MiB for an 8 MiB frame) are kept for the next frame's model. Allocating them again for every frame left the allocator's free space in pieces, and memory kept growing.

Page model (src/Uorc/Page.lex.tex)

  • Requirements useTheCores and progressWithinFrame, with gated measures coresUnused (0) and progressGap (at most 2 s).
  • browserThroughput states the before and after numbers above. boundedMemory states the decode bound.
  • The throughput measures now run on 10 MiB.

Acceptance (local, headless Chromium)

All 18 cases pass, along with 11 transitions and 10 measures:

Measure Result Limit
Cores unused 0 0
Progress gap 552 ms 2000 ms
Longest page task 0 ms 200 ms
Cancel 155 ms 3000 ms
Ready 340 ms 5000 ms
Differing archives 0 0
Worker memory 227 MiB 256 MiB

The branch history also contains #132's commits through a merge that keeps this tree, because a force-push was not available. The diff against main is only this follow-up.


Generated by Claude Code

…sure v2

Uorc.Neuromorphic (src/Uorc/Neuromorphic.lex.tex) states the plasticity of
every learning part of the ncm coder as the Prism neuromorphic standard, in
the codec's own integer (offset) form, with kernel-checked bounds:

- hebbianBoundsHold: every mixer weight stays within +-2^22 under the
  bounded Hebbian rule; potentiation never lowers and depression never
  raises a weight.
- plasticityBoundsHold: the plasticity 24 + 104*4096/(4096+pos) lies in
  [24, 128] per 65536 (closed vectors 128, 76, 25, 24 by decide).
- laneFitHolds: |x * err * plasticity| <= 1,073,217,536 < 2^31, so the
  SIMD kernels' 32-bit lanes cannot overflow, and one bit changes a weight
  by at most 16,376.
- slot plasticity: rates 131072/(2n+3) lie in [510, 43690], a slot never
  steps past its gap to the coded bit, and consolidates at count 127.
- refinementBoundsHold: probability-map points stay within 16 bits.

The products are proved through `scale`, the product by its recursion
(scaleIsMultiply bridges it to `*`), with each bound carried in one
induction, since omega has no nonlinear arithmetic.

The native coder reads the model's constants from ncm::standard (byte
identical output); tests/neuromorphic.rs holds them equal to the model's and
the theorem inventory complete. Registered as UC-NEU-01 (kernel_theorem).

The Pages site becomes a Prism browser closure v2 (PrismPM SPEC 1.2.1):
index.html, app.css, app.js, uorc.js, uorc_bg.wasm, provenance.json and the
ten Hologram brand files (fonts, licences, icons, manifest.webmanifest),
flat and same-origin, installable, with no remote load. The brand files are
vendored byte-identical to PrismPM's (vendor/hologram-brand).
scripts/site.mjs assembles and checks the closure (provenance.json names
every file's SHA-256); release.yml uses it and `just test` runs its tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3MmVJi8mhN3UZ6rgwH
…-5h3yzm

# Conflicts:
#	.github/workflows/release.yml
#	scripts/ci-receipt.sh
…h browser acceptance

src/Uorc/Page.lex.tex states the page in Prism's product-modeling vocabulary
(Foundation.Arch stakeholders, features, journeys and UML state machine;
Foundation.Qual requirements and measures; the text/3 workbench's operations,
fields, examples, error hints, sections, usage, links and labels), registered
as UC-WEB-01. repo-model checks it as PrismPM checks its own (PP2014-PP2017,
one initial state, closed triggers, reachability, closed sets) and renders
web/index.html from it; app.js interprets the embedded model.

The codec runs in a module worker (uorc.js), with progress, Cancel, typed
error hints and recovery from a stopped worker. In-memory decoding no longer
spawns threads, which trapped WebAssembly on Verify, every example and every
folder archive. The wasm client builds with SIMD128 (1.34x, same bytes).

scripts/page-acceptance.mjs drives the built site in headless Chrome over the
DevTools protocol: every journey, all 17 acceptance cases, all 11 transitions,
controls checked against the model in every state, gated quality measures.
CI runs it in pages-client.yml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
The page acceptance runs as obligation `pages` (factory/graph.json): its
identity covers the client set, the wasm crate, web/, the brand files and the
two site scripts, so other changes find its receipt and skip it. The release
gate needs it, and factory.yml records its receipts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
The sequential one-core decode path stays ahead of the threaded scheduler,
which now counts cores per block. The acceptance's repeat input is a
repeated passage: under #130 three all-zero frames are coded as ncm frames,
not reference frames, so they no longer exercise reference decoding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
compressTreeBlob and decompressTreeBlob run a file tree a frame at a time
(FrameEncoder::objects over the tree layout; decoded frames split at their
object lengths), so a folder of any size never sits whole in wasm memory.
The wasm test checks the streamed tree equals the library's compress_tree
bytes and restores the same entries. The model's features name the new
calls, and the acceptance checks that every uorc.* call a feature names
exists in the shipped uorc.js.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
The formal and scripts sets exclude the page acceptance and its workflow;
the pages obligation and its gate edge sit beside #131's per-leg inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
…e model

#131's legs run on a checkout materialized to their inputs, and the formal
set excluded web/, so reproducibility found the check-model projection
web/index.html missing (URP004). The page is rendered from
src/Uorc/Page.lex.tex, so it belongs with the formal inputs; the rest of
web/ stays out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
…hives

New acceptance case decompress-released (feature decompressArchive): every
archive in crates/uorc/tests/compat is restored in the browser to its
original bytes. Measured against ncm v4 (#133): 18 cases pass; compress
178 KB/s, decompress 467 KB/s, worker memory 215 MiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
The rest legs fail with ENOSPC while PrismPM's template check builds the
regeneration copy in the SDK container's 2 GiB /tmp; locally the copy and
its build peak at 0.72 GB both at the last green revision and now. Print
what holds /tmp at that point so the cause is measured, not guessed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
Measured in the rest leg: when PrismPM's native regeneration starts,
/tmp (2 GiB tmpfs in the SDK container) already holds 1.2 GB, all of it
/tmp/prismpm-cargo-1001, the debug xtask build audit-bootstrap leaves
there. The regeneration copies the tree (0.13 GB) and builds it from
scratch (0.72 GB at peak, measured in a 2 GiB tmpfs), so the lexlean
rlib write hits ENOSPC. Running the template check first gives it an
empty /tmp; both checks still run and must pass. Drops the /tmp report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
…the template check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
…target

Measured in the rest leg: the SDK container's /tmp is a 2 GiB tmpfs and
CARGO_HOME (/tmp/prismpm-cargo-1001) already holds 1.1 GiB of it when the
template check starts. Native regeneration copies the tree there (0.13 GB)
and built it from scratch beside the copy (0.69 GiB at peak), which ran
out of space writing the lexlean rlib. With CARGO_TARGET_DIR at the
workspace's target on disk the build reuses audit-bootstrap's compiled
dependencies, rebuilds the workspace crates from the staged sources, and
/tmp holds only the copy (measured in a 2 GiB tmpfs: 0.13 GB, 4 s instead
of 43 s). Restores the original order and drops the diagnostics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
…de the staged copy

PrismPM runs model-write in a copy of the tree staged in the SDK's /tmp
(a 2 GiB tmpfs of which CARGO_HOME holds 1.1 GiB) with a cleared
environment that keeps only PATH, the toolchain homes, HOME and
XDG_CACHE_HOME, so CARGO_TARGET_DIR (8032234) never reached it. The
template check now leaves .tmp/model-write-target, which is copied with
the tree, and model-write builds into that directory (the workspace's
target on disk) when the note is there and into target otherwise.
Simulated with the same cleared environment in a 2 GiB tmpfs: /tmp holds
only the 0.13 GB copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
…of its own

On e6399ae the template check passed, but the next recipe (validate) ran
the xtask binary the staged copy had built into the workspace's target:
cargo took it as current, and it read sources at the copy's deleted
/tmp path. The note now names target/regeneration, still on the
workspace's disk and shared with nothing else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
The portfolio's strategy coding is lifted out of compress_pieces as
compress_strategy and choose (same bounds, same tie rule, same raw
fallback), and FrameEncoder gains plan/plan_end, which cut frames and
finish repeats without coding the rest (Planned::code codes one). The
wasm crate exports plan_frame, plan_end, code_frame (one strategy or the
whole frame) and choose_frame, so the page can code frames and strategies
on several workers and write the library's bytes; a test checks that
stream equals uorc::compress. ncm calls a thread-local progress hook once
per 64 KiB of each coding or decoding pass; the wasm crate forwards it to
the page's uorc.progress import. All 64 uorc tests pass unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
The page's worker plans each frame with the library and a pool of workers
(one per core, at most 16, two per GiB of device memory, one shared module)
codes the frame's two portfolio strategies side by side; the library's rule
keeps one, so archives equal uorc::compress. ncm reports progress every
64 KiB, so the bar moves within an 8 MiB frame.

The page model adds requirements useTheCores and progressWithinFrame and
gated measures coresUnused (0) and progressGap (at most 2 s), and states the
throughput before and after on the same 10 MiB (4 vCPU, headless Chromium):
compress 168 -> 369 KB/s, decompress 563 -> 561 KB/s (every frame of that
input is a window frame, so decoding stays sequential, as natively).
Acceptance: 18 cases pass, 10 measures within their gates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
…the output

The archive layer joined the window's frames into one buffer and the
model copied that buffer again into its history. The model now takes the
history as pieces and copies the frames once, and the decoded output gives
back the history's share of its buffer (the next frame's window holds the
output). Same bytes: the uorc tests pass.

Measured in Node (WebAssembly, SIMD128), streams of distinct 8 MiB frames:
decode memory 241/257/257 MiB before for 3/6/12 frames, 237/237 MiB after
for 3/6 frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
…oving

ncm: in WebAssembly, whose memory never shrinks, a model's tables (about
100 MiB for an 8 MiB frame) are kept for the next model of the same size
instead of being allocated again; that had left the allocator's free space
in pieces and memory grew with the frame count. Measured in Node on
distinct 8 MiB frames: decode 226 / 250 / 250 MiB for 3 / 6 / 12 frames
(241 / 257 / 257 before). Same bytes.

Progress (gate progressGap, 2 s): the acceptance found two stalls over 2 s.
The page's worker now codes alone until two pool workers have loaded, so
the first file does not wait on their start; a short last frame, coded
when the stream ends, now reports its passes; ncm reports every 16 KiB;
and a frame's share keeps moving through passes beyond the usual three.
Longest stall now 552 ms; 18 cases pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3N6Mcveqrwnu6XBiYk
@afflom
afflom merged commit fc151ec into main Nov 11, 2026
17 checks passed
afflom pushed a commit that referenced this pull request Nov 11, 2026
…lient.md, the page's evidence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QdiL6gUZQx4FxLkEH
afflom pushed a commit that referenced this pull request Nov 11, 2026
#140 moved the encoder to planned frames and coded strategies, so the
portfolio-as-table commit this branch carried for lane 3 leaves this PR
(lane 3 brings it with the engine). frontier codes a codec point through
uorc::compress_with_strategies over FrameEncoder's planned frames; the
shipped plan writes the shipped stream (tested), and the offline corpus
re-decides to the same claim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QshCSRwiddS64zU3j
afflom pushed a commit that referenced this pull request Nov 11, 2026
…ine file

Merge main into the product model lane. model.json now reads frontier.toml's
nested [[claim.members]] tables and model/engine.toml as uorc/engine/1
(rule30_state, [[strategy]] words named by [[weight]] index, [[generator]]
weights). The theorems table is labelled for the performance rows it lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3PSU8BqT6iazmFzCy1
afflom pushed a commit that referenced this pull request Nov 11, 2026
compress_pieces and compress_frames take the portfolio table again on
main's per-strategy coding; compress_strategy reads the shipped table.
portfolio::STRATEGIES (decoded from weights.rs at compile time) and
portfolio::plan stay for callers of the shipped portfolio.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3LeJgXkUvvAfeAGt9A
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants