Skip to content

Export cost follows what a change touches: parallel certificates, shared process store - #186

Open
afflom wants to merge 4 commits into
mainfrom
claude/project-thread-ubnvx2
Open

afflom wants to merge 4 commits into
mainfrom
claude/project-thread-ubnvx2

Conversation

@afflom

@afflom afflom commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Requested by Alex · project thread

Result (measured, 4 vCPU / 16 GB)

client export wall time certificates stage
before (main a1357b5, pin 560b649) 196 min 183.3 min
after, cold store (pin c06c769) 154 min (9249 s) 148 min
after, warm store, no change (pin c06c769) 5.1 min (303 s) 10 s

The warm run published the same result (e5f0d61a…) as the cold run, with 281 processes restored from the store and 3 run. Of the 5.1 min that remains, certificate generation takes 171 s (Rust, in-process), which is the next measured distance. The first warm run took 7.8 min (466 s), because it restored from a store on the network mount that had not been read yet.

In the cold run the decompress and search B certificates (8.9 GB and 7.7 GB peak) compiled and replayed one after the other, because together they exceed the declared memory. Those two set the cold time.

Where the export's time went before

stage min
certificate generation (Rust, one thread) 4.9
elaboration + replay + audit 3.6
extraction 3.3
certificates (9, one after another, ~1 GB of Lean) 183.3

Every certificate A imported all 78 modules, and every certificate was named after the project's semantic id, so any change re-proved all nine.

What changes

The pin moves to LexLean claude/export-parallel-store-2 at c06c769 (LexLean#558, on 2bf5c1e):

  • Parallel certificates. Certificates compile in the topological frontiers of their imports, at the verification width and within the memory budget (SPEC 22.11 item 6).
  • Process store (LEXLEAN_VERIFY_STORE, SPEC 22.13). The key is the tool, the Lean executable digest, the normalized argv, and the Merkle κ of the module text and everything its imports reach. Restored files are digest-checked; the published bytes are the same with and without the store.
  • Store keys bound to their context. Each key also carries the program digest, the digest of every file the toolchain loads, the Lake configuration and the .lake tree, and the normalized environment. A module that neither the stage nor the toolchain holds is a miss, and so is every module that imports it.
  • Extraction and probe are reused by κ, and certificate A is generated beside B and E. Certificate generation went from 344 s to 198 s.
  • Certificates named by their root's lowered program. Certificate A imports only the modules its terms name, so a root that a change does not reach keeps its certificates.

scripts/client-export.mjs takes every scheduling input from the host or from the last measurement: width = os.availableParallelism(), memory = MemAvailable, hints = generated/rust/export-hints.json (rewritten after every export), store = UORC_EXPORT_STORE (default .lexlean/store, pruned least recently used within free space). Lanes share /mnt/project-files/export-store. The prune walk skips entries that another export is writing or pruning at the same time.

The approach follows the PrismPM factory's identity-by-closure receipts (#678/#681/#682) and verdant-engine's input-addressed κ cache.

On LexLean's examples/production-coverage the attestation is identical at width 1, at width 2 with a cold store, and with a warm store (683 s, 387 s, 12 s; 24 s after a one-module edit, with 5 of 298 certificate processes re-run).

https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q


Generated by Claude Code

…hared process store, schedule from the host

The client export spent 183 of its 196 minutes (main a1357b5, 4 vCPU, 16 GB)
compiling the 9 preservation certificates one after another, and every
change re-proved all of them. The pin moves to LexLean
claude/export-parallel-store-2 (2bf5c1e plus two commits):

- certificates compile in the frontiers of their imports at the export
  width, within the memory budget (SPEC 22.11 item 6);
- every Lean process goes through a store keyed by the content identity of
  its module and all it imports (SPEC 22.13), digest-checked on restore;
- certificates are named by their root's lowered program, and certificate A
  imports only the modules its terms name, so a root a change does not
  reach keeps its certificates and is not proved again.

client-export.mjs runs it at the host's width (availableParallelism) and
available memory, schedules from generated/rust/export-hints.json (the last
export's per-process wall time and peak, rewritten after each export), uses
the store UORC_EXPORT_STORE (default .lexlean/store; lanes share
/mnt/project-files/export-store), and prunes it least recently used so it
never holds more than the free space it leaves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
…tore

Cold export on this branch: 9169 s (baseline 3 h 16 min at width 1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
…olerate concurrent store writers

The client export under the final LexLean revision: cold 154 min (the two
largest certificates run one after another under the memory gate), warm
no-change 5.0 min with the same result id. The store walk skips entries
another export is writing or pruning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants