Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support by erneestoc · Pull Request #2497 · TraceMachina/nativelink

erneestoc · 2026-07-02T20:55:40Z

Implements the server side of the REAPI blob split/splice extension (remote-apis#282) used by Bazel's --experimental_remote_cache_chunking (Bazel 8.7.0+ / 9.1.0+).

Fixes #2496.

What this adds

Vendored protocol: SplitBlob/SpliceBlob RPCs, ChunkingFunction, FastCdc2020Params, RepMaxCdcParams, and CacheCapabilities fields 8–12, verbatim from upstream remote-apis.
SpliceBlob re-assembles chunked uploads: validates chunk sizes/counts, checks all chunks exist (touching them), streams them through an incremental hasher into the CAS with reads pipelined 10-at-a-time — the digest is verified before EOF so a mismatch aborts the upload uncommitted — then persists the chunk layout in a configurable index store. The blob is fully materialized so ByteStream/FindMissingBlobs stay correct for non-chunking clients.
SplitBlob serves stored layouts (validated to sum to the blob size, so corrupt or truncated index entries are never served), and otherwise chunks blobs on demand with FastCDC 2020 — this is the path that matters for RBE, since worker-uploaded action outputs are never spliced by a client. Unusable layouts (evicted chunks, decode failures, transient existence-check errors) fall back to re-chunking.
Native proxy forwarding: grpc-store-backed instances forward both RPCs verbatim to the backend (instance-name rewriting + retries), making NativeLink relays transparent for chunking.
Capabilities advertise split_blob_support/splice_blob_support and FastCDC 2020 params per instance (collected across all server blocks, so split-listener deployments work), gated behind the new opt-in experimental_chunking CAS service config.
Observability: ChunkingMetrics — splice/split request totals, split hit/miss/on-demand counters, byte totals, and digest verification failures (tracked precisely via an explicit flag, not error-code inference).

Conformance

The chunker is the fastcdc crate at 3.2.1 — the exact version the REAPI spec names as a compliant implementation — used with the spec-mandated normalization level 2 (the crate default level 1 does not match). fastcdc_conformance_test.rs validates chunk offsets, lengths, SHA-256s, and gear-hash fingerprints against the official fastcdc2020_test_vectors.txt from remote-apis, for both seed 0 and seed 666, in both in-memory and streaming form. This is the guarantee that server-produced chunks dedupe byte-for-byte against Bazel-produced chunks. The canonical fixture image is the one already vendored at nativelink-util/tests/data/SekienAkashita.jpg for the existing FastCDC tests (its SHA-256 is asserted in the test).

Note: the vendored nativelink-util/src/fastcdc.rs used by DedupStore is not FastCDC-2020-compliant and is deliberately untouched — deployed dedup indexes depend on its exact boundaries; the two implementations coexist.

Safety / hardening

Digests are never trusted: splice verifies the re-assembled digest before commit; per-chunk content lengths are validated against their digests.
index_store == cas_store is rejected at startup (layouts stored under blob digests would overwrite blob content).
Setting index_store on a grpc-store instance is rejected (the backend owns layouts there).
Per-chunk size cap (16 MiB) and a configurable per-instance max_chunk_count (default 50k) bound memory, layout size, and response size; layout reads are capped consistently with the configured count and truncation is detected by the size-consistency check.
Off by default; with experimental_chunking unset, behavior is byte-for-byte identical to before (new RPCs return Unimplemented, capabilities advertise false).

Design decisions worth reviewer attention

Materialize-on-splice: the spliced blob is written whole into the CAS rather than serving reads from the layout. This keeps every existing read path correct and lets storage-level dedup compose naturally when cas_store is a DedupStore; the cost is chunks + blob both stored. A ChunkingStore that serves reads from layouts is a possible future direction.
Chunk/index lifetime coupling is best-effort: existence checks touch chunks, but stores answering from existence caches may not promote backing entries, and layouts can outlive chunks. Degradation is graceful (fallback to re-chunking or full download), never incorrect. A CompletenessCheckingStore-style wrapper is the analogous future fix.
ChunkingMetrics follows the ByteStreamMetrics pattern (not yet wired into a metrics root — same as existing service metrics).

Testing

23 service tests: splice/split round trip, digest & size mismatch rejection, missing-chunk NotFound, on-demand chunking (single- and multi-chunk with reassembly verification), layout reuse, unusable-layout fallback, absent-blob NotFound, disabled-instance Unimplemented, config rejection cases, and metric assertions throughout.
2 conformance tests against upstream vectors.
GrpcStore forwarding tested against a fake CAS backend over a real gRPC round trip (passthrough + instance-name rewriting).
bazel test green across nativelink-service, nativelink-store, nativelink-config, nativelink-util (63 tests) with clippy + rustfmt aspects; cargo fmt --check clean; bazel build //:nativelink succeeds.

Notes

Verified end-to-end against a real Bazel client (see PR comment for full numbers): incremental uploads of a ~30 MB artifact drop 99.4% for a single-point edit (183 KB) and 83.9% for six scattered edits (4.96 MB) — in line with published production CDC figures — with fetched bytes SHA-verified identical through both the ByteStream and SplitBlob paths.
Blobs exceeding the configurable max_chunk_count (default 50k, ~25 GiB at the default 512 KiB average) are served without chunking: SplitBlob returns NOT_FOUND and clients fall back to a regular download. Raise the knob for larger blobs, keeping client gRPC message limits in mind.
The chunking flag requires Bazel 8.7+/9.1+ on the client; the repo's current Bazel pin (9.0.2) predates it, so the LRE E2E jobs will start exercising this path once the pin moves past 9.1. Recommend 9.1.1+ specifically: 9.1.0 has an upstream client bug that corrupts outputs when --experimental_remote_cache_chunking is combined with --disk_cache (fixed in 9.1.1; reproducible against any chunking server).

🤖 Generated with Claude Code

This change is

Implements the server side of the remote-apis blob split/splice extension used by Bazel's --experimental_remote_cache_chunking (Bazel 8.7.0+/9.1.0+), fixes TraceMachina#2496. - Vendor SplitBlob/SpliceBlob RPCs, ChunkingFunction, FastCdc2020Params and CacheCapabilities fields 8-12 from upstream remote-apis. - SpliceBlob re-assembles chunked uploads: verifies chunk existence and the spliced digest before committing, materializes the blob so non-chunking clients stay correct, and persists the chunk layout in a configurable index store. Chunk reads are pipelined while hashing stays in chunk order. - SplitBlob serves stored layouts (validated against the blob size so corrupt or truncated index entries are never served), or chunks blobs on demand with FastCDC 2020 (fastcdc crate, normalization level 2) so outputs uploaded whole by remote execution workers also get chunked downloads. Unusable layouts fall back to re-chunking. - Capabilities advertise split/splice support and FastCDC 2020 parameters per instance, collected across all server blocks, gated behind the new opt-in experimental_chunking CAS service config (off by default; zero behavior change when unset). - Reject foot-gun configs at startup: index_store == cas_store (chunk layouts stored under blob digests would overwrite blob content) and chunking on grpc proxy stores (would download/re-upload entire blobs instead of forwarding RPCs). - Conformance-test the chunker against the official REAPI fastcdc2020_test_vectors.txt (offsets, lengths, sha256s and gear fingerprints, seeds 0 and 666, in-memory and streaming). - Add ChunkingMetrics (splice/split totals, hit/miss/on-demand rates, byte counters, digest verification failures). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

For grpc-store-backed CAS instances the chunking RPCs are now forwarded verbatim to the backend (with instance-name rewriting and the store's usual retry handling) instead of being rejected at startup. This makes NativeLink relays transparent for content-defined chunking: one RPC in, one RPC out, the backend owns chunking and the layout index. - Add GrpcStore::split_blob/splice_blob following the existing find_missing_blobs/batch_*/get_tree forwarding pattern. - Shortcut to the proxy in the CAS handlers before any local chunking machinery is consulted, mirroring the other four CAS RPCs. - Make experimental_chunking.index_store optional: required for locally chunked instances, rejected for grpc-store instances where the backend owns the chunk layouts. The capabilities service still advertises split/splice + FastCDC params from the same config block, so relay operators set avg_chunk_size_bytes to match their backend. - Test forwarding against a fake CAS backend over a real gRPC round trip (verifies passthrough and instance-name rewriting) and the new constructor rules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

vercel · 2026-07-02T20:55:47Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview, Comment	Jul 7, 2026 6:21am
nativelink-aidm	Ready	Preview, Comment	Jul 7, 2026 6:21am

- Reuse the FastCDC test fixture already vendored at nativelink-util/tests/data/SekienAkashita.jpg (and already excluded from the forbid-binary-files hook) instead of adding a duplicate binary copy; export it from nativelink-util for the conformance test. - Format nativelink-service/Cargo.toml per taplo. - Replace the hardcoded 50k chunk cap with a per-instance experimental_chunking.max_chunk_count knob (default 50000). Blobs above the cap are served without chunking; the layout read cap is derived from the configured count so the two can never disagree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

MarcusSorealheis

One comment, really.

For new configuration options, we'd like to clarify that this is optional and include a configuration example. See ./nativelink-config and lots of examples.

Otherwise, this one is definitely on the correct path.

Address review feedback: state explicitly that experimental_chunking is optional (with unchanged behavior when unset) and add a complete, test-validated configuration example at nativelink-config/examples/chunking_cas.json5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

erneestoc · 2026-07-06T21:11:09Z

Thanks for the review! Addressed in ed98063:

Added a complete configuration example at nativelink-config/examples/chunking_cas.json5 (validated by the existing test_example_parsing config test), with each field annotated as optional/required and its default.
The experimental_chunking doc comment now states explicitly that the option is optional and that behavior is byte-for-byte unchanged when it is unset, and points to the example.

Resolves lockfile conflicts: Cargo.lock and MODULE.bazel.lock taken from main and regenerated by cargo/bazel to re-include the fastcdc dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

erneestoc · 2026-07-06T21:49:16Z

End-to-end verification against a real Bazel client (NativeLink from this branch + --experimental_remote_cache_chunking, wire bytes measured at the TCP socket with a byte-counting proxy). Each row: incremental re-upload after a small change, chunking off vs on.

Artifact (change)	Upload, chunking off	Upload, chunking on	savings
30 MB text, 1 edit at front (CDC best case)	30.7 MB	183 KB	99.4%
30 MB text, 6 scattered edits	30.7 MB	4.96 MB	83.9%
16.8 MB tar layer, 1 file changed	16.8 MB	2.1 MB	87.5%
3.4 MB clang-linked binary, 1 source file changed	3.4 MB	653 KB	80.8%
7.2 MB gzip of the tar (change near end of stream)	7.2 MB	3.3 MB	54.2%

Representative artifacts (tar layers, linked binaries, scattered edits) cluster at ~80–88%, matching BuildBuddy's published production figure of ~85% deduped bytes. Compressed artifacts are position-dependent: everything after the first changed byte re-uploads (a front-of-stream change would approach 0% savings) — inherent to CDC, not this implementation. Download side mirrors upload: a fresh client re-fetching the edited text artifact downloaded 178 KB instead of 30.7 MB.

Correctness verified end-to-end: fetched artifacts are byte-identical to expected (SHA-256) via both the plain ByteStream path (validates splice materialization) and the chunked SplitBlob path. Server-side stores show the expected mechanics (shared chunks + layout entries, no errors). Capability gating verified: with the server not advertising chunking, a flag-enabled client falls back to full uploads.

One upstream finding: Bazel 9.1.0's --experimental_remote_cache_chunking combined with --disk_cache reassembles corrupt outputs (reproducible against any server; the RPCs the server serves are identical and correct). Bazel 9.1.1 fixes it — worth recommending 9.1.1+/8.7+ in docs.

…lice

Now that the repo's Bazel is 9.1.1 (which supports --experimental_remote_cache_chunking), exercise the SplitBlob/SpliceBlob paths end-to-end in the existing integration-tests job: enable experimental_chunking in the docker-compose CAS config and add chunking_cache_test.sh, which uploads a ~6.9MB artifact as chunks, asserts the server registered a chunk layout, then re-fetches it through the chunked download path and verifies it is byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

MarcusSorealheis

We need the web/ documentation as well, as that's where most customers live as opposed to the repo.

Bazel 9.1.1 leaves digest_function unset in SplitBlob/SpliceBlob even when running with --digest_function=blake3 (surfaced by the new CI chunking integration test, which runs with the repo's blake3 default). REAPI length-based inference cannot disambiguate SHA256 from BLAKE3 (both 32 bytes), so: - SpliceBlob hashes the re-assembled blob with both candidates when the field is unset and accepts whichever reproduces the expected digest. - SplitBlob's on-demand chunking infers the blob's digest function with an extra content pass so chunk digests use the right function. Explicitly-set digest functions keep the single-hasher fast path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…hunking (FastCDC-2020) Completes the runtime half of upstream TraceMachina#2497 (commit df9b2cf) onto the v1.6.1 merge. The base merge already vendored the config (experimental_chunking / CasChunkingConfig), regenerated proto (SplitBlob/SpliceBlob/FastCdc2020Params), the fastcdc v3.2.1 + tokio-util deps, digest_hasher::digest_hasher_func_from_context, verify_store's use of it, the fastcdc conformance test + REAPI vectors, the Bazel BUILD entries, the chunking_cas.json5 example, and the integration scaffolding — but left the cas_server SplitBlob/SpliceBlob handlers as Status::unimplemented stubs, hardcoded capabilities split/splice support to false, and dropped the handler + grpc-forward tests. This commit implements the real handlers. Invariant being violated: a CAS instance that opts into experimental_chunking must serve REAPI SplitBlob/SpliceBlob with byte-exact FastCDC-2020 chunking and advertise that support; the base merge left it unimplemented/hardcoded-off. Mechanism that violates it: cas_server.rs split_blob/splice_blob returned Status::unimplemented, and capabilities_server.rs hardcoded split_blob_support/splice_blob_support = false regardless of config. Mechanism that re-establishes it after the fix: cas_server.rs real inner_split_blob/inner_splice_blob/chunk_blob_on_demand wired to the instance's cas_store chain + configured index_store (built in CasServer::new with the same-store and grpc-store foot-gun rejections); grpc_store.rs split_blob/splice_blob forward the RPCs verbatim for grpc-backed instances; capabilities_server.rs advertises split/splice + FastCDC params gated on chunking_params.is_some() (populated iff experimental_chunking is set). Composite invariants this fix interacts with: - FL-688 ack-gate / WorkerProxyStore CAS chain: chunking is disabled in the production config (experimental_chunking: None), so chunking_instances is empty and every chunking store op is short-circuited by chunking_instance() returning Unimplemented — zero interaction with the ack-gate / mirror path in production. - No per-RPC timeouts (operator directive): the grpc split/splice forwarders follow the find_missing_blobs pattern with NO per-RPC deadline; the local handlers add none. Liveness stays on transport keepalive. - Bounded network buffers: per-chunk size capped at MAX_SPLICE_CHUNK_SIZE (16 MiB), in-flight chunk reads/writes bounded at CHUNK_CONCURRENCY (10), chunk_digests / SplitBlobResponse bounded by the per-instance max_chunk_count, layout reads capped at max_layout_size(); no fsync. Test that proves the fix re-establishes the invariant: fastcdc2020_matches_reapi_test_vectors (byte-exact vs REAPI vectors), splice_and_split_round_trip, and chunking_enabled_instance_advertises_split_splice_and_fastcdc_params. FastCDC reconciliation: TraceMachina#2497 uses the EXTERNAL fastcdc crate v3.2.1 v2020 module (AsyncStreamCDC / FastCDC, Normalization::Level2), which is distinct from our homegrown nativelink-util/src/fastcdc.rs (the dedup_store chunker). They coexist; our fastcdc.rs is untouched. Behavior changes (enumerated against production composition): - Production CAS instances (experimental_chunking: None): split_blob/ splice_blob still return Code::Unimplemented (via chunking_instance()), and capabilities still advertise split/splice = false. Only the Unimplemented error MESSAGE text changes ("Blob chunking is not enabled for instance '<name>'" vs the old stub string); the Code is unchanged and conformant clients never call the RPCs (support advertised false). No chunking store op is reachable in production. - Opt-in instances (experimental_chunking set, non-grpc cas_store): split_blob/splice_blob now serve real chunk layouts / re-assemble blobs; capabilities advertise support + FastCDC params. New behavior, off by default. - grpc-store-backed CAS instances with experimental_chunking (no index_store): split_blob/splice_blob now forward to the backend instead of Unimplemented. Not a production topology (prod cas_store is FilesystemStore via FastSlow). Verified: fastcdc conformance 2/2 byte-exact; cas_server_test 28/28 (12 new chunking); capabilities chunking 2/2; grpc_store_test 17/17 (1 new forward); workspace build --features quic green. 5 mutation-verify cycles (capabilities gate, splice digest verify, max_chunk_count, FastCDC normalization level, grpc instance-name rewrite) each fail with the test's bespoke assertion and restore green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

These test targets never compiled under `chunked_fast_slow[,test-utils]`; the breaks pre-date and are unrelated to the v1-chunked-path removal (two review cadres confirmed byte-identical base<->HEAD for the removal). Test-only; no production code changed. Fixes (all verified by compiling first, then re-running green): 1. Missing `bypass_dedup_threshold_bytes` field (added by TraceMachina#2497 FastCDC) in one `FastSlowSpec` literal: - fast_slow_store_334_backpressure_preservation_test.rs:170 (E0063) -> add `bypass_dedup_threshold_bytes: 0` (u64 default, matching every other literal in-tree). (The dispatch also named fast_slow_str_key_...; that file already carried the field at HEAD -- its real break was TraceMachina#2 below.) 2. `StoreLike::update` returns `Result<u64, Error>` but five test helpers still declare `Result<(), Error>` (E0308). Discard the byte count at the terminal expression (`.map(|_| ())`) -- least-invasive, preserves each helper's declared signature and every caller: - store/fast_slow_str_key_skips_chunked_dispatch_test.rs:142,417 - store/fast_slow_store_334_backpressure_preservation_test.rs:153 - service/chunked_stable_digests_push_test.rs:237 - service/bazel_facing_internal_chunking_test.rs:232 - service/chunked_p25_p27_e2e_test.rs:193 3. `chunked_commit_soft_warn_test` had no `[[test]]` block, so a whole-suite `cargo test --features chunked_fast_slow` (no test-utils) tried to compile it (its body is `#![cfg(feature = "chunked_fast_slow")]`) against test-utils-gated items instead of auto-skipping (E0432/E0599 x15). Add the entry with `required-features = ["chunked_fast_slow", "test-utils"]`, matching siblings. 4. (NOT in dispatch, found by compiling) chunked_b1_writev_test.rs:66 (E0425): the `skip_if_no_io_uring` guard -- explicitly documented to skip "when the io-uring feature is compiled out" -- called `nativelink_util::fs:: is_io_uring_available()`, which only exists under `#[cfg(all(feature = "io-uring", target_os = "linux"))]`, so the guard itself could not compile without io-uring. cfg-guard the call and default `available = false` on the complement, honoring the guard's documented intent. Invariant being violated: every declared `#[[test]]`/helper under `chunked_fast_slow[,test-utils]` must compile in the feature configs its `required-features` (or file-level cfg) permit. Mechanism that violates it: three drifted APIs -- TraceMachina#2497's new FastSlowSpec field, `StoreLike::update`'s `Result<u64>` return, and the io-uring-gated fs fn -- plus a missing Cargo `[[test]]` gate, left seven test files uncompilable. Mechanism that re-establishes it: add the field/`.map(|_| ())`/cfg-guard/`[[test]]` entry so each target compiles (and auto-skips) in exactly the configs it declares. Composite invariants this fix interacts with: none -- no runtime path, store chain, ack/pin/eviction semantics, or buffering is touched. Test that proves the fix: the targets themselves now compile and run green (fast_slow_str_key... 3 ok, fast_slow_store_334... 10 ok, chunked_p25_p27... 2 ok, chunked_stable_digests... 14 ok; bazel_facing... 20 ok + 1 pre-existing tracing-test global-buffer isolation flake, passes in isolation, unrelated). Behavior changes: none. All edits are test-only (test bodies, a test helper's return-value discard, a test's cfg guard, and a Cargo `[[test]]` declaration). No production source, config, or runtime path changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…g field The v1 WriteChunked worker-upload path was deleted; workers always use v2 (`update_via_chunked_inner` dispatches WriteChunkedV2 unconditionally). The `GrpcSpec.chunked_v2_writes_enabled` flag was parsed-but-ignored after that removal -- no runtime path read it. Remove the field, its docs, every struct literal that set it, and the stale comments that named it. DEPLOYED-CONFIG ORDERING: `GrpcSpec` is `#[serde(deny_unknown_fields)]`, so a config that STILL carries `chunked_v2_writes_enabled` is now a HARD parse error (worker fails to boot). Deployed worker configs carry `chunked_v2_writes_enabled: true` and MUST be stripped of the key BEFORE this build ships. The parent owns that config migration + deploy ordering. Invariant walk: - Invariant violated: the config schema must expose only knobs a runtime path honors -- a parsed-but-ignored field implies a v1/v2 write-path selector that no longer exists, misleading operators and reviewers. - Mechanism that violates it: `GrpcSpec.chunked_v2_writes_enabled` (nativelink-config/src/stores.rs:1847, pre-removal) was deserialized but never read after the v1 dispatcher was deleted. - Mechanism that re-establishes it: delete the field from `GrpcSpec` and every struct literal; `deny_unknown_fields` (stores.rs:1659) now rejects any lingering key, forcing operators to remove the dead knob. - Composite invariants this interacts with: deny_unknown_fields config-parse safety (deployed configs must be stripped first -- parent-owned deploy ordering); the sibling `chunked_writes_enabled` field is retained and unchanged; the v2 dispatch path is untouched (already unconditional). - Proving tests: grpc_spec_rejects_removed_chunked_v2_writes_enabled_key (deny-unknown-field reject contract) and grpc_spec_parses_without_chunked_v2_writes_enabled_key (absence-OK). Behavior changes (each vs production composition): - nativelink-config GrpcSpec deserialization: a config setting `chunked_v2_writes_enabled` now FAILS to parse (was: accepted-and-ignored). Production composition: live worker configs set it `true`, so they must be stripped of the key before this ships or the worker will not boot. This is the ONLY runtime behavior change; it is intentional and the reason for the deploy-ordering note above. - Write path: NONE. `update_via_chunked_inner` already dispatched v2 unconditionally before this change (the field was already ignored), so the data plane is unaffected. - Test/bench struct-literal removals + comment rewords: no runtime behavior. Verification: - cargo check --bin nativelink --release --features quic,pprof,chunked_fast_slow -> clean (only a pre-existing unrelated tokio_unstable cfg warning). - cargo test -p nativelink-config -- grpc_spec -> new reject/absence tests pass; mutation (re-add field) -> reject test fails with its bespoke message -> restored -> green. - git grep -w chunked_v2_writes_enabled: zero struct-literal / field-read refs; only the intentional reject-test key string + doc comments remain. Pre-existing (NOT introduced here): nativelink-worker test `backfill_is_worker_header_seam_test` fails to compile at base 1010596 -- its `impl ContentAddressableStorage` mock lacks `split_blob`/`splice_blob` added by b1dc254 (TraceMachina#2497). The diff to that file here is the single field-line removal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… (D1-D4 + example) Chunking stays OFF by default (experimental_chunking: None in prod). These four follow-ups + the example fix land BEFORE anyone flips it on. Invariant being violated: a chunking-capable build must be OBSERVABLE (poisoning-rejection visible on /metrics), must not do unbounded/wasted work on over-cap blobs, must not forward chunking RPCs an operator never enabled, and must fail fast on a store topology whose write path is unvalidated — while a chunking-OFF instance stays byte-for-byte unaffected. Mechanism that violates it: D3 cas_server.rs — ChunkingMetrics (incl. splice_verification_failures) was a per-instance MetricsComponent tree never registered → dark on /metrics. D4 cas_server.rs chunk_blob_on_demand — max_chunk_count checked AFTER try_collect + AFTER writing every chunk → orphans + O(blob_size) waste. D2 cas_server.rs grpc_store_for_instance — forwarded split/splice for ANY grpc-backed instance, even without experimental_chunking set. D1 cas_server.rs CasServer::new — no guard against enabling chunking on a WorkerProxyStore-wrapped cas_store, whose SpliceBlob server-originated store.update flows through the FL-688 ack-gate (worker-upload-only, unexercised for server streams). Mechanism that re-establishes it: D3 process-wide chunking_metrics_singleton() + register_chunking_metrics(), called once in nativelink.rs; CasServer::new wires every instance's counters to that same Arc. Rendered names cas_split_*/cas_splice_*. D4 in-stream guard: enumerate the FastCDC stream, return NotFound once chunk_index >= max_chunk_count BEFORE hashing/storing; the early Err drops rx and cancels the blob read. Post-collect check removed. D2 grpc_chunking_instances set records opted-in grpc instances; grpc_store_for_instance returns None otherwise → handlers answer Unimplemented (matching advertised split/splice = false). D1 error_if! at CasServer::new when the outermost driver downcasts to WorkerProxyStore and experimental_chunking is set (mirrors the existing same-store / grpc-index rejections). Composite invariants this fix interacts with: - FL-688 ack-gate: D1 fails startup rather than splice through it; prod cas_STORE (WPS-wrapped) can no longer silently enable chunking. - No per-RPC timeouts: unchanged; no deadlines added. - Bounded buffers: D4 tightens the bound (writes at most max_chunk_count chunks on an over-cap blob instead of the full set); no new buffers. - No fsync: unchanged. Test that proves the fix re-establishes the invariant: chunking_metrics_render_prometheus_exposes_names / _pins_values (D3), split_over_cap_blob_short_circuits_without_writing_all_chunks (D4), grpc_forward_gated_on_chunking_config (D2), chunking_on_worker_proxy_store_rejected (D1), split_and_splice_disabled_return_unimplemented (prod-off unchanged). Every contract mutation-verified (see below); all bite with bespoke messages. Example: nativelink-config/examples/chunking_cas.json5 now wraps CAS_MAIN_STORE in a VerifyStore (verify_hash/verify_size) so the normal ByteStream upload path is integrity-checked — previously splice's verify-before-EOF was the ONLY gate. Behavior changes (enumerated against production composition — prod runs experimental_chunking: None on a WorkerProxyStore-wrapped cas_STORE): - D3: /metrics gains 9 counter lines cas_split_*/cas_splice_* (process-global, registered once). In prod they read 0 (no chunking traffic). Request handling unchanged. Verified: register+render tests; split_and_splice_disabled still returns Unimplemented. - D4: over-cap on-demand split now errors after <= max_chunk_count chunks and aborts the blob read. UNREACHABLE in prod (chunking off). When enabled: fewer writes + no O(blob_size) waste; same NotFound (message still contains "max_chunk_count"). Verified: split_over_cap_blob_short_circuits. - D2: a grpc-backed CAS instance WITHOUT experimental_chunking now returns Unimplemented for split/splice instead of forwarding. Prod cas_STORE is not a grpc store and chunking is off → this path is unreachable in prod. Opted-in grpc instances still forward. Verified: grpc_forward_gated_on_chunking_config. - D1: CasServer::new now REJECTS startup if experimental_chunking is set AND the cas_store outermost driver is a WorkerProxyStore. In prod chunking is off → no rejection; the guard only fires the moment someone tries to enable chunking on the exact (unvalidated) prod topology. Verified: chunking_on_worker_proxy_store_rejected. - CasServer::new delegates to new_with_chunking_metrics(singleton); existing callers unchanged. No other runtime behavior change. Mutation verification (each restored after): - D1 guard commented -> chunking_on_worker_proxy_store_rejected RED "expected chunking on a WorkerProxyStore-wrapped cas_store to be rejected". - D2 gate commented -> grpc_forward_gated_on_chunking_config: no-chunk instance forwards to the unreachable backend and blocks (>400s) vs <1s Unimplemented green — gate is load-bearing. - D4 in-stream guard disabled + old post-collect check restored -> split_over_cap_blob_short_circuits RED "over-cap split wrote 14 chunks (full set is 14); expected the stream to short-circuit and write strictly fewer". - D3 register_chunking_metrics gutted -> _exposes_names RED "TraceMachina#2497 D3 dark on /metrics: chunking metric `cas_splice_requests_total` ABSENT". - D3 splice_verification_failures publish! commented -> _pins_values RED "TraceMachina#2497 D3: expected `cas_splice_verification_failures 5` on the /metrics render". New metric names on /metrics: cas_splice_requests_total, cas_splice_already_exists, cas_splice_verification_failures, cas_splice_bytes_total, cas_split_requests_total, cas_split_hits, cas_split_misses, cas_split_chunked_on_demand, cas_split_bytes_total. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…sm + D3 singleton E2E) Tier-2 cadre fix-up on 473e9be (both items non-blocking; feature stays OFF). Invariant-walk: - Invariant violated: (D4) an over-cap SplitBlob must return REAPI Code::NotFound for ALL over-cap blobs, not only those that fit the buf channel; (D3) the chunking metrics must be provably reachable from the production producer through the registered consumer. - Mechanism violating it: (D4) chunk_blob_on_demand's in-stream cap guard drops rx to abort the in-flight get_part; for a blob larger than the 1024-slot buf channel the read is still streaming, fails with Code::Internal, and Error::merge (nativelink-error/src/lib.rs:688-703) prefers the read code -> RPC returns Internal, not NotFound. (D3) no test drove the production new() singleton through render_prometheus, so re-pointing the producer at a per-server Arc shipped green. - Mechanism re-establishing it: (D4) an over_cap AtomicBool set by the guard; after the join we force NotFound whenever it fired, regardless of the merged code (cas_server.rs). (D3) an E2E test binding CasServer::new (singleton) -> real splice_blob -> register_chunking_metrics + render_prometheus. - Composite invariants: D1 over-cap chunks land on a non-WPS store (unchanged — still up-to-cap, eviction-reclaimable); no fsync; no lock across await (AtomicBool mirrors the existing verification_failed pattern in inner_splice_blob); prod-OFF byte-unaffected (chunking None -> guard never reached; the D4 override and D3 test are chunking-on only). - Test proving it: split_over_cap_large_streaming_blob_returns_not_found (multi-chunk streamed MemoryStore feed so the read blocks mid-stream); chunking_metrics_singleton_renders_production_producer_counter. Design note: chose the in-stream code override alone over the red-team's pre-stream lower-bound reject. The guard already bounds the read (drops rx after the cap) and writes (~max_chunk_count), so a pre-stream reject saves only marginal work; it would also be mutation-invisible (the existing 16 KiB test would silently shift onto it, losing its in-stream-guard sensitivity) unless a dedicated zero-write test were added. The override alone makes over-cap deterministically NotFound for large streaming blobs and keeps the existing 16 KiB test's meaning intact. Behavior changes (against production composition, chunking is OFF in prod): - CasServer::split_blob on a chunking-enabled instance: an over-cap on-demand split of a blob larger than the buf channel now returns Code::NotFound ("no split information available") instead of Code::Internal. Only reachable when experimental_chunking is set (no prod instance sets it). All other paths byte-identical. - No runtime change on any chunking-OFF path: handlers still return Unimplemented; only the 9 always-zero chunking metric lines render. - Tests only: two added (D4 large-streaming NotFound; D3 singleton-aliasing E2E render). No production API/signature change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ble (D1-D4 + example, Tier-2 APPROVE + fix-up) Chunking stays OFF (experimental_chunking: None) — prod byte-unaffected (only 9 zero cas_* metric lines). D3 un-dark ChunkingMetrics (singleton+register), D1 fail-fast WPS guard, D4 over-cap short-circuit + deterministic NotFound, D2 grpc-forward gate, VerifyStore example. 35 tests, mutation-verified.

…ocal_worker_test Chunk 1 (TraceMachina#2497 SplitBlob/SpliceBlob + FL-1383 targetkey) added the `targetkey: Option<TargetKey>` field to `ActionInfo` (nativelink-util/src/action_messages.rs:321) and mechanically added `targetkey: None` to every exhaustive struct literal in util/config/service, but missed the 14 construction sites in this worker test file (chunk 1's review only ran util/config/service targets, not the worker crate). Result: 14x `error[E0063]: missing field targetkey in initializer of ActionInfo`. Invariant walk: - Invariant violated: every exhaustive `ActionInfo { .. }` literal must name all struct fields (Rust exhaustiveness); the new `targetkey` field left 14 worker-test literals non-exhaustive → crate does not compile. - Mechanism that violates it: chunk 1 added `ActionInfo.targetkey` (action_messages.rs:321) without updating the 14 sites in nativelink-worker/tests/local_worker_test.rs. - Mechanism that re-establishes it: add `targetkey: None,` after each `unique_qualifier` block at the 14 sites, matching the mechanical pattern applied in every other crate (e.g. scheduler_m1v2_ranker_test.rs:121). - Composite invariants this interacts with: none — `targetkey` is `None` on the current fleet and inert by construction (see field doc, action_messages.rs); the only other `}), \n };`-closing literal in the file (RetryInfo at line ~2055) was deliberately NOT touched. - Test that proves it: `cargo test -p nativelink-worker --features test-utils --test local_worker_test` now compiles and runs (22/24 pass). Behavior changes: none. `targetkey: None` is inert (the field is None on the current fleet and consumed by no code yet); no production or test runtime behavior changes. Purely a compile-fix to a test file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…AS split/splice stubs) Two accumulated merge debts blocked the full `nativelink-worker --features test-utils` suite from compiling; these are mechanical, inert-by-construction fixes with no runtime behavior change. Invariant being violated: the worker test-utils suite must compile so its seam tests can run; two merges left exhaustive struct literals and a tonic service impl short of trait/field members added upstream. Mechanism that violates it: chunk-1 merge added `ActionInfo::targetkey` (an inert `Option`, consumed by no code) making three ActionInfo literals non-exhaustive; TraceMachina#2497 added `SplitBlob`/`SpliceBlob` RPCs to the REAPI `ContentAddressableStorage` service trait, leaving the test-only `HeaderCapturingCasServer` impl missing two required methods. Mechanism that re-establishes it: add `targetkey: None` to each ActionInfo literal (ac_write_detach_test.rs:71, kill_upload_tail_publish_seam_test.rs:93, tree_proto_cache_test.rs:181); add `split_blob`/`splice_blob` tonic stubs returning `Status::unimplemented` to the CAS service impl (backfill_is_worker_header_seam_test.rs:127), matching the file's existing `batch_read_blobs`/`get_tree` unimplemented-stub pattern. Composite invariants this fix interacts with: none — `targetkey: None` is consumed by no production code; the two new stubs are on a test-only gRPC server that never receives SplitBlob/SpliceBlob (its find_missing/batch_update seam is the only exercised path). Test that proves the fix re-establishes the invariant: the full `cargo test -p nativelink-worker --features test-utils --no-run` compiles the previously-broken test binaries (tree_proto_cache_test, kill_upload_tail_publish_seam_test, ac_write_detach_test, backfill_is_worker_header_seam_test) with 0 errors. Behavior changes: none. `targetkey: None` matches the value every other ActionInfo literal already carries and no code reads the field. The two CAS stubs return `unimplemented` on RPCs the test server never receives; the tested backfill header seam (find_missing_blobs + batch_update_blobs) is unchanged. Note (design-vs-code drift): the dispatch located the split_blob/splice_blob gap at `utils/mock_running_actions_manager.rs:30` as a Store/StoreDriver impl; the actual gap is the tonic `ContentAddressableStorage` service impl in `backfill_is_worker_header_seam_test.rs:127`. mock_running_actions_manager.rs has no Store impl and needed no change. backfill_is_worker_header_seam_test.rs has no ActionInfo literal (the dispatch expected a targetkey fix there). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…into worker call-sites The §8 eviction/sweep APIs (PortableIncrContext::sweep_stale_contender_dirs, evict_warm_dirs_over_budget, DEFAULT_WARM_DIR_BUDGET_BYTES) landed on main but were never CALLED — the parallel split left the call-sites open. This wires them, flag-gated INERT (a no-op on the entire fleet where the context is None). Invariant-walk: - Invariant violated: the on-disk portable-incr warm-dir pool at FIXED_PREFIX must stay bounded, and CONTENDER dirs orphaned by a crashed prior run must be reaped — but the APIs that enforce this were dead code (never invoked). - Mechanism that violates it: the §8 APIs existed only as uncalled pub methods (portable_incr.rs:569/592); nothing on the worker startup or post-action path called them. - Mechanism that re-establishes it: local_worker.rs new_local_worker calls portable_incr_startup_sweep(ctx) once before accepting actions; RunningAction ::cleanup calls maybe_evict_warm_dirs_post_action(DEFAULT_WARM_DIR_BUDGET_BYTES) after each portable action. Both run the blocking API under spawn_blocking and are gated so no cost is paid when the feature is off. - Composite invariants this interacts with: the §5 EXECROOT_OWNERSHIP lease (eviction never removes a leased live dir — enforced inside the API), the §9 contender cold-discard on cleanup (untouched; eviction only reaps warm OWNER dirs, discard reaps CONTENDER dirs), and the fleet-INERT gate (context is None on the whole fleet, so both call-sites early-return with no spawn_blocking). - Proving test: post_action_evict_wiring_fires_for_portable_action + post_action_evict_wiring_skips_non_portable_action + startup_sweep_wiring_reaps_contender_when_context_present (mutation-verified). Behavior changes (each vs its production composition): - new_local_worker: adds one awaited call that, on the fleet (portable_incr disabled → context None), returns immediately with no filesystem touch and no spawn_blocking. When the feature is enabled, it reaps orphaned contender dirs once at startup; a failure is logged and swallowed (never blocks startup). - RunningActionImpl::cleanup: adds one awaited call that, for every non-portable action (the entire fleet, portable_execroot None), returns at its first line with no context access. For a portable action on a portable-enabled worker it LRU-evicts warm dirs over the 20 GiB budget; a failure is logged and swallowed (never fails the action). No change to the contender-discard path, do_cleanup, the ack/store chain, or any timeout/deadline. - No fsync/sync-write primitive added. No new unbounded buffer (the eviction candidate Vec is inside the pre-existing, capped API). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… call-sites; import cosmetics Closes the before-canary pair-b follow-ups from the Tier-3 cadre review (.claude/reviews/2b825527/code-perf-security-testing.md): one integration-test coverage gap (testing-czar MINOR) + two cosmetic import nits (code MINOR-1/2). Feature remains flag-gated INERT (fleet runs incr_seed_index_store=None). Invariant being violated: every production call-site gate must have a test that RED-fails when the gate is neutered; the FL-1383 publish gate (inner_upload_results) and fetch call (inner_prepare_action) had only unit-tested building blocks (plan_seed_publish / fetch_and_materialize_seed) — the WIRING itself was unexercised, so a mutation to either gate passed CI silently. Mechanism that violates it: the publish gate at running_actions_manager.rs:6683 (`if let (Some(publish), Some(index_store))` → update_oneshot + note_index_published) and the fetch gate at :5274 (`if let (Some(index_store), Some(targetkey))` → fetch_and_materialize_seed) were reachable only through the full create_and_add_action→prepare→execute→upload composition, which no test drove. Mechanism that re-establishes it: two #[nativelink_test] integration tests drive the REAL RunningActionImpl + manager composition with portable_incr ENABLED and an incr_seed_index_store installed (mirroring new_local_worker's set_portable_incr + set_incr_seed_index_store), asserting (a) a successful portable action lands an index update_oneshot at hash(targetkey) AND bumps incr_index_publish, and (b) a pre-seeded index hit materializes the -incr seed at <execroot>/<stem>-incr. Composite invariants this fix interacts with: portable_incr §11 targetkey derive-vs-carrier verify (drives the output_paths sort ordering: primary must sort before the -incr dir, hence aaa.rlib/zzz-incr); §7 wipe seed-survival; §12 metrics process-singleton; the CAS + index store composition; plan_seed_publish as the pub bridge to the private index_action_digest key/value shape. Test that proves the fix re-establishes the invariant: portable_action_success_publishes_seed_index_and_bumps_counter (publish gate) and portable_action_with_seeded_index_fetches_and_materializes (fetch gate). Mutation verification (each RED with its bespoke message, then restored green): - publish gate first-tuple → None::<SeedPublish>: index-store assertion RED ("...no entry means the `if let (Some(publish), Some(index_store))` wire never fired: NotFound"). - note_index_published() gated off at runtime: counter assertion RED ("note_index_published() MUST increment incr_index_publish exactly once ... delta 0 ... DARK on /metrics"). - fetch gate second-tuple → None::<&TargetKey>: materialize assertion RED ("inner_prepare_action MUST call fetch_and_materialize_seed after the wipe ... the fetch call-site never fired"). Behavior changes: NONE. running_actions_manager.rs changes are a pure cosmetic reorder (moved `use crate::incr_seed_fetch` after the external-crate imports and `const INCR_SEED_FETCH_TIMEOUT` below the import block) with zero runtime effect, verified: nativelink-worker lib (216) + portable_incr_execroot_test (33) green. Everything else is test-only. No fsync/sync-write; no new network buffers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…n tests + nits); write Bazel naming-contract handoff Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…er snippets, canary runbook); clarify residency-gossip Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ted declared path (full-empty wipe) Two load-bearing pre-canary divergences from the Bazel/rules_rust chunk-4 exchange (both cold-not-wrong but would reuse 0% / DARK). Flag-gated INERT. §2 — targetkey exclusion. TargetKey::derive excluded no outputs, so it keyed on the lexicographically-smallest raw output_paths entry. A rustc pipelined action declares <label>-incr / -incr-metadata / -incr-unused-inputs.txt; since '-' (0x2D) < '.' (0x2E) the bytewise-min is <label>-incr — config-blind, diverging from the carrier + the KAT. derive now excludes any entry whose BASENAME contains "-incr" (catches all three; no legit .rlib/.rmeta/.d basename contains it) before sorting; all-excluded => None. verify_against_command_outputs re-derives via the same derive(), so the worker-side integrity check now agrees with the client on the .rlib key. §3 — nested seed path + full-empty wipe (option A). seed_dest_dir returned a TOP-LEVEL <stem>-incr, but rustc reads the NESTED declared output bazel-out/cfg/bin/<pkg>/<label>-incr => cold; and the chunk-2 wipe preserved only top-level *-incr names, so a nested seed lived under bazel-out and was deleted. Now seed_dest_dir joins the action's OWN declared nested -incr output_paths entry onto the execroot (option A: the index OutputDirectory.path stays the .rlib collision guard), the wipe is FULL-EMPTY (nothing preserved), and the out-of-band fetch is MOVED to run after [B2] input-materialize + [C] output-dir creation so the nested parent exists (and [B2] clonefiles into an empty execroot). Invariant being violated: a portable-incr action's fleet-shared targetkey AND its on-disk seed path must both agree with the Bazel client so rustc reuse fires. Mechanism that violates it: targetkey.rs::derive keyed on -incr; seed_dest_dir returned a top-level <stem>-incr; portable_incr wipe preserved only top-level *-incr (deleting the nested seed under bazel-out). Mechanism that re-establishes it: derive excludes -incr basenames -> key on the .rlib; seed_dest_dir joins the declared nested -incr output; full-empty wipe + fetch-after-[C] materializes the seed fresh at the nested path. Composite invariants this fix interacts with: §11 verify_against_command_outputs (re-derives via derive => same exclusion, key matches carrier); §6.2 plan_seed_publish / select_incr_seed_folder (now share the ends_with("-incr") predicate; behavior preserved); §9 delete containment (unchanged; assert_under_prefix still gates every delete, symlink-safe); the macOS clonefile fast path (an empty execroot restores it). Test that proves the fix: targetkey_kat_excludes_incr_and_keys_on_rlib (§2 KAT, both hashes b3sum-verified) + seed_dest_dir_uses_declared_nested_incr_output + wipe_full_empty_removes_all_including_incr + portable_action_with_seeded_index_fetches_and_materializes (production composition). Behavior changes (all INERT on the fleet: every changed path is gated on Some(PortableIncrContext)/portable_targetkey, which is None everywhere): - targetkey.rs::derive: when output_paths contains an -incr basename the derived key/primary_output moves from the -incr entry to the smallest non-incr entry; all-incr => None (was Some(-incr key)). Reached only via the portable-incr worker verify/stash path and any future targetkey caller — none on the live fleet. - portable_incr wipe: Owner execroot is now FULL-EMPTY wiped each build (was: preserve top-level *-incr). Reached only when portable_execroot is Some (off). - running_actions_manager inner_prepare_action: the seed fetch MOVED from before [B2] to after [C]; execroot wipe is full-empty. Reached only when portable_execroot is Some (off). The normal-mode create_dir path is byte-identical. - fs_util.rs try_clonefile: NO behavior change — comments/log strings only (dropped the now-moot FL-1383 seed-preservation reasoning + a removed test cite). - Non-portable actions (the entire fleet): byte-identical. No fsync/sync-write primitive added. No per-RPC timeout added (the seed fetch keeps its existing bounded overall deadline, invariant TraceMachina#10). No new unbounded buffer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tory (§2/§3 cadre fix-up) Cadre fix-up on 20efa2f: pair-a FIXES-REQUIRED (TraceMachina#2 working_directory drop), pair-a/red-team KAT edge case, pair-b test-gap nits. Feature INERT off-fleet (triple-gated: incr_seed_index_store None, portable_targetkey None, no -incr output). Invariant being violated: the fetched `-incr` seed must land at exactly the path where this action's rustc reads `-Cincremental` — i.e. where `prepare_output_directory` created the seed's parent. Mechanism that violates it: `incr_seed_fetch.rs` `seed_dest_dir` joined the declared `-incr` output_path directly onto the execroot, DROPPING `Command.working_directory`, while `prepare_output_directory` (`running_actions_manager.rs:3673-3677`) builds `{execroot}/{working_directory}/{output_path}`. They agree only when working_directory ∈ {"", "."}; any non-empty value darkens reuse (seed materialized where rustc never reads → `incr_index_fetch_hit` climbs, `incr_reuse_fired` flat) or fails cold (parent absent). Mechanism that re-establishes it after the fix: `seed_dest_dir` now takes `working_directory` and joins with the SAME formula as `prepare_output_directory` (single source of truth); the call-site (`running_actions_manager.rs:5399-5433`) threads `command.working_directory`. The execroot-escape guard (reject `is_absolute()`/`Component::ParentDir`) is kept load-bearing. Composite invariants this fix interacts with: (a) §2 key-agreement — the worker `TargetKey::derive` `-incr` exclusion is byte-identical to the client `contains("-incr")` predicate (else hard FailedPrecondition at `verify_against_command_outputs`); (b) §3 option A — the index `OutputDirectory.path` stays the `.rlib`, unchanged; (c) INERTNESS — the fetch gate reads `portable_targetkey` (None off-fleet) so the wd change is dormant until a portable execroot is planned. Test that proves the fix re-establishes the invariant: `seed_dest_dir_honors_non_empty_working_directory` (wd="k8-fastbuild/bin" → dest `execroot/k8-fastbuild/bin/pkg/foo-incr`); mutation to execroot-only join RED-fails it while the empty/"." cases stay green. Fixes: - FIX 1 (pair-a TraceMachina#2): `seed_dest_dir` working_directory-aware + non-empty-wd test. - FIX 2 (pair-a/red-team): targetkey KAT case for a crate literally named `foo-incr` (outputs `libfoo-incr-<hash>.rlib`/.rmeta both contain `-incr`) → all excluded → derive None → cold-SAFE; documents the worker side of the `contains("-incr")` contract edge (an `ends_with` client would diverge). - FIX 3a (pair-b): dedicated inertness test — a NON-portable action with an installed index store + matching pre-seed does NOT materialize the seed (A/B partner of the portable materialize test). - FIX 3c (pair-b): bespoke messages on the two path-traversal asserts in `seed_dest_dir_uses_declared_nested_incr_output`. Behavior changes: - `seed_dest_dir` (portable seed-fetch path only): the seed dest now honors `Command.working_directory` (prior → new: `{execroot}/{output_path}` → `{execroot}/{working_directory}/{output_path}`). Verified safe in compositions: portable-incr fetch gate — reached ONLY when a portable execroot was planned (`portable_targetkey` Some) AND an `incr_seed_index` store is installed, neither true on the fleet → INERT off-fleet. For working_directory ∈ {"", "."} the join is byte-equivalent to the prior behavior (the `.` component normalizes against the absolute execroot), so the existing empty/"." integration + unit tests stay green. - No other runtime behavior change. No fsync/sync-write primitive added. No lock held across `.await` (the fetch keeps the `state.lock().clone()`-then- drop-before-await pattern). No new buffering field on a network path. Verified: `cargo test -p nativelink-util -p nativelink-worker` (targetkey_test 12, portable_incr_execroot_test 32, util lib 280, worker lib 217; 541 total) all green; TDD mutation cycles RED-verified for FIX 1, FIX 2, FIX 3c with bespoke messages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…client omits segment → server blake3 default); §2/§3 wd-fix landed 29af854 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…cr_reuse_fired canary) Chunk 4 worker side: read THIS action's own declared `<label>-incr-reuse` marker (a SIBLING of the `-incr` tree the client's process_wrapper writes every action: content `1` = rustc incremental reuse fired, `0` = cold) and bump the process-singleton `incr_reuse_fired` counter only on content `1`. Cross-repo contract locked with rules_rust 2026-07-18. Flag-gated INERT. Invariant being violated: `incr_reuse_fired` is registered-but-DARK — the one FL-1383 §12 counter with no worker producer, so 0 events cannot be told apart from "reuse never fired" (the dark-counter trap). Mechanism that violates it: nothing in `inner_upload_results` ever reads the client-produced `-incr-reuse` marker, so `incr_reuse_fired` stays 0 forever even when rustc reuse fires on-fleet. Mechanism that re-establishes it after the fix: running_actions_manager.rs inner_upload_results reads the marker at the working_directory-aware declared path BEFORE the upload loop renames outputs into the store, and calls `note_reuse_fired()` (incr_seed_fetch.rs) when the trimmed content is `1`. Composite invariants this fix interacts with: (a) `-incr-reuse` ends in `-reuse` not `-incr`, so it is excluded from neither the seed selector (`seed_dest_dir` `ends_with("-incr")`) nor the targetkey (`derive` `contains("-incr")`) — no key pollution, distinct from the seed dir; (b) the working_directory-aware `{execroot}/{working_directory}/{output_path}` join — the SAME single-source-of-truth `prepare_output_directory`/`seed_dest_dir` use (the wd-drop just fixed for the seed path must not be re-introduced); (c) upload's `update_with_whole_file` RENAMES outputs out of the execroot (filesystem_store.rs:540), so the read MUST precede the loop; (d) best-effort — an absent/unreadable/malformed marker LOGS (warn) and leaves the counter at 0, never failing the action; (e) portable-gated on `portable_targetkey` (None for every fleet action) — off-fleet the whole block is skipped. Test that proves the fix re-establishes the invariant: portable_action_reuse_marker_bumps_incr_reuse_fired_only_on_content_1 (content 1 -> +1; content 0 -> +0; absent -> +0), driving the real portable RunningActionImpl composition with a non-empty working_directory. Behavior changes: - RunningActionImpl::inner_upload_results (portable action, portable_targetkey Some, declaring an `-incr-reuse` output): NEW -> reads the marker before the upload loop and bumps incr_reuse_fired when content trims to `1`. Verified safe: the `-incr-reuse` file is still uploaded to CAS as a normal declared output (fs::read does not consume it); read is best-effort (no action-fail path); no new buffering, no fsync, no lock across .await (the state lock is released into a bool before the async read). On-fleet this stays INERT until rules_rust chunk-4 emits the marker (find() returns None -> no read). - RunningActionImpl::inner_upload_results (non-portable / off-fleet, the DEPLOYED config): unchanged except one added `self.state.lock()` + `.is_some()` bool read per action (the same pattern the existing seed-fetch site at :5399 already uses); the read/bump block is skipped entirely. - incr_seed_fetch::note_reuse_fired: NEW pub fn; bumps the same singleton the render test already pins. No other caller. incr_reuse_fired name unchanged (render pin `incr_seed_metrics_render_pins_all_counter_names` still green). - All other stores/paths: none. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ad_to_directory (input-tree mechanism), hidden side-input; for cadre Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… hardlink = CAS corruption; no cache value; lost guards) — keep materialize_tree, parallelize in-place instead Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…_SEEDED carrier only on Materialized seed Bazel-client contract item TraceMachina#4: the in-action process_wrapper must reliably skip its local-tool seed path (setup() early-return) on remote execution when — and only when — a genuine -incr seed was materialized on the worker. Seed directory presence alone cannot distinguish the branch (a partial/cold path can leave nothing OR the wrong thing), so the worker emits an explicit child-env signal that process_wrapper keys on. Invariant walk (5 slots): - Invariant violated: process_wrapper may skip its local-tool seed setup ONLY when a real -incr seed is present in the remote execroot. - Mechanism that violates it: before this change the SeedOutcome from fetch_and_materialize_seed was only info!-logged and discarded, so the child had no reliable signal — process_wrapper could not tell Materialized from a cold NoSeed/Collision/TimedOut, risking a skip against a non-existent seed. - Mechanism that re-establishes it: inner_prepare_action stashes portable_incr_seeded=true ONLY on SeedOutcome::Materialized; inner_execute injects NL_PORTABLE_INCR_SEEDED=1 + NL_INCR_TARGETKEY=<64-hex> into the spawned CHILD env (via portable_incr_seed_child_env) gated on that flag AND a present portable_targetkey. Cold outcomes inject nothing. - Composite invariants this interacts with: action-digest / AC-key stability (vars added to command_builder AFTER env_clear + the command_proto env loop, NEVER to command_proto, so they never enter the REAPI Command / action digest); targetkey identity (NL_INCR_TARGETKEY == TargetKey::key(), the same hex the incr_seed_index writes); env ordering (injected last so no action-supplied var can clobber it). - Test that proves it: portable_incr_seed_env_tests::{ materialized_seed_injects_both_carrier_vars, cold_seed_injects_nothing, non_portable_action_injects_nothing}. Behavior changes: - RunningActionImpl (portable + allowlisted + seed Materialized): child process now additionally receives NL_PORTABLE_INCR_SEEDED=1 and NL_INCR_TARGETKEY=<hex> in its environment. Verified safe: child-env only (not in command_proto → no action-digest / AC-key change); rustc ignores both vars; only process_wrapper reads them. INERT on the current fleet (portable_targetkey is None fleet-wide, so portable_incr_seeded is never set true → no var injected). - All non-portable actions (the whole fleet) and any cold seed outcome (NoSeed/Collision/TimedOut): none. No child-env, action-digest, ack-semantics, or store-chain behavior changes; no new network buffering; no per-RPC timeout; no fsync/sync-write primitive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ag→child-env wiring + strip client-forged reserved carriers Cadre fix-up on c7ed3483 (both review pairs APPROVE the mechanism). Addresses the CONVERGENT must-fix (the prepare→execute wiring was untested) and the pair-b sole-authority MINOR (client could forge the reserved signal on a cold outcome). Invariant being violated: the worker-authority `-incr` seed signal must reach the child process ONLY on a genuine `SeedOutcome::Materialized` seed and must NEVER be forgeable by the client — but (a) the SeedOutcome→flag→child-env wiring had no test (inverting the prepare-site guard shipped green, 220/220), and (b) a client-supplied `NL_PORTABLE_INCR_SEEDED` in the action's own REAPI env survived into the child on a COLD outcome, so process_wrapper would skip a non-existent seed → wrong/cold build. Mechanism that violates it: (a) only the PURE helper `portable_incr_seed_child_env(seeded: bool, …)` was tested — the prepare-site `matches!(outcome, Materialized)` guard (running_actions_manager.rs:5486), the `state.portable_incr_seeded` stash, and the `command_builder.env` injection (:5688) had zero coverage; (b) the action-env loop (:5671) applied client env verbatim, so a forged reserved name passed straight through. Mechanism that re-establishes it after the fix: (a) an END-TO-END wiring test spawns the REAL child and reads the env it actually received, pinning Materialized→present / cold→absent AND the digest non-leak (byte-identical action digest seeded-vs-cold); (b) the action-env loop now STRIPS the reserved `NL_PORTABLE_INCR_SEEDED`/`NL_INCR_TARGETKEY` names on the portable path (`portable_targetkey.is_some()`) before the worker injects its own only on Materialized — the worker is the SOLE authority. Both reserved names are now shared consts (`NL_PORTABLE_INCR_SEEDED_ENV`/`NL_INCR_TARGETKEY_ENV`) referenced by both the injector and the strip so they can never drift. Composite invariants this fix interacts with: child-env-only (carrier never enters command_proto / the REAPI Command / action digest — the strip makes command_proto a reserved-name-free zone, so a carrier routed through it is evicted); fleet-INERT (portable_targetkey None fleet-wide → strip predicate short-circuits AND injection empty → child env byte-identical); no lock across `.await`; no fsync; carrier bounded (≤2-pair Vec by construction). Test that proves the fix re-establishes the invariant: `portable_seed_wiring_injects_carrier_into_child_env_only_when_materialized` (guard + digest-leak) and `portable_cold_strips_client_supplied_reserved_carrier` (strip). Mutation-verified: invert :5486 guard → wiring RED (Materialized run loses carrier); route injection through command_proto → wiring RED (strip evicts it); remove the strip → strip-test RED (child sees `client-forged-1`). Behavior changes: - FIX 2 (reserved-name strip): on the portable-incr path only (`portable_targetkey.is_some()`), any action-declared env var named `NL_PORTABLE_INCR_SEEDED` or `NL_INCR_TARGETKEY` is dropped before spawn; the worker re-injects its own values ONLY on a Materialized seed. Production composition: an allowlisted portable-incr action on a worker with the feature enabled. Off-fleet INERT — `portable_targetkey` is None for every action on the live fleet and every non-portable action, so the strip predicate short-circuits and the child env is BYTE-IDENTICAL to before; only a portable action literally declaring one of the two reserved names changes, and the change is the intended sole-authority strip. - No other runtime behavior change. The names-into-consts refactor emits the identical strings (verified by the unchanged passing helper tests); the wiring test is test-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…-file fetch at concurrency 16 The ONLY optimization the 5/5 design cadre left standing (it refuted the download_to_directory hardlink route: hardlinking a writable -incr to the read-only CAS blob corrupts CAS fleet-wide and drops the seed's guards). So this parallelizes the per-file fetch loop IN PLACE, touching no guard. Invariant being violated: materialize_tree serialized N independent per-file CAS blob reads (wall time = Σ per-blob read latency), leaving the §6.5 server-CAS fall-through reads un-overlapped though each file is independent. Mechanism that violates it: incr_seed_fetch.rs:350 `for file in &plan.files` awaits each fetch_and_write_file before starting the next. Mechanism that re-establishes it: incr_seed_fetch.rs materialize_files_parallel drives the SAME per-file futures via buffer_unordered(PARALLEL_MATERIALIZE_ CONCURRENCY=16); each future still runs the full blake3 verify + O_NOFOLLOW|O_EXCL byte-copy; the FIRST failure drops the stream (aborting remaining in-flight fetches) and the caller wipes the temp tree before returning cold. Composite invariants this fix interacts with: cold-not-wrong short-circuit (any cold/error -> NoSeed, no partial dir); no-partial-dir (files still write into the sibling temp dir; swap_into_place stays after ALL files complete); per-blob blake3 verify (unchanged, inside fetch_and_write_file); path-validation + symlink-reject (plan-time, before the loop, untouched); overall D1 deadline (still wraps the whole now-parallel materialize); worker-memory bound (was 1x largest file, now 16x); inner_prepare_action Send bound (helper owns per-file items so the buffered futures capture only Copy refs -> future stays Send). Test that proves the fix re-establishes the invariant: hit_materializes_many_files_parallel (40 files > the 16 window all land with correct content+mode) plus the unchanged file_digest_mismatch_is_cold_no_partial_dir (mutation-verified: commenting the blake3 verify -> RED with Materialized != NoSeed, proving the guard survives parallelization). Behavior changes: - materialize_tree per-file fetch+verify+write: SEQUENTIAL (1 blob resident) -> BOUNDED-PARALLEL (up to 16 blobs resident, buffer_unordered(16)). In the CAS chain (WorkerProxyStore->...->cas_FAST_SLOW), outcome is unchanged: same Materialized/NoSeed/Collision/TimedOut, same cold-not-wrong, same no-partial-dir. - Peak seed-materialize residency: 1x -> 16x the largest single -incr file. -incr files are per-CGU rustc artifacts (small); annotated UNBOUNDED-OK on the read site + the const doc. - On failure, the reported rel_path/reason in the warn is the FIRST-TO-FAIL file, not first-in-plan-order (nondeterministic across runs). The OUTCOME (cold, wiped, no-partial-dir) is identical either way. - No per-RPC timeout added (internal-RPC policy). No fsync/sync-write. No change to the index fetch, Tree read, plan-build, skeleton, swap, or the D1 timeout handler. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…first failure + partial-abort test The chunk-3 parallel-materialize path dropped the buffer_unordered stream on the FIRST per-file failure to abort the rest. write_file_nofollow runs its byte-copy on spawn_blocking, which is NOT cancellable, so an early drop orphaned the <=15 other in-flight blocking writes: they kept creating files under the temp dir and raced the caller's remove_dir_all -> intermittent ENOTEMPTY. cleanup_or_err then returned Err (not the cold SeedOutcome, and skipping the incr_seed_present_but_cold counter) and a half-written temp dir LEAKED on disk. Switch to DRAIN: capture the first error, poll the stream to completion so every in-flight write quiesces, then return the first error. Mirrors the .collect().await parallel-BFS precedent in running_actions_manager (whose futures are read-only, so its drain was already safe; ours does a spawn_blocking write, making the abort actively unsafe). Invariant being violated: a per-file materialize failure must resolve to a COLD SeedOutcome with the temp dir fully wiped and NO partial dir at dest (and no leaked temp dir), never an intermittent Err. Mechanism that violates it: incr_seed_fetch.rs materialize_files_parallel dropped the buffer_unordered stream on the first Err, orphaning uncancellable spawn_blocking writes that raced remove_dir_all -> ENOTEMPTY -> Err + leaked temp dir. Mechanism that re-establishes it after the fix: materialize_files_parallel now drains the stream to completion (first error captured, later results discarded) so all in-flight writes quiesce before the caller wipes the temp tree. Composite invariants this fix interacts with: memory bound (buffer_unordered still caps in-flight at PARALLEL_MATERIALIZE_CONCURRENCY=16 throughout the drain; each blob dropped after its write); cold-no-partial-dir at dest (dest only ever touched by the success-path rename in swap_into_place); no-fsync (write_file_nofollow unchanged); internal-RPC no-per-step-timeout (unchanged: the overall deadline in fetch_and_materialize_seed still bounds the whole op). Test that proves the fix: parallel_one_bad_file_among_many_is_cold_no_partial_dir (40 files, one digest-mismatched; asserts NoSeed + !dest.exists() + no leftover temp dir), run 5x with zero ENOTEMPTY; mutation (comment cleanup_or_err) -> RED with "no partial temp dir may survive a parallel first-failure abort". Behavior changes: - incr_seed_fetch::materialize_files_parallel: on a per-file failure, now DRAINS the remaining <=15 in-flight fetch+writes to completion before returning the first error, instead of dropping the stream to abort them. Verified safe in the worker inner_prepare_action composition (running_actions_manager.rs:5411, which already treats an Err as "proceed cold"): extra in-flight work is bounded (<=16 small -incr blobs), the returned outcome is unchanged (first error -> cold), the temp dir is then wiped, and dest never appears. The extra work only occurs on the cold path (rare) and eliminates the ENOTEMPTY leak + spurious Err + uncounted-cold. - All-success path: unchanged (drain and abort are behaviorally identical when no file fails). - New test only (parallel_one_bad_file_among_many_is_cold_no_partial_dir); no other runtime behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…vs-abort test + scope drain doc-comment Review fb4e4d4d (review-pair-a): distsys READY-TO-MERGE, red-team Landable Needs-justification. The drain fix is correct; this closes red-team's two asks so it merges. Test + doc-comment only — production drain logic byte-for-byte unchanged. Invariant being violated: on a first per-file failure the parallel materialize must leave NO in-flight uncancellable spawn_blocking write orphaned to race the temp-dir wipe (cold-wipe-no-partial-dir), and the test suite must RELIABLY fail if that guarantee is removed. Mechanism that violates it (the mutation guarded against): materialize_files_parallel returning on the first error (drop the buffer_unordered stream) instead of draining — incr_seed_fetch.rs:648-655. Mechanism that re-establishes it: the drain loop polls buffer_unordered to None, so every in-flight future (each resolving only after its write JoinHandle awaits) has quiesced before the fn returns and the caller wipes. Composite invariants this interacts with: the 16-cap (buffer_unordered holds throughout the drain); the D1 outer-timeout cancellation path (NOT covered by the drain — now scoped out of the doc-comment as a known residual, parent follow-up); per-file blake3/O_NOFOLLOW|O_EXCL guards (unchanged). Test that proves it: parallel_drain_awaits_inflight_not_abort_on_first_error. FIX 1 (red-team blind-spot 2): the shipped physical-race test (parallel_one_bad_file_among_many_is_cold_no_partial_dir) is timing-flaky green under in-memory stores — a buggy abort would mostly also pass it. Added a DETERMINISTIC, race-free discriminator: one bad file (immediate digest mismatch) among good files whose get_part HANGS FOREVER (reusing SelectiveStore). The correct DRAIN awaits the hung in-flight fetches and can only exit via the outer deadline -> TimedOut; a buggy abort returns NoSeed the instant the bad file errors. Because the good fetches hang unconditionally, the outcome is fixed by the code path, not by write/wipe timing. The hung stage is the fetch (a CAS wrapper cannot gate the uncancellable write) but the drain-loop await-all-futures structure it pins is identical for a future stuck in fetch or in write — a faithful, deterministic proxy for the write-orphan guarantee. Mutation-verified: reverting drain->abort makes it RED on all 5/5 runs with its bespoke message (left: NoSeed / right: TimedOut), 0.01s each (evidence: /tmp/parallelize-fix-mutation-red.log). Also annotated the older physical-race test as timing-dependent, pointing to this deterministic one. FIX 2 (distsys MINOR + red-team A1): the materialize_files_parallel doc-comment claimed the cold-wipe-no-partial-dir contract "holds deterministically" without qualification. Scoped that guarantee to the INTERNAL first-error path and added a one-line note that the outer fetch_and_materialize_seed timeout cancellation path is a known residual (dest untouched; worst case a leaked .<name>.incrtmp sibling self-healed by the §7 full-empty execroot wipe; parent follow-up). The timeout-path restructure is NOT attempted here. Behavior changes: NONE. materialize_files_parallel drain loop is byte-for-byte unchanged (git diff shows no edit to the loop body). All edits are one new test, a doc-comment on an existing test, and the materialize_files_parallel doc-comment. No fsync; no lock across .await; no sleep-as-synchronization (the discriminator uses an unconditionally-hung fetch, not a sleep); no new network-path buffer. Flag-gated INERT feature (not yet wired into a live path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…-holding workers) — for design cadre Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…proto-free, ingestion Feed A, Directory-digest fix, perf claim corrected); impl GATED on measurement Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…on for the residency measurement gate Add OBSERVE-ONLY read/write-split + fast-vs-slow blob-read classification to `incr_seed_fetch::materialize_tree`, so the §10 residency-gossip measurement gate (design `docs/fl1383-residency-gossip-design.md` §6) numbers fall out of real canary materializations. No behavior change: timing brackets + counter adds only. Signal chosen (design §6 priority (a)>(b)>(c)): per-blob READ-latency classification (b). (a) FastSlowStore exposes only PROCESS-AGGREGATE counters (populate_spawn_count etc.) — racy under the 16-wide buffer_unordered fan-out and concurrent actions, so no clean per-blob signal. (b) reuses the read span already taken for the read/write split → ZERO added store call, no downcast to FastSlowStore (robust to store composition), and it directly measures the read_delta the gate weighs. (c) fast-tier has() rejected: adds a call and couples to a FastSlowStore downcast. Threshold FAST_TIER_READ_LATENCY_THRESHOLD_NS = 1ms; raw read_ns/bytes sums are ALSO emitted so the divider is recalibratable from real data (it is a heuristic, flagged for assumption-auditor). Invariant being violated: none is BEING violated — this closes an observability gap. The invariant established: a successful materialize's per-blob read time, write time, byte volume, and fast-tier-hit-vs-slow-fetch split are recorded on the registered §12 process-singleton so the gate decision rests on measured canary data, not assumption (materialize_tree previously recorded only the Materialized/cold three-state, dark to the read-vs-write and residency-proxy questions). Mechanism that would violate it (pre-change): materialize_tree (incr_seed_fetch.rs) timed nothing per file and emitted only the incr_seed_materialized one-shot counter — the gate numbers were unobtainable without a bespoke bench. Mechanism that re-establishes it: fetch_and_write_file brackets get_part_unchunked and write_file_nofollow in Instant deltas → FileTiming; materialize_files_parallel folds each success into a MaterializeTiming; materialize_tree calls record_materialize_timing on the all-success path → the incr_seed_materialize_* + fast/slow singleton counters (registered in bin/nativelink.rs under "incr_seed_index"). Composite invariants this interacts with: (1) the drain-don't-abort first-error contract — timing is folded from Ok results in the same drain loop, the first failure still captured, no early return added; (2) the no-partial-dir / bounded- deadline guarantees — emission is on the success path only, after swap_into_place; (3) the §12 non-dark render contract — new counters ride the same singleton + render test. Test that proves it: materialize_records_read_write_ns_and_classifies_fast_vs_slow (non-zero read/write ns, bytes, >=1 fast-tier hit, >=1 slow fetch via a 25ms DelayStore) + incr_seed_metrics_render_pins_all_counter_names (5 new names). Behavior changes (enumerated against production composition): - materialize_tree control flow / materialized bytes / outcome: NONE. The Instant deltas and counter fetch_adds are O(1) per file, read no store, take no lock, add no await/fsync, and never branch materialize control flow. The read/write timing brackets only the already-present get_part_unchunked / write_file_nofollow awaits. Byte-identical materialize. - New registered counters incr_seed_materialize_{read_ns,write_ns,bytes}, incr_seed_{fast_tier_hit,slow_fetch}_blobs on the existing IncrSeedMetrics singleton (already registered in bin/nativelink.rs — no bin change). INERT on the fleet: they read 0 until portable seeds flow (materialize only runs for seeded portable actions), then accrue only from SUCCESSFUL materializations. - No new unbounded buffer (MaterializeTiming is 5 stack u64s; test fakes only). No new RPC timeout/deadline. No fsync/sync primitive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… every portable action + strips client FL_INCR_TOOL (fix cold-action process_wrapper ENOENT) Invariant being violated: on ANY portable-incr action the in-action process_wrapper must NOT run its own local FL_INCR_TOOL seed setup — the worker is the SOLE authority for -incr seeding. Previously the worker injected a skip signal ONLY on SeedOutcome::Materialized, so on a COLD/NoSeed portable action (the steady state while incr_seed_index is sparse) it injected nothing, the in-action process_wrapper ran its local seed setup off an empty FL_INCR_TOOL PathBuf -> ENOENT -> exit 1 -> cold build. Mechanism that violates it: running_actions_manager.rs portable_incr_seed_child_env returned an EMPTY vec on every non-Materialized outcome, and the env-strip loop did not touch the client's FL_INCR_TOOL — so a cold portable action carried no worker skip-signal and kept the client's local-tool var. Mechanism that re-establishes it: portable_incr_seed_child_env now emits the BROAD NL_PORTABLE_INCR_MANAGED=1 whenever a portable targetkey is present (independent of seed outcome), while the NARROW NL_PORTABLE_INCR_SEEDED=1 + NL_INCR_TARGETKEY pair stay Materialized-only; the env-strip loop additionally drops any client-supplied FL_INCR_TOOL on portable actions (worker supersedes the client's local tool). Composite invariants this interacts with: - MANAGED must NOT imply SEEDED — a false SEEDED on cold makes process_wrapper reuse a seed that isn't there; the narrow pair remains gated on the prepare-site Materialized guard (state.portable_incr_seeded). - all carriers are CHILD-env only (command_builder), never command_proto .environment_variables — the client action digest is byte-identical with vs without seeding (no cache poisoning). - sole-authority strip-then-inject: MANAGED/SEEDED/TARGETKEY are stripped from the client env before the worker injects its own; a client cannot forge them. - a NON-portable action's child env is byte-UNCHANGED (strip scoped to portable_targetkey.is_some(); helper returns empty on targetkey None). Test that proves the fix: - managed_fires_on_every_portable_action_seeded_only_on_materialized (seam, COLD -> MANAGED present + SEEDED/TARGETKEY absent; Materialized -> all three; plus digest byte-identical seeded-vs-cold). - strip_removes_client_fl_incr_tool_on_portable_only (portable strips; non-portable preserves + injects nothing). - pure-helper unit tests materialized_seed_injects_managed_and_both_carrier_vars, cold_seed_injects_managed_but_not_narrow_carriers, non_portable_action_injects_nothing. Mutation-verified: MANAGED-broad, SEEDED-narrow prepare guard, FL_INCR_TOOL strip clause, and non-portable strip-scope each go RED with their bespoke messages. Behavior changes (LIVE on the fleet: portable_incr.enabled=true + broad action_output_allowlist=["bazel-out/"] -> portable_targetkey is Some for allowlisted rustc actions): - portable action child env: GAINS NL_PORTABLE_INCR_MANAGED=1 on EVERY portable action (previously only the Materialized subset carried any -incr signal), and LOSES any client-supplied FL_INCR_TOOL. Narrow SEEDED/TARGETKEY semantics UNCHANGED (Materialized-only). All child-env only. - non-portable action child env (portable_targetkey None): byte-UNCHANGED (strip scoped to portable; helper injects nothing). - action digest / AC key: byte-UNCHANGED — all carriers land on command_builder, never command_proto; the digest is computed client-side and never recomputed. - no store-chain, ack-semantics, flow-control, or RPC-timeout change; no new buffering field on any network path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…omment + record skip-tool review The (c) assertion compares two client-frozen action digests → it only proves setup symmetry, not worker-side digest-non-leak (the worker never re-hashes command_builder). Reworded to state what it actually proves; the real guarantee is structural + the child-env dump assertions. Persist da8f311 Tier-2 cadre (pair-a distsys READY / red-team RECONSIDER-PREMISE-on-assumption; pair-b four-hat APPROVE, no FIXES-REQUIRED) + backlog entry (allowlist-mismatch REFUTED, real cause = Materialized-only signal). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ample), LRU 32768, reserve-undeclared flag (default on) Operator-directed policy change (no design cadre run per directive 2026-07-18). Five changes to the scheduler resource-profile / Phase-3 memory-reservation system. Invariant being violated: the memory reservation must be sized on the window p95 (operator policy), and an undeclared action with a learned footprint must reserve it — the prior monotone-max over-reservation and the "undeclared == free" gap violate the operator's packing policy. Mechanism that violates it: resource_profile.rs `Agg` used a monotone log2-bucket histogram whose reservation basis was `memory_tail_kb` (max), and `phase3_compute_effective_action_info` (api_worker_scheduler.rs) returned None for any action not already declaring `memory_kb` (`_ => return None`). Mechanism that re-establishes it: `SampleWindow` (20-sample FIFO ring) with `memory_p95_kb()` threaded through `TieredTail::Trusted.p95_kb`; RAISE/DOWN/inject all read p95; a new `is_undeclared` branch injects `memory_kb`=p95 store-once. Composite invariants this fix interacts with: store-once ledger symmetry (reduce/restore read the stored clone, never the mutating map); the K>=20 trust gate still gates every move/inject; `down_lowering_trusted` staleness gate preserved; the worker `memory_gate` free-floor NAK remains the SOLE OOM backstop (commented at the enforce site); RAISE/inject starvation clamp keeps reservations schedulable; persistence version gate discards the old schema. Test that proves the fix re-establishes the invariant: window_evicts_oldest_beyond_twenty, window_p95_below_max_drops_single_spike, undeclared_injection_reserves_p95_and_counts, phase3_applied_counters_track_raise_and_down, down_overcommit_lowers_reservation_below_declared, raise_reserves_the_tail_via_store_once_ledger, snapshot_entries_and_load_round_trip_preserves_trusted_key. Changes: 1. resource_profile.rs: LogHistogram -> SampleWindow (last <=20 raw samples/dim, FIFO). p50/p95/max recomputed on demand. memory_p95_kb() is the reservation statistic; memory_tail_kb() (window max) kept for observe/accuracy only. 2. LRU cap PROFILE_MAP_MAX_KEYS 16384 -> 32768 (footprint ~32 MiB). 3. Persistence: ProfileEntrySnapshot Vec<u32> histograms -> Vec<u64> samples; SNAPSHOT_VERSION 1 -> 2. A pre-TraceMachina#2497 snapshot version-mismatches -> discarded (re-learn); load stays infallible. 4. Enforce p95 basis: RAISE reserves p95; DOWN reserves clamp(p95, floor, declared); phase3_down_effective_kb simplified (p50xmargin/tier removed; PHASE3_COARSE_MARGIN_PENALTY deleted). 5. New flag phase3_reserve_undeclared_enabled (default TRUE): undeclared/zero memory_kb action + trusted profile -> inject memory_kb=p95 (starvation-clamped). Plus registered counters phase3_{raise,down}_applied, phase3_undeclared_injected, phase3_noop_ineligible on SchedulerMetrics (render on /metrics), incremented only on an actually-applied override / None return. Behavior changes (against production composition = SimpleScheduler ProfileMap): * Profile estimator monotone-max histogram -> 20-sample sliding window; the reservation statistic is now window p95 (forgets a spike after 20 fresher samples). Affects ONLY the profile map's stats; observe metrics still read window max. VERIFIED safe: worker memory_gate free-floor NAK is the retained OOM backstop for a p95 under-estimate (unchanged). * LRU cap 16384 -> 32768: larger retained working set. No functional change. * Persistence schema+version bump: on restart with resource_profile_persist_path set (default None = OFF in prod), any existing v1 file is discarded -> one-time re-warm. No data-plane effect. * RAISE/DOWN reservation basis max/p50-margin -> p95: gated by phase3_raise_enabled / phase3_down_overcommit_enabled, BOTH default OFF in prod -> NO runtime change until an operator enables. When enabled: RAISE reserves a lower (p95<max) value; DOWN reserves clamp(p95,floor,declared). * Undeclared->p95 injection (phase3_reserve_undeclared_enabled default TRUE): once profiles are K-mature, an undeclared action with a trusted profile now reserves p95 (was: no reservation). The injected Minimum affects matching (intended). Kill-switch = set flag false. Backstop = worker free-floor NAK. * New counters: observability only, no scheduling effect. Design-vs-code drift (flagged, not silently adapted): the observe/counterfactual path (inject_observe_would_raise, down_opportunity, accuracy classification) still uses window MAX (memory_tail_kb) as the tail, while the enforce path now reserves p95. The observe would-raise counterfactual therefore over-counts vs the applied RAISE rate. Left unchanged per the surgical/operator-directive scope; the new phase3_*_applied counters give the true applied rate. Operator to decide whether to migrate the observe counterfactual to p95. No design-review cadre was run (operator policy directive). The worker-side memory_gate free-floor NAK is the retained OOM backstop and was NOT touched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…to p95 (match enforce) Operator follow-up to 5eb692b: the observe/counterfactual path still keyed on window MAX (memory_tail_kb) while enforce reserves on p95, so would_raise over-counted vs the applied RAISE rate. Migrate every observe/counterfactual site to the SAME window p95 the enforce phase reserves on. Invariant being violated: the observe counterfactual must key on the SAME statistic the enforce phase reserves on (window p95), so `inject_observe_would_raise` is a faithful proxy for `phase3_raise_applied` and the accuracy falsifier scores the reservation we actually stand. Mechanism that violates it: observe_inject_counterfactual / the dispatch stash / classify_prediction_accuracy all read `tail_kb` (window max), diverging from the enforce path's p95 basis — would_raise fired on max>declared (over-counting) and the accuracy under/covered falsifier scored max, not the reserved p95. Mechanism that re-establishes it: the observe would_raise/down_opportunity split now keys on `p95_kb` (p95 > declared -> would_raise; p95 <= declared -> down_opportunity, mutually exclusive, mirroring the enforce RAISE/DOWN branch); the dispatch stash carries p95_kb (TieredTail::LowSample gained a p95_kb field for parity); classify_prediction_accuracy scores the stashed p95 against actual. Composite invariants this fix interacts with: the K>=20 gate still suppresses a <K stash (SkippedLowSample); the fine/coarse tier marker still threads through; observe remains OBSERVE-ONLY (no reservation/gate change); window max (memory_tail_kb) is kept only for the peek_tail observability helper. Test that proves the fix re-establishes the invariant: observe_keys_on_p95_not_max_spread_window (a spread window with max>declared but p95<=declared now counts as down_opportunity, not would_raise), down_opportunity_metric_folds_declared_over_p50, the accuracy/stash suite (dispatch_prediction_marks_coarse_tier_on_fallback, accuracy_dispatch_time_not_hindsight, lowsample_stash_production_path_classifies_skipped_low_sample). Changes: * resource_profile.rs: TieredTail::LowSample gains `p95_kb` (lookup_tiered fills it from peek_tier_stats); parity with Trusted so any dispatch stash carries p95. * api_worker_scheduler.rs observe_inject_counterfactual: destructure p95_kb (not tail_kb); would_raise fires on p95 > declared; down_opportunity now GATED on p95 <= declared (mutually exclusive with would_raise, mirroring enforce). Sample log field tail_stat_kb -> p95_stat_kb. * Dispatch stash: Trusted/LowSample -> (tier, p95_kb, samples). * classify_prediction_accuracy + PredictionAccuracy docs/locals: tail -> p95 (value-generic logic unchanged; the stashed value is now p95). Behavior changes (against production composition = SimpleScheduler observe path, ships ON): * inject_observe_would_raise now fires on p95 > declared (was max > declared): fewer events, exactly matching the enforce RAISE predicate. would_raise is again a faithful proxy for phase3_raise_applied. * down_opportunity_* now folded ONLY when p95 <= declared (was: every trusted lookup): the DOWN-side counterpart, mutually exclusive with would_raise. * accuracy_predicted_covered/_under now compare the dispatch-time p95 (not max) against the actual peak: the falsifier falsifies the statistic we reserve on. * Dispatch stash value is p95 (was max). For identical-sample windows p95==max, so existing stash-value assertions are unchanged; only spread windows differ. * No reservation/gate/dispatch decision changes (observe stays OBSERVE-ONLY). The worker memory_gate free-floor NAK remains the sole OOM backstop. No design-review cadre run (operator policy directive continues). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

erneestoc and others added 2 commits July 2, 2026 13:41

vercel Bot deployed to Preview – nativelink-aidm July 2, 2026 20:56 View deployment

vercel Bot deployed to Preview – nativelink July 2, 2026 20:56 View deployment

vercel Bot deployed to Preview – nativelink-aidm July 2, 2026 22:39 View deployment

vercel Bot deployed to Preview – nativelink July 2, 2026 22:40 View deployment

MarcusSorealheis reviewed Jul 3, 2026

View reviewed changes

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 21:11 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 21:12 View deployment

Merge branch 'main' into experimental-cdc-split-splice

a03db5c

Resolves lockfile conflicts: Cargo.lock and MODULE.bazel.lock taken from main and regenerated by cargo/bazel to re-include the fastcdc dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 21:16 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 21:17 View deployment

Format chunking example per formatjson5

a65ed71

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 21:40 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 21:41 View deployment

erneestoc requested a review from MarcusSorealheis July 6, 2026 21:56

erneestoc marked this pull request as ready for review July 6, 2026 22:51

erneestoc and others added 2 commits July 6, 2026 16:37

Merge remote-tracking branch 'tm/main' into experimental-cdc-split-sp…

0942bca

…lice

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 23:40 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 23:41 View deployment

MarcusSorealheis requested changes Jul 6, 2026

View reviewed changes

vercel Bot deployed to Preview – nativelink-aidm July 6, 2026 23:56 View deployment

vercel Bot deployed to Preview – nativelink July 6, 2026 23:56 View deployment

rejuvenile added a commit to rejuvenile/nativelink that referenced this pull request Aug 19, 2026

TraceMachina#2497 FL-1383 §10: residency-gossip design (route to seed…

5f14568

…-holding workers) — for design cadre Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Navigation Menu

Uh oh!

Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support - #2497

Add REAPI content-defined chunking (SplitBlob/SpliceBlob) support#2497
palfrey merged 15 commits into
TraceMachina:mainfrom
erneestoc:experimental-cdc-split-splice

erneestoc commented Jul 2, 2026 •

edited by MarcusSorealheis

Loading

Uh oh!

vercel Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

MarcusSorealheis left a comment •

edited

Loading

Uh oh!

erneestoc commented Jul 6, 2026

Uh oh!

erneestoc commented Jul 6, 2026 •

edited

Loading

Uh oh!

MarcusSorealheis left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

erneestoc commented Jul 2, 2026 • edited by MarcusSorealheis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

Conformance

Safety / hardening

Design decisions worth reviewer attention

Testing

Notes

Uh oh!

vercel Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcusSorealheis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erneestoc commented Jul 6, 2026

Uh oh!

erneestoc commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcusSorealheis left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

erneestoc commented Jul 2, 2026 •

edited by MarcusSorealheis

Loading

vercel Bot commented Jul 2, 2026 •

edited

Loading

MarcusSorealheis left a comment •

edited

Loading

erneestoc commented Jul 6, 2026 •

edited

Loading