Summary
Bazel 8.7.0 / 9.1.0+ ship --experimental_remote_cache_chunking (bazelbuild/bazel#28437), which uses content-defined chunking (FastCDC 2020) to upload/download only the chunks of a large blob that changed, instead of re-transferring the whole blob whenever its digest changes. The protocol surface — SplitBlob / SpliceBlob RPCs on ContentAddressableStorage, plus ChunkingFunction negotiation and capability advertisement — is standardized upstream in remote-apis (bazelbuild/remote-apis#282, follow-up in #357). BuildBuddy is currently the only server implementation (see their write-up); they report ~85% of written bytes deduplicated for eligible files, and Bazel's benchmarks showed ~40% less data uploaded and ~40% smaller local disk cache.
This issue proposes implementing the server side in NativeLink.
Why NativeLink is well positioned
NativeLink already has production CDC machinery at the storage layer:
DedupStore (nativelink-store/src/dedup_store.rs) chunks blobs with a vendored FastCDC implementation (nativelink-util/src/fastcdc.rs, 64k/256k/512k min/avg/max defaults), stores chunks individually, and persists a serialized chunk-digest index keyed by the original blob digest.
That is almost exactly the data model SplitBlob/SpliceBlob expose over the wire — but today it is invisible to REAPI, so it saves storage without saving network transfer. Wire-level chunking is where most of the user-visible win is (large linked binaries, archives, container layers re-transferred after one-line source changes).
Current gaps
- The vendored
nativelink-proto/build/bazel/remote/execution/v2/remote_execution.proto predates the extension: no SplitBlob/SpliceBlob RPCs, no ChunkingFunction, no split_blob_support/splice_blob_support in CacheCapabilities.
cas_server.rs implements only the classic four CAS RPCs; capabilities_server.rs advertises nothing chunking-related.DedupStore's chunk digests are internal BLAKE3 hashes regardless of the outer digest function, so its index can't be returned to clients as-is — the protocol requires chunks to be ordinary, client-addressable CAS blobs under the negotiated digest function.
Proposed phased plan
- Proto + capabilities. Vendor the updated
remote_execution.proto; add split_blob_support / splice_blob_support and supported ChunkingFunction values to GetCapabilities, gated behind config and off by default. Bazel only activates chunking when the server advertises it, so this is safe to ship incrementally.
SpliceBlob (write path). Accept an ordered chunk-digest list for a blob, verify all chunks exist, verify the concatenation actually hashes to blob_digest before registering the mapping (otherwise this is a cache-poisoning vector), and store the blob→chunks index.SplitBlob (read path). Return the stored layout when a splice was registered; optionally chunk on demand server-side using the existing FastCDC util for blobs that were uploaded whole — this benefits mixed fleets where some clients upload without chunking.- (Optional)
DedupStore unification. Evaluate sharing chunking/index machinery between DedupStore and the wire protocol so a chunk-uploaded blob is stored deduplicated without re-chunking.
Open design questions
- Chunk lifetime / eviction coupling. Chunks must not be evicted while an index referencing them is alive. This needs a design that works across NativeLink's composable store topologies (evicting memory stores, S3 backends, sharded stores) — similar in spirit to the ActionResult→blob existence dependency, but inside the CAS.
- Where the blob→chunks index lives. A dedicated keyspace/store vs. a CAS blob keyed off the original digest; interaction with
FindMissingBlobs semantics (does a registered splice make the blob "exist"?).
- Chunking parameters. Server-advertised FastCDC 2020 parameters must match what Bazel uses, or cross-client dedup silently degrades;
DedupStore's current defaults may differ from Bazel's.
- Digest functions. Chunk digests must use the request's digest function (SHA256/BLAKE3), unlike
DedupStore's internal fixed-BLAKE3 hashing.
References
Summary
Bazel 8.7.0 / 9.1.0+ ship
--experimental_remote_cache_chunking(bazelbuild/bazel#28437), which uses content-defined chunking (FastCDC 2020) to upload/download only the chunks of a large blob that changed, instead of re-transferring the whole blob whenever its digest changes. The protocol surface —SplitBlob/SpliceBlobRPCs onContentAddressableStorage, plusChunkingFunctionnegotiation and capability advertisement — is standardized upstream in remote-apis (bazelbuild/remote-apis#282, follow-up in #357). BuildBuddy is currently the only server implementation (see their write-up); they report ~85% of written bytes deduplicated for eligible files, and Bazel's benchmarks showed ~40% less data uploaded and ~40% smaller local disk cache.This issue proposes implementing the server side in NativeLink.
Why NativeLink is well positioned
NativeLink already has production CDC machinery at the storage layer:
DedupStore(nativelink-store/src/dedup_store.rs) chunks blobs with a vendored FastCDC implementation (nativelink-util/src/fastcdc.rs, 64k/256k/512k min/avg/max defaults), stores chunks individually, and persists a serialized chunk-digest index keyed by the original blob digest.That is almost exactly the data model
SplitBlob/SpliceBlobexpose over the wire — but today it is invisible to REAPI, so it saves storage without saving network transfer. Wire-level chunking is where most of the user-visible win is (large linked binaries, archives, container layers re-transferred after one-line source changes).Current gaps
nativelink-proto/build/bazel/remote/execution/v2/remote_execution.protopredates the extension: noSplitBlob/SpliceBlobRPCs, noChunkingFunction, nosplit_blob_support/splice_blob_supportinCacheCapabilities.cas_server.rsimplements only the classic four CAS RPCs;capabilities_server.rsadvertises nothing chunking-related.DedupStore's chunk digests are internal BLAKE3 hashes regardless of the outer digest function, so its index can't be returned to clients as-is — the protocol requires chunks to be ordinary, client-addressable CAS blobs under the negotiated digest function.Proposed phased plan
remote_execution.proto; addsplit_blob_support/splice_blob_supportand supportedChunkingFunctionvalues toGetCapabilities, gated behind config and off by default. Bazel only activates chunking when the server advertises it, so this is safe to ship incrementally.SpliceBlob(write path). Accept an ordered chunk-digest list for a blob, verify all chunks exist, verify the concatenation actually hashes toblob_digestbefore registering the mapping (otherwise this is a cache-poisoning vector), and store the blob→chunks index.SplitBlob(read path). Return the stored layout when a splice was registered; optionally chunk on demand server-side using the existing FastCDC util for blobs that were uploaded whole — this benefits mixed fleets where some clients upload without chunking.DedupStoreunification. Evaluate sharing chunking/index machinery betweenDedupStoreand the wire protocol so a chunk-uploaded blob is stored deduplicated without re-chunking.Open design questions
FindMissingBlobssemantics (does a registered splice make the blob "exist"?).DedupStore's current defaults may differ from Bazel's.DedupStore's internal fixed-BLAKE3 hashing.References