Support streaming split/splice APIs for large blobs

[tyler-french]

TL;DR: Unary split/splice forces large chunk manifests into a single size-limited message and makes the receiver wait for the complete manifest before processing it; streaming keeps messages bounded and enables incremental processing, improving scalability and performance for large blobs.

This PR adds GetChunkMapping and RegisterChunkMapping, streaming replacements for split and splice. This lets clients send the ordered chunk list across multiple messages instead of fitting it all into one unary request or response.

The names make the direction of each operation explicit and describe what the RPCs exchange: an ordered blob-to-chunk digest mapping. They do not transfer chunk data, so names such as GetChunks or PutChunks would be misleading.

Streaming also gives servers a chance to process the mapping as it arrives. For splice, this can avoid doing all of the work in one final RPC after every chunk has already been uploaded. Servers can still choose to wait until the stream is complete and process it like the unary path.

This solves the issues described in #376.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds streamed equivalents of CAS blob split/splice operations to avoid unary message size limits while exposing capability flags for feature discovery.

Changes:

• Document unary SplitBlob/SpliceBlob size limitations and recommend streaming alternatives.
• Add SplitChunks (server-streaming) and SpliceChunks (client-streaming) RPCs plus request/response messages.
• Extend CacheCapabilities with support flags for the new streaming RPCs.

Files not reviewed (2)

• build/bazel/remote/execution/v2/remote_execution.pb.go: Language not supported
• build/bazel/remote/execution/v2/remote_execution_grpc.pb.go: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[EdSchouten]

I got added as a reviewer, so here's my take: just like how chunking has been added to the protocol in its current form, I'm not a fan of this. Not having streaming, but requiring a recursive construction of very large files using Prolly trees is in my opinion the way to go.

That said, if people actually on the working group think that this is the way to go, go for it! :-)

[tyler-french]

I got added as a reviewer, so here's my take: just like how chunking has been added to the protocol in its current form, I'm not a fan of this. Not having streaming, but requiring a recursive construction of very large files using Prolly trees is in my opinion the way to go.

That said, if people actually on the working group think that this is the way to go, go for it! :-)

@EdSchouten Thanks, that makes sense. I agree that a recursive/Merkle-style representation such as Prolly trees is probably the better long-term model for representing extremely large blobs, since it bounds individual node sizes and avoids one giant flat chunk list.

My goal with this PR is narrower: provide a transport-level alternative to #376 without making clients materialize or dereference a chunk-list object in CAS. Streaming lets clients send the existing ordered chunk mapping in bounded batches, and it lets servers begin verification as chunks become available instead of waiting until every chunk has been uploaded and then receiving the whole mapping in one unary SpliceBlob request.

I also see this as compatible with a future recursive representation. A server could still store the received mapping internally as a Prolly tree or some other recursive structure. The protocol question here is whether clients need to understand and construct that representation, or whether they can just stream the chunk references they already have.

Longer term, I would prefer for the streaming APIs to become the primary split/splice shape for clients that support them, with the existing unary APIs remaining as the compatibility/simple-case path. For cases where the chunk list fits in one request, sending it as a single stream message should be performance-equivalent to a unary gRPC request in practice, while still leaving the same API able to scale when the list grows. A recursive representation also fits into that model: the stream does not require the server to store a flat list internally, it just gives the server live visibility into the composition progress. The server can choose to persist that mapping as a Prolly tree, a flat manifest, or another internal structure, while still getting the benefits of streaming transport: bounded message sizes, earlier validation, and the ability to pipeline verification while the client is still uploading or discovering chunks, since chunking is inherently sequential.

Some recursive composition can be approximated with the current API today by splicing subranges and then splicing those intermediate blobs, but standardizing that as the preferred model seems like it would need more protocol work: tree node encoding, fanout rules, digest/validation semantics, and how chunking_function applies at each level. Streaming is a smaller additive step that addresses the immediate unary message-size and final-RPC verification problem while leaving room for that design.

[tyler-french]

@EdSchouten One open question I have for a Prolly tree design is how it preserves the existing full-blob digest semantics. In REAPI, the digest in an ActionResult is the digest of the output file contents, and CAS lookups are keyed by that full content digest. So even if the server stores the blob as a recursive tree internally, both the client and server eventually need to know that the tree represents exactly the bytes for the full blob digest.

If the Prolly root digest is a digest of tree-node bytes, it is not a substitute for the file's content digest. If the key remains the full blob digest, the server still has to verify the leaves in order by hashing the reconstructed content, or trust a mapping it has already validated. And if a server only has chunks/tree nodes under one chunking strategy, it is not obvious how it can know that a different client-provided tree already represents the same blob without either having a stored full-digest-to-tree mapping or walking/rehashing the content.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 3 changed files in this pull request and generated 2 comments.

Files not reviewed (2)

• build/bazel/remote/execution/v2/remote_execution.pb.go: Language not supported
• build/bazel/remote/execution/v2/remote_execution_grpc.pb.go: Language not supported

[moroten]

The streaming approach keeps chunk-list transport in the RPC layer instead. That has a few advantages:

• It avoids using CAS as a protocol transport for RPC metadata.

That is true for this specific RPC, but many other parts of the protocol already work this way.

• It does not require clients to add extra read/write logic for chunk-list objects.

I agree that it would probably be easier to adapt most servers and clients.

• It lets servers receive the ordered chunk mapping incrementally instead of only at the final splice call.

The speed improvement I can see is that splicing can finish faster after the last chunk has been uploaded. The work still has to be done, so it is a matter of cutting latency, instead of loading GB of data at the end, it can be loaded while the client is uploading the chunks. @tyler-french Do you see the latency as a problem in your current clusters?

My biggest concern is that the servers may have to hold an infinite number of connections open, waiting for more chunks to arrive.

• What timeout should be allowed? For Bazel, we will still have the be quicker than the global --remote_timeout.
• If a splice call times out, the server has to throw away all work. With a unary call with a chunk-list the server can process the whole splicing and store the result, so timeout should not be any problem (unless server internal networking is slow).
• Minor note: For clients to handle timeouts, they need to keep the full list of chunks until splicing is done. There will not be any benefit in lower memory consumption on client side.
• With a unary call, the splicing can be deduplicated on server side.

To address my concerns, I might implement the streaming splicing by waiting for the last message and then do the splicing anyway, effectively converting it into the chunk-list version.

• It keeps the old unary APIs simple and fully usable by older clients.

The chunk-list approach also keeps the old API fully usable by older clients.

• It gives clients a capability-gated path for large blobs without changing the semantics of existing fields.

Either we have the feature of handling large blobs capability gated or we can put it as requirement on Split/Splice API from a certain version of the REv2 API.

[tyler-french]

@moroten (and cc @meroton-benjamin) Thanks, these are good concerns. I am also happy to support #376 if that is where we land. I needed to add streaming support on our side anyway, so this felt like a good time to see whether it could also solve the large chunk-list problem.

I think the main point is that streaming does not remove the splice work. It gives clients a bounded-message way to send the chunk list, and it lets servers start seeing the mapping earlier if they want to. A client does not have to keep the stream open while chunks are still uploading. It can upload the chunks first, then call SpliceChunks with the full chunk list in one or a few messages. That makes this usable as an alternative to the indirect chunk-list approach, without requiring the chunk list itself to be stored and fetched through CAS.

The speed improvement I can see is that splicing can finish faster after the last chunk has been uploaded. The work still has to be done, so it is a matter of cutting latency, instead of loading GB of data at the end, it can be loaded while the client is uploading the chunks. @tyler-french Do you see the latency as a problem in your current clusters?

Yes. We have hit issues with blobs around 1 GB. Today the client uploads chunks with BatchUpdateBlobs, then calls SpliceBlob after all chunks are available. Since the server only sees the mapping at the end, all splice work is deferred into one final RPC, and we have seen that call time out.

I agree that the work still has to be done. The benefit is mostly message sizing and tail latency. With a fast chunker, I would expect an incremental SpliceChunks flow to take about as long as Write for the same blob.

My biggest concern is that the servers may have to hold an infinite number of connections open, waiting for more chunks to arrive.

That risk is real, but it is similar to Write. Servers still need normal RPC controls, such as deadlines, idle timeouts, max concurrent streams, and per instance limits.

Also, clients that want to avoid long idle splice streams can wait until all chunks are uploaded, then send the mapping in one or a few SpliceChunks messages. That gives a similar call shape to PR #376, but keeps the chunk-list transport in the RPC layer.

What timeout should be allowed? For Bazel, we will still have the be quicker than the global --remote_timeout.

I think this should follow the normal RPC deadline model. For Bazel, the call still needs to complete inside --remote_timeout. Probably something similar to whatever the Write call was for the blob.

If a splice call times out, the server has to throw away all work. With a unary call with a chunk-list the server can process the whole splicing and store the result, so timeout should not be any problem (unless server internal networking is slow).

This is the main tradeoff. If the stream fails before the server commits the splice, the client has to retry. After the chunks are uploaded, that retry is much cheaper than retrying the original upload.

There is also a useful fallback. If the chunk list fits in one message, the client can retry with unary SpliceBlob. If it does not, it can retry SpliceChunks with smaller batches or only a few messages. So the client has a bit more control over getting things to succeed

Minor note: For clients to handle timeouts, they need to keep the full list of chunks until splicing is done. There will not be any benefit in lower memory consumption on client side.

Agreed. Keeping a []digests is pretty cheap. You could even just keep []offsets to save even more memory.

With a unary call, the splicing can be deduplicated on server side.

I got this feedback, and I think it makes sense the request requires blob_digest. It should deduplicate well now by routing by this.

To address my concerns, I might implement the streaming splicing by waiting for the last message and then do the splicing anyway, effectively converting it into the chunk-list version.

That seems like a valid implementation. It would still avoid the unary message size limit and let clients use one API shape for larger blobs.

The chunk-list approach also keeps the old API fully usable by older clients.

Agreed. I did not mean that the chunk-list approach breaks older clients.

The difference I see is that PR #376 adds another representation to the existing unary API shape. Newer clients then need logic to write, read, and interpret chunk-list objects. With streaming, the old unary APIs stay as the simple path, and the large-blob path is a separate capability-gated RPC. A client that is not aware of indirection may fail if the SplitBlob response sets it.

[moroten]

@tyler-french I would be happy with the streaming approach as well. Both approaches have their pros and cons, but the main problem is that the whole API refers to blobs as whole files, not a Prolly tree.

Maybe set the decision next monthly? I'll add some code comments as well.

[moroten]

Updated SpliceChunks so blob_digest is required only on the final request. A client may not know the full blob digest when it begins chunking, so this allows chunking, uploading, and incremental splice verification to proceed concurrently.

How does the client know that it does need to upload the blob at all? I thought FindMissingBlobs would be called on the full blob before calling SpliceChunks but this indicates that the recommended process is to split the blob and stream the chunk digests. Before sending a chunk digest, it first must go through FindMissingBlobs for potential upload.

My original though was to first run FindMissingBlobs on the digest of the full blob before starting SpliceChunks.

Does calling many FindMissingBlobs really increase the efficiency compared to #376?

Earlier requests do not need the blob digest for routing because each chunk is routed independently by its own digest. Only the final commit needs blob_digest to validate and route storage of the resulting manifest.

Well, that depends on the implementation of your server. Some servers might want to shard out the splicing process depending on the final blob_digest.

With the blob_digest up front, concurrent uploads can be aborted with ALREADY_EXISTS once the first upload finished.

[tyler-french]

@moroten good points. I moved blob_digest back to the first request. You're right that clients should generally call FMB for the full blob first, and sending the digest up front gives servers more control over routing and early deduplication. It could also short circuit if it it already has the blob but FMB never got called.

[meroton-benjamin]

Having implemented a client/server pair with this RPC I think we need to add guidance towards how this should be used to to upload a large blob. The naïve implementation, where you calculate a chunk, check if it exists, uploads it if it doesn't exist and sends a message with the chunk to the streaming rpc is incredibly inefficient and splicing is very intensive server side.

In essence a well behaved client should do the following for a large blob:

Step 1:

• Calculate the blob digest and determine if the blob exists or not

Step 2:

• Calculate a N of chunks of the blob where N is as many as can be comfortably held by the client, if calculating from a situation where you can do random access (e.g. when uploading a blob from disk) this may be something like 1MB worth of digests.
• Determine which if any of the chunks are missing from the CAS with FMB
• Upload those missing chunks
• Repeat until all chunks exist in CAS

Step 3:

• From the chunk list calculated earlier split into safe message sizes (e.g. below 1MB) and stream the splice chunks call.

While the api formally will work for clients which don't follow the above recipe, in practice server implementations will struggle to be able to serve it. A naïve implementation from above would from a very modest 1GB blob of random data create the equivalent of 10000 unary rpc calls.

[meroton-benjamin]

Could we rename this into SplitIntoChunks as SplitChunks doesn't really make any sense.

[armandomontanez]

On this note: everyone I've discussed these RPCs with gets confused about the directionality of split/splice.

Maybe we could be more clear with something like PutSplitBlob/GetSplitBlob so directionality is clearer. Even if we want to be pedantic, RegisterSplitBlob/RetrieveSplitBlob are much clearer.

[tyler-french]

We could even go as simple as Get and Set 🤔

[tyler-french]

@armandomontanez @meroton-benjamin per-request, I renamed the APIs:
- GetChunkMapping
- RegisterChunkMapping
I think this makes the intention much more clear how to use these.

@tjgq @fmeum @sluongng if you all also have time to take another look!

[fmeum]

Symmetry is an argument for this, but perhaps we shouldn't carry these defaults forward to new APIs?

[tyler-french]

We can do that, but I'd like that to be separate from this PR. If we want to still do it, we can make this change before a release, and focus the discussion there. This API is flexible until we publish a release

[EdSchouten]

Indeed; this sentence should not be added to new RPCs. Also notice how the exact list of digest functions differs per RPC/field. Take ExecuteRequest:

  // If the digest function used is one of MD5, MURMUR3, SHA1, SHA256,
  // SHA384, SHA512, or VSO, the client MAY leave this field unset.

Now look at ExecuteOperationMetadata:

  // If the digest function used is one of BLAKE3, MD5, MURMUR3, SHA1,
  // SHA256, SHA256TREE, SHA384, SHA512, or VSO, the server MAY leave
  // this field unset.

We only list digest functions here that could be used in combination with those messages whose use predates the introduction of the digest_function field. Because BLAKE3 was added AFTER ExecuteRequest.digest_function was added, it's never valid to use BLAKE3 and leave ExecuteRequest.digest_function unset.

So by copying the phrasing from just one of these places you're doing something totally arbitrary. For a new RPC this should simply not be added. Symmetry is not a valid argument.

[tyler-french]

Fair, updated

[tyler-french]

@tjgq @sluongng friendly ping here -

I think we should go ahead and try to get this merged. Leaving it open is causing a follow-up churn as we continue to iterate on CDC, and I want to avoid making this PR a grab-bag CDC related changes. Its in a good shape now, but I'd like any additional follow-ups to be raised by in separate PRs, to keep this one scoped.

We can wait to do a version release until we're confident in the shape of the spec, so merging the PR doesn't prevent changes to these unreleased APIs.