investigate nvsnap checkpoint/restore for Dynamo aggregated and disaggregated modes

[balajinvda]

Context

We were asked how nvsnap checkpoint/restore behaves for Dynamo deployments in
aggregated and disaggregated mode. nvsnap currently has no Dynamo integration:
grep -ri dynamo src/compute-plane-services/nvsnap/ returns nothing. This issue
tracks the investigation before we make any claim about support.

What is already established

Dynamo topology (from NVIDIA Dynamo docs):

• Aggregated: one worker performs both prefill and decode in a single process
  and pod.
• Disaggregated: separate prefill and decode pods. The prefill worker's KV
  cache is transferred into the decode worker's VRAM.

The transport is NIXL, which supports RDMA/InfiniBand, RoCE via UCX, TCP
fallback, NVMe-oF and S3. Dynamo docs state deployments instantiate the UCX
backend and that NIXL stages transfer metadata through the pod's /dev/shm.

The in-repo sample at
examples/function-samples/helmchart-samples/dynamo-operator-sample deploys the
disaggregated shape:

--disaggregation-mode prefill
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
--kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:20080",...}'
--tensor-parallel-size <nodeCount * gpus>

with etcd and NATS for coordination.

Relevant nvsnap constraints:

• CRIU cannot dump processes using RDMA (criu fails for RDMA programs checkpoint-restore/criu#267), and
  restoring InfiniBand verbs hits queue-pair and memory-region collisions that
  upstream CRIU does not solve.
• nvsnap has no NIXL, UCX or RDMA handling. The only RDMA-adjacent code
  externalizes the gdrdrv device node, which is not the same as checkpointing
  live RDMA state.
• Multi-GPU process checkpoint does not work today. Workers with tensor
  parallelism above 1 can only use the cache-directory capture path.
• External TCP sockets are destroyed before checkpoint; intra-pod connections
  are preserved.
• Dynamo publishes KV events over ZMQ, which is the class of state our
  interception library exists to repair after restore.

Assessment to validate

• Aggregated with TP=1 is closest to what already works: one pod, one process
  tree, GPU state via cuda-checkpoint. Open work is coordination re-registration
  and the ZMQ event socket.
• Aggregated with TP above 1 is blocked by the existing multi-GPU limitation.
• Disaggregated is a distributed snapshot problem rather than a harder
  single-pod one: two pods, live GPU-to-GPU transport, in-flight transfers, and
  cluster-wide registration.

The question that decides the design

Can Dynamo workers rejoin after an abrupt restart? They run under Kubernetes, so
they very likely tolerate pod loss: the router re-routes and the worker
re-registers. If that holds, we may not need to checkpoint transport state at
all. We could drop NIXL, ZMQ and coordination state at capture and let it
re-establish on restore, which is what we already do for external TCP.

Dynamo's published guides do not document failure or restart semantics, so this
must be answered from the Dynamo source rather than inferred.

Tasks

• Determine from Dynamo source whether workers re-register after an abrupt
  restart, and what the router does with in-flight requests.
• Determine whether NIXL connections are long-lived or established per
  request. This materially changes the disaggregated answer.
• Stand up an aggregated TP=1 Dynamo worker and attempt a capture/restore
  end to end.
• Establish what a restored worker must re-do: coordination registration,
  ZMQ event publisher, NIXL agent metadata in /dev/shm.
• Decide whether the cache-directory path alone is the near-term answer for
  Dynamo, since it is engine-agnostic and topology-agnostic.
• Record the interception-library dependency, given the intent to reduce
  reliance on patched ZMQ.

Out of scope for now

Disaggregated checkpoint/restore across pods. Record findings, but do not design
for it until the restart-semantics question above is answered.

[balajinvda]

Multi-GPU: what actually blocks it

Relevant because Dynamo workers run with tensor parallelism above 1 in most
real deployments, so aggregated-mode support is gated on this.

cuda-checkpoint --launch-job (NVIDIA driver 610 and higher, released
2026-08-03) enables cuIpcGetMemHandle-based CUDA IPC support. That is one
layer of peer state, not all of it. Per NVIDIA's own documentation and issue
tracker:

Layer	Status
CUDA IPC memory handles	addressed by --launch-job, driver 610+
NCCL communicators	not solved; on the NCCL Q2 2026 roadmap (NVIDIA/nccl#2090)
CUDA graphs containing NCCL collectives	not solved; graphs embed ncclComm_t, stale after rebuild
NVLS communicators	incompatible with cuda-checkpoint (NVIDIA/nccl#2337)

nvsnap already handles the NCCL deadlock by calling ncclCommAbort on tracked
communicators during quiesce, because locking one GPU otherwise freezes its NCCL
ring buffers and the other GPUs block on collectives. So --launch-job would
likely remove the IPC blocker while leaving us severing and rebuilding NCCL,
which is what we already do.

The CUDA-graph row is the one that decides how close multi-GPU is. Inference
engines rely on CUDA graphs for decode. A graph restored with dead communicator
handles gives a process that comes back and then fails on first collective,
which is worse than failing to restore.

To be tested separately on a single node with 2 to 4 GPUs and driver 610. A
second node is not needed and would confuse the result, since it adds network
transports and CRIU cannot checkpoint RDMA at all.

Validating a Dynamo workload

The repo already carries a deployable disaggregated topology at
examples/function-samples/helmchart-samples/dynamo-operator-sample, so we do
not need to invent one.

The trap this must avoid

Dynamo is built to survive worker loss. The router re-routes, workers
re-register. That means a failed restore can be invisible: the request still
succeeds, served by a different worker, and the test passes.

This is the same class of error as a restore test that silently measured a cold
start. The mitigation is the same. Assert the mechanism, not the outcome:

• run exactly one worker of the type under test, so there is nothing to route
  around, and
• assert the request was served by the restored worker specifically, not merely
  that a request succeeded.

Any Dynamo restore result that does not do both should be treated as unproven.

Ladder

Each rung is a stop-and-decide point, and every rung is run repeatedly rather
than once. A single pass cannot distinguish working from lucky.

0. Deploy the sample topology unchanged. Confirm the operator, etcd, NATS and
   the CRDs come up, and record what a healthy deployment looks like.
1. Aggregated, TP=1, cache-directory capture. Weights and compiled kernels only,
   no process state. Lowest risk, and it proves the plumbing.
2. Aggregated, TP=1, full criu-v2 process capture. Expected work: coordination
   re-registration, the ZMQ KV-events publisher, and NIXL agent metadata staged
   in /dev/shm.
3. Aggregated, TP above 1. Expected to be blocked by the multi-GPU section
   above. Run it to confirm the failure mode matches, not to succeed.
4. Disaggregated, capture and restore the decode worker only, with prefill left
   live. Does the restored worker re-register, and does a request that requires
   a prefill-to-decode KV transfer complete through it?
5. Disaggregated, capture both workers. This is a distributed snapshot and is
   the point at which in-flight KV transfers matter.

Checks that must pass at each rung

• The restored worker appears in coordination state, not just Running in
  Kubernetes.
• A request is served by the restored worker, verified by worker identity.
• Output is correct, not merely present.
• For rung 4 and 5, a prefill-to-decode transfer completes end to end.
• Timings are compared against a cold start of the same topology on the same
  hardware.

Prerequisite

Whether Dynamo workers rejoin cleanly after an abrupt restart is still the
question that decides how much of this is even necessary. If they do, we can
drop NIXL, ZMQ and coordination state at capture and let it re-establish, the
way external TCP is already dropped. That is answerable from the Dynamo source
and should be settled before rung 2 is designed in detail.

[balajinvda]

Validation plan written up at docs/proposals/dynamo-cr-validation.md (on branch nvsnap/pidns-dump-root, PR #1027).

Verified from the Dynamo operator source that component-level annotations on a DynamoGraphDeployment reach pod metadata: GetDCDKubeAnnotations in deploy/operator/internal/dynamo/v1beta1_helpers.go merges the alpha component annotations and podTemplate.annotations, and its comments state the destination is generated pod metadata. So nvsnap.io/restore-from on a component will trigger the webhook on an operator-created pod, which is what makes restore-in-place viable.

That matters because our test harness deletes the source pod and creates its own placeholder. Against an operator-managed workload the operator would reconcile, create a fresh cold worker, and the test would pass while proving nothing.