[ozbarshalom]

Refs #627. Targets the branch of #456. It does two things: gives the guides one flow you follow top to bottom, with the recipe chosen once at the start, and makes the eight recipes follow one scheme, so a 1 GPU and an 8 GPU run differ only where the GPU count forces it.

The flow

• Guides index is one flow. After the shared setup (install, example files, namespace, model cache), you choose a recipe once by setting FRAMEWORK, MODEL and GPUS from one table, and paste one block that works out the files and resource names. Every later command uses that choice: prepare the ConfigMap, checkpoint (Option A PodSnapshot or Option B SnapshotJob), restore, and confirm the restore by the pod's nvidia.com/Restored condition. A closing section points people with their own engine or image to the workload contract.
• No text substitution in commands. The two placeholders in the manifests (the agent image in multi-GPU source manifests, the checkpoint name in restore profiles) are filled in by editing the one line the page names and showing it, not with envsubst or sed.
• WORKLOAD_NAMESPACE names the namespace the replicas run in, which is not the namespace Snapshot is installed in.
• Multi-GPU recipes are one short section: all GPUs on one node, the agent image for a Deployment source, and the first-run time. The framework guides keep the engine-specific detail as reference.
• The multi-GPU pod shape moved to the workload contract.
• The guides have no clean-up section, no failure section and no inference request.

One scheme for every recipe

• Model cache in every recipe, one PVC named model-cache defined once in docs/guides/model-cache-pvc.yaml. The vLLM and TensorRT-LLM Qwen3 recipes gain the init container, offline env and PVC the others already had.
• Names <framework>-<model>-source and -snapshot, hyphens only (deepseek-v4.1-flash is now deepseek-v4-1-flash), and <framework>-1gpu-restored and -8gpu-restored.
• Restore profiles are 1-gpu.yaml and 8-gpu.yaml, and all carry nvidia.com/restore-from: ${SNAPSHOT_NAME}. No profile needs hand editing.
• Deadlines and probes follow the GPU class only: Deployments 1800 or 3600, SnapshotJob 3600 or 7200, readiness threshold 1800 or 3600. vLLM and SGLang had no Deployment deadline, and vLLM probed at 1200.
• TensorRT-LLM SnapshotJobs set the same fsGroup as its Deployments and restore profiles. The controller merges its seccomp setting into an existing security context, so this is safe.

Other consumers of the recipes

• Benchmarks deploy the vLLM Qwen3 recipe as a template and swap in seven other models. The engine now drops the recipe's model cache plumbing, so the pods it builds are unchanged apart from the renamed app label and the readiness threshold. I built them from the head of feat(examples): document and exercise multi-GPU workloads #456 and from this branch for three models and two pull policies and diffed them.
• e2e points every framework at the shared PVC manifest and 1-gpu.yaml. The CI shared NFS claim now defaults to shared-model-cache, so a model-cache claim from a guide-download run cannot shadow it. SNAPSHOT_E2E_MODEL_CACHE_* and the benchmark modelCacheMode key are unchanged.

How it was checked

On a B200 node, with chart 0.0.0-g7e282f0e (this branch's base, 7e282f0e) installed, where the agent registers its NRI plugin and so runs the restore proxy, using the manifests in this branch:

• Single GPU: vLLM Qwen3 0.6B, source Deployment and PodSnapshot, then restore with restore/1-gpu.yaml. PID 1 in the restored container was /snapshot-control/snapshot-restore-proxy, the pod reported nvidia.com/Restored=True, and the replica answered a /generate request.
• Two GPUs: vLLM Qwen3 0.6B at tensor-parallel 2, a copy generated from the 8-GPU GLM 5.3 manifests (so it carries the nvidia.com/snapshot-multi-gpu annotation, the snapshot-cuda-install init container and the launcher), through a SnapshotJob, then restore with the 8-GPU profile at two GPUs. Same results.

The new flow, run from the page. A fresh clone of this branch, the commands extracted from the guides index and run in its order, with the two "edit this line" steps done by changing the line the page names. Same chart, same node. Two environment deviations, both in my clone of the repository and not in any step: the model-cache claim points at an existing PVC, because this cluster's Kyverno policy rejects a private PVC named model-cache, and the ConfigMaps are named ozbarshalom-<engine>-app, because the namespace already held incompatible ones.

• Single GPU: vLLM Qwen3 0.6B, Option A (Deployment + PodSnapshot). Pass: the checkpoint became Ready, PID 1 in the restored container was /snapshot-control/snapshot-restore-proxy, and nvidia.com/Restored was True.
• Two GPUs: vLLM Qwen3 0.6B at tensor-parallel 2 (the copy generated from the 8-GPU manifests), Option B (SnapshotJob), including the multi-GPU step that reads the agent image. Pass, with the same proxy PID 1 and Restored=True. Only 2 GPUs were free, and Option A needs 2 for the source plus 2 for the restored replica at once, so this one used the SnapshotJob.
• TensorRT-LLM single GPU, after the review fix that removes TRTLLM_MAX_NUM_TOKENS and TRTLLM_MAX_BATCH_SIZE (they repeated app.py's defaults) and raises TRTLLM_FREE_GPU_MEMORY_FRACTION from 0.10 to 0.75, as in the GLM recipes: both Option A and Option B pass, with the restore proxy as PID 1 and Restored=True.
• The CRDs were updated from this chart before the runs.

Also checked:

• Consistency checks over all 24 manifests: naming, deadlines, probes, cache wiring, init container and engine agree on model and revision, the two lanes run the same workload, and no stale name is left anywhere. I mutated the recipes and docs seven ways and each check failed as it should.
• All eight SnapshotJob recipes and the guide's PodSnapshot validate against the chart's CRDs.
• The three pinned images exist and pull anonymously on amd64 and arm64. The three models are public and ungated, and their pinned revisions exist.
• The e2e unit tests that run offline (365 pass; the 2 that need a live cluster are not run), and every relative link and anchor.
• All 36 command blocks in the guides parse, and the recipe block derives the right manifests and names for all eight recipes in bash and zsh.

Not run:

• the new flow for SGLang, for TensorRT-LLM at two GPUs, and Option A at two GPUs. The earlier fake-kubectl run of 16 flows no longer applies, because the commands changed;
• the eight-GPU recipes (GLM 5.3, DeepSeek V4.1 Flash), and the model-download init container (the model was already in the cache);
• the GPU e2e suites and make check test.

The same guides for Snapshot 0.2.0 are in #658, which targets release/0.2. 0.2.0 has no restore proxy, so its restore profiles run an inert placeholder command instead of the source command. They were run there on a 0.2.0 install.

For #456, not changed here

• The README installs chart 0.2.0, which does not contain the restore proxy (the NRI plugin that makes it PID 1) or the no-proxy fallback, and the restore profiles here rely on them. The renamed nvidia.com/snapshot-multi-gpu annotation and the cuinterpose fixes are in 0.2.0. This branch is based on 7e282f0e, which is published as chart 0.0.0-g7e282f0e with matching images. The version wording needs settling before this merges.
• e2e/snapshot_e2e/upgrade/scenarios.py sets RESTORE_PROXY_RELEASE = (0, 2), "the first release whose agent restores through the restore proxy", but the v0.2.0 tag does not have it.
• docs/limitations.md still says path remapping does not "establish multi-GPU workload support", which now reads oddly.
• There is no TensorRT-LLM DeepSeek V4.1 Flash recipe, so the matrix is 2 + 3 + 3.
• The quickest recipe now needs a ReadWriteMany model cache PVC (2 TiB request), and SGLang's single-GPU recipe moves from its own 20Gi ReadWriteOnce claim to it. On a cluster whose policy provides a shared model cache, a private PVC named model-cache can be rejected (it was on the dev cluster used for testing), which fails step 4 of the guides index there.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/snapshot/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Enterprise
• Run ID: 3900aace-a5e4-409c-bf68-afad1d2c42db

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}