Problem
nvsnap behaviour is configured through roughly 40 distinct NVSNAP_* environment
variables read across the agent, webhook, restore entrypoint and helper binaries.
Thirty of them appear nowhere in the Helm chart, so the only way to learn what a
running cluster is set to is to exec into a DaemonSet and read the pod spec.
This is not theoretical. During the recent restore-reliability investigation
(#925) time was lost suspecting NVSNAP_CUDA_PARALLELISM precisely because its
effective value was not visible anywhere. A configuration surface you cannot read
is one you end up guessing about.
The most significant example is NVSNAP_CRIU_V2
(internal/agent/checkpoint_v2.go:85): which capture engine runs is decided by an
environment variable with no chart surface, no schema and no validation.
A recent change of ours added a 41st and it has already been removed
(NVSNAP_DUMP_PIDNS_ROOT, commit 625f556) rather than merged, on the grounds
that an env switch which changes what a capture contains is invisible, untyped
and outlives the experiment that introduced it. That reasoning applies more
broadly.
Proposed categories
Not every one of these is wrong. They should be sorted before anything is
changed:
-
Process-to-process wiring, legitimate as env. The agent or webhook sets these
for a child it launches; they are not user configuration.
NVSNAP_ORIG_COMMAND, NVSNAP_ORIG_CWD, NVSNAP_CHECKPOINT_HOST_PATH,
NVSNAP_CAPTURE_NODE, NVSNAP_AGENT_TOKEN, NVSNAP_VERSION.
-
Behaviour switches that change what the system does. These should be agent
flags plumbed through chart values, with the value visible in the rendered
spec.
NVSNAP_CRIU_V2, NVSNAP_DEFAULT_CAPTURE_PATH, NVSNAP_CAPTURE_PLAN,
NVSNAP_NO_OVERLAY, NVSNAP_LAZY_PAGES, NVSNAP_COMPRESS_CHECKPOINT,
NVSNAP_CRIU_V2_COMPRESS, NVSNAP_REPLAY_MOUNTS, NVSNAP_ALLOW_WHOLE_ROOTFS.
-
Tuning knobs. Chart values with documented defaults and range validation.
NVSNAP_CUDA_PARALLELISM, NVSNAP_PEER_FETCH_CONCURRENCY,
NVSNAP_PREP_DEADLINE, NVSNAP_PREP_POLL, NVSNAP_KEEPALIVE_SECONDS.
-
Debug-only. Keep as env, but name them consistently and confirm they are inert
by default.
NVSNAP_STRACE_ENABLED, NVSNAP_DEBUG_IO_URING, NVSNAP_DISABLE_IO_URING_REINIT.
-
Deployment wiring that belongs in values.
NVSNAP_L2_*, NVSNAP_REPLICATION_*, NVSNAP_BLOB_STORE_URL,
NVSNAP_CATALOG_URL, NVSNAP_WEBHOOK_AGENT_BASE_URL, NVSNAP_FSSTORE_PATH,
NVSNAP_POD_CACHE_DIR, NVSNAP_WORKLOADS_DIR.
-
Dead. Some of these are likely unused; delete rather than migrate.
Tasks
Notes
Reproduce the inventory with:
grep -rnoE 'os\.Getenv\("NVSNAP_[A-Z0-9_]+"\)' --include=*.go internal/ cmd/
The startup-logging and unknown-variable tasks are worth doing first even if the
migration is slow: both make the current state visible without changing any
behaviour.
Problem
nvsnap behaviour is configured through roughly 40 distinct
NVSNAP_*environmentvariables read across the agent, webhook, restore entrypoint and helper binaries.
Thirty of them appear nowhere in the Helm chart, so the only way to learn what a
running cluster is set to is to exec into a DaemonSet and read the pod spec.
This is not theoretical. During the recent restore-reliability investigation
(#925) time was lost suspecting
NVSNAP_CUDA_PARALLELISMprecisely because itseffective value was not visible anywhere. A configuration surface you cannot read
is one you end up guessing about.
The most significant example is
NVSNAP_CRIU_V2(
internal/agent/checkpoint_v2.go:85): which capture engine runs is decided by anenvironment variable with no chart surface, no schema and no validation.
A recent change of ours added a 41st and it has already been removed
(
NVSNAP_DUMP_PIDNS_ROOT, commit 625f556) rather than merged, on the groundsthat an env switch which changes what a capture contains is invisible, untyped
and outlives the experiment that introduced it. That reasoning applies more
broadly.
Proposed categories
Not every one of these is wrong. They should be sorted before anything is
changed:
Process-to-process wiring, legitimate as env. The agent or webhook sets these
for a child it launches; they are not user configuration.
NVSNAP_ORIG_COMMAND,NVSNAP_ORIG_CWD,NVSNAP_CHECKPOINT_HOST_PATH,NVSNAP_CAPTURE_NODE,NVSNAP_AGENT_TOKEN,NVSNAP_VERSION.Behaviour switches that change what the system does. These should be agent
flags plumbed through chart values, with the value visible in the rendered
spec.
NVSNAP_CRIU_V2,NVSNAP_DEFAULT_CAPTURE_PATH,NVSNAP_CAPTURE_PLAN,NVSNAP_NO_OVERLAY,NVSNAP_LAZY_PAGES,NVSNAP_COMPRESS_CHECKPOINT,NVSNAP_CRIU_V2_COMPRESS,NVSNAP_REPLAY_MOUNTS,NVSNAP_ALLOW_WHOLE_ROOTFS.Tuning knobs. Chart values with documented defaults and range validation.
NVSNAP_CUDA_PARALLELISM,NVSNAP_PEER_FETCH_CONCURRENCY,NVSNAP_PREP_DEADLINE,NVSNAP_PREP_POLL,NVSNAP_KEEPALIVE_SECONDS.Debug-only. Keep as env, but name them consistently and confirm they are inert
by default.
NVSNAP_STRACE_ENABLED,NVSNAP_DEBUG_IO_URING,NVSNAP_DISABLE_IO_URING_REINIT.Deployment wiring that belongs in values.
NVSNAP_L2_*,NVSNAP_REPLICATION_*,NVSNAP_BLOB_STORE_URL,NVSNAP_CATALOG_URL,NVSNAP_WEBHOOK_AGENT_BASE_URL,NVSNAP_FSSTORE_PATH,NVSNAP_POD_CACHE_DIR,NVSNAP_WORKLOADS_DIR.Dead. Some of these are likely unused; delete rather than migrate.
Tasks
NVSNAP_prefixed reads.var as a deprecated fallback for one release.
be inspected from logs rather than by exec.
NVSNAP_*variable, so a typo is loudrather than silently ignored.
Notes
Reproduce the inventory with:
The startup-logging and unknown-variable tasks are worth doing first even if the
migration is slow: both make the current state visible without changing any
behaviour.