Support in-place Helm values updates for deployed functions

[estroz]

Is this related to a problem? Please describe.

Today, an NVCF Helm function owner has only two ways to change configuration (Helm values) on a running function: scale the instances to 0 and deploy a new function version, or run a blue-green deployment. The first introduces downtime of minutes to hours depending on deploy time. The second keeps old and new versions running simultaneously, doubling GPU consumption for a period, which is untenable for full-cluster deployments.

Describe the solution you'd like

Expose Helm values updates as a first-class lifecycle operation on a deployed function, reusing existing primitives:

• NVCF API: extend the existing GPU specification update endpoint to accept a revision object carrying the new configuration (Helm values). The update is accepted any time after ICMS has accepted the initial instance request, regardless of lifecycle state, and never creates a net-new instance. The entire configuration document is the source of truth; prior values are not merged.
• Revisions: the NVCF API stores the most recent revision and the most recent successfully applied revision per GPU specification, so owners can view the current state and roll back to the last known-good revision.
• ICMS: accepts an update action carrying the new values and target instances, resolves targets, and forwards to the managing NVCA instances.
• NVCA: idempotently applies new values to the running workload via its in-place upgrade mechanism, without recreating the worker pod or NVCF infrastructure pods.
• Lifecycle: report a new "updating" instance state, semantically equivalent to active, while the rollout is in flight. Function-level status is unaffected.

This is essentially an in-place helm upgrade against the live instances, leveraging Kubernetes/operator (e.g. Dynamo Operator under First Class Operators) support for rolling updates.

Describe alternatives you've considered

• Scale instances to 0 and deploy a new function version (current guidance) - introduces downtime.
• Blue-green deployment (current guidance) - doubles GPU usage for the cutover period, untenable at full-cluster scale.

Additional context

• Scope: applies to Helm functions only; container functions are out of scope. Tasks (short-lived, non-serving) are out of scope. FCO manifest updates will be handled in a follow-up SDD.
• Constraint: updates must not change the worker sidecar contract (inference service name, port, health endpoint); such changes require a redeploy instead.

Tasks

• NVCF API: accept updated configuration on the GPU specification update endpoint
• NVCF API: allow updates once instances are past initial deploy, including while deploying
• NVCF API: generate a revision identifier and record it as the most recent revision on each update
• NVCF API: report affected instances in a new "updating" state, equivalent to active, without affecting function-level status
• NVCF API: inform ICMS of the updated GPU specification
• NVCF API: store only the most recent revision and most recent successfully applied revision per GPU specification, with read APIs to view them
• NVCF API: treat the configuration document as opaque and complete (no merge with prior values)
• ICMS: accept an update action with new values and target instances, resolve targets, forward to managing NVCA instances via a dedicated instance-update queue
• NVCA: apply new values idempotently to the running workload without recreating the worker pod or infra pods
• NVCA: ignore update targets for instances it does not manage (safe under at-least-once delivery)
• NVCA: wait for worker readiness before reporting the update as fulfilled
• Ensure updates never create a net-new instance and always target the full instance set for a GPU specification (no canary/partial targeting)
• Support manual rollback via a roll-forward GPU spec update using the last known-good revision
• Handle temporary and permanent update failure states, including revision marked failed on timeout
• Update user-facing docs for the new update/rollback flow and the "updating" instance state

---

By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.