First Class Operators in NVCF: support Dynamo/NIM Operator CRDs as function/task workload types

[estroz]

Is this related to a problem? Please describe.

NVCF function and task deployment is scoped to container images or Helm charts, both of which NVCF renders and applies end to end. Kubernetes operators such as Dynamo Operator and NIM Operator work differently: a long-lived cluster process watches CRD objects and autonomously creates workloads on behalf of users, with its own control loop and broader cluster API access than NVCF grants today. There is currently no first-class way for a function or task owner to deploy a workload driven by one of these operators without hand-packaging the operator's custom resources into a Helm chart, and NVCF has no visibility into or control over objects an operator creates asynchronously after that chart is applied.

Describe the solution you'd like

Support Dynamo Operator and NIM Operator as first-class NVCF workload types, covering:

• A direct manifest submission path so function/task owners can submit operator custom resources directly in the create/deploy API, instead of only via Helm chart packaging.
• NVCA MiniService controller support for an "operator mode" that applies operator manifests, watches both those objects and the secondary objects the operator's control loop creates, and aggregates their readiness into instance status.
• Metadata injection (labels, annotations, env vars) onto objects the operator creates on NVCF's behalf, since NVCA does not create them directly.
• Idempotent, in-place updates to a running instance's operator configuration (the kubectl apply pattern), instead of only full redeploys.
• Cluster-side guardrails: single-tenancy enforcement for any cluster hosting operator workloads, and a formal onboarding process (including a hard no-wildcard RBAC gate) before a new operator can be enabled in any NVCF cluster.

Todo

• NVCF/NVCT API: accept operatorTypes and manifests fields on function/task creation, exclusive with container image/Helm chart fields
• NVCF/NVCT API: accept manifestMergePatches and serviceName/servicePort on function/task deployment
• NVCF/NVCT API: validate operator manifests against a per-operator-type CRD kind allowlist built from configuration
• NVCF API: support the GPU specification update endpoint accepting manifestMergePatches for idempotent instance updates (builds on Support in-place Helm values updates for deployed functions #1013)
• NVCF/NVCT configuration: registry of supported operator types (name, validation policy, extra Kubernetes types), starting with Dynamo and NIM
• NVCA: dynamic GVK status mapping via the Karta library for operator-managed objects (already wired in for Dynamo)
• NVCA: extend the MiniService CRD with operatorTypes, manifests, manifestMergePatches spec fields
• NVCA: MiniService controller operator mode - apply merged manifests via Server-Side Apply and aggregate readiness of operator-created objects
• NVCA: extend the mutating admission webhook to inject NVCF labels/annotations/env vars onto objects created by operator control loops (webhook server and generic pod-spec mutation already exist and can be extended)
• Helm ReVal: accept and structurally validate operator manifests and merge patches against configured CRD schemas
• Helm ReVal/NVCA: merge manifestMergePatches into base manifests (JSON Merge Patch) prior to apply
• ICMS: accept and pass through new launch specification fields (OperatorTypes, Manifests, ServiceName, ServicePort, OperatorVersions, ManifestMergePatches)
• ICMS: temporary cluster-attribute-based filtering per operator type (e.g. DynamoOperatorSupport, NIMOperatorSupport) as a stopgap until validation policy support lands
• Validation policy support in ICMS and NVCF/NVCT for cluster selection by operator type (see Helm Validation Policy for BYOC clusters #880)
• Single-tenancy enforcement mechanism for dataplane clusters hosting operator workloads
• Operator RBAC governance: onboarding review process with a hard no-wildcard-RBAC gate
• Operator onboarding template/checklist document (install, manage, observability requirements)
• Self-hosted Dynamo Operator installation via Helmfile (disabled by default)
• Self-hosted NIM Operator installation via Helmfile
• Managed cluster installation and SBOM configuration for supported operators
• Observability: metrics for MiniService controller operator mode and the webhook (phase transitions, install duration, secondary object readiness ratio)
• Troubleshooting playbooks per operator (operator pod not running, MiniService stuck installing, PSS violation, single-tenancy violation, webhook unavailable)
• Documentation for operator-based function/task creation and deployment

Describe alternatives you've considered

Installing one operator per workload namespace was considered and rejected: operators are slow to start (adding deployment overhead), their CRDs are cluster-scoped and cannot be independently versioned per namespace, and not all operators support this topology (Dynamo does not recommend it). Continuing to require Helm-chart packaging of operator CRDs remains an option for operators that don't need first-class treatment, but does not give NVCF visibility into or control over objects the operator's control loop creates.

Additional context

Related open issues: #113 (Dynamo Operator integration guide), #880 (Helm Validation Policy for BYOC clusters - the validation-policy dependency this work builds on), #969 (first-class BYOO chart configuration), #983 (Dynamo/NVCF responsibility boundary diagram), #1013 (in-place Helm values updates for deployed functions - the idempotent update mechanism operator manifest patches build on).

---

By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.