-
Notifications
You must be signed in to change notification settings - Fork 51
feat(self-managed): add highAvailability Helmfile schema and value mapping #996
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -24,6 +24,77 @@ tolerations: | |
| {{- end -}} | ||
| {{- end -}} | ||
|
|
||
| {{/* | ||
| highAvailability helpers. | ||
|
|
||
| global.yaml.gotmpl maps highAvailability.* onto chart values (replicaCount, | ||
| affinity, topology spread, PDB, update strategy). | ||
|
|
||
| When highAvailability.enabled is false, callers keep existing env/chart defaults. | ||
| When enabled, profile selects effective sizing: | ||
|
|
||
| production — Full P0 defaults; requires ≥3 schedulable nodes | ||
| staging — Same shape; soft anti-affinity for smaller pools | ||
| single-node — Single replicas; hard anti-affinity disabled (local / CI / BDD) | ||
|
|
||
| In-scope charts MUST expose the required value hooks. The local BDD fixture | ||
| SHALL remain on single-node. | ||
| */}} | ||
| {{- define "nvcf.ha.isMultiNode" -}} | ||
| {{- $enabled := dig "highAvailability" "enabled" false . -}} | ||
| {{- $profile := dig "highAvailability" "profile" "single-node" . -}} | ||
| {{- if and $enabled (has $profile (list "production" "staging")) -}}true{{- else -}}false{{- end -}} | ||
|
Comment on lines
+43
to
+46
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift Make the
Select Also applies to: 61-86 🤖 Prompt for AI Agents |
||
| {{- end -}} | ||
|
|
||
| {{- define "nvcf.ha.tier1ReplicaCount" -}} | ||
| {{- if eq (include "nvcf.ha.isMultiNode" .) "true" -}} | ||
| {{- dig "highAvailability" "tier1" "replicaCount" 2 . -}} | ||
| {{- else -}} | ||
| 1 | ||
| {{- end -}} | ||
| {{- end -}} | ||
|
|
||
| {{/* Soft/hard pod anti-affinity on hostname for a Helm release instance name. | ||
| Context: dict "Values" $.Values "instance" "<release-instance>" */}} | ||
| {{- define "nvcf.ha.tier1Affinity" -}} | ||
| {{- if eq (include "nvcf.ha.isMultiNode" .Values) "true" -}} | ||
| {{- $paa := dig "highAvailability" "tier1" "podAntiAffinity" dict .Values -}} | ||
| {{- if dig "enabled" true $paa -}} | ||
| {{- $type := dig "type" "preferred" $paa -}} | ||
| affinity: | ||
| podAntiAffinity: | ||
| {{- if eq $type "required" }} | ||
| requiredDuringSchedulingIgnoredDuringExecution: | ||
| - labelSelector: | ||
| matchExpressions: | ||
| - key: app.kubernetes.io/instance | ||
| operator: In | ||
| values: | ||
| - {{ .instance | quote }} | ||
| topologyKey: kubernetes.io/hostname | ||
| {{- else }} | ||
| preferredDuringSchedulingIgnoredDuringExecution: | ||
| - weight: 100 | ||
| podAffinityTerm: | ||
| labelSelector: | ||
| matchExpressions: | ||
| - key: app.kubernetes.io/instance | ||
| operator: In | ||
| values: | ||
| - {{ .instance | quote }} | ||
| topologyKey: kubernetes.io/hostname | ||
| {{- end }} | ||
| {{- end -}} | ||
| {{- end -}} | ||
| {{- end -}} | ||
|
|
||
| {{- $haEnabled := dig "highAvailability" "enabled" false .Values }} | ||
| {{- $haProfile := dig "highAvailability" "profile" "single-node" .Values }} | ||
| {{- if and $haEnabled (not (has $haProfile (list "production" "staging" "single-node"))) }} | ||
| {{- fail (printf "highAvailability.profile must be production, staging, or single-node, got %q" $haProfile) }} | ||
| {{- end }} | ||
| {{- $haMultiNode := eq (include "nvcf.ha.isMultiNode" .Values) "true" }} | ||
|
|
||
| cassandra: | ||
| global: | ||
| {{- if .Values.global.imagePullSecrets }} | ||
|
|
@@ -36,13 +107,20 @@ cassandra: | |
| defaultStorageClass: {{ .Values.global.storageClass }} | ||
| {{- end }} | ||
|
|
||
| replicaCount: {{ dig "cassandra" "replicaCount" 3 .Values }} | ||
| replicaCount: {{ if $haEnabled }}{{ if $haMultiNode }}{{ dig "highAvailability" "cassandra" "replicaCount" 3 .Values }}{{ else }}1{{ end }}{{ else }}{{ dig "cassandra" "replicaCount" 3 .Values }}{{ end }} | ||
| resourcesPreset: {{ dig "cassandra" "resourcesPreset" "xlarge" .Values }} | ||
|
|
||
| {{- if and $haEnabled $haMultiNode }} | ||
| {{- with dig "highAvailability" "cassandra" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- else }} | ||
| {{- with dig "cassandra" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- end }} | ||
|
|
||
| {{- with include "nvcf.nodeSelector" (dict "type" "cassandra" "selectors" .Values.global.nodeSelectors) }} | ||
| {{- . | nindent 2 }} | ||
|
|
@@ -140,7 +218,7 @@ openbao: | |
| {{- with include "nvcf.tolerations" (dict "type" "vault" "tolerations" .Values.global.tolerations) }} | ||
| {{- . | nindent 4 }} | ||
| {{- end }} | ||
| replicas: {{ .Values.openbao.injector.replicas }} | ||
| replicas: {{ if $haEnabled }}{{ if $haMultiNode }}{{ dig "highAvailability" "openbao" "injector" "replicas" 2 .Values }}{{ else }}1{{ end }}{{ else }}{{ .Values.openbao.injector.replicas }}{{ end }} | ||
| {{- $nvcfUiEnabled := dig "addons" "nvcfUi" "enabled" false .Values }} | ||
| {{- with dig "openbao" "injector" "webhook" dict .Values }} | ||
| webhook: | ||
|
|
@@ -178,6 +256,10 @@ openbao: | |
| size: {{ .Values.global.storageSize | default "10Gi" }} | ||
|
|
||
| ha: | ||
| {{- if $haEnabled }} | ||
| enabled: {{ dig "highAvailability" "openbao" "ha" "enabled" true .Values }} | ||
| replicas: {{ if $haMultiNode }}{{ dig "highAvailability" "openbao" "ha" "replicas" 3 .Values }}{{ else }}1{{ end }} | ||
| {{- end }} | ||
| {{- with dig "openbao" "server" "ha" "disruptionBudget" dict .Values }} | ||
| disruptionBudget: | ||
| {{- toYaml . | nindent 8 }} | ||
|
|
@@ -228,16 +310,33 @@ nats: | |
|
|
||
| {{- if .Values.global.storageClass }} | ||
| config: | ||
| {{- if $haEnabled }} | ||
| cluster: | ||
| enabled: true | ||
| replicas: {{ if $haMultiNode }}{{ dig "highAvailability" "nats" "replicas" 3 .Values }}{{ else }}1{{ end }} | ||
| {{- end }} | ||
| jetstream: | ||
| fileStore: | ||
| pvc: | ||
| storageClassName: {{ .Values.global.storageClass }} | ||
| {{- else if $haEnabled }} | ||
| config: | ||
| cluster: | ||
| enabled: true | ||
| replicas: {{ if $haMultiNode }}{{ dig "highAvailability" "nats" "replicas" 3 .Values }}{{ else }}1{{ end }} | ||
| {{- end }} | ||
|
|
||
| {{- if and $haEnabled $haMultiNode }} | ||
| {{- with dig "highAvailability" "nats" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- else }} | ||
| {{- with dig "nats" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- end }} | ||
|
|
||
| apikeys: | ||
| fullnameOverride: api-keys | ||
|
|
@@ -342,6 +441,9 @@ natsAuthCalloutService: | |
| {{- $nvctApiGrpcRouteHostnames := dig "ingress" "gatewayApi" "routes" "nvctApi" "grpc" "hostnames" (list) .Values | default (list (printf "tasks-grpc.%s" .Values.global.domain)) }} | ||
| api: | ||
| fullnameOverride: nvcf-api | ||
| {{- if $haEnabled }} | ||
| replicaCount: {{ include "nvcf.ha.tier1ReplicaCount" .Values }} | ||
| {{- end }} | ||
| {{- if .Values.global.imagePullSecrets }} | ||
| imagePullSecrets: | ||
| {{- toYaml .Values.global.imagePullSecrets | nindent 4 }} | ||
|
|
@@ -360,6 +462,9 @@ api: | |
| {{- with include "nvcf.tolerations" (dict "type" "controlplane" "tolerations" .Values.global.tolerations) }} | ||
| {{- . | nindent 2 }} | ||
| {{- end }} | ||
| {{- with include "nvcf.ha.tier1Affinity" (dict "Values" .Values "instance" "api") }} | ||
| {{- . | nindent 2 }} | ||
| {{- end }} | ||
| {{- if $llmEnabled }} | ||
| remoteConfig: | ||
| configData: | ||
|
|
@@ -436,6 +541,9 @@ api: | |
|
|
||
| invocation: | ||
| fullnameOverride: invocation-service | ||
| {{- if $haEnabled }} | ||
| replicaCount: {{ include "nvcf.ha.tier1ReplicaCount" .Values }} | ||
| {{- end }} | ||
| {{- if .Values.global.imagePullSecrets }} | ||
| imagePullSecrets: | ||
| {{- toYaml .Values.global.imagePullSecrets | nindent 4 }} | ||
|
|
@@ -449,6 +557,9 @@ invocation: | |
| {{- with include "nvcf.tolerations" (dict "type" "controlplane" "tolerations" .Values.global.tolerations) }} | ||
| {{- . | nindent 2 }} | ||
| {{- end }} | ||
| {{- with include "nvcf.ha.tier1Affinity" (dict "Values" .Values "instance" "invocation-service") }} | ||
| {{- . | nindent 2 }} | ||
| {{- end }} | ||
|
|
||
| env: | ||
| # Observability | ||
|
|
@@ -470,10 +581,17 @@ invocation: | |
| baggageAttributeAllowlist: | ||
| {{- toYaml . | nindent 6 }} | ||
| {{- end }} | ||
| {{- if and $haEnabled $haMultiNode }} | ||
| {{- with dig "highAvailability" "tier1" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- else }} | ||
| {{- with dig "invocation" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- end }} | ||
|
|
||
| nvctApi: | ||
| fullnameOverride: nvct-api | ||
|
|
@@ -515,6 +633,9 @@ nvctApi: | |
|
|
||
| grpcproxy: | ||
| fullnameOverride: grpc-proxy | ||
| {{- if $haEnabled }} | ||
| replicaCount: {{ include "nvcf.ha.tier1ReplicaCount" .Values }} | ||
| {{- end }} | ||
| {{- if .Values.global.imagePullSecrets }} | ||
| imagePullSecrets: | ||
| {{- toYaml .Values.global.imagePullSecrets | nindent 4 }} | ||
|
|
@@ -528,6 +649,13 @@ grpcproxy: | |
| {{- with include "nvcf.tolerations" (dict "type" "controlplane" "tolerations" .Values.global.tolerations) }} | ||
| {{- . | nindent 2 }} | ||
| {{- end }} | ||
| {{- if $haMultiNode }} | ||
| {{- $grpcAffinity := include "nvcf.ha.tier1Affinity" (dict "Values" .Values "instance" "grpc-proxy") }} | ||
| {{- if $grpcAffinity }} | ||
| deployment: | ||
| {{- $grpcAffinity | nindent 4 }} | ||
| {{- end }} | ||
| {{- end }} | ||
| {{- with $grpcProxyWorkerConnectBaseURL }} | ||
| workerConnectBaseURL: {{ . | quote }} | ||
| {{- end }} | ||
|
|
@@ -549,10 +677,17 @@ grpcproxy: | |
| RATE_LIMIT_ENABLED: "true" | ||
| RATE_LIMIT_ADDR: "http://ratelimiter.nvcf.svc.cluster.local:7777" | ||
| {{- end }} | ||
| {{- if and $haEnabled $haMultiNode }} | ||
| {{- with dig "highAvailability" "tier1" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- else }} | ||
| {{- with dig "grpcproxy" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- end }} | ||
|
|
||
| rateLimiter: | ||
| {{- if .Values.global.imagePullSecrets }} | ||
|
|
@@ -667,6 +802,9 @@ sis: | |
|
|
||
| adminIssuerProxy: | ||
| fullnameOverride: admin-token-issuer-proxy | ||
| {{- if $haEnabled }} | ||
| replicaCount: {{ include "nvcf.ha.tier1ReplicaCount" .Values }} | ||
| {{- end }} | ||
| {{- if .Values.global.imagePullSecrets }} | ||
| imagePullSecrets: | ||
| {{- toYaml .Values.global.imagePullSecrets | nindent 4 }} | ||
|
|
@@ -680,17 +818,27 @@ adminIssuerProxy: | |
| {{- with include "nvcf.tolerations" (dict "type" "controlplane" "tolerations" .Values.global.tolerations) }} | ||
| {{- . | nindent 2 }} | ||
| {{- end }} | ||
| {{- with include "nvcf.ha.tier1Affinity" (dict "Values" .Values "instance" "admin-token-issuer-proxy") }} | ||
| {{- . | nindent 2 }} | ||
| {{- end }} | ||
| gateway: | ||
| enabled: true | ||
| namespace: {{ .Values.ingress.gatewayApi.gateways.shared.namespace }} | ||
| gatewayRef: | ||
| name: {{ required "ingress.gatewayApi.gateways.shared.name is required" .Values.ingress.gatewayApi.gateways.shared.name }} | ||
| hostname: "api-keys.{{ .Values.global.domain }}" | ||
| path: "/v1/admin/keys" | ||
| {{- if and $haEnabled $haMultiNode }} | ||
| {{- with dig "highAvailability" "tier1" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- else }} | ||
| {{- with dig "adminIssuerProxy" "podDisruptionBudget" dict .Values }} | ||
| podDisruptionBudget: | ||
| {{- toYaml . | nindent 4 }} | ||
| {{- end }} | ||
| {{- end }} | ||
|
|
||
| stateMetrics: | ||
| enabled: false | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 3271
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 50368
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 50367
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 4401
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 43971
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 36370
🏁 Script executed:
Repository: NVIDIA/nvcf
Length of output: 50367
Map or remove the unused durability settings.
global.yaml.gotmpldoes not mapreplicationFactor,replicationStrategy,consistencyLevel, orjetstream.replicaFactor.The Cassandra init hook hard-codes
NetworkTopologyStrategyand derives replication fromcassandra.replicaCount. NATS stream creation usesNatsProperties.replicas, notjetstream.replicaFactor.Map these settings to their consumers, or remove them from the public configuration. Add render assertions. Update the relevant architecture or sequence diagram if the mapping changes runtime behavior.
🤖 Prompt for AI Agents