Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion deploy/gpustack-operator/chart/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ Kubernetes: `>=1.23.0-0`
| deviceManager.tolerations | list | `[{"operator":"Exists"}]` | Tolerations for the device-manager DaemonSets. Defaults tolerate every taint so a manager schedules on any node carrying the matching accelerator. |
| deviceManager.extraArgs | list | `[]` | Extra command-line arguments appended to each device-manager container args. |
| deviceManager.env | object | `{}` | Extra environment variables for the device-manager, as a name/value map. |
| modelManager.enabled | bool | `true` | Deploy the model-manager: the CSI node plugin that delivers a Hugging Face ModelArtifact's weights from each node's cache, and its CSIDriver `model.csi.gpustack.ai`. When enabled, the worker seeds the `model-artifact-delivery-mode` Setting with `Node`. A seed fills only a Setting the cluster does not have yet, so a later upgrade never overrides an administrator's choice. Upgrading from a version without that Setting rolls every ModelDeployment on a Hugging Face artifact once, to node delivery; set the Setting to `Engine` before upgrading, or upgrade with this disabled, to avoid it. |
| modelManager.enabled | bool | `true` | Deploy the model-manager: the CSI node plugin that delivers a Hugging Face ModelArtifact's weights from each node's cache, and its CSIDriver `model.csi.gpustack.ai`. When enabled, the chart declares `GPUSTACK_MODEL_ARTIFACT_DELIVERY_MODE=Node`, so the chart owns `model-artifact-delivery-mode`. The worker reasserts `Node` on every start and reverts a runtime edit to the Setting. `modelManager.enabled` is the only signal that reaches this value; the worker flag `--model-manager-service-account` defaults to a non-empty name and cannot indicate whether the plugin is deployed. With the plugin enabled, upgrading from a version without the Setting rolls every ModelDeployment on a Hugging Face artifact once, to node delivery. |
| modelManager.rootPath | string | `"/var/lib/gpustack/models"` | The node's model cache directory, a hostPath. Mount a dedicated filesystem here: on one shared with kubelet the plugin caps its high watermark below kubelet's eviction and image collection thresholds. Changing it points the plugin at an empty cache; nothing is migrated or removed from the old path. |
| modelManager.kubeletDir | string | `"/var/lib/kubelet"` | kubelet's root directory, the convention the bundled CSI drivers follow. The plugin mounts its pods directory with Bidirectional propagation so its bind mounts reach the Pods. |
| modelManager.securePort | int | `32444` | Secure serving port of the model-manager container: metrics, readiness and liveness. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,8 @@ spec:
- name: GPUSTACK_LEDGER_RELEASE_TERMINATED_PODS
value: {{ .Values.ledgerReleaseTerminatedPods | quote }}
{{- if .Values.modelManager.enabled }}
# Seeds the delivery Setting where the cluster has none yet; a stored value always wins.
# Declares the chart-owned delivery mode; the worker asserts it on every start, reverting
# any stored value that differs from this declaration at the next worker start.
- name: GPUSTACK_MODEL_ARTIFACT_DELIVERY_MODE
value: Node
{{- end }}
Expand Down
2 changes: 1 addition & 1 deletion deploy/gpustack-operator/chart/values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -1223,7 +1223,7 @@
"properties": {
"enabled": {
"default": true,
"description": "Deploy the model-manager: the CSI node plugin that delivers a Hugging Face ModelArtifact's\nweights from each node's cache, and its CSIDriver `model.csi.gpustack.ai`.\n\nWhen enabled, the worker seeds the `model-artifact-delivery-mode` Setting with `Node`. A seed\nfills only a Setting the cluster does not have yet, so a later upgrade never overrides an\nadministrator's choice. Upgrading from a version without that Setting rolls every\nModelDeployment on a Hugging Face artifact once, to node delivery; set the Setting to `Engine`\nbefore upgrading, or upgrade with this disabled, to avoid it.",
"description": "Deploy the model-manager: the CSI node plugin that delivers a Hugging Face ModelArtifact's\nweights from each node's cache, and its CSIDriver `model.csi.gpustack.ai`.\n\nWhen enabled, the chart declares `GPUSTACK_MODEL_ARTIFACT_DELIVERY_MODE=Node`, so the chart\nowns `model-artifact-delivery-mode`. The worker reasserts `Node` on every start and reverts\na runtime edit to the Setting. `modelManager.enabled` is the only signal that reaches this\nvalue; the worker flag `--model-manager-service-account` defaults to a non-empty name and\ncannot indicate whether the plugin is deployed. With the plugin enabled, upgrading from a\nversion without the Setting rolls every ModelDeployment on a Hugging Face artifact once,\nto node delivery.",
"title": "enabled",
"type": "boolean"
},
Expand Down
12 changes: 7 additions & 5 deletions deploy/gpustack-operator/chart/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -433,11 +433,13 @@ modelManager:
# -- Deploy the model-manager: the CSI node plugin that delivers a Hugging Face ModelArtifact's
# weights from each node's cache, and its CSIDriver `model.csi.gpustack.ai`.
#
# When enabled, the worker seeds the `model-artifact-delivery-mode` Setting with `Node`. A seed
# fills only a Setting the cluster does not have yet, so a later upgrade never overrides an
# administrator's choice. Upgrading from a version without that Setting rolls every
# ModelDeployment on a Hugging Face artifact once, to node delivery; set the Setting to `Engine`
# before upgrading, or upgrade with this disabled, to avoid it.
# When enabled, the chart declares `GPUSTACK_MODEL_ARTIFACT_DELIVERY_MODE=Node`, so the chart
# owns `model-artifact-delivery-mode`. The worker reasserts `Node` on every start and reverts
# a runtime edit to the Setting. `modelManager.enabled` is the only signal that reaches this
# value; the worker flag `--model-manager-service-account` defaults to a non-empty name and
# cannot indicate whether the plugin is deployed. With the plugin enabled, upgrading from a
# version without the Setting rolls every ModelDeployment on a Hugging Face artifact once,
# to node delivery.
enabled: true
# -- The node's model cache directory, a hostPath. Mount a dedicated filesystem here: on one
# shared with kubelet the plugin caps its high watermark below kubelet's eviction and image
Expand Down
6 changes: 3 additions & 3 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- Use for: giving model weights a stable identity; node caching and delivery; prefetch, peer sync, progress views and cache operations.
- Aliases: `ModelArtifact`, `NodeModelStore`, `ModelStore`, `ModelPrefetch`, `model cache`, `Hugging Face`
- Guides: [Model Delivery](modules/model-delivery/_index.md); the artifact contract in [Model Artifact](modules/model-delivery/artifact.md); the node cache in [Node Model Store](modules/model-delivery/node-store.md); the runbook in [Model Store Operations](modules/model-delivery/operations.md); a measured example in [Model Prefetch Walkthrough](walkthroughs/model-delivery/prefetch.md).
- Specs: [model artifact](../specs/2026-09-25-model-artifact.md) · [node model store](../specs/2026-09-25-node-model-store.md) · [prefetch](../specs/2026-09-27-model-prefetch.md) · [peer sync](../specs/2026-09-27-model-peer-sync.md) · [image source](../specs/2026-09-27-model-image-source.md)
- Specs: [model artifact](../specs/2026-09-25-model-artifact.md) · [node model store](../specs/2026-09-25-node-model-store.md) · [prefetch](../specs/2026-09-27-model-prefetch.md) · [peer sync](../specs/2026-09-27-model-peer-sync.md) · [image source](../specs/2026-09-27-model-image-source.md) · [settings sync semantics](../specs/2026-10-19-settings-env-sync-and-default-tracking.md)
- Code: Hub resolution in [pkg/modelartifact](../pkg/modelartifact/) · the per-node plugin and cache in [pkg/modelmanager](../pkg/modelmanager/) · cache configuration in [pkg/modelstore](../pkg/modelstore/)
- Related: [Model Deployment](#model-deployment) (deployments mount or download artifacts) · [KV Cache](#kv-cache) (the weight identity enters KV keys) · [GPU Instances](#gpu-instances) (an Instance can mount weights)
- Skills: [gpustack-operator-e2e](../.agents/skills/gpustack-operator-e2e/SKILL.md) (node-delivery cases)
Expand Down Expand Up @@ -140,8 +140,8 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- Use for: choosing chart or image mode; vendored subcharts and their patches; vendor prerequisites; upgrade and migration paths.
- Aliases: `Helm chart`, `subchart`, `chart mode`, `image mode`, `values.yaml`
- Guides: choices in [Installation Modes](operate/installation-modes.md); node drivers in [Vendor Prerequisites](getting-started/vendor-prerequisites.md); transfer in [Migrating to Bundled Subcharts](operate/migration/to-subcharts.md).
- Specs: [bundled apps subchart split](../specs/2026-07-28-bundled-apps-subchart-split.md)
- Code: [deploy/gpustack-operator/chart](../deploy/gpustack-operator/chart/) · vendoring in [hack/deps.sh](../hack/deps.sh) · image-mode in-cluster install in [pkg/worker/kuberess](../pkg/worker/kuberess/)
- Specs: [bundled apps subchart split](../specs/2026-07-28-bundled-apps-subchart-split.md) · [settings sync semantics](../specs/2026-10-19-settings-env-sync-and-default-tracking.md)
- Code: [deploy/gpustack-operator/chart](../deploy/gpustack-operator/chart/) · vendoring in [hack/deps.sh](../hack/deps.sh) · image-mode in-cluster install in [pkg/worker/kuberess](../pkg/worker/kuberess/) · the settings sync on every worker start in [pkg/setting](../pkg/setting/) and [pkg/worker/settings](../pkg/worker/settings/)
- Related: [Development](#development) (chart targets and subchart patches) · [Devices Manager](#devices-manager) (the Device Manager DaemonSets the chart deploys)
- Skills: [gpustack-operator-chart-e2e](../.agents/skills/gpustack-operator-chart-e2e/SKILL.md) (install, rollout and uninstall verification) · [gpustack-operator-chart-subcharts-manage](../.agents/skills/gpustack-operator-chart-subcharts-manage/SKILL.md) (add, patch or bump a vendored subchart)

Expand Down
21 changes: 10 additions & 11 deletions docs/modules/model-delivery/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ hands each mount its artifact's Secret.

| Layer | Carrier | Who changes it | What |
| --- | --- | --- | --- |
| Deploy time | chart values; in image mode the worker's overlay, which sets only the switch | installer | the switch, the cache root, the kubelet directory, images, resources and placement, the delivery seed |
| Deploy time | chart values; in image mode the worker's overlay, which sets only the switch | installer | the switch, the cache root, the kubelet directory, images, resources and placement, the delivery mode |
| Cluster runtime | [Settings](../../reference/settings.md#online-adjustable-settings) in the operator namespace | administrator | delivery mode, Hub endpoint, proxy, no-proxy, CA bundle, watermarks, download concurrency and bandwidth |
| Object | the `ModelArtifact` | tenant | source, revision, patterns, Secret; immutable |

Expand All @@ -57,7 +57,7 @@ writes chooses where a node connects. The Settings this adds:

| Setting | Default | Check |
| --- | --- | --- |
| `model-artifact-delivery-mode` | `Engine`; the chart seeds `Node` with the plugin | `Engine` or `Node`; `Node` needs the CSIDriver |
| `model-artifact-delivery-mode` | `Engine`; the chart declares `Node` with the plugin, and the worker reasserts it | `Engine` or `Node`; `Node` needs the CSIDriver |
| `model-store-high-watermark` | `80` | integer, above the low watermark, at most `95` |
| `model-store-low-watermark` | `70` | integer, at least `1`, below the high watermark |
| `model-store-download-concurrency` | `8` | concurrent requests per node, `1` to `64` |
Expand Down Expand Up @@ -199,16 +199,15 @@ names refuses to start, so a Kueue upgrade checks it first.

## Upgrade notes

**From the version before node delivery.** With the plugin enabled, the upgrade seeds
`model-artifact-delivery-mode=Node`, and every `ModelDeployment` using a hub artifact rolls
once, to Node delivery. A seed fills only a Setting the cluster does not have yet, so there are two
ways to avoid the roll:
**From the version before node delivery.** With `modelManager.enabled: true`, the chart declares
`GPUSTACK_MODEL_ARTIFACT_DELIVERY_MODE=Node`. The worker reasserts that value on every start, so
a runtime edit of `model-artifact-delivery-mode` to `Engine` is reverted at the next worker start;
it does not avoid the upgrade roll. Every `ModelDeployment` using a hub artifact rolls once, to
Node delivery.

1. set the Setting to `Engine` before upgrading, which the seed then leaves alone; or
2. upgrade with `--set modelManager.enabled=false`, which seeds nothing and keeps `Engine`.

Either way you can switch later, at a time of your choosing. A later `helm upgrade` never overrides
the Setting.
To keep `Engine` through the upgrade, set `modelManager.enabled=false`; this also removes the
plugin and its CSIDriver. If you enable the plugin later, the chart declares `Node` again and the
worker writes it at its next start.

**To the version with the placement preference.** Kueue restarts with `TASRespectNodeAffinityPreferred`
on. Running replicas are not touched; new node-delivered Pods carry the preference. A Pod in a TAS
Expand Down
Loading
Loading