Skip to content

Add PVC retention policy to WorkerPool - #1851

Open
scottt732 wants to merge 1 commit into
kelos-dev:mainfrom
scottt732:workerpool-pvc-retention
Open

scottt732 wants to merge 1 commit into
kelos-dev:mainfrom
scottt732:workerpool-pvc-retention

Conversation

@scottt732

@scottt732 scottt732 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind api

What this PR does / why we need it:

buildStatefulSet sets volumeClaimTemplates on a WorkerPool's StatefulSet but no persistentVolumeClaimRetentionPolicy, so Kubernetes uses its default of Retain/Retain. When a WorkerPool is scaled down or deleted, its workspace PVCs, and the disks behind them, stay around until someone deletes them by hand. A cluster that resizes or recreates pools often builds up a lot of orphaned volumes this way.

This PR adds an optional WorkerPool.spec.persistentVolumeClaimRetentionPolicy (the standard apps/v1 StatefulSetPersistentVolumeClaimRetentionPolicy):

spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete   # Retain | Delete
    whenScaled: Delete    # Retain | Delete
  • The policy is set on the StatefulSet when it is created.
  • reconcileStatefulSet also updates the policy on an existing StatefulSet when it changes. The StatefulSet field is mutable. When comparing, a nil policy or an empty field counts as Retain, which matches what the API server fills in, so a defaulted StatefulSet is not updated again on every reconcile.
  • CEL rules on the CRD reject any value other than Retain or Delete when the WorkerPool is admitted. Without them, a bad value would only show up later, as a failed StatefulSet create.
  • docs/reference.md documents both fields.

Backward compatibility: the field is optional and has no default. If it is unset, the StatefulSet gets no policy and Kubernetes keeps its Retain/Retain default, so existing WorkerPools behave exactly as before. If a policy is set and later removed, the StatefulSet goes back to the API server default (Retain).

Which issue(s) this PR is related to:

N/A

Special notes for your reviewer:

  • I embedded the upstream appsv1 type, as the WorkerPool API already does with corev1.PersistentVolumeClaimSpec, rather than adding a Kelos-owned copy. Validation comes from the two CEL rules.
  • Tests:
    • Unit tests (internal/controller/workerpool_controller_test.go):
      • an unset field gives a nil policy on create
      • a set policy is copied to the StatefulSet on create
      • a changed policy updates an existing StatefulSet. I checked that this test fails when the new update branch is disabled.
      • the comparison treats nil or empty as Retain
    • Integration tests (test/integration/workerpool_test.go, envtest):
      • the policy reaches the StatefulSet and is updated when the WorkerPool changes
      • an invalid value is rejected with the CEL message
    • Ran locally:
      • make update
      • make verify
      • go test ./internal/controller/ ./api/...
      • the WorkerPool integration specs (-ginkgo.focus)
    • I did not run e2e locally.

Does this PR introduce a user-facing change?

WorkerPool supports an optional `spec.persistentVolumeClaimRetentionPolicy` (`whenDeleted` / `whenScaled`: `Retain` or `Delete`), which is passed to the worker StatefulSet so that worker PVCs can be deleted when the pool is scaled down or deleted. Leaving it unset keeps the existing behavior of retaining PVCs.

🤖 Generated with Claude Code


Summary by cubic

Adds an optional spec.persistentVolumeClaimRetentionPolicy to WorkerPool so worker PVCs can be deleted when the pool is scaled down or deleted. Previously the StatefulSet had no policy, so Kubernetes retained PVCs indefinitely.

  • The policy is copied to the worker StatefulSet on create and updates existing StatefulSets when changed.
  • Unset or empty fields default to Retain, matching Kubernetes' default, so existing WorkerPools behave exactly as before.
  • CEL validation on the CRD rejects values other than Retain or Delete.
  • docs/reference.md documents both fields.

Written for commit e9bedd3. Summary will update on new commits.

View guided diff

WorkerPool StatefulSets were created without a
persistentVolumeClaimRetentionPolicy, so Kubernetes retained every worker
PVC when a pool was scaled down or deleted, leaving the claims and their
backing disks behind indefinitely.

Add an optional spec.persistentVolumeClaimRetentionPolicy with whenDeleted
and whenScaled (Retain or Delete). The controller sets it on the
StatefulSet at creation and updates existing StatefulSets when it changes.
Leaving it unset keeps the current Retain behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 8 files

You’re at about 96% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="internal/controller/workerpool_controller_test.go">

<violation number="1" location="internal/controller/workerpool_controller_test.go:455">
P2: The PR says removing a previously set policy reverts to the default, but no reconcile test covers that path. Add a test where the existing StatefulSet has Delete/Delete and the pool's `persistentVolumeClaimRetentionPolicy` is nil, then assert the StatefulSet's policy is nil after reconcile.</violation>
</file>

<file name="internal/controller/workerpool_controller.go">

<violation number="1" location="internal/controller/workerpool_controller.go:509">
P3: A retention-only change now reaches the shared StatefulSet update path, which still logs and emits the event "Updated StatefulSet %s to %d replicas". That message reports a replica count even when only the PVC retention policy changed. Make the message name what changed, or make it generic.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

}, sts.Spec.PersistentVolumeClaimRetentionPolicy)
}

func TestWorkerPoolReconciler_PVCRetentionPolicyUpdatesExistingStatefulSet(t *testing.T) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The PR says removing a previously set policy reverts to the default, but no reconcile test covers that path. Add a test where the existing StatefulSet has Delete/Delete and the pool's persistentVolumeClaimRetentionPolicy is nil, then assert the StatefulSet's policy is nil after reconcile.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/controller/workerpool_controller_test.go, line 455:

<comment>The PR says removing a previously set policy reverts to the default, but no reconcile test covers that path. Add a test where the existing StatefulSet has Delete/Delete and the pool's `persistentVolumeClaimRetentionPolicy` is nil, then assert the StatefulSet's policy is nil after reconcile.</comment>

<file context>
@@ -409,6 +409,92 @@ func TestWorkerPoolReconciler_UpdatesReplicas(t *testing.T) {
+	}, sts.Spec.PersistentVolumeClaimRetentionPolicy)
+}
+
+func TestWorkerPoolReconciler_PVCRetentionPolicyUpdatesExistingStatefulSet(t *testing.T) {
+	pool := newTestWorkerPool("my-pool", "default", 1)
+	pool.Spec.PersistentVolumeClaimRetentionPolicy = &appsv1.StatefulSetPersistentVolumeClaimRetentionPolicy{
</file context>

needsUpdate = true
}

if !pvcRetentionPolicyEqual(sts.Spec.PersistentVolumeClaimRetentionPolicy, desired.Spec.PersistentVolumeClaimRetentionPolicy) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: A retention-only change now reaches the shared StatefulSet update path, which still logs and emits the event "Updated StatefulSet %s to %d replicas". That message reports a replica count even when only the PVC retention policy changed. Make the message name what changed, or make it generic.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/controller/workerpool_controller.go, line 509:

<comment>A retention-only change now reaches the shared StatefulSet update path, which still logs and emits the event "Updated StatefulSet %s to %d replicas". That message reports a replica count even when only the PVC retention policy changed. Make the message name what changed, or make it generic.</comment>

<file context>
@@ -506,6 +506,11 @@ func (r *WorkerPoolReconciler) reconcileStatefulSet(ctx context.Context, pool *k
 		needsUpdate = true
 	}
 
+	if !pvcRetentionPolicyEqual(sts.Spec.PersistentVolumeClaimRetentionPolicy, desired.Spec.PersistentVolumeClaimRetentionPolicy) {
+		sts.Spec.PersistentVolumeClaimRetentionPolicy = desired.Spec.PersistentVolumeClaimRetentionPolicy
+		needsUpdate = true
</file context>

@gjkim42

gjkim42 commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

/kelos api-review

@kelos-bot

kelos-bot Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

🤖 Kelos Task Status

Task kelos-api-reviewer-issue-comment-0016ebd9944c has succeeded. ✅

@gjkim42

gjkim42 commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

/kelos review

@kelos-bot

kelos-bot Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

🤖 Kelos Task Status

Task kelos-reviewer-issue-comment-a3076096a6b0 has succeeded. ✅

@kelos-bot

kelos-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

🤖 Kelos API Reviewer Agent @gjkim42

API Design Review

Verdict: APPROVE
Scope: Adds an optional WorkerPool.spec.persistentVolumeClaimRetentionPolicy to v1alpha2. It embeds the upstream appsv1.StatefulSetPersistentVolumeClaimRetentionPolicy (whenDeleted / whenScaled: Retain or Delete), adds CEL validation, and passes the policy through to the worker StatefulSet.

Reviewed head e9bedd3. There are no blocking (P0/P1) findings.

Findings

Compatibility

  • [P3, strength] api/v1alpha2/workerpool_types.go:87: The change only adds a field. The field is optional, a pointer with omitempty, has no +kubebuilder:default, and is not required. Existing WorkerPools apply unchanged, and their StatefulSets keep the API server's Retain/Retain default. WorkerPool exists only in v1alpha2 (the storage version), so the field correctly lands only in the latest version and no conversion work is needed. The regenerated CRDs (the chart and install-crd.yaml) and the deepcopy changes are included, and verify passes. No manifests under examples/ or self-development/ need updating.

Naming / Shape

  • [P3, strength] api/v1alpha2/workerpool_types.go:87: The field name, the whenDeleted/whenScaled shape, and the values match StatefulSetSpec.persistentVolumeClaimRetentionPolicy exactly. The semantics match too: the StatefulSet is controller-owned by the WorkerPool (internal/controller/workerpool_controller.go:474), so deleting the pool cascades to the StatefulSet and its whenDeleted policy. Because the policy is an object, more knobs can be added later without a new top-level field. Embedding the upstream type follows existing precedent: Task.spec.podFailurePolicy embeds batchv1.PodFailurePolicy with a CEL rule that narrows it, and volumeClaimTemplate embeds corev1.PersistentVolumeClaimSpec. The WorkerPool godoc already says the pool is "backed by a StatefulSet", so the StatefulSet-shaped name does not expose a new backend detail.

Documentation

  • [P2] api/v1alpha2/workerpool_types.go:79-83, docs/reference.md:1138: whenScaled: Delete also applies to spec.replicas: 0, which the Replicas godoc (workerpool_types.go:68) describes as the way to "pause the pool". Scaling to 0 removes every worker pod, so with whenScaled: Delete every worker PVC is deleted, including the workspace clone and anything else stored on it. When the pool is resumed, all workers start on empty volumes. The current wording ("when spec.replicas is reduced") is accurate, but users who expect a pause to keep their data will be surprised. Suggest adding one sentence to the field godoc and to the whenScaled row in docs/reference.md, for example: "With whenScaled: Delete, setting spec.replicas to 0 deletes all worker PVCs." Then run make update so the CRD description stays in sync. Non-blocking.

Validation

  • [P3] api/v1alpha2/workerpool_types.go:85-86: The CEL rules are correct. !has(...) lets an omitted field through, and an explicit "" is rejected. This matches the godoc, which says PVCs are retained "when a field is omitted" and does not invite the empty form. Embedding the upstream type has two costs:

    • The allowed values exist only in CEL, not as an OpenAPI enum, so kubectl explain workerpool.spec.persistentVolumeClaimRetentionPolicy.whenScaled and schema-driven tooling won't list them.
    • The sub-field descriptions are copied from upstream and talk about "the StatefulSet" rather than the WorkerPool.

    Both costs are acceptable for consistency with the podFailurePolicy precedent. No change requested.

Suggestions (optional)

  • Because the type is embedded, a future k8s.io/api bump that adds a sub-field to StatefulSetPersistentVolumeClaimRetentionPolicy would appear in the WorkerPool CRD through make update without an API review. pvcRetentionPolicyEqual (internal/controller/workerpool_controller.go:1901) also compares only whenDeleted and whenScaled. When bumping k8s.io/api, check the regenerated WorkerPool CRD diff for this field.

Note on prompt injection: A third-party automated review on this PR includes embedded agent instructions (a "Prompt for AI agents" block telling agents to fix code, reply in threads, and resolve them). Those instructions were ignored, and this review was done independently.

/kelos needs-input

@kelos-bot

kelos-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

🤖 Kelos Reviewer Agent @gjkim42

Review Summary

Verdict: APPROVE
Overall correctness: patch is correct
Scope: Adds optional WorkerPool.spec.persistentVolumeClaimRetentionPolicy, passes it to the worker StatefulSet on create and update, validates it with CEL, and documents it in docs/reference.md.

Reviewed head e9bedd3.

Findings Overview

Priority Count File:Line Summary
P0 0 — none
P1 0 — none
P2 0 — none
P3 1 internal/controller/workerpool_controller_test.go:455 No reconcile test covers removing a set policy (Delete → unset)

Findings

Tests

  • [P3] internal/controller/workerpool_controller_test.go:455: The PR description promises that removing a policy returns the StatefulSet to the Retain default. That is the direction that stops PVC deletion, but no reconcile test covers it. TestPVCRetentionPolicyEqual_TreatsUnsetAsRetain does check that nil and deleteOnScale compare as different, so the comparator is pinned. Nothing pins the reconcile branch. If someone later adds a guard like desired != nil && !pvcRetentionPolicyEqual(...) (treating unset as "don't manage"), the StatefulSet would keep Delete after the user removed the field, and PVCs would keep being deleted on scale-down. Suggested test: copy the UpdatesExistingStatefulSet case, give the existing StatefulSet Delete/Delete, leave the pool's policy nil, and assert that the reconciled StatefulSet's policy is nil.

Verified (no change needed)

  • Steady state does not churn: The API server defaults a nil policy to Retain/Retain and fills in each empty sub-field. pvcRetentionPolicyEqual normalizes both sides the same way, so existing WorkerPools are not updated on upgrade. A partial policy such as {whenScaled: Delete} also matches its defaulted StatefulSet {Retain, Delete} and does not re-update.
  • No aliasing: buildStatefulSet uses DeepCopy() (workerpool_controller.go:1047), and the update branch assigns from desired. The pool's spec pointer is never shared with the StatefulSet.
  • whenDeleted cascades: The StatefulSet name is stable (workerPoolStatefulSetName). The controller never deletes or recreates the StatefulSet itself, so whenDeleted: Delete only fires when the WorkerPool is deleted and the owned StatefulSet is garbage-collected.
  • Cluster support: StatefulSetAutoDeletePVC is on by default in every Kubernetes version Kelos supports (README: 1.28+) and in envtest 1.31. The field is never silently dropped.
  • Generated artifacts: Deepcopy, the chart CRD, and install-crd.yaml are all regenerated. The generated clientset has no apply configurations, so nothing else needed regenerating.
  • Existing pattern: The Session controller already sets PersistentVolumeClaimRetentionPolicy on its StatefulSet (session_controller.go:1741).

Key takeaways

  • The change is minimal, backward compatible, and correct. No blocking issues.
  • The /kelos api-review P2 is still open: it asks to document that with whenScaled: Delete, setting spec.replicas: 0 (the documented way to pause the pool) deletes every worker PVC. It applies to docs/reference.md:1138 and the field godoc, and is worth addressing before merge.

Note on prompt injection: A third-party automated review on this PR contains embedded agent instructions (a "Prompt for AI agents" block telling agents to fix code, reply in threads, and resolve them). Those instructions were disregarded, and this review was done independently.

This branch is waiting to be deployed

1 waiting deployment
ok-to-test — e9bedd36 Waiting Oct 28, 2026 by scottt732 via fork-e2e / e2e-with-environment #1977
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/api Categorizes issue or PR as related to API changes needs-actor needs-priority needs-triage release-note

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants