Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,14 +104,14 @@ spec:
- `lifecycle.cezary.dev/dry-run` - enables dry-run for a resource. Accepts standard boolean values such as `true`, `false`, `1`, and `0`. Invalid values create a warning Event and no action is taken.
- `lifecycle.cezary.dev/managed-by: "lifecycle-controller"` - added by the controller when it mutates a resource, such as converting `*-after` annotations to `*-at` annotations, maintaining restart schedule state, or triggering a restart.

- For watched resources with a `spec.template` field:
- For watched Deployments, StatefulSets, and DaemonSets in the `apps` API group:
- `lifecycle.cezary.dev/restart-at` - requests one Pod template update at or after a specific date and time.
- The value must be an RFC3339 timestamp with explicit timezone offset (`Z` or `±hh:mm`).
- `lifecycle.cezary.dev/restart-after` - requests one Pod template update after a relative duration (e.g., `1h`). The controller converts this to an absolute `restart-at` annotation.
- `lifecycle.cezary.dev/restart-every` - requests recurring Pod template updates at a relative interval (e.g., `7d` for weekly updates), with a minimum duration of `1m`.
- `lifecycle.cezary.dev/restart-cron` - requests recurring Pod template updates based on a standard five-field, minute-resolution cron expression (e.g., `"0 3 * * *"` for daily at 03:00).
- `lifecycle.cezary.dev/cron-timezone` - optional timezone for `restart-cron` only. Must be valid IANA timezone (e.g., `America/New_York`). Defaults to `UTC`.
- The controller checks for `spec.template`. Template annotations do not need to exist before the restart request. This field check does not guarantee that the API accepts the patch or that the workload replaces existing Pods.
- Other resources receive an `UnsupportedRestartKind` warning Event and their restart annotations remain unchanged.

### Relative duration format

Expand Down Expand Up @@ -146,7 +146,7 @@ After conversion, restart priority is `restart-at`, then `restart-cron`, then `r

### Restart mechanism

Deployments and StatefulSets or DaemonSets with `RollingUpdate` can replace Pods after a template change. StatefulSets and DaemonSets with `OnDelete`, and ReplicaSets, do not automatically replace existing Pods for this change. Paused Deployments and StatefulSet partitions can also prevent or limit Pod replacement. The controller does not wait for a rollout to complete.
Restarts are supported only for native Deployments, StatefulSets, and DaemonSets, while deletion remains available for all watched resources.

- Triggering a restart - the controller writes `lifecycle.cezary.dev/restartedAt: "<timestamp>"` into the resource's `spec.template.metadata.annotations`. The workload controller determines how to apply the changed template.
- The same pod template mutation also adds `lifecycle.cezary.dev/managed-by: "lifecycle-controller"` to `spec.template.metadata.annotations`.
Expand Down Expand Up @@ -180,7 +180,7 @@ Resource filters select which API types to watch. The controller reads, deletes,

### RBAC

By default, the Helm chart watches standard workloads, ConfigMaps, Services, Ingresses, NetworkPolicies, PersistentVolumeClaims, HorizontalPodAutoscalers, PodDisruptionBudgets, Namespaces, and PersistentVolumes. It grants `get`, `list`, `watch`, `patch`, `update`, and `delete` for those resources, plus `Event` permissions and permissions for enabled leader-election and secure-metrics features.
By default, the Helm chart watches standard workloads, ConfigMaps, Services, Ingresses, NetworkPolicies, PersistentVolumeClaims, HorizontalPodAutoscalers, PodDisruptionBudgets, Namespaces, and PersistentVolumes. It grants `get`, `list`, `watch`, `patch`, `update`, and `delete` for those resources, plus Event permissions and permissions for enabled leader-election and secure-metrics features.

A list of filtered namespaces creates a `Role` and `RoleBinding` in each watched namespace. Namespace patterns or leaving the list unset results in a `ClusterRole`. Namespace objects still require cluster permissions.

Expand Down
4 changes: 2 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,10 @@ require (
github.com/prometheus/client_golang v1.24.1
github.com/robfig/cron/v3 v3.0.1
k8s.io/api v0.37.1
k8s.io/apiextensions-apiserver v0.37.1
k8s.io/apimachinery v0.37.1
k8s.io/client-go v0.37.1
k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3
sigs.k8s.io/controller-runtime v0.25.2
)

Expand Down Expand Up @@ -94,13 +96,11 @@ require (
google.golang.org/protobuf v1.36.12 // indirect
gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
k8s.io/apiextensions-apiserver v0.37.1 // indirect
k8s.io/apiserver v0.37.1 // indirect
k8s.io/component-base v0.37.1 // indirect
k8s.io/klog/v2 v2.140.0 // indirect
k8s.io/kube-openapi v0.0.0-20260721132016-d427ff9ee9ad // indirect
k8s.io/streaming v0.37.1 // indirect
k8s.io/utils v0.0.0-20260707023825-cf1189d6abe3 // indirect
sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.37.0 // indirect
sigs.k8s.io/json v0.0.0-20260909141634-11ed52e25bc5 // indirect
sigs.k8s.io/randfill v1.0.0 // indirect
Expand Down
105 changes: 24 additions & 81 deletions internal/controller/lifecycle_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -101,17 +101,6 @@ func planRecurringRestart(schedule recurringSchedule, lastRestart, now time.Time
return scheduledAt, schedule.coalescedAnchor(lastRestart, now), true
}

func requeueForNextOccurrence(next time.Time) ctrl.Result {
if next.IsZero() {
return ctrl.Result{}
}
requeueAfter := time.Until(next)
if requeueAfter <= 0 {
requeueAfter = time.Nanosecond
}
return ctrl.Result{RequeueAfter: requeueAfter}
}

// LifecycleReconciler reconciles objects with lifecycle annotations.
type LifecycleReconciler struct {
client.Client
Expand All @@ -121,6 +110,7 @@ type LifecycleReconciler struct {
GlobalDryRun bool
discovery preferredResourceDiscovery
coverage *coverageState
now func() time.Time
}

// +kubebuilder:rbac:groups=*,resources=*,verbs=get;list;watch;delete;update;patch
Expand All @@ -143,11 +133,11 @@ func (r *LifecycleReconciler) getReferenceTime(obj client.Object, logger logr.Lo
} else {
logger.Info("Using 'applyTimestamp' (reconciliation time) as reference point")
}
return time.Now().UTC()
return r.currentTime().UTC()
default:
logger.Info("Invalid reference-point specified, falling back to 'applyTimestamp'", "value", referencePoint)
r.Recorder.Eventf(obj, "Warning", "InvalidAnnotationValue", "Invalid value for %s: '%s', falling back to 'applyTimestamp'", ReferencePointAnnotation, referencePoint)
return time.Now().UTC()
return r.currentTime().UTC()
}
}

Expand Down Expand Up @@ -298,7 +288,7 @@ func (r *LifecycleReconciler) handleDeletion(ctx context.Context, obj *unstructu
return ctrl.Result{}, nil
}

if time.Now().After(deleteAtTime) {
if !r.currentTime().Before(deleteAtTime) {
logger.Info("Deleting resource based on delete-at annotation", "targetTime", deleteAtTime.String())
if !isDryRun {
uid := obj.GetUID()
Expand All @@ -318,9 +308,9 @@ func (r *LifecycleReconciler) handleDeletion(ctx context.Context, obj *unstructu
}
return ctrl.Result{}, nil
} else {
requeueAfter := time.Until(deleteAtTime)
logger.Info("Resource deletion scheduled for a future time", "requeueAfter", requeueAfter)
return ctrl.Result{RequeueAfter: requeueAfter}, nil
result := r.requeueForNextOccurrence(deleteAtTime)
logger.Info("Resource deletion scheduled for a future time", "requeueAfter", result.RequeueAfter)
return result, nil
}
}

Expand All @@ -329,15 +319,19 @@ func (r *LifecycleReconciler) handleDeletion(ctx context.Context, obj *unstructu

// handleRestart implements the full restart logic with precedence.
func (r *LifecycleReconciler) handleRestart(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, logger logr.Logger) (ctrl.Result, error) {
_, found, err := unstructured.NestedFieldNoCopy(obj.Object, "spec", "template")
if err != nil || !found {
logger.Info("Resource is not a pod-spawner, skipping restart action.", "resource", client.ObjectKeyFromObject(obj))
if !supportsRestart(obj) {
logger.Info("Skipping restart for an unsupported resource kind", "gvk", obj.GroupVersionKind())
r.Recorder.Event(obj, "Warning", "UnsupportedRestartKind", "Restart requires an apps Deployment, StatefulSet, or DaemonSet; no action taken")
return ctrl.Result{}, nil
}
if !hasRestartTemplate(obj) {
logger.Info("Skipping restart because the Pod template is missing", "resource", client.ObjectKeyFromObject(obj))
r.Recorder.Event(obj, "Warning", "NotPodSpawner", "Restart annotations are present but resource does not have a spec.template field.")
return ctrl.Result{}, nil
}

annotations := obj.GetAnnotations()
now := time.Now()
now := r.currentTime()

if restartAfterStr := annotations[RestartAfterAnnotation]; restartAfterStr != "" {
if annotations[ReferencePointAnnotation] == ReferencePointCreationTimestamp && annotations[RestartAtAnnotation] != "" {
Expand Down Expand Up @@ -394,7 +388,7 @@ func (r *LifecycleReconciler) handleRestart(ctx context.Context, obj *unstructur
r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for restart-at annotation: %v", err)
return ctrl.Result{}, nil
}
if now.After(restartTime) {
if !now.Before(restartTime) {
logger.Info("Triggering one-time restart based on restart-at annotation", "restartTime", restartTime)
if err := r.triggerRestart(ctx, obj, isDryRun, func(annotations map[string]string) {
delete(annotations, RestartAtAnnotation)
Expand All @@ -403,9 +397,9 @@ func (r *LifecycleReconciler) handleRestart(ctx context.Context, obj *unstructur
}
return ctrl.Result{}, nil
} else {
requeueAfter := time.Until(restartTime)
logger.Info("One-time restart scheduled for a future time", "requeueAfter", requeueAfter)
return ctrl.Result{RequeueAfter: requeueAfter}, nil
result := r.requeueForNextOccurrence(restartTime)
logger.Info("One-time restart scheduled for a future time", "requeueAfter", result.RequeueAfter)
return result, nil
}
}

Expand All @@ -427,7 +421,7 @@ func (r *LifecycleReconciler) handleRestart(ctx context.Context, obj *unstructur
r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid cron expression for restart-cron: %v", err)
return ctrl.Result{}, nil
}
if schedule.Next(time.Now()).IsZero() {
if schedule.Next(r.currentTime()).IsZero() {
err := fmt.Errorf("cron expression has no future occurrence")
logger.Error(err, "invalid cron expression", "cron", cronStr)
r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid cron expression for restart-cron: %v", err)
Expand All @@ -452,7 +446,7 @@ func (r *LifecycleReconciler) handleRestart(ctx context.Context, obj *unstructur
// reconcileRecurringRestart handles the stateful logic for both cron and interval restarts.
func (r *LifecycleReconciler) reconcileRecurringRestart(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, scheduleType string, schedule recurringSchedule, logger logr.Logger) (ctrl.Result, error) {
annotations := obj.GetAnnotations()
now := time.Now()
now := r.currentTime()
lastRestartStr := annotations[LastRestartTimestamp]

if lastRestartStr == "" {
Expand All @@ -468,7 +462,7 @@ func (r *LifecycleReconciler) reconcileRecurringRestart(ctx context.Context, obj
logger.Error(err, "failed to initialize last-restart-timestamp")
return ctrl.Result{}, err
}
return requeueForNextOccurrence(schedule.Next(now)), nil
return r.requeueForNextOccurrence(schedule.Next(now)), nil
}

lastRestartTime, err := time.Parse(time.RFC3339, lastRestartStr)
Expand All @@ -490,61 +484,10 @@ func (r *LifecycleReconciler) reconcileRecurringRestart(ctx context.Context, obj
if isDryRun {
return ctrl.Result{}, nil
}
return requeueForNextOccurrence(schedule.Next(coalescedAnchor)), nil
return r.requeueForNextOccurrence(schedule.Next(coalescedAnchor)), nil
} else {
result := requeueForNextOccurrence(nextScheduledRestart)
result := r.requeueForNextOccurrence(nextScheduledRestart)
logger.Info("Next recurring restart is scheduled", "type", scheduleType, "at", nextScheduledRestart, "requeueAfter", result.RequeueAfter)
return result, nil
}
}

// triggerRestart applies the rollout marker and its acknowledgement in one patch.
func (r *LifecycleReconciler) triggerRestart(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, acknowledge func(map[string]string), logger logr.Logger) error {
restartedAtTime := time.Now().UTC().Format(time.RFC3339)
logger.Info("Attempting to trigger restart", "restartedAt", restartedAtTime)

if isDryRun {
logger.Info("[DRY-RUN] Would trigger restart by setting template annotation", "annotation", RestartedAtTemplate, "value", restartedAtTime)
r.Recorder.Event(obj, "Normal", "DryRunRestart", "Dry-run: Resource would be restarted now.")
return nil
}

base := obj.DeepCopy()
annotations := obj.GetAnnotations()
if annotations == nil {
annotations = make(map[string]string)
}
acknowledge(annotations)
obj.SetAnnotations(annotations)
markManagedBy(obj)

templateAnnotations, found, err := unstructured.NestedStringMap(obj.Object, "spec", "template", "metadata", "annotations")
if err != nil {
logger.Error(err, "failed to get spec.template.metadata.annotations")
r.Recorder.Eventf(obj, "Warning", "RestartFailed", "Could not read pod template annotations: %v", err)
return err
}
if !found || templateAnnotations == nil {
templateAnnotations = make(map[string]string)
}

templateAnnotations[RestartedAtTemplate] = restartedAtTime
templateAnnotations[ManagedByAnnotation] = ManagedByValue
err = unstructured.SetNestedStringMap(obj.Object, templateAnnotations, "spec", "template", "metadata", "annotations")
if err != nil {
logger.Error(err, "failed to set restartedAt annotation on pod template")
r.Recorder.Eventf(obj, "Warning", "RestartFailed", "Could not set pod template annotations: %v", err)
return err
}

patch := client.MergeFromWithOptions(base, client.MergeFromWithOptimisticLock{})
if err := r.Patch(ctx, obj, patch); err != nil {
logger.Error(err, "failed to patch object to trigger restart")
r.Recorder.Eventf(obj, "Warning", "RestartFailed", "Could not patch object to trigger restart: %v", err)
return err
}

logger.Info("Successfully patched object to trigger restart")
r.Recorder.Event(obj, "Normal", "RestartTriggered", "Triggered a rolling restart of the resource.")
return nil
}
11 changes: 9 additions & 2 deletions internal/controller/lifecycle_predicate.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@ var lifecycleActionAnnotations = []string{
}

// LifecyclePredicate limits the shared workqueue to lifecycle-relevant events.
// Controller-owned state updates are intentionally excluded: reconcile results
// drive the next pass without requiring the controller's own writes to enqueue it.
// Restart state changes also enqueue pending actions.
func LifecyclePredicate() predicate.Predicate {
return predicate.Funcs{
CreateFunc: func(e event.CreateEvent) bool {
Expand Down Expand Up @@ -59,6 +58,10 @@ func lifecycleConfigurationChanged(e event.UpdateEvent) bool {
(hasRelativeAction(oldAnnotations) || hasRelativeAction(newAnnotations)) {
return true
}
if (hasRecurringRestart(oldAnnotations) || hasRecurringRestart(newAnnotations)) &&
oldAnnotations[LastRestartTimestamp] != newAnnotations[LastRestartTimestamp] {
return true
}
return oldAnnotations[CronTimezoneAnnotation] != newAnnotations[CronTimezoneAnnotation] &&
(oldAnnotations[RestartCronAnnotation] != "" || newAnnotations[RestartCronAnnotation] != "")
}
Expand All @@ -79,3 +82,7 @@ func hasLifecycleAction(obj client.Object) bool {
func hasRelativeAction(annotations map[string]string) bool {
return annotations[DeleteAfterAnnotation] != "" || annotations[RestartAfterAnnotation] != ""
}

func hasRecurringRestart(annotations map[string]string) bool {
return annotations[RestartCronAnnotation] != "" || annotations[RestartEveryAnnotation] != ""
}
6 changes: 5 additions & 1 deletion internal/controller/lifecycle_predicate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ func TestLifecyclePredicateFiltersWorkqueueEvents(t *testing.T) {
{name: "generic with lifecycle action", got: p.Generic(event.GenericEvent{Object: actionable}), want: false},
{name: "metadata update", got: p.Update(updateEvent(actionable, objectWithNameAndAnnotations("renamed", actionable.GetAnnotations()))), want: false},
{name: "spec update", got: p.Update(updateEvent(actionable, objectWithSpecAndAnnotations(map[string]interface{}{"replicas": int64(2)}, actionable.GetAnnotations()))), want: false},
{name: "template addition with restart action", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{RestartAtAnnotation: "2030-01-01T00:00:00Z"}), objectWithSpecAndAnnotations(map[string]interface{}{"template": map[string]interface{}{}}, map[string]string{RestartAtAnnotation: "2030-01-01T00:00:00Z"}))), want: false},
{name: "unrelated annotation update", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", "example.com/note": "changed"}))), want: false},
{name: "action addition", got: p.Update(updateEvent(objectWithAnnotations(nil), actionable)), want: true},
{name: "action value update", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "2h"}))), want: true},
Expand All @@ -36,7 +37,10 @@ func TestLifecyclePredicateFiltersWorkqueueEvents(t *testing.T) {
{name: "reference-point update with absolute action", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{DeleteAtAnnotation: "2030-01-01T00:00:00Z"}), objectWithAnnotations(map[string]string{DeleteAtAnnotation: "2030-01-01T00:00:00Z", ReferencePointAnnotation: ReferencePointCreationTimestamp}))), want: false},
{name: "timezone update with cron action", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{RestartCronAnnotation: "0 * * * *"}), objectWithAnnotations(map[string]string{RestartCronAnnotation: "0 * * * *", CronTimezoneAnnotation: "Europe/Warsaw"}))), want: true},
{name: "timezone update without cron action", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", CronTimezoneAnnotation: "Europe/Warsaw"}))), want: false},
{name: "internal last-restart update", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{RestartEveryAnnotation: "1h"}), objectWithAnnotations(map[string]string{RestartEveryAnnotation: "1h", LastRestartTimestamp: "2030-01-01T00:00:00Z"}))), want: false},
{name: "recurring state initialization", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{RestartEveryAnnotation: "1h"}), objectWithAnnotations(map[string]string{RestartEveryAnnotation: "1h", LastRestartTimestamp: "2030-01-01T00:00:00Z"}))), want: true},
{name: "cron state removal", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{RestartCronAnnotation: "0 * * * *", LastRestartTimestamp: "invalid"}), objectWithAnnotations(map[string]string{RestartCronAnnotation: "0 * * * *"}))), want: true},
{name: "state update with one-time restart", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{RestartAtAnnotation: "2030-01-01T00:00:00Z"}), objectWithAnnotations(map[string]string{RestartAtAnnotation: "2030-01-01T00:00:00Z", LastRestartTimestamp: "2030-01-01T00:00:00Z"}))), want: false},
{name: "state update with deletion", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", LastRestartTimestamp: "2030-01-01T00:00:00Z"}))), want: false},
{name: "internal managed-by update", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", ManagedByAnnotation: ManagedByValue}))), want: false},
}

Expand Down
Loading
Loading