diff --git a/README.md b/README.md index 61483ce..69d04a9 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ A Kubernetes controller that schedules resource deletion and Pod template updates through time-based annotations. It supports namespaced and cluster-scoped resources that expose the required API operations. -Use it to schedule resource deletion or request application restarts without a separate CronJob or wrapper object. Pod replacement depends on the workload's update behavior. See [Restart mechanism](#restart-mechanism). +Use it to schedule resource deletion or request application restarts without a separate CronJob or wrapper object. Pod replacement depends on the workload's update behavior. See [Restart mechanism](docs/controller-behavior.md#restart-mechanism). ## Quick start @@ -101,6 +101,7 @@ spec: - The value must be an RFC3339 timestamp with explicit timezone offset (`Z` or `±hh:mm`). - This can be applied directly to a `Namespace` to trigger its deletion. Kubernetes will handle the subsequent removal of all resources within that namespace. - `lifecycle.cezary.dev/delete-after` - relative TTL (e.g., `5m`, `1h`, `3d`). The controller calculates a deletion time from the selected reference point, saves it as `lifecycle.cezary.dev/delete-at`, and removes `delete-after`. + - `lifecycle.cezary.dev/catch-up` - overrides the global policy for a missed deletion or restart. Accepts `never`, `always`, or a positive duration (e.g., `15m`). See [Missed actions](docs/controller-behavior.md#missed-actions). - `lifecycle.cezary.dev/dry-run` - enables dry-run for a resource. Accepts standard boolean values such as `true`, `false`, `1`, and `0`. Invalid values create a warning Event and no action is taken. - `lifecycle.cezary.dev/managed-by: "lifecycle-controller"` - added by the controller when it mutates a resource, such as converting `*-after` annotations to `*-at` annotations, maintaining restart schedule state, or triggering a restart. @@ -115,70 +116,15 @@ spec: ### Relative duration format -`delete-after`, `restart-after`, and `restart-every` require exactly one positive integer and unit, without signs or spaces. Valid units are `s`, `m`, `h`, and `d`. Values such as `1d`, `25h`, and `90m` are valid. +`delete-after`, `restart-after`, `restart-every`, and duration values for `catch-up` require exactly one positive integer and unit, without signs or spaces. Valid units are `s`, `m`, `h`, and `d`. Values such as `1d`, `25h`, and `90m` are valid. -Zero, compound values (e.g., `1d2h`), arithmetic expressions (e.g., `1d-25h`), decimal values (e.g., `1.5h`), and values that exceed the Go duration limit are invalid. If the selected action has an invalid duration, the controller records an `InvalidAnnotation` warning Event and leaves the resource unchanged. +Zero, compound values (e.g., `1d2h`), arithmetic expressions (e.g., `1d-25h`), decimal values (e.g., `1.5h`), and values that exceed the Go duration limit are invalid. Invalid values record an `InvalidAnnotation` warning Event and leaves the resource unchanged. ## Controller behavior -### Relative timers +Actions can run after their scheduled time. Restarts update the Pod template. The workload's update strategy determines whether Pods are replaced. See [Controller behavior](docs/controller-behavior.md) for more information. -Relative timers (`delete-after`, `restart-after`) start from the selected reference point. By default, `applyTimestamp` means the time when the controller processes the annotation, not the time when a client applies the manifest. Cluster load or controller downtime can delay processing. - -Re-applying a manifest that restores the `-after` annotation resets the derived `-at` deadline when the controller processes it. This is the default "keep-alive" behavior. - -Use `lifecycle.cezary.dev/reference-point: "creationTimestamp"` to calculate the deadline from `metadata.creationTimestamp`. Re-applying `-after` does not reset an existing `-at` deadline with this reference point. - -Use `delete-at` or `restart-at` for a fixed timestamp, or `restart-cron` for a recurring calendar schedule. Actions can run after their scheduled time. These annotations do not guarantee exact-time execution or completion. - -### Precedence - -If a resource mixes `restart-*` and `delete-*` action annotations, the controller records a warning Event and takes no action. - -Within one action family, the controller processes a non-empty `delete-after` or `restart-after` before the corresponding `-at` annotation: - -| Reference point | When both `-after` and `-at` are present | -| --- | --- | -| `applyTimestamp` (default) | Convert `-after` to a new `-at` value, replacing the existing deadline. | -| `creationTimestamp` | Keep the existing `-at` value and remove `-after`. | - -After conversion, restart priority is `restart-at`, then `restart-cron`, then `restart-every`. For deletion, the controller acts on `delete-at`. - -### Restart mechanism - -Restarts are supported only for native Deployments, StatefulSets, and DaemonSets, while deletion remains available for all watched resources. - -- Triggering a restart - the controller writes `lifecycle.cezary.dev/restartedAt: ""` into the resource's `spec.template.metadata.annotations`. The workload controller determines how to apply the changed template. - - The same pod template mutation also adds `lifecycle.cezary.dev/managed-by: "lifecycle-controller"` to `spec.template.metadata.annotations`. -- State tracking for recurring restarts - for `restart-every` and `restart-cron` schedules, the controller uses a top-level `lifecycle.cezary.dev/last-restart-timestamp: ""` annotation as the anchor for calculating the next restart. - - Initialization - if `last-restart-timestamp` is missing, the controller sets it to the current time. - - Reconciliation - the controller calculates the next occurrence from the schedule and timestamp. When due, it updates the Pod template and timestamp, then schedules the next occurrence. Missed occurrences produce only one template update. -- Cleanup - the restart patch also removes the one-time `restart-at` annotation or updates recurring schedule state. This acknowledges the template update, not rollout completion. - -### Dry-run - -Dry-run logs planned actions without changing resources. Enable it globally with the `--dry-run` flag or the `controllerManager.dryRun` Helm value, or per resource with the `lifecycle.cezary.dev/dry-run` annotation. - -In dry-run, relative and recurring schedules are logged without saving state or immediately requeueing the same occurrence. - -## Scope and permissions - -### Configuration flags - -By default, the controller discovers resources across all namespaces and watches those that support `get`, `list`, `watch`, `patch`, and `delete`. It skips subresources and APIs that lack these verbs. An explicit watch pattern that selects an unsupported API causes a discovery error unless an ignore rule excludes that API. - -Resource filters select which API types to watch. The controller reads, deletes, and restarts resources only in the filtered namespaces. Namespace patterns limit deletions and restarts but require cluster-wide reads. - -- `--watch-resource` (repeatable) - glob pattern for resources to watch. - - Format - `.` for grouped APIs (e.g., `deployments.apps`) or `` for core APIs (e.g., `pods`). - - If not provided, all eligible resources are watched unless excluded by ignore rules. Literal `*.*` is not equivalent to this default: it does not match core keys such as `pods`. - - Broad patterns can select APIs that lack the required verbs. Prefer exact resource names. -- `--ignore-resource` (repeatable) - glob pattern for resources to strictly ignore. Takes precedence over watch rules. -- `--watch-namespace` (repeatable) - glob pattern for namespaces to watch (e.g. `default`, `dev-*`). - - If provided, the controller deletes or restarts only objects in matching namespaces. Namespace objects also require `--watch-resource=namespaces`. Other cluster-scoped resources are excluded. -- `--ignore-namespace` (repeatable) - glob pattern for namespaces to strictly ignore. Takes precedence over watch rules. - -### RBAC +## RBAC By default, the Helm chart watches standard workloads, ConfigMaps, Services, Ingresses, NetworkPolicies, PersistentVolumeClaims, HorizontalPodAutoscalers, PodDisruptionBudgets, Namespaces, and PersistentVolumes. It grants `get`, `list`, `watch`, `patch`, `update`, and `delete` for those resources, plus Event permissions and permissions for enabled leader-election and secure-metrics features. diff --git a/charts/lifecycle-controller/templates/deployment.yaml b/charts/lifecycle-controller/templates/deployment.yaml index 3065cb5..a91784f 100644 --- a/charts/lifecycle-controller/templates/deployment.yaml +++ b/charts/lifecycle-controller/templates/deployment.yaml @@ -53,6 +53,19 @@ spec: - --enable-http2 {{- end }} - --dry-run={{ .Values.controllerManager.dryRun }} + {{- $timing := .Values.controllerManager.actionTiming | default dict }} + {{- $restartTiming := $timing.restart | default dict }} + {{- $deleteTiming := $timing.delete | default dict }} + {{- $restartCatchUp := "always" }} + {{- $deleteCatchUp := "always" }} + {{- if hasKey $restartTiming "catchUp" }} + {{- $restartCatchUp = $restartTiming.catchUp }} + {{- end }} + {{- if hasKey $deleteTiming "catchUp" }} + {{- $deleteCatchUp = $deleteTiming.catchUp }} + {{- end }} + - {{ printf "--restart-catch-up=%v" $restartCatchUp | quote }} + - {{ printf "--delete-catch-up=%v" $deleteCatchUp | quote }} - --zap-log-level={{ .Values.controllerManager.logLevel }} {{- range (include "lifecycle-controller.watchResources" . | fromJsonArray) }} - --watch-resource={{ . }} diff --git a/charts/lifecycle-controller/tests/action_timing_test.yaml b/charts/lifecycle-controller/tests/action_timing_test.yaml new file mode 100644 index 0000000..39dfe33 --- /dev/null +++ b/charts/lifecycle-controller/tests/action_timing_test.yaml @@ -0,0 +1,80 @@ +suite: Action timing configuration +templates: + - templates/deployment.yaml + +tests: + - it: preserves catch-up by default for both actions + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --restart-catch-up=always + - contains: + path: spec.template.spec.containers[0].args + content: --delete-catch-up=always + + - it: configures restart and deletion independently + set: + controllerManager.actionTiming.restart.catchUp: never + controllerManager.actionTiming.delete.catchUp: 15m + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --restart-catch-up=never + - contains: + path: spec.template.spec.containers[0].args + content: --delete-catch-up=15m + + - it: accepts a day duration and an explicit always policy + set: + controllerManager.actionTiming.restart.catchUp: 1d + controllerManager.actionTiming.delete.catchUp: always + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --restart-catch-up=1d + - contains: + path: spec.template.spec.containers[0].args + content: --delete-catch-up=always + + - it: defaults missing action timing to always for preserved release values + set: + controllerManager.actionTiming: null + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --restart-catch-up=always + - contains: + path: spec.template.spec.containers[0].args + content: --delete-catch-up=always + + - it: defaults missing action maps to always + set: + controllerManager.actionTiming.restart: null + controllerManager.actionTiming.delete: null + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --restart-catch-up=always + - contains: + path: spec.template.spec.containers[0].args + content: --delete-catch-up=always + + - it: defaults a missing policy to always + set: + controllerManager.actionTiming.restart.catchUp: null + controllerManager.actionTiming.delete.catchUp: null + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --restart-catch-up=always + - contains: + path: spec.template.spec.containers[0].args + content: --delete-catch-up=always + + - it: passes an empty policy to startup validation + set: + controllerManager.actionTiming.restart.catchUp: "" + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --restart-catch-up= diff --git a/charts/lifecycle-controller/tests/rbac_watchresources_test.yaml b/charts/lifecycle-controller/tests/rbac_watchresources_test.yaml index be42893..c5ce2c5 100644 --- a/charts/lifecycle-controller/tests/rbac_watchresources_test.yaml +++ b/charts/lifecycle-controller/tests/rbac_watchresources_test.yaml @@ -43,6 +43,8 @@ tests: - --metrics-bind-address=:8443 - --metrics-secure=true - --dry-run=false + - --restart-catch-up=always + - --delete-catch-up=always - --zap-log-level=info - --watch-resource=pods - --watch-resource=deployments.apps diff --git a/charts/lifecycle-controller/values.yaml b/charts/lifecycle-controller/values.yaml index 73e4df3..3d20624 100644 --- a/charts/lifecycle-controller/values.yaml +++ b/charts/lifecycle-controller/values.yaml @@ -56,6 +56,14 @@ controllerManager: # Enable global dry-run mode (logs actions without executing them) dryRun: false + # Default policy for missed actions: always. Also accepts never or a positive duration. + # The lifecycle.cezary.dev/catch-up annotation overrides these defaults. + actionTiming: + restart: + catchUp: "always" + delete: + catchUp: "always" + # Log level for the controller (debug, info, error) logLevel: info diff --git a/cmd/main.go b/cmd/main.go index 9f75cf8..bd23502 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -63,6 +63,7 @@ func main() { var watchResources, ignoreResources arrayFlags var watchNamespaces, ignoreNamespaces arrayFlags var globalDryRun bool + var deleteCatchUp, restartCatchUp string flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+ "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.") @@ -91,6 +92,10 @@ func main() { flag.Var(&ignoreNamespaces, "ignore-namespace", "Glob pattern for namespaces to ignore. Can be repeated.") flag.BoolVar(&globalDryRun, "dry-run", false, "Enable dry-run mode for all resources. When set, actions are logged but not executed.") + flag.StringVar(&deleteCatchUp, "delete-catch-up", "always", + "Default deletion catch-up policy: always, never, or a positive duration (e.g. 15m).") + flag.StringVar(&restartCatchUp, "restart-catch-up", "always", + "Default restart catch-up policy: always, never, or a positive duration (e.g. 15m).") opts := zap.Options{} @@ -98,6 +103,16 @@ func main() { flag.Parse() ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts))) + deletePolicy, err := controller.ParseCatchUpPolicy(deleteCatchUp) + if err != nil { + setupLog.Error(err, "invalid --delete-catch-up") + os.Exit(1) + } + restartPolicy, err := controller.ParseCatchUpPolicy(restartCatchUp) + if err != nil { + setupLog.Error(err, "invalid --restart-catch-up") + os.Exit(1) + } // if the enable-http2 flag is false (the default), http/2 should be disabled // due to its vulnerabilities. More specifically, disabling http/2 will @@ -210,12 +225,14 @@ func main() { "ignoreNamespaces", ignoreNamespaces, ) setupLog.Info("Global dry-run mode", "enabled", globalDryRun) + setupLog.Info("Action catch-up policies", "delete", deleteCatchUp, "restart", restartCatchUp) reconciler := &controller.LifecycleReconciler{ Client: mgr.GetClient(), Scheme: mgr.GetScheme(), Config: scopeConfig, GlobalDryRun: globalDryRun, + ActionTiming: controller.ActionTimingConfig{Delete: deletePolicy, Restart: restartPolicy}, } if err := reconciler.SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", "controller", "Lifecycle") diff --git a/docs/controller-behavior.md b/docs/controller-behavior.md new file mode 100644 index 0000000..8b88ddd --- /dev/null +++ b/docs/controller-behavior.md @@ -0,0 +1,111 @@ +# Controller behavior + +## Configuration flags + +By default, the controller watches resources across all namespaces. It skips subresources and APIs that do not support `get`, `list`, `watch`, `patch`, and `delete`. Discovery fails if a `--watch-resource` pattern selects an API without these verbs, unless `--ignore-resource` excludes it. + +Resource filters select which API types to watch. Namespace filters limit deletions and restarts. Exact namespace watch lists also restrict reads of namespaced resources. Watch lists with glob patterns require reads across all namespaces. + +Use these flags to select resources and namespaces: + +- `--watch-resource` (repeatable) - select resource types with a glob pattern. + - Use `.` for grouped APIs (e.g., `deployments.apps`) or `` for core APIs (e.g., `pods`). + - If omitted, watch all eligible resources except those excluded by `--ignore-resource`. The pattern `*.*` does not match core resources such as `pods`. + - Use exact resource names to avoid selecting unsupported APIs. +- `--ignore-resource` (repeatable) - exclude resource types with a glob pattern, overrides `--watch-resource`. +- `--watch-namespace` (repeatable) - limit deletions and restarts to namespaces that match a glob pattern (e.g., `default` or `dev-*`). + - To act on Namespace objects, also set `--watch-resource=namespaces`. Other cluster-scoped resources are excluded when this flag is set. +- `--ignore-namespace` (repeatable) - exclude namespaces with a glob pattern, overrides `--watch-namespace`. + +Use these flags to set the default policies for missed actions: + +- `--restart-catch-up` - set the policy for missed restarts. Accepts `never`, `always`, or a positive duration (e.g., `15m`). Default: `always`. Helm value: `controllerManager.actionTiming.restart.catchUp`. +- `--delete-catch-up` - set the policy for missed deletions. Accepts `never`, `always`, or a positive duration (e.g., `15m`). Default: `always`. Helm value: `controllerManager.actionTiming.delete.catchUp`. + +## Scheduling + +### Relative timers + +Relative timers (`delete-after`, `restart-after`) start from the selected reference point. By default, `applyTimestamp` means the time when the controller processes the annotation, not the time when a client applies the manifest. Cluster load or controller downtime can delay processing. + +Re-applying a manifest that restores the `-after` annotation resets the derived `-at` deadline when the controller processes it. This is the default "keep-alive" behavior. + +Use `lifecycle.cezary.dev/reference-point: "creationTimestamp"` to calculate the deadline from `metadata.creationTimestamp`. Re-applying `-after` does not reset an existing `-at` deadline with this reference point. + +Use `delete-at` or `restart-at` for a fixed timestamp, or `restart-cron` for a recurring calendar schedule. Actions can run after their scheduled time. These annotations do not guarantee exact-time execution or completion. + +### Recurring schedules + +For `restart-every` and `restart-cron` schedules, the controller uses a top-level `lifecycle.cezary.dev/last-restart-timestamp: ""` annotation as the anchor for calculating the next restart. + +- Initialization - if `last-restart-timestamp` is missing, the controller sets it to the current time. +- Reconciliation - the controller calculates the next occurrence from the schedule and timestamp. When due, it updates the Pod template and timestamp, then schedules the next occurrence. Missed occurrences produce only one template update. + +## Restart mechanism + +Restarts are supported only for native Deployments, StatefulSets, and DaemonSets in the `apps` API group. Deletion remains available for all watched resources. Other resources receive an `UnsupportedRestartKind` warning Event and their restart annotations remain unchanged. + +Deployments and StatefulSets or DaemonSets with `RollingUpdate` can replace Pods after a template change. StatefulSets and DaemonSets with `OnDelete` do not automatically replace existing Pods for this change. Paused Deployments and StatefulSet partitions can also prevent or limit Pod replacement. The controller does not wait for a rollout to complete. + +- Triggering a restart - the controller writes `lifecycle.cezary.dev/restartedAt: ""` into the resource's `spec.template.metadata.annotations`. The workload controller determines how to apply the changed template. + - The same pod template mutation also adds `lifecycle.cezary.dev/managed-by: "lifecycle-controller"` to `spec.template.metadata.annotations`. +- Cleanup - the restart patch also removes the one-time `restart-at` annotation or updates recurring schedule state. This acknowledges the template update, not rollout completion. + +## Dry-run + +Dry-run logs planned actions without changing resources. Enable it globally with the `--dry-run` flag or the `controllerManager.dryRun` Helm value, or per resource with the `lifecycle.cezary.dev/dry-run` annotation. + +In dry-run, relative and recurring schedules are logged without saving state or immediately requeueing the same occurrence. + +## Precedence + +If a resource mixes `restart-*` and `delete-*` action annotations, the controller records a warning Event and takes no action. + +Within one action family, the controller processes a non-empty `delete-after` or `restart-after` before the corresponding `-at` annotation: + +| Reference point | When both `-after` and `-at` are present | +| --- | --- | +| `applyTimestamp` (default) | Convert `-after` to a new `-at` value, replacing the existing deadline. | +| `creationTimestamp` | Keep the existing `-at` value and remove `-after`. | + +After conversion, restart priority is `restart-at`, then `restart-cron`, then `restart-every`. For deletion, the controller acts on `delete-at`. + +## Missed actions + +Use a catch-up policy to control actions after their scheduled time. The policy applies to deletion, one-time restarts, and recurring restarts. The default is `always` for both actions. + +| Policy | Behavior | +| --- | --- | +| `always` | Execute a due action, with no limit on delay. Missed recurring occurrences produce one restart. | +| A duration, such as `15m` | Execute only if a due occurrence is no more than this duration late. The boundary is inclusive. The deadline uses the scheduled time and does not reset on retries. | +| `never` | Execute an occurrence only if this controller process observed or created it before its due time. Skip an overdue occurrence first seen in this process. | + +`never` does not disable future actions. The controller keeps future occurrences in process memory. A matching occurrence stays eligible when its timer is due, including retries after API errors. A controller process restart or leader failover loses this memory. An overdue occurrence first seen by the new process is skipped. Queue or API delays within the same process do not expire an occurrence that was observed before its due time. Use a duration policy to limit those delays. + +For a recurring schedule, a duration policy checks for a recent due occurrence. For example, a controller that returns at 03:10 after several missed daily 03:00 restarts can request one restart with `catch-up: "15m"`. It does not replay the missed restarts. If no due occurrence is in the window, it skips them and keeps the next scheduled time. + +Set separate global defaults in Helm values: + +```yaml +controllerManager: + actionTiming: + restart: + catchUp: "15m" + delete: + catchUp: "always" +``` + +Use `lifecycle.cezary.dev/catch-up` to override the default on a resource: + +```yaml +metadata: + annotations: + lifecycle.cezary.dev/restart-at: "2026-10-10T03:00:00Z" + lifecycle.cezary.dev/catch-up: "never" +``` + +If the annotation is absent, the resource uses the global default for its selected action. An invalid global value prevents startup. An invalid annotation records an `InvalidAnnotation` warning Event and blocks the action. + +A skipped action records an `ActionSkipped` Event after the controller saves its acknowledgement. The controller removes the selected one-time `delete-at` or `restart-at` annotation; a skipped deletion leaves the resource in place. For a skipped recurring restart, it advances the schedule state without changing the Pod template. The `catch-up` annotation and recurring schedules stay in place. To request a skipped one-time action again, apply a new action annotation. + +The policy starts from the computed deadline for relative timers. With `applyTimestamp`, controller downtime before conversion delays the start of the timer. With `creationTimestamp`, the computed deadline can already be overdue when the controller first reads the resource. diff --git a/internal/controller/catch_up.go b/internal/controller/catch_up.go new file mode 100644 index 0000000..8ee468d --- /dev/null +++ b/internal/controller/catch_up.go @@ -0,0 +1,192 @@ +package controller + +import ( + "context" + "fmt" + "sync" + "time" + + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +type catchUpMode uint8 + +const ( + catchUpAlways catchUpMode = iota + catchUpNever + catchUpDeadline + deleteAction = "delete" + restartAction = "restart" +) + +// CatchUpPolicy controls eligibility after an action's scheduled time. +// The zero value permits catch-up without a deadline. +type CatchUpPolicy struct { + mode catchUpMode + window time.Duration +} + +// ParseCatchUpPolicy accepts always, never, or one positive integer with s, m, h, or d. +func ParseCatchUpPolicy(value string) (CatchUpPolicy, error) { + switch value { + case "always": + return CatchUpPolicy{}, nil + case "never": + return CatchUpPolicy{mode: catchUpNever}, nil + default: + window, err := parseExtendedDuration(value) + if err != nil { + return CatchUpPolicy{}, fmt.Errorf("invalid catch-up policy %q: expected always, never, or a positive duration: %w", value, err) + } + return CatchUpPolicy{mode: catchUpDeadline, window: window}, nil + } +} + +// ActionTimingConfig supplies the default catch-up policy for each action. +type ActionTimingConfig struct { + Delete CatchUpPolicy + Restart CatchUpPolicy +} + +type occurrenceIdentity struct { + uid types.UID + action string + policy CatchUpPolicy + config [4]string +} + +type armedOccurrence struct { + identity occurrenceIdentity + due time.Time +} + +type actionTimers struct { + mu sync.Mutex + armed map[resourceRequest]armedOccurrence +} + +func timingRequest(obj client.Object) resourceRequest { + return resourceRequest{NamespacedName: client.ObjectKeyFromObject(obj), GVK: obj.GetObjectKind().GroupVersionKind()} +} + +func timingIdentity(obj client.Object, action string, policy CatchUpPolicy) occurrenceIdentity { + a := obj.GetAnnotations() + identity := occurrenceIdentity{uid: obj.GetUID(), action: action, policy: policy} + switch { + case action == deleteAction: + identity.config = [4]string{DeleteAtAnnotation, a[DeleteAtAnnotation]} + case a[RestartAtAnnotation] != "": + identity.config = [4]string{RestartAtAnnotation, a[RestartAtAnnotation]} + case a[RestartCronAnnotation] != "": + identity.config = [4]string{RestartCronAnnotation, a[RestartCronAnnotation], a[CronTimezoneAnnotation], a[LastRestartTimestamp]} + default: + identity.config = [4]string{RestartEveryAnnotation, a[RestartEveryAnnotation], a[LastRestartTimestamp]} + } + return identity +} + +func (r *LifecycleReconciler) forgetTiming(obj client.Object) { + r.forgetRequestTiming(timingRequest(obj)) +} + +func (r *LifecycleReconciler) forgetRequestTiming(req resourceRequest) { + r.timers.mu.Lock() + defer r.timers.mu.Unlock() + delete(r.timers.armed, req) +} + +func (r *LifecycleReconciler) catchUpPolicy(obj client.Object, action string, dryRun bool) (CatchUpPolicy, bool) { + policy := r.ActionTiming.Restart + if action == deleteAction { + policy = r.ActionTiming.Delete + } + if value, found := obj.GetAnnotations()[CatchUpAnnotation]; found { + var err error + policy, err = ParseCatchUpPolicy(value) + if err != nil { + r.forgetTiming(obj) + r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "%v for %s; taking no action", err, CatchUpAnnotation) + return CatchUpPolicy{}, false + } + } + if policy.mode != catchUpNever || dryRun { + r.forgetTiming(obj) + } + return policy, true +} + +// observeOccurrence arms future work only in this process. A matching arm +// survives API errors and queue delays until the occurrence is acknowledged. +func (r *LifecycleReconciler) observeOccurrence(obj client.Object, action string, policy CatchUpPolicy, due, now time.Time, dryRun bool) bool { + switch policy.mode { + case catchUpAlways: + return true + case catchUpDeadline: + return !now.After(due.Add(policy.window)) + } + if dryRun { + return false + } + key := timingRequest(obj) + identity := timingIdentity(obj, action, policy) + r.timers.mu.Lock() + defer r.timers.mu.Unlock() + armed, found := r.timers.armed[key] + if found && (armed.identity != identity || !armed.due.Equal(due)) { + delete(r.timers.armed, key) + found = false + } + if due.After(now) { + if r.timers.armed == nil { + r.timers.armed = make(map[resourceRequest]armedOccurrence) + } + r.timers.armed[key] = armedOccurrence{identity: identity, due: due} + return true + } + return found +} + +func (r *LifecycleReconciler) armAbsolute(obj client.Object, action string, policy CatchUpPolicy, annotation string, observedAt time.Time) { + due, err := time.Parse(time.RFC3339, obj.GetAnnotations()[annotation]) + if err == nil { + r.observeOccurrence(obj, action, policy, due, observedAt, false) + } +} + +// occurrenceWithinWindow checks recent work without enumerating missed history. +func occurrenceWithinWindow(schedule recurringSchedule, lastRestart, now time.Time, window time.Duration) (time.Time, bool) { + cutoff := now.Add(-window) + if interval, ok := schedule.(intervalRecurringSchedule); ok { + latest := interval.coalescedAnchor(lastRestart, now) + return latest, latest.After(lastRestart) && !latest.Before(cutoff) + } + lower := cutoff.Add(-time.Nanosecond) // Include the exact deadline boundary. + if lastRestart.After(lower) { + lower = lastRestart + } + occurrence := schedule.Next(lower) + return occurrence, !occurrence.IsZero() && !occurrence.After(now) +} + +// skipAction consumes one occurrence with an optimistic patch. It never changes +// the Pod template and cannot acknowledge a concurrent schedule edit. +func (r *LifecycleReconciler) skipAction(ctx context.Context, obj *unstructured.Unstructured, action string, due time.Time, dryRun bool, acknowledge func(map[string]string)) error { + if dryRun { + r.Recorder.Eventf(obj, "Normal", "DryRunActionSkipped", "Dry-run: Would skip %s scheduled at %s under the catch-up policy.", action, due.UTC().Format(time.RFC3339Nano)) + return nil + } + base := obj.DeepCopy() + a := obj.GetAnnotations() + acknowledge(a) + obj.SetAnnotations(a) + markManagedBy(obj) + if err := r.Patch(ctx, obj, client.MergeFromWithOptions(base, client.MergeFromWithOptimisticLock{})); err != nil { + r.Recorder.Eventf(obj, "Warning", "ActionSkipFailed", "Could not acknowledge skipped %s: %v", action, err) + return err + } + r.forgetTiming(obj) + r.Recorder.Eventf(obj, "Normal", "ActionSkipped", "Skipped %s scheduled at %s under the catch-up policy.", action, due.UTC().Format(time.RFC3339Nano)) + return nil +} diff --git a/internal/controller/catch_up_never_test.go b/internal/controller/catch_up_never_test.go new file mode 100644 index 0000000..e0248ff --- /dev/null +++ b/internal/controller/catch_up_never_test.go @@ -0,0 +1,352 @@ +package controller + +import ( + "context" + "errors" + "reflect" + "testing" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/client-go/tools/record" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" +) + +var catchUpTestStart = time.Date(2026, 10, 5, 3, 0, 0, 0, time.UTC) + +func TestCatchUpNeverUnusedConfigurationKeepsSelectedArm(t *testing.T) { + for _, tt := range []struct { + action, modifier, value string + }{ + {DeleteAtAnnotation, ReferencePointAnnotation, ReferencePointCreationTimestamp}, + {RestartAtAnnotation, ReferencePointAnnotation, ReferencePointCreationTimestamp}, + {RestartAtAnnotation, CronTimezoneAnnotation, "Europe/Warsaw"}, + {RestartAtAnnotation, RestartCronAnnotation, "0 3 * * *"}, + {RestartAtAnnotation, RestartEveryAnnotation, "1h"}, + {RestartAtAnnotation, LastRestartTimestamp, catchUpTestStart.Format(time.RFC3339)}, + } { + t.Run(tt.action+" "+tt.modifier, func(t *testing.T) { + now := catchUpTestStart + due := now.Add(time.Minute) + obj := testDeployment("selected-arm", map[string]string{tt.action: due.Format(time.RFC3339)}) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + catchUpReconcile(t, r, obj) + catchUpUpdate(t, r, obj, func(current *unstructured.Unstructured) { + a := current.GetAnnotations() + a[tt.modifier] = tt.value + current.SetAnnotations(a) + }) + now = due.Add(time.Second) + catchUpReconcile(t, r, obj) + catchUpAssertExecuted(t, r, obj, tt.action) + }) + } +} + +func catchUpNeverReconciler(t *testing.T, obj *unstructured.Unstructured, now *time.Time, funcs interceptor.Funcs) *LifecycleReconciler { + t.Helper() + policy, err := ParseCatchUpPolicy("never") + if err != nil { + t.Fatalf("parse never policy: %v", err) + } + return &LifecycleReconciler{ + Client: fake.NewClientBuilder().WithObjects(obj.DeepCopy()).WithInterceptorFuncs(funcs).Build(), + Recorder: record.NewFakeRecorder(64), + ActionTiming: ActionTimingConfig{Delete: policy, Restart: policy}, + now: func() time.Time { return *now }, + } +} + +func catchUpReconcile(t *testing.T, r *LifecycleReconciler, obj *unstructured.Unstructured) { + t.Helper() + _, err := r.Reconcile(context.Background(), resourceRequest{ + NamespacedName: client.ObjectKeyFromObject(obj), + GVK: obj.GroupVersionKind(), + }) + if err != nil { + t.Fatalf("reconcile: %v", err) + } +} + +func catchUpFetch(t *testing.T, r *LifecycleReconciler, obj *unstructured.Unstructured) *unstructured.Unstructured { + t.Helper() + return getObject(t, r, client.ObjectKeyFromObject(obj)) +} + +func catchUpUpdate(t *testing.T, r *LifecycleReconciler, obj *unstructured.Unstructured, change func(*unstructured.Unstructured)) { + t.Helper() + current := catchUpFetch(t, r, obj) + change(current) + if err := r.Update(context.Background(), current); err != nil { + t.Fatalf("update resource: %v", err) + } +} + +func catchUpAssertSkipped(t *testing.T, r *LifecycleReconciler, obj *unstructured.Unstructured, action string) { + t.Helper() + current := catchUpFetch(t, r, obj) + if _, exists := current.GetAnnotations()[action]; exists { + t.Fatalf("skipped request still present: %s", action) + } + if got := restartedAt(t, current); got != "" { + t.Fatalf("skipped request changed template: %q", got) + } +} + +func catchUpAssertExecuted(t *testing.T, r *LifecycleReconciler, obj *unstructured.Unstructured, action string) { + t.Helper() + if action == RestartAtAnnotation { + assertOneTimeRestartAcknowledged(t, catchUpFetch(t, r, obj)) + return + } + current := obj.DeepCopy() + err := r.Get(context.Background(), client.ObjectKeyFromObject(obj), current) + if !apierrors.IsNotFound(err) { + t.Fatalf("get deleted resource: %v, want NotFound", err) + } +} + +func TestCatchUpNeverArmedOneTimeActionsExecute(t *testing.T) { + for _, action := range []string{DeleteAtAnnotation, RestartAtAnnotation} { + for _, source := range []string{"global", "annotation"} { + t.Run(action+"/"+source, func(t *testing.T) { + now := catchUpTestStart + due := now.Add(time.Minute) + obj := testDeployment("armed-action", map[string]string{action: due.Format(time.RFC3339)}) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + if source == "annotation" { + r.ActionTiming = ActionTimingConfig{} + catchUpUpdate(t, r, obj, func(current *unstructured.Unstructured) { + annotations := current.GetAnnotations() + annotations[CatchUpAnnotation] = "never" + current.SetAnnotations(annotations) + }) + } + catchUpReconcile(t, r, obj) + now = due.Add(250 * time.Millisecond) + catchUpReconcile(t, r, obj) + catchUpAssertExecuted(t, r, obj, action) + }) + } + } +} + +func TestCatchUpNeverRecoverySkipsOneTimeActions(t *testing.T) { + for _, action := range []string{DeleteAtAnnotation, RestartAtAnnotation} { + t.Run(action, func(t *testing.T) { + now := catchUpTestStart + due := now.Add(time.Minute) + obj := testDeployment("recovered-action", map[string]string{action: due.Format(time.RFC3339)}) + setTemplate(t, obj) + original := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + catchUpReconcile(t, original, obj) + now = due.Add(time.Second) + recovered := catchUpNeverReconciler(t, catchUpFetch(t, original, obj), &now, interceptor.Funcs{}) + catchUpReconcile(t, recovered, obj) + catchUpAssertSkipped(t, recovered, obj, action) + }) + } +} + +func TestCatchUpNeverAPIFailureRetainsArmedAction(t *testing.T) { + for _, action := range []string{DeleteAtAnnotation, RestartAtAnnotation} { + t.Run(action, func(t *testing.T) { + now := catchUpTestStart + due := now.Add(time.Minute) + obj := testDeployment("retry-action", map[string]string{action: due.Format(time.RFC3339)}) + setTemplate(t, obj) + failure := errors.New("temporary API failure") + fail := true + calls := 0 + funcs := interceptor.Funcs{} + if action == DeleteAtAnnotation { + funcs.Delete = func(ctx context.Context, c client.WithWatch, target client.Object, opts ...client.DeleteOption) error { + calls++ + if fail { + return failure + } + return c.Delete(ctx, target, opts...) + } + } else { + funcs.Patch = func(ctx context.Context, c client.WithWatch, target client.Object, patch client.Patch, opts ...client.PatchOption) error { + calls++ + if fail { + return failure + } + return c.Patch(ctx, target, patch, opts...) + } + } + r := catchUpNeverReconciler(t, obj, &now, funcs) + catchUpReconcile(t, r, obj) + now = due.Add(time.Second) + _, err := r.Reconcile(context.Background(), resourceRequest{NamespacedName: client.ObjectKeyFromObject(obj), GVK: obj.GroupVersionKind()}) + if !errors.Is(err, failure) { + t.Fatalf("failed action error = %v, want temporary API failure", err) + } + fail = false + now = now.Add(10 * time.Minute) + catchUpReconcile(t, r, obj) + catchUpAssertExecuted(t, r, obj, action) + if calls != 2 { + t.Fatalf("action calls = %d, want 2", calls) + } + }) + } +} + +func TestCatchUpNeverChangedIdentityDoesNotReuseArm(t *testing.T) { + for _, change := range []string{"UID", "deadline", "policy"} { + t.Run(change, func(t *testing.T) { + now := catchUpTestStart + due := now.Add(time.Minute) + obj := testDeployment("changed-action", map[string]string{RestartAtAnnotation: due.Format(time.RFC3339)}) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + catchUpReconcile(t, r, obj) + catchUpUpdate(t, r, obj, func(current *unstructured.Unstructured) { + annotations := current.GetAnnotations() + switch change { + case "UID": + current.SetUID("replacement") + case "deadline": + annotations[RestartAtAnnotation] = due.Add(-time.Second).Format(time.RFC3339) + case "policy": + annotations[CatchUpAnnotation] = "always" + } + current.SetAnnotations(annotations) + }) + if change == "policy" { + catchUpReconcile(t, r, obj) + catchUpUpdate(t, r, obj, func(current *unstructured.Unstructured) { + annotations := current.GetAnnotations() + annotations[CatchUpAnnotation] = "never" + current.SetAnnotations(annotations) + }) + } + now = due.Add(time.Second) + catchUpReconcile(t, r, obj) + catchUpAssertSkipped(t, r, obj, RestartAtAnnotation) + }) + } +} + +func TestCatchUpNeverRelativeConversionArmsActions(t *testing.T) { + for _, action := range []string{DeleteAfterAnnotation, RestartAfterAnnotation} { + t.Run(action, func(t *testing.T) { + now := catchUpTestStart + obj := testDeployment("relative-action", map[string]string{action: "1m"}) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + catchUpReconcile(t, r, obj) + absolute := DeleteAtAnnotation + if action == RestartAfterAnnotation { + absolute = RestartAtAnnotation + } + converted := catchUpFetch(t, r, obj) + if got := converted.GetAnnotations()[absolute]; got != now.Add(time.Minute).Format(time.RFC3339) { + t.Fatalf("converted deadline = %q, want one minute from processing", got) + } + now = now.Add(time.Minute + time.Second) + catchUpReconcile(t, r, obj) + catchUpAssertExecuted(t, r, obj, absolute) + }) + } +} + +func TestCatchUpNeverRecurringInitializationArmsNextOccurrence(t *testing.T) { + for _, schedule := range []struct{ annotation, value string }{ + {RestartEveryAnnotation, "1h"}, + {RestartCronAnnotation, "0 * * * *"}, + } { + t.Run(schedule.annotation, func(t *testing.T) { + now := catchUpTestStart + obj := testDeployment("initial-recurring", map[string]string{schedule.annotation: schedule.value}) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + catchUpReconcile(t, r, obj) + now = now.Add(time.Hour + time.Second) + catchUpReconcile(t, r, obj) + assertRestartMarker(t, catchUpFetch(t, r, obj)) + }) + } +} + +func TestCatchUpNeverRecurringSkipPreservesCadence(t *testing.T) { + now := catchUpTestStart.Add(3*time.Hour + 20*time.Minute) + obj := testDeployment("missed-recurring", map[string]string{ + RestartEveryAnnotation: "1h", + LastRestartTimestamp: catchUpTestStart.Format(time.RFC3339), + }) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + catchUpReconcile(t, r, obj) + current := catchUpFetch(t, r, obj) + wantAnchor := catchUpTestStart.Add(3 * time.Hour).Format(time.RFC3339) + if got := current.GetAnnotations()[LastRestartTimestamp]; got != wantAnchor { + t.Fatalf("skip anchor = %q, want %q", got, wantAnchor) + } + if got := restartedAt(t, current); got != "" { + t.Fatalf("skipped recurring occurrence changed template: %q", got) + } + if got := current.GetAnnotations()[RestartEveryAnnotation]; got != "1h" { + t.Fatalf("recurring schedule = %q, want 1h", got) + } + now = catchUpTestStart.Add(4*time.Hour + time.Second) + catchUpReconcile(t, r, obj) + assertRestartMarker(t, catchUpFetch(t, r, obj)) +} + +func TestCatchUpNeverChangedRecurringScheduleDoesNotReuseArm(t *testing.T) { + now := catchUpTestStart + obj := testDeployment("changed-recurring", map[string]string{ + RestartEveryAnnotation: "1h", + LastRestartTimestamp: catchUpTestStart.Format(time.RFC3339), + }) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + catchUpReconcile(t, r, obj) + catchUpUpdate(t, r, obj, func(current *unstructured.Unstructured) { + annotations := current.GetAnnotations() + annotations[RestartEveryAnnotation] = "2h" + current.SetAnnotations(annotations) + }) + now = catchUpTestStart.Add(2*time.Hour + time.Second) + catchUpReconcile(t, r, obj) + if got := restartedAt(t, catchUpFetch(t, r, obj)); got != "" { + t.Fatalf("changed schedule reused old arm: %q", got) + } +} + +func TestCatchUpNeverDryRunDoesNotWriteOrArm(t *testing.T) { + for _, action := range []string{DeleteAtAnnotation, RestartAtAnnotation} { + t.Run(action, func(t *testing.T) { + now := catchUpTestStart + due := now.Add(time.Minute) + obj := testDeployment("dry-run-action", map[string]string{ + action: due.Format(time.RFC3339), + DryRunAnnotation: "true", + }) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + before := catchUpFetch(t, r, obj) + catchUpReconcile(t, r, obj) + now = due.Add(time.Second) + catchUpReconcile(t, r, obj) + if after := catchUpFetch(t, r, obj); !reflect.DeepEqual(before.Object, after.Object) { + t.Fatal("dry-run changed resource") + } + catchUpUpdate(t, r, obj, func(current *unstructured.Unstructured) { + annotations := current.GetAnnotations() + delete(annotations, DryRunAnnotation) + current.SetAnnotations(annotations) + }) + catchUpReconcile(t, r, obj) + catchUpAssertSkipped(t, r, obj, action) + }) + } +} diff --git a/internal/controller/catch_up_policy_test.go b/internal/controller/catch_up_policy_test.go new file mode 100644 index 0000000..85b4831 --- /dev/null +++ b/internal/controller/catch_up_policy_test.go @@ -0,0 +1,313 @@ +package controller + +import ( + "context" + "errors" + "reflect" + "strconv" + "strings" + "testing" + "time" + + "github.com/go-logr/logr" + "github.com/robfig/cron/v3" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/client-go/tools/record" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" +) + +func TestCatchUpPolicyParsing(t *testing.T) { + for _, value := range []string{"always", "never", "1s", "15m", "2h", "7d"} { + t.Run(value, func(t *testing.T) { + if _, err := ParseCatchUpPolicy(value); err != nil { + t.Fatal(err) + } + }) + } + for _, value := range []string{"", "none", "Always", " NEVER", "0s", "-1m", "+1m", "1.5m", "1h30m", "15 m", "999999999999999999d"} { + t.Run("invalid "+value, func(t *testing.T) { + if _, err := ParseCatchUpPolicy(value); err == nil { + t.Fatalf("accepted invalid catch-up policy %q", value) + } + }) + } +} + +func TestCatchUpPolicyDefaultsAndOverrides(t *testing.T) { + never, err := ParseCatchUpPolicy("never") + if err != nil { + t.Fatal(err) + } + deadline, err := ParseCatchUpPolicy("15m") + if err != nil { + t.Fatal(err) + } + r := &LifecycleReconciler{ActionTiming: ActionTimingConfig{Delete: never, Restart: deadline}} + obj := objectWithAnnotations(nil) + for _, tt := range []struct { + action string + want CatchUpPolicy + }{{deleteAction, never}, {restartAction, deadline}} { + got, valid := r.catchUpPolicy(obj, tt.action, false) + if !valid || got != tt.want { + t.Fatalf("%s default = %+v, %t; want %+v", tt.action, got, valid, tt.want) + } + } + obj.SetAnnotations(map[string]string{CatchUpAnnotation: "always"}) + for _, action := range []string{deleteAction, restartAction} { + got, valid := r.catchUpPolicy(obj, action, false) + if !valid || got != (CatchUpPolicy{}) { + t.Fatalf("annotation did not override %s default: %+v, %t", action, got, valid) + } + } + got, valid := (&LifecycleReconciler{}).catchUpPolicy(objectWithAnnotations(nil), deleteAction, false) + if !valid || got != (CatchUpPolicy{}) { + t.Fatalf("zero config = %+v, %t; want always", got, valid) + } +} + +func TestCatchUpInvalidAnnotationBlocksAllMutations(t *testing.T) { + for _, action := range []string{DeleteAtAnnotation, RestartAtAnnotation, DeleteAfterAnnotation, RestartEveryAnnotation} { + for _, value := range []string{"", "none", "0s"} { + t.Run(action+value, func(t *testing.T) { + obj := testDeployment("invalid-policy", map[string]string{action: "1h", CatchUpAnnotation: value}) + setTemplate(t, obj) + tracked := &mutationTrackingClient{Client: fake.NewClientBuilder().WithObjects(obj.DeepCopy()).Build()} + recorder := record.NewFakeRecorder(8) + r := &LifecycleReconciler{Client: tracked, Recorder: recorder} + before := obj.DeepCopy() + result, err := r.reconcileLogic(t.Context(), obj, logr.Discard()) + if err != nil || !result.IsZero() { + t.Fatalf("result=%+v err=%v", result, err) + } + assertNoTargetMutation(t, before, obj, tracked) + if event := <-recorder.Events; !strings.HasPrefix(event, "Warning InvalidAnnotation ") { + t.Fatalf("event=%q", event) + } + }) + } + } +} + +func TestCatchUpDeadlineOneTimeBoundaries(t *testing.T) { + now := time.Date(2026, 10, 4, 12, 0, 0, 0, time.UTC) + for _, action := range []string{deleteAction, restartAction} { + for _, tt := range []struct { + name string + late time.Duration + skip bool + }{{"inside", 14 * time.Minute, false}, {"boundary", 15 * time.Minute, false}, {"expired", 15*time.Minute + time.Nanosecond, true}} { + t.Run(action+" "+tt.name, func(t *testing.T) { + annotation := DeleteAtAnnotation + if action == restartAction { + annotation = RestartAtAnnotation + } + obj := testDeployment("deadline", map[string]string{annotation: now.Add(-tt.late).Format(time.RFC3339Nano), CatchUpAnnotation: "15m"}) + setTemplate(t, obj) + tracked := &mutationTrackingClient{Client: fake.NewClientBuilder().WithObjects(obj.DeepCopy()).Build()} + r := &LifecycleReconciler{Client: tracked, Recorder: record.NewFakeRecorder(8), now: func() time.Time { return now }} + if _, err := r.reconcileLogic(t.Context(), obj, logr.Discard()); err != nil { + t.Fatal(err) + } + if action == deleteAction && !tt.skip { + if tracked.deletes != 1 { + t.Fatalf("deletes=%d", tracked.deletes) + } + return + } + persisted := getObject(t, r, client.ObjectKeyFromObject(obj)) + if _, found := persisted.GetAnnotations()[annotation]; found { + t.Fatal("one-time occurrence not acknowledged") + } + _, marker, err := unstructured.NestedString(persisted.Object, "spec", "template", "metadata", "annotations", RestartedAtTemplate) + if err != nil || marker != (action == restartAction && !tt.skip) || tracked.deletes != 0 { + t.Fatalf("marker=%t deletes=%d err=%v", marker, tracked.deletes, err) + } + }) + } + } +} + +func TestCatchUpRecurringWindow(t *testing.T) { + anchor := time.Date(2026, 10, 1, 3, 0, 0, 0, time.UTC) + cronSchedule, err := cron.ParseStandard("CRON_TZ=UTC 0 3 * * *") + if err != nil { + t.Fatal(err) + } + for _, schedule := range []recurringSchedule{intervalRecurringSchedule{duration: 24 * time.Hour}, cronRecurringSchedule{Schedule: cronSchedule}} { + for _, tt := range []struct { + name string + now time.Time + want bool + }{ + {"recent after old misses", anchor.Add(72*time.Hour + 10*time.Minute), true}, + {"inclusive boundary", anchor.Add(72*time.Hour + 15*time.Minute), true}, + {"expired", anchor.Add(72*time.Hour + 15*time.Minute + time.Nanosecond), false}, + {"future anchor", anchor.Add(-time.Minute), false}, + {"already acknowledged", anchor, false}, + } { + t.Run(reflect.TypeOf(schedule).Name()+" "+tt.name, func(t *testing.T) { + _, got := occurrenceWithinWindow(schedule, anchor, tt.now, 15*time.Minute) + if got != tt.want { + t.Fatalf("eligible=%t, want %t", got, tt.want) + } + }) + } + } +} + +func TestCatchUpRecurringWindowTimezoneAndDST(t *testing.T) { + schedule, err := cron.ParseStandard("CRON_TZ=Europe/Warsaw 30 2 * * *") + if err != nil { + t.Fatal(err) + } + for _, tt := range []struct { + name, now string + want bool + }{ + {"before spring jump", "2026-03-28T01:40:00Z", true}, + {"spring nonexistent occurrence", "2026-03-29T01:40:00Z", false}, + {"after spring jump", "2026-03-30T00:40:00Z", true}, + {"first fall occurrence", "2026-10-25T00:40:00Z", true}, + {"second fall occurrence", "2026-10-25T01:40:00Z", true}, + } { + t.Run(tt.name, func(t *testing.T) { + now, err := time.Parse(time.RFC3339, tt.now) + if err != nil { + t.Fatal(err) + } + _, got := occurrenceWithinWindow(cronRecurringSchedule{Schedule: schedule}, now.Add(-7*24*time.Hour), now, 15*time.Minute) + if got != tt.want { + t.Fatalf("eligible=%t, want %t", got, tt.want) + } + }) + } +} + +func TestCatchUpDeadlineExpiresDuringRetry(t *testing.T) { + now := time.Date(2026, 10, 4, 12, 0, 0, 0, time.UTC) + obj := testDeployment("retry-deadline", map[string]string{RestartAtAnnotation: now.Add(-14 * time.Minute).Format(time.RFC3339), CatchUpAnnotation: "15m"}) + setTemplate(t, obj) + failure := errors.New("API unavailable") + calls := 0 + r := &LifecycleReconciler{ + Client: fake.NewClientBuilder().WithObjects(obj.DeepCopy()).WithInterceptorFuncs(interceptor.Funcs{ + Patch: func(ctx context.Context, c client.WithWatch, target client.Object, patch client.Patch, opts ...client.PatchOption) error { + calls++ + if calls == 1 { + return failure + } + return c.Patch(ctx, target, patch, opts...) + }, + }).Build(), + Recorder: record.NewFakeRecorder(8), now: func() time.Time { return now }, + } + if _, err := r.reconcileLogic(t.Context(), obj, logr.Discard()); !errors.Is(err, failure) { + t.Fatalf("first error=%v", err) + } + now = now.Add(2 * time.Minute) + obj = getObject(t, r, client.ObjectKeyFromObject(obj)) + if _, err := r.reconcileLogic(t.Context(), obj, logr.Discard()); err != nil { + t.Fatal(err) + } + got := getObject(t, r, client.ObjectKeyFromObject(obj)) + if _, found := got.GetAnnotations()[RestartAtAnnotation]; found { + t.Fatal("expired retry not consumed") + } + if _, found, _ := unstructured.NestedString(got.Object, "spec", "template", "metadata", "annotations", RestartedAtTemplate); found { + t.Fatal("retry restarted after deadline") + } +} + +func TestCatchUpRecurringDeadlineReconcile(t *testing.T) { + anchor := time.Date(2026, 10, 1, 3, 0, 0, 0, time.UTC) + for _, action := range []string{RestartCronAnnotation, RestartEveryAnnotation} { + for _, late := range []time.Duration{10 * time.Minute, 20 * time.Minute} { + t.Run(action+late.String(), func(t *testing.T) { + now := anchor.Add(72*time.Hour + late) + value := "24h" + if action == RestartCronAnnotation { + value = "0 3 * * *" + } + obj := testDeployment("recurring-deadline", map[string]string{action: value, LastRestartTimestamp: anchor.Format(time.RFC3339), CatchUpAnnotation: "15m"}) + setTemplate(t, obj) + r := &LifecycleReconciler{Client: fake.NewClientBuilder().WithObjects(obj.DeepCopy()).Build(), Recorder: record.NewFakeRecorder(8), now: func() time.Time { return now }} + result, err := r.reconcileLogic(t.Context(), obj, logr.Discard()) + if err != nil || result.RequeueAfter <= 0 { + t.Fatalf("result=%+v error=%v", result, err) + } + persisted := getObject(t, r, client.ObjectKeyFromObject(obj)) + if persisted.GetAnnotations()[LastRestartTimestamp] == anchor.Format(time.RFC3339) || persisted.GetAnnotations()[action] != value { + t.Fatal("recurring state not advanced or schedule removed") + } + _, marker, err := unstructured.NestedString(persisted.Object, "spec", "template", "metadata", "annotations", RestartedAtTemplate) + if err != nil || marker != (late <= 15*time.Minute) { + t.Fatalf("marker=%t err=%v", marker, err) + } + if late > 15*time.Minute { + before, _, _ := unstructured.NestedMap(obj.Object, "spec", "template") + after, _, _ := unstructured.NestedMap(persisted.Object, "spec", "template") + if !reflect.DeepEqual(before, after) { + t.Fatal("skip changed template") + } + } + }) + } + } +} + +func TestCatchUpDryRunExpiredActionDoesNotAcknowledge(t *testing.T) { + for _, action := range []string{DeleteAtAnnotation, RestartAtAnnotation, RestartEveryAnnotation} { + t.Run(action, func(t *testing.T) { + now := time.Now().UTC() + value := now.Add(-time.Hour).Format(time.RFC3339) + if action == RestartEveryAnnotation { + value = "1h" + } + obj := testDeployment("dry-skip", map[string]string{action: value, CatchUpAnnotation: "1s", LastRestartTimestamp: now.Add(-90 * time.Minute).Format(time.RFC3339)}) + setTemplate(t, obj) + before := obj.DeepCopy() + _, tracked, recorder := reconcileForDryRunTest(t, obj, true) + assertNoTargetMutation(t, before, obj, tracked) + if event := <-recorder.Events; !strings.Contains(event, "DryRunActionSkipped") { + t.Fatalf("event=%q", event) + } + }) + } +} + +func TestCatchUpNotFoundAndNoActionForgetArms(t *testing.T) { + for _, missing := range []bool{false, true} { + t.Run("missing="+strconv.FormatBool(missing), func(t *testing.T) { + now := time.Now().UTC() + obj := testDeployment("removed-action", map[string]string{RestartAtAnnotation: now.Add(time.Minute).Format(time.RFC3339), CatchUpAnnotation: "never"}) + setTemplate(t, obj) + objects := []client.Object{obj.DeepCopy()} + if missing { + objects = nil + } + r := &LifecycleReconciler{Client: fake.NewClientBuilder().WithObjects(objects...).Build(), Recorder: record.NewFakeRecorder(8), now: func() time.Time { return now }} + if _, err := r.reconcileLogic(t.Context(), obj.DeepCopy(), logr.Discard()); err != nil { + t.Fatal(err) + } + if missing { + if _, err := r.Reconcile(t.Context(), timingRequest(obj)); err != nil { + t.Fatal(err) + } + } else { + noAction := obj.DeepCopy() + noAction.SetAnnotations(nil) + if _, err := r.reconcileLogic(t.Context(), noAction, logr.Discard()); err != nil { + t.Fatal(err) + } + } + policy, _ := ParseCatchUpPolicy("never") + due, _ := time.Parse(time.RFC3339, obj.GetAnnotations()[RestartAtAnnotation]) + if r.observeOccurrence(obj, restartAction, policy, due, due.Add(time.Minute), false) { + t.Fatal("removed resource/action retained eligibility") + } + }) + } +} diff --git a/internal/controller/catch_up_skip_test.go b/internal/controller/catch_up_skip_test.go new file mode 100644 index 0000000..fc27b97 --- /dev/null +++ b/internal/controller/catch_up_skip_test.go @@ -0,0 +1,211 @@ +package controller + +import ( + "context" + "errors" + "fmt" + "sync" + "testing" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" +) + +func catchUpSkipObject(t *testing.T, action string) *unstructured.Unstructured { + t.Helper() + annotations := map[string]string{} + if action == RestartEveryAnnotation || action == RestartCronAnnotation { + annotations[action] = "1h" + if action == RestartCronAnnotation { + annotations[action] = "0 * * * *" + } + annotations[LastRestartTimestamp] = catchUpTestStart.Format(time.RFC3339) + } else { + annotations[action] = catchUpTestStart.Add(time.Minute).Format(time.RFC3339) + } + obj := testDeployment("skip-action", annotations) + setTemplate(t, obj) + return obj +} + +func TestCatchUpSkipConflictPreservesConcurrentSchedule(t *testing.T) { + for _, action := range []string{DeleteAtAnnotation, RestartAtAnnotation, RestartEveryAnnotation, RestartCronAnnotation} { + t.Run(action, func(t *testing.T) { + now := catchUpTestStart.Add(3*time.Hour + time.Minute) + obj := catchUpSkipObject(t, action) + changedValue := now.Add(time.Hour).Format(time.RFC3339) + switch action { + case RestartEveryAnnotation: + changedValue = "2h" + case RestartCronAnnotation: + changedValue = "0 */2 * * *" + } + inject := true + funcs := interceptor.Funcs{ + Patch: func(ctx context.Context, c client.WithWatch, target client.Object, patch client.Patch, opts ...client.PatchOption) error { + if inject { + inject = false + current := obj.DeepCopy() + if err := c.Get(ctx, client.ObjectKeyFromObject(obj), current); err != nil { + return err + } + annotations := current.GetAnnotations() + annotations[action] = changedValue + annotations["example.com/concurrent"] = "preserved" + current.SetAnnotations(annotations) + if err := c.Update(ctx, current); err != nil { + return err + } + } + return c.Patch(ctx, target, patch, opts...) + }, + } + r := catchUpNeverReconciler(t, obj, &now, funcs) + _, err := r.Reconcile(context.Background(), timingRequest(obj)) + if !apierrors.IsConflict(err) { + t.Fatalf("skip patch error = %v, want conflict", err) + } + current := catchUpFetch(t, r, obj) + if got := current.GetAnnotations()[action]; got != changedValue { + t.Fatalf("schedule after conflict = %q, want %q", got, changedValue) + } + if got := current.GetAnnotations()["example.com/concurrent"]; got != "preserved" { + t.Fatalf("concurrent annotation = %q, want preserved", got) + } + if got := restartedAt(t, current); got != "" { + t.Fatalf("skip conflict changed template: %q", got) + } + if action == RestartEveryAnnotation || action == RestartCronAnnotation { + if got := current.GetAnnotations()[LastRestartTimestamp]; got != catchUpTestStart.Format(time.RFC3339) { + t.Fatalf("conflicted skip changed anchor: %q", got) + } + } + }) + } +} + +func TestCatchUpCommittedSkipIsNotReplayedAfterLostResponse(t *testing.T) { + for _, action := range []string{DeleteAtAnnotation, RestartAtAnnotation, RestartEveryAnnotation, RestartCronAnnotation} { + t.Run(action, func(t *testing.T) { + now := catchUpTestStart.Add(3*time.Hour + time.Minute) + obj := catchUpSkipObject(t, action) + lostResponse := errors.New("response lost after skip committed") + patches := 0 + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{ + Patch: func(ctx context.Context, c client.WithWatch, target client.Object, patch client.Patch, opts ...client.PatchOption) error { + patches++ + if err := c.Patch(ctx, target, patch, opts...); err != nil { + return err + } + return lostResponse + }, + }) + _, err := r.Reconcile(context.Background(), timingRequest(obj)) + if !errors.Is(err, lostResponse) { + t.Fatalf("skip patch error = %v, want lost response", err) + } + catchUpReconcile(t, r, obj) + if patches != 1 { + t.Fatalf("skip patch calls = %d, want 1", patches) + } + current := catchUpFetch(t, r, obj) + if action == RestartEveryAnnotation || action == RestartCronAnnotation { + wantAnchor := catchUpTestStart.Add(3 * time.Hour).Format(time.RFC3339) + if action == RestartCronAnnotation { + wantAnchor = now.Format(time.RFC3339) + } + if got := current.GetAnnotations()[LastRestartTimestamp]; got != wantAnchor { + t.Fatalf("skip anchor = %q, want %q", got, wantAnchor) + } + if got := restartedAt(t, current); got != "" { + t.Fatalf("committed skip changed template: %q", got) + } + } else { + catchUpAssertSkipped(t, r, obj, action) + } + }) + } +} + +func TestCatchUpNeverBlockedConfigurationClearsArm(t *testing.T) { + for _, blocked := range []string{"policy", "deadline", "dry-run", "conflict", "no action", "no annotations"} { + t.Run(blocked, func(t *testing.T) { + now := catchUpTestStart + due := now.Add(time.Minute) + obj := testDeployment("blocked-configuration", map[string]string{RestartAtAnnotation: due.Format(time.RFC3339)}) + setTemplate(t, obj) + r := catchUpNeverReconciler(t, obj, &now, interceptor.Funcs{}) + catchUpReconcile(t, r, obj) + catchUpUpdate(t, r, obj, func(current *unstructured.Unstructured) { + annotations := current.GetAnnotations() + switch blocked { + case "policy": + annotations[CatchUpAnnotation] = "invalid" + case "deadline": + annotations[RestartAtAnnotation] = "invalid" + case "dry-run": + annotations[DryRunAnnotation] = "invalid" + case "conflict": + annotations[DeleteAtAnnotation] = due.Format(time.RFC3339) + case "no action": + delete(annotations, RestartAtAnnotation) + annotations["example.com/other"] = "retained" + case "no annotations": + annotations = nil + } + current.SetAnnotations(annotations) + }) + catchUpReconcile(t, r, obj) + catchUpUpdate(t, r, obj, func(current *unstructured.Unstructured) { + current.SetAnnotations(map[string]string{RestartAtAnnotation: due.Format(time.RFC3339)}) + }) + now = due.Add(time.Second) + catchUpReconcile(t, r, obj) + catchUpAssertSkipped(t, r, obj, RestartAtAnnotation) + }) + } +} + +func TestCatchUpNeverConcurrentResourcesExecuteArmedActions(t *testing.T) { + now := catchUpTestStart + due := now.Add(time.Minute) + const count = 32 + objects := make([]*unstructured.Unstructured, count) + stored := make([]client.Object, count) + for i := range objects { + action := RestartAtAnnotation + if i%2 == 0 { + action = DeleteAtAnnotation + } + objects[i] = testDeployment(fmt.Sprintf("concurrent-action-%d", i), map[string]string{action: due.Format(time.RFC3339)}) + setTemplate(t, objects[i]) + stored[i] = objects[i].DeepCopy() + } + r := catchUpNeverReconciler(t, objects[0], &now, interceptor.Funcs{}) + r.Client = fake.NewClientBuilder().WithObjects(stored...).Build() + reconcileAll := func() { + var group sync.WaitGroup + for _, obj := range objects { + group.Go(func() { + if _, err := r.Reconcile(context.Background(), timingRequest(obj)); err != nil { + t.Errorf("reconcile %s: %v", obj.GetName(), err) + } + }) + } + group.Wait() + } + reconcileAll() + now = due.Add(time.Millisecond) + reconcileAll() + for i, obj := range objects { + action := RestartAtAnnotation + if i%2 == 0 { + action = DeleteAtAnnotation + } + catchUpAssertExecuted(t, r, obj, action) + } +} diff --git a/internal/controller/lifecycle_controller.go b/internal/controller/lifecycle_controller.go index 209076a..3eb77c5 100644 --- a/internal/controller/lifecycle_controller.go +++ b/internal/controller/lifecycle_controller.go @@ -3,12 +3,10 @@ package controller import ( "context" "fmt" - "math" "strconv" "time" "github.com/go-logr/logr" - "github.com/robfig/cron/v3" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" @@ -23,6 +21,7 @@ import ( const ( CronTimezoneAnnotation = "lifecycle.cezary.dev/cron-timezone" DryRunAnnotation = "lifecycle.cezary.dev/dry-run" + CatchUpAnnotation = "lifecycle.cezary.dev/catch-up" ReferencePointAnnotation = "lifecycle.cezary.dev/reference-point" DeleteAtAnnotation = "lifecycle.cezary.dev/delete-at" DeleteAfterAnnotation = "lifecycle.cezary.dev/delete-after" @@ -43,64 +42,6 @@ type resourceRequest struct { GVK schema.GroupVersionKind } -// recurringSchedule describes the two decisions needed by recurring restart -// reconciliation: the next occurrence and the anchor to persist after missed -// occurrences are coalesced. -type recurringSchedule interface { - Next(time.Time) time.Time - coalescedAnchor(lastRestart, now time.Time) time.Time -} - -type intervalRecurringSchedule struct { - duration time.Duration -} - -func (s intervalRecurringSchedule) Next(anchor time.Time) time.Time { - return anchor.Add(s.duration) -} - -func (s intervalRecurringSchedule) coalescedAnchor(lastRestart, now time.Time) time.Time { - return now.Add(-elapsedModulo(lastRestart, now, s.duration)) -} - -func elapsedModulo(start, end time.Time, interval time.Duration) time.Duration { - // time.Time.Sub saturates at the largest duration, so retain only each - // chunk's modulo while advancing across anchors farther than ~292 years. - var remainder time.Duration - for cursor := start; cursor.Before(end); { - chunk := end.Sub(cursor) - next := end - if chunk == time.Duration(math.MaxInt64) { - next = cursor.Add(chunk) - } - - chunk %= interval - if remainder >= interval-chunk { - remainder -= interval - chunk - } else { - remainder += chunk - } - cursor = next - } - return remainder -} - -type cronRecurringSchedule struct { - cron.Schedule -} - -func (cronRecurringSchedule) coalescedAnchor(_ time.Time, now time.Time) time.Time { - return now -} - -func planRecurringRestart(schedule recurringSchedule, lastRestart, now time.Time) (scheduledAt, coalescedAnchor time.Time, due bool) { - scheduledAt = schedule.Next(lastRestart) - if now.Before(scheduledAt) { - return scheduledAt, lastRestart, false - } - return scheduledAt, schedule.coalescedAnchor(lastRestart, now), true -} - // LifecycleReconciler reconciles objects with lifecycle annotations. type LifecycleReconciler struct { client.Client @@ -108,9 +49,11 @@ type LifecycleReconciler struct { Recorder eventRecorder Config ScopeConfig GlobalDryRun bool + ActionTiming ActionTimingConfig + now func() time.Time + timers actionTimers discovery preferredResourceDiscovery coverage *coverageState - now func() time.Time } // +kubebuilder:rbac:groups=*,resources=*,verbs=get;list;watch;delete;update;patch @@ -170,6 +113,7 @@ func (r *LifecycleReconciler) Reconcile(ctx context.Context, req resourceRequest obj.SetGroupVersionKind(req.GVK) if err := r.Get(ctx, req.NamespacedName, obj); err != nil { if apierrors.IsNotFound(err) { + r.forgetRequestTiming(req) logger.Info("object not found, likely deleted", "gvk", req.GVK) return ctrl.Result{}, nil } @@ -183,6 +127,7 @@ func (r *LifecycleReconciler) Reconcile(ctx context.Context, req resourceRequest func (r *LifecycleReconciler) reconcileLogic(ctx context.Context, obj client.Object, logger logr.Logger) (ctrl.Result, error) { annotations := obj.GetAnnotations() if len(annotations) == 0 { + r.forgetTiming(obj) return ctrl.Result{}, nil } @@ -201,6 +146,7 @@ func (r *LifecycleReconciler) reconcileLogic(ctx context.Context, obj client.Obj isDryRun, err := r.dryRunEnabled(obj) if err != nil { logger.Error(err, "invalid dry-run annotation; taking no action") + r.forgetTiming(obj) r.Recorder.Eventf(obj, "Warning", "InvalidAnnotationValue", "%v; taking no action", err) return ctrl.Result{}, nil } @@ -213,6 +159,7 @@ func (r *LifecycleReconciler) reconcileLogic(ctx context.Context, obj client.Obj hasRestartAnno := annotations[RestartAtAnnotation] != "" || annotations[RestartAfterAnnotation] != "" || annotations[RestartCronAnnotation] != "" || annotations[RestartEveryAnnotation] != "" if hasDeleteAnno && hasRestartAnno { + r.forgetTiming(obj) logger.Info("Conflict: Resource has both delete and restart annotations. Taking no action.", "resource", client.ObjectKeyFromObject(obj)) r.Recorder.Event(obj, "Warning", "ConflictingAnnotations", "Resource has both delete and restart annotations.") return ctrl.Result{}, nil @@ -225,269 +172,6 @@ func (r *LifecycleReconciler) reconcileLogic(ctx context.Context, obj client.Obj return r.handleRestart(ctx, u, isDryRun, logger) } + r.forgetTiming(obj) return ctrl.Result{}, nil } - -func (r *LifecycleReconciler) handleDeletion(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, logger logr.Logger) (ctrl.Result, error) { - annotations := obj.GetAnnotations() - - if deleteAfterStr := annotations[DeleteAfterAnnotation]; deleteAfterStr != "" { - if annotations[ReferencePointAnnotation] == ReferencePointCreationTimestamp && annotations[DeleteAtAnnotation] != "" { - logger.Info("Ignoring delete-after because delete-at is already set with creationTimestamp reference point") - if isDryRun { - logger.Info("[DRY-RUN] Would remove redundant delete-after annotation") - return ctrl.Result{}, nil - } - delete(annotations, DeleteAfterAnnotation) - obj.SetAnnotations(annotations) - markManagedBy(obj) - if err := r.Update(ctx, obj); err != nil { - logger.Error(err, "failed to remove redundant delete-after annotation") - return ctrl.Result{}, err - } - return ctrl.Result{RequeueAfter: time.Nanosecond}, nil - } - - duration, err := parseExtendedDuration(deleteAfterStr) - if err != nil { - logger.Error(err, "invalid duration format for delete-after annotation", "value", deleteAfterStr) - r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for delete-after annotation: %v", err) - return ctrl.Result{}, nil - } - referenceTime := r.getReferenceTime(obj, logger) - deletionTime := referenceTime.Add(duration) - - if isDryRun { - logger.Info("[DRY-RUN] Would convert delete-after to delete-at without modifying the resource", "deleteAfter", deleteAfterStr, "calculatedDeleteAt", deletionTime.UTC().Format(time.RFC3339)) - r.Recorder.Eventf(obj, "Normal", "DryRunDelete", "Dry-run: Resource would be scheduled for deletion at %s.", deletionTime.UTC().Format(time.RFC3339)) - return ctrl.Result{}, nil - } - - logger.Info("Converting delete-after to delete-at", "deleteAfter", deleteAfterStr, "calculatedDeleteAt", deletionTime.UTC().Format(time.RFC3339)) - newAnnotations := obj.GetAnnotations() - if newAnnotations == nil { - newAnnotations = make(map[string]string) - } - newAnnotations[DeleteAtAnnotation] = deletionTime.UTC().Format(time.RFC3339) - delete(newAnnotations, DeleteAfterAnnotation) - obj.SetAnnotations(newAnnotations) - markManagedBy(obj) - - if err := r.Update(ctx, obj); err != nil { - logger.Error(err, "failed to update object with delete-at annotation") - return ctrl.Result{}, err - } - return ctrl.Result{RequeueAfter: time.Nanosecond}, nil - } - - if deleteAtStr := annotations[DeleteAtAnnotation]; deleteAtStr != "" { - deleteAtTime, err := time.Parse(time.RFC3339, deleteAtStr) - if err != nil { - logger.Error(err, "invalid format for delete-at annotation", "value", deleteAtStr) - r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for delete-at annotation: %v", err) - return ctrl.Result{}, nil - } - - if !r.currentTime().Before(deleteAtTime) { - logger.Info("Deleting resource based on delete-at annotation", "targetTime", deleteAtTime.String()) - if !isDryRun { - uid := obj.GetUID() - resourceVersion := obj.GetResourceVersion() - preconditions := client.Preconditions{UID: &uid, ResourceVersion: &resourceVersion} - if err := r.Delete(ctx, obj, preconditions); err != nil { - if !apierrors.IsNotFound(err) { - logger.Error(err, "failed to delete object") - r.Recorder.Eventf(obj, "Warning", "DeletionFailed", "Failed to delete resource: %v", err) - return ctrl.Result{}, err - } - } - r.Recorder.Eventf(obj, "Normal", "Deleted", "Resource deleted based on delete-at annotation: %s", deleteAtStr) - } else { - logger.Info("[DRY-RUN] Would delete resource now") - r.Recorder.Event(obj, "Normal", "DryRunDelete", "Dry-run: Resource would be deleted now.") - } - return ctrl.Result{}, nil - } else { - result := r.requeueForNextOccurrence(deleteAtTime) - logger.Info("Resource deletion scheduled for a future time", "requeueAfter", result.RequeueAfter) - return result, nil - } - } - - return ctrl.Result{}, nil -} - -// handleRestart implements the full restart logic with precedence. -func (r *LifecycleReconciler) handleRestart(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, logger logr.Logger) (ctrl.Result, error) { - if !supportsRestart(obj) { - logger.Info("Skipping restart for an unsupported resource kind", "gvk", obj.GroupVersionKind()) - r.Recorder.Event(obj, "Warning", "UnsupportedRestartKind", "Restart requires an apps Deployment, StatefulSet, or DaemonSet; no action taken") - return ctrl.Result{}, nil - } - if !hasRestartTemplate(obj) { - logger.Info("Skipping restart because the Pod template is missing", "resource", client.ObjectKeyFromObject(obj)) - r.Recorder.Event(obj, "Warning", "NotPodSpawner", "Restart annotations are present but resource does not have a spec.template field.") - return ctrl.Result{}, nil - } - - annotations := obj.GetAnnotations() - now := r.currentTime() - - if restartAfterStr := annotations[RestartAfterAnnotation]; restartAfterStr != "" { - if annotations[ReferencePointAnnotation] == ReferencePointCreationTimestamp && annotations[RestartAtAnnotation] != "" { - logger.Info("Ignoring restart-after because restart-at is already set with creationTimestamp reference point") - if isDryRun { - logger.Info("[DRY-RUN] Would remove redundant restart-after annotation") - return ctrl.Result{}, nil - } - delete(annotations, RestartAfterAnnotation) - obj.SetAnnotations(annotations) - markManagedBy(obj) - if err := r.Update(ctx, obj); err != nil { - logger.Error(err, "failed to remove redundant restart-after annotation") - return ctrl.Result{}, err - } - return ctrl.Result{RequeueAfter: time.Nanosecond}, nil - } - - duration, err := parseExtendedDuration(restartAfterStr) - if err != nil { - logger.Error(err, "invalid duration format for restart-after annotation", "value", restartAfterStr) - r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for restart-after annotation: %v", err) - return ctrl.Result{}, nil - } - referenceTime := r.getReferenceTime(obj, logger) - restartTime := referenceTime.Add(duration) - - if isDryRun { - logger.Info("[DRY-RUN] Would convert restart-after to restart-at without modifying the resource", "restartAfter", restartAfterStr, "calculatedRestartAt", restartTime.UTC().Format(time.RFC3339)) - r.Recorder.Eventf(obj, "Normal", "DryRunRestart", "Dry-run: Resource would be scheduled for restart at %s.", restartTime.UTC().Format(time.RFC3339)) - return ctrl.Result{}, nil - } - - logger.Info("Converting restart-after to restart-at", "restartAfter", restartAfterStr, "calculatedRestartAt", restartTime.UTC().Format(time.RFC3339)) - newAnnotations := obj.GetAnnotations() - if newAnnotations == nil { - newAnnotations = make(map[string]string) - } - newAnnotations[RestartAtAnnotation] = restartTime.UTC().Format(time.RFC3339) - delete(newAnnotations, RestartAfterAnnotation) - obj.SetAnnotations(newAnnotations) - markManagedBy(obj) - if err := r.Update(ctx, obj); err != nil { - logger.Error(err, "failed to update object with restart-at annotation") - return ctrl.Result{}, err - } - return ctrl.Result{RequeueAfter: time.Nanosecond}, nil - } - - if restartAtStr := annotations[RestartAtAnnotation]; restartAtStr != "" { - restartTime, err := time.Parse(time.RFC3339, restartAtStr) - if err != nil { - logger.Error(err, "invalid format for restart-at annotation", "value", restartAtStr) - r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for restart-at annotation: %v", err) - return ctrl.Result{}, nil - } - if !now.Before(restartTime) { - logger.Info("Triggering one-time restart based on restart-at annotation", "restartTime", restartTime) - if err := r.triggerRestart(ctx, obj, isDryRun, func(annotations map[string]string) { - delete(annotations, RestartAtAnnotation) - }, logger); err != nil { - return ctrl.Result{}, err - } - return ctrl.Result{}, nil - } else { - result := r.requeueForNextOccurrence(restartTime) - logger.Info("One-time restart scheduled for a future time", "requeueAfter", result.RequeueAfter) - return result, nil - } - } - - if cronStr := annotations[RestartCronAnnotation]; cronStr != "" { - cronTimezone := annotations[CronTimezoneAnnotation] - if cronTimezone == "" { - cronTimezone = "UTC" - } - if _, err := time.LoadLocation(cronTimezone); err != nil { - logger.Error(err, "invalid cron timezone", "timezone", cronTimezone) - r.Recorder.Eventf(obj, "Warning", "InvalidTimezone", "Invalid timezone '%s' for restart-cron, taking no action.", cronTimezone) - return ctrl.Result{}, nil - } - - parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow) - schedule, err := parser.Parse(fmt.Sprintf("CRON_TZ=%s %s", cronTimezone, cronStr)) - if err != nil { - logger.Error(err, "invalid cron expression", "cron", cronStr) - r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid cron expression for restart-cron: %v", err) - return ctrl.Result{}, nil - } - if schedule.Next(r.currentTime()).IsZero() { - err := fmt.Errorf("cron expression has no future occurrence") - logger.Error(err, "invalid cron expression", "cron", cronStr) - r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid cron expression for restart-cron: %v", err) - return ctrl.Result{}, nil - } - return r.reconcileRecurringRestart(ctx, obj, isDryRun, "cron", cronRecurringSchedule{Schedule: schedule}, logger) - } - - if everyStr := annotations[RestartEveryAnnotation]; everyStr != "" { - duration, err := parseRestartEvery(everyStr) - if err != nil { - logger.Error(err, "invalid duration for restart-every", "duration", everyStr) - r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid duration for restart-every: %v", err) - return ctrl.Result{}, nil - } - return r.reconcileRecurringRestart(ctx, obj, isDryRun, "interval", intervalRecurringSchedule{duration: duration}, logger) - } - - return ctrl.Result{}, nil -} - -// reconcileRecurringRestart handles the stateful logic for both cron and interval restarts. -func (r *LifecycleReconciler) reconcileRecurringRestart(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, scheduleType string, schedule recurringSchedule, logger logr.Logger) (ctrl.Result, error) { - annotations := obj.GetAnnotations() - now := r.currentTime() - lastRestartStr := annotations[LastRestartTimestamp] - - if lastRestartStr == "" { - logger.Info("Initializing schedule by setting last-restart-timestamp", "type", scheduleType) - if isDryRun { - logger.Info("[DRY-RUN] Would initialize recurring restart state without modifying the resource", "type", scheduleType, "annotation", LastRestartTimestamp, "value", now.UTC().Format(time.RFC3339)) - return ctrl.Result{}, nil - } - annotations[LastRestartTimestamp] = now.UTC().Format(time.RFC3339Nano) - obj.SetAnnotations(annotations) - markManagedBy(obj) - if err := r.Update(ctx, obj); err != nil { - logger.Error(err, "failed to initialize last-restart-timestamp") - return ctrl.Result{}, err - } - return r.requeueForNextOccurrence(schedule.Next(now)), nil - } - - lastRestartTime, err := time.Parse(time.RFC3339, lastRestartStr) - if err != nil { - logger.Error(err, "failed to parse last-restart-timestamp", "value", lastRestartStr) - r.Recorder.Eventf(obj, "Warning", "InvalidState", "Could not parse last-restart-timestamp: %v", err) - return ctrl.Result{}, nil - } - - nextScheduledRestart, coalescedAnchor, restartDue := planRecurringRestart(schedule, lastRestartTime, now) - - if restartDue { - logger.Info("Triggering recurring restart", "type", scheduleType, "scheduledAt", nextScheduledRestart) - if err := r.triggerRestart(ctx, obj, isDryRun, func(annotations map[string]string) { - annotations[LastRestartTimestamp] = coalescedAnchor.UTC().Format(time.RFC3339Nano) - }, logger); err != nil { - return ctrl.Result{}, err - } - if isDryRun { - return ctrl.Result{}, nil - } - return r.requeueForNextOccurrence(schedule.Next(coalescedAnchor)), nil - } else { - result := r.requeueForNextOccurrence(nextScheduledRestart) - logger.Info("Next recurring restart is scheduled", "type", scheduleType, "at", nextScheduledRestart, "requeueAfter", result.RequeueAfter) - return result, nil - } -} diff --git a/internal/controller/lifecycle_delete.go b/internal/controller/lifecycle_delete.go new file mode 100644 index 0000000..63d13d4 --- /dev/null +++ b/internal/controller/lifecycle_delete.go @@ -0,0 +1,119 @@ +package controller + +import ( + "context" + "time" + + "github.com/go-logr/logr" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +func (r *LifecycleReconciler) handleDeletion(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, logger logr.Logger) (result ctrl.Result, err error) { + defer func() { + if err == nil && result.IsZero() { + r.forgetTiming(obj) + } + }() + policy, valid := r.catchUpPolicy(obj, deleteAction, isDryRun) + if !valid { + return ctrl.Result{}, nil + } + now := r.currentTime() + annotations := obj.GetAnnotations() + + if deleteAfterStr := annotations[DeleteAfterAnnotation]; deleteAfterStr != "" { + if annotations[ReferencePointAnnotation] == ReferencePointCreationTimestamp && annotations[DeleteAtAnnotation] != "" { + logger.Info("Ignoring delete-after because delete-at is already set with creationTimestamp reference point") + if isDryRun { + logger.Info("[DRY-RUN] Would remove redundant delete-after annotation") + return ctrl.Result{}, nil + } + delete(annotations, DeleteAfterAnnotation) + obj.SetAnnotations(annotations) + markManagedBy(obj) + if err := r.Update(ctx, obj); err != nil { + logger.Error(err, "failed to remove redundant delete-after annotation") + return ctrl.Result{}, err + } + r.armAbsolute(obj, deleteAction, policy, DeleteAtAnnotation, now) + return ctrl.Result{RequeueAfter: time.Nanosecond}, nil + } + + duration, err := parseExtendedDuration(deleteAfterStr) + if err != nil { + logger.Error(err, "invalid duration format for delete-after annotation", "value", deleteAfterStr) + r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for delete-after annotation: %v", err) + return ctrl.Result{}, nil + } + referenceTime := r.getReferenceTime(obj, logger) + deletionTime := referenceTime.Add(duration) + + if isDryRun { + logger.Info("[DRY-RUN] Would convert delete-after to delete-at without modifying the resource", "deleteAfter", deleteAfterStr, "calculatedDeleteAt", deletionTime.UTC().Format(time.RFC3339)) + r.Recorder.Eventf(obj, "Normal", "DryRunDelete", "Dry-run: Resource would be scheduled for deletion at %s.", deletionTime.UTC().Format(time.RFC3339)) + return ctrl.Result{}, nil + } + + logger.Info("Converting delete-after to delete-at", "deleteAfter", deleteAfterStr, "calculatedDeleteAt", deletionTime.UTC().Format(time.RFC3339)) + newAnnotations := obj.GetAnnotations() + if newAnnotations == nil { + newAnnotations = make(map[string]string) + } + newAnnotations[DeleteAtAnnotation] = deletionTime.UTC().Format(time.RFC3339) + delete(newAnnotations, DeleteAfterAnnotation) + obj.SetAnnotations(newAnnotations) + markManagedBy(obj) + + if err := r.Update(ctx, obj); err != nil { + logger.Error(err, "failed to update object with delete-at annotation") + return ctrl.Result{}, err + } + r.armAbsolute(obj, deleteAction, policy, DeleteAtAnnotation, now) + return ctrl.Result{RequeueAfter: time.Nanosecond}, nil + } + + if deleteAtStr := annotations[DeleteAtAnnotation]; deleteAtStr != "" { + deleteAtTime, err := time.Parse(time.RFC3339, deleteAtStr) + if err != nil { + logger.Error(err, "invalid format for delete-at annotation", "value", deleteAtStr) + r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for delete-at annotation: %v", err) + return ctrl.Result{}, nil + } + + allowed := r.observeOccurrence(obj, deleteAction, policy, deleteAtTime, now, isDryRun) + if !now.Before(deleteAtTime) { + if !allowed { + return ctrl.Result{}, r.skipAction(ctx, obj, deleteAction, deleteAtTime, isDryRun, func(a map[string]string) { + delete(a, DeleteAtAnnotation) + }) + } + logger.Info("Deleting resource based on delete-at annotation", "targetTime", deleteAtTime.String()) + if !isDryRun { + uid := obj.GetUID() + resourceVersion := obj.GetResourceVersion() + preconditions := client.Preconditions{UID: &uid, ResourceVersion: &resourceVersion} + if err := r.Delete(ctx, obj, preconditions); err != nil { + if !apierrors.IsNotFound(err) { + logger.Error(err, "failed to delete object") + r.Recorder.Eventf(obj, "Warning", "DeletionFailed", "Failed to delete resource: %v", err) + return ctrl.Result{}, err + } + } + r.Recorder.Eventf(obj, "Normal", "Deleted", "Resource deleted based on delete-at annotation: %s", deleteAtStr) + } else { + logger.Info("[DRY-RUN] Would delete resource now") + r.Recorder.Event(obj, "Normal", "DryRunDelete", "Dry-run: Resource would be deleted now.") + } + return ctrl.Result{}, nil + } else { + result := r.requeueForNextOccurrence(deleteAtTime) + logger.Info("Resource deletion scheduled for a future time", "requeueAfter", result.RequeueAfter) + return result, nil + } + } + + return ctrl.Result{}, nil +} diff --git a/internal/controller/lifecycle_predicate.go b/internal/controller/lifecycle_predicate.go index 71ba3f6..1b20b99 100644 --- a/internal/controller/lifecycle_predicate.go +++ b/internal/controller/lifecycle_predicate.go @@ -54,6 +54,11 @@ func lifecycleConfigurationChanged(e event.UpdateEvent) bool { if oldAnnotations[DryRunAnnotation] != newAnnotations[DryRunAnnotation] { return true } + oldCatchUp, oldFound := oldAnnotations[CatchUpAnnotation] + newCatchUp, newFound := newAnnotations[CatchUpAnnotation] + if oldCatchUp != newCatchUp || oldFound != newFound { + return true + } if oldAnnotations[ReferencePointAnnotation] != newAnnotations[ReferencePointAnnotation] && (hasRelativeAction(oldAnnotations) || hasRelativeAction(newAnnotations)) { return true diff --git a/internal/controller/lifecycle_predicate_test.go b/internal/controller/lifecycle_predicate_test.go index 5a1ac1e..b85c613 100644 --- a/internal/controller/lifecycle_predicate_test.go +++ b/internal/controller/lifecycle_predicate_test.go @@ -33,6 +33,10 @@ func TestLifecyclePredicateFiltersWorkqueueEvents(t *testing.T) { {name: "action value update", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "2h"}))), want: true}, {name: "action removal", got: p.Update(updateEvent(actionable, objectWithAnnotations(nil))), want: true}, {name: "dry-run update with action", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", DryRunAnnotation: "true"}))), want: true}, + {name: "catch-up update with action", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", CatchUpAnnotation: "never"}))), want: true}, + {name: "empty catch-up addition with action", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", CatchUpAnnotation: ""}))), want: true}, + {name: "catch-up removal with action", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", CatchUpAnnotation: "never"}), actionable)), want: true}, + {name: "catch-up update without action", got: p.Update(updateEvent(irrelevant, objectWithAnnotations(map[string]string{CatchUpAnnotation: "never"}))), want: false}, {name: "reference-point update with relative action", got: p.Update(updateEvent(actionable, objectWithAnnotations(map[string]string{DeleteAfterAnnotation: "1h", ReferencePointAnnotation: ReferencePointCreationTimestamp}))), want: true}, {name: "reference-point update with absolute action", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{DeleteAtAnnotation: "2030-01-01T00:00:00Z"}), objectWithAnnotations(map[string]string{DeleteAtAnnotation: "2030-01-01T00:00:00Z", ReferencePointAnnotation: ReferencePointCreationTimestamp}))), want: false}, {name: "timezone update with cron action", got: p.Update(updateEvent(objectWithAnnotations(map[string]string{RestartCronAnnotation: "0 * * * *"}), objectWithAnnotations(map[string]string{RestartCronAnnotation: "0 * * * *", CronTimezoneAnnotation: "Europe/Warsaw"}))), want: true}, diff --git a/internal/controller/lifecycle_restart.go b/internal/controller/lifecycle_restart.go new file mode 100644 index 0000000..7993ec0 --- /dev/null +++ b/internal/controller/lifecycle_restart.go @@ -0,0 +1,228 @@ +package controller + +import ( + "context" + "fmt" + "time" + + "github.com/go-logr/logr" + "github.com/robfig/cron/v3" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +// handleRestart implements the full restart logic with precedence. +func (r *LifecycleReconciler) handleRestart(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, logger logr.Logger) (result ctrl.Result, err error) { + defer func() { + if err == nil && result.IsZero() { + r.forgetTiming(obj) + } + }() + policy, valid := r.catchUpPolicy(obj, restartAction, isDryRun) + if !valid { + return ctrl.Result{}, nil + } + if !supportsRestart(obj) { + logger.Info("Skipping restart for an unsupported resource kind", "gvk", obj.GroupVersionKind()) + r.Recorder.Event(obj, "Warning", "UnsupportedRestartKind", "Restart requires an apps Deployment, StatefulSet, or DaemonSet; no action taken") + return ctrl.Result{}, nil + } + if !hasRestartTemplate(obj) { + logger.Info("Skipping restart because the Pod template is missing", "resource", client.ObjectKeyFromObject(obj)) + r.Recorder.Event(obj, "Warning", "NotPodSpawner", "Restart annotations are present but resource does not have a spec.template field.") + return ctrl.Result{}, nil + } + + annotations := obj.GetAnnotations() + now := r.currentTime() + + if restartAfterStr := annotations[RestartAfterAnnotation]; restartAfterStr != "" { + if annotations[ReferencePointAnnotation] == ReferencePointCreationTimestamp && annotations[RestartAtAnnotation] != "" { + logger.Info("Ignoring restart-after because restart-at is already set with creationTimestamp reference point") + if isDryRun { + logger.Info("[DRY-RUN] Would remove redundant restart-after annotation") + return ctrl.Result{}, nil + } + delete(annotations, RestartAfterAnnotation) + obj.SetAnnotations(annotations) + markManagedBy(obj) + if err := r.Update(ctx, obj); err != nil { + logger.Error(err, "failed to remove redundant restart-after annotation") + return ctrl.Result{}, err + } + r.armAbsolute(obj, restartAction, policy, RestartAtAnnotation, now) + return ctrl.Result{RequeueAfter: time.Nanosecond}, nil + } + + duration, err := parseExtendedDuration(restartAfterStr) + if err != nil { + logger.Error(err, "invalid duration format for restart-after annotation", "value", restartAfterStr) + r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for restart-after annotation: %v", err) + return ctrl.Result{}, nil + } + referenceTime := r.getReferenceTime(obj, logger) + restartTime := referenceTime.Add(duration) + + if isDryRun { + logger.Info("[DRY-RUN] Would convert restart-after to restart-at without modifying the resource", "restartAfter", restartAfterStr, "calculatedRestartAt", restartTime.UTC().Format(time.RFC3339)) + r.Recorder.Eventf(obj, "Normal", "DryRunRestart", "Dry-run: Resource would be scheduled for restart at %s.", restartTime.UTC().Format(time.RFC3339)) + return ctrl.Result{}, nil + } + + logger.Info("Converting restart-after to restart-at", "restartAfter", restartAfterStr, "calculatedRestartAt", restartTime.UTC().Format(time.RFC3339)) + newAnnotations := obj.GetAnnotations() + if newAnnotations == nil { + newAnnotations = make(map[string]string) + } + newAnnotations[RestartAtAnnotation] = restartTime.UTC().Format(time.RFC3339) + delete(newAnnotations, RestartAfterAnnotation) + obj.SetAnnotations(newAnnotations) + markManagedBy(obj) + if err := r.Update(ctx, obj); err != nil { + logger.Error(err, "failed to update object with restart-at annotation") + return ctrl.Result{}, err + } + r.armAbsolute(obj, restartAction, policy, RestartAtAnnotation, now) + return ctrl.Result{RequeueAfter: time.Nanosecond}, nil + } + + if restartAtStr := annotations[RestartAtAnnotation]; restartAtStr != "" { + restartTime, err := time.Parse(time.RFC3339, restartAtStr) + if err != nil { + logger.Error(err, "invalid format for restart-at annotation", "value", restartAtStr) + r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid format for restart-at annotation: %v", err) + return ctrl.Result{}, nil + } + allowed := r.observeOccurrence(obj, restartAction, policy, restartTime, now, isDryRun) + if !now.Before(restartTime) { + if !allowed { + return ctrl.Result{}, r.skipAction(ctx, obj, restartAction, restartTime, isDryRun, func(a map[string]string) { + delete(a, RestartAtAnnotation) + }) + } + logger.Info("Triggering one-time restart based on restart-at annotation", "restartTime", restartTime) + if err := r.triggerRestart(ctx, obj, isDryRun, func(annotations map[string]string) { + delete(annotations, RestartAtAnnotation) + }, logger); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{}, nil + } else { + result := r.requeueForNextOccurrence(restartTime) + logger.Info("One-time restart scheduled for a future time", "requeueAfter", result.RequeueAfter) + return result, nil + } + } + + if cronStr := annotations[RestartCronAnnotation]; cronStr != "" { + cronTimezone := annotations[CronTimezoneAnnotation] + if cronTimezone == "" { + cronTimezone = "UTC" + } + if _, err := time.LoadLocation(cronTimezone); err != nil { + logger.Error(err, "invalid cron timezone", "timezone", cronTimezone) + r.Recorder.Eventf(obj, "Warning", "InvalidTimezone", "Invalid timezone '%s' for restart-cron, taking no action.", cronTimezone) + return ctrl.Result{}, nil + } + + parser := cron.NewParser(cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow) + schedule, err := parser.Parse(fmt.Sprintf("CRON_TZ=%s %s", cronTimezone, cronStr)) + if err != nil { + logger.Error(err, "invalid cron expression", "cron", cronStr) + r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid cron expression for restart-cron: %v", err) + return ctrl.Result{}, nil + } + if schedule.Next(now).IsZero() { + err := fmt.Errorf("cron expression has no future occurrence") + logger.Error(err, "invalid cron expression", "cron", cronStr) + r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid cron expression for restart-cron: %v", err) + return ctrl.Result{}, nil + } + return r.reconcileRecurringRestart(ctx, obj, isDryRun, "cron", cronRecurringSchedule{Schedule: schedule}, logger) + } + + if everyStr := annotations[RestartEveryAnnotation]; everyStr != "" { + duration, err := parseRestartEvery(everyStr) + if err != nil { + logger.Error(err, "invalid duration for restart-every", "duration", everyStr) + r.Recorder.Eventf(obj, "Warning", "InvalidAnnotation", "Invalid duration for restart-every: %v", err) + return ctrl.Result{}, nil + } + return r.reconcileRecurringRestart(ctx, obj, isDryRun, "interval", intervalRecurringSchedule{duration: duration}, logger) + } + + return ctrl.Result{}, nil +} + +// reconcileRecurringRestart handles the stateful logic for both cron and interval restarts. +func (r *LifecycleReconciler) reconcileRecurringRestart(ctx context.Context, obj *unstructured.Unstructured, isDryRun bool, scheduleType string, schedule recurringSchedule, logger logr.Logger) (ctrl.Result, error) { + policy, valid := r.catchUpPolicy(obj, restartAction, isDryRun) + if !valid { + return ctrl.Result{}, nil + } + annotations := obj.GetAnnotations() + now := r.currentTime() + lastRestartStr := annotations[LastRestartTimestamp] + + if lastRestartStr == "" { + logger.Info("Initializing schedule by setting last-restart-timestamp", "type", scheduleType) + if isDryRun { + logger.Info("[DRY-RUN] Would initialize recurring restart state without modifying the resource", "type", scheduleType, "annotation", LastRestartTimestamp, "value", now.UTC().Format(time.RFC3339)) + return ctrl.Result{}, nil + } + annotations[LastRestartTimestamp] = now.UTC().Format(time.RFC3339Nano) + obj.SetAnnotations(annotations) + markManagedBy(obj) + if err := r.Update(ctx, obj); err != nil { + logger.Error(err, "failed to initialize last-restart-timestamp") + return ctrl.Result{}, err + } + r.observeOccurrence(obj, restartAction, policy, schedule.Next(now), now, false) + return r.requeueForNextOccurrence(schedule.Next(now)), nil + } + + lastRestartTime, err := time.Parse(time.RFC3339, lastRestartStr) + if err != nil { + logger.Error(err, "failed to parse last-restart-timestamp", "value", lastRestartStr) + r.forgetTiming(obj) + r.Recorder.Eventf(obj, "Warning", "InvalidState", "Could not parse last-restart-timestamp: %v", err) + return ctrl.Result{}, nil + } + + nextScheduledRestart, coalescedAnchor, restartDue := planRecurringRestart(schedule, lastRestartTime, now) + allowed := r.observeOccurrence(obj, restartAction, policy, nextScheduledRestart, now, isDryRun) + if restartDue && policy.mode == catchUpDeadline { + var recent time.Time + recent, allowed = occurrenceWithinWindow(schedule, lastRestartTime, now, policy.window) + if allowed { + nextScheduledRestart = recent + } + } + + if restartDue { + acknowledge := func(a map[string]string) { + a[LastRestartTimestamp] = coalescedAnchor.UTC().Format(time.RFC3339Nano) + } + if !allowed { + if err := r.skipAction(ctx, obj, restartAction, nextScheduledRestart, isDryRun, acknowledge); err != nil { + return ctrl.Result{}, err + } + } else { + logger.Info("Triggering recurring restart", "type", scheduleType, "scheduledAt", nextScheduledRestart) + if err := r.triggerRestart(ctx, obj, isDryRun, acknowledge, logger); err != nil { + return ctrl.Result{}, err + } + } + if isDryRun { + return ctrl.Result{}, nil + } + r.forgetTiming(obj) + r.observeOccurrence(obj, restartAction, policy, schedule.Next(coalescedAnchor), now, false) + return r.requeueForNextOccurrence(schedule.Next(coalescedAnchor)), nil + } else { + result := r.requeueForNextOccurrence(nextScheduledRestart) + logger.Info("Next recurring restart is scheduled", "type", scheduleType, "at", nextScheduledRestart, "requeueAfter", result.RequeueAfter) + return result, nil + } +} diff --git a/internal/controller/lifecycle_schedule.go b/internal/controller/lifecycle_schedule.go new file mode 100644 index 0000000..eacc1b4 --- /dev/null +++ b/internal/controller/lifecycle_schedule.go @@ -0,0 +1,66 @@ +package controller + +import ( + "math" + "time" + + "github.com/robfig/cron/v3" +) + +// recurringSchedule describes the two decisions needed by recurring restart +// reconciliation: the next occurrence and the anchor to persist after missed +// occurrences are coalesced. +type recurringSchedule interface { + Next(time.Time) time.Time + coalescedAnchor(lastRestart, now time.Time) time.Time +} + +type intervalRecurringSchedule struct { + duration time.Duration +} + +func (s intervalRecurringSchedule) Next(anchor time.Time) time.Time { + return anchor.Add(s.duration) +} + +func (s intervalRecurringSchedule) coalescedAnchor(lastRestart, now time.Time) time.Time { + return now.Add(-elapsedModulo(lastRestart, now, s.duration)) +} + +func elapsedModulo(start, end time.Time, interval time.Duration) time.Duration { + // time.Time.Sub saturates at the largest duration, so retain only each + // chunk's modulo while advancing across anchors farther than ~292 years. + var remainder time.Duration + for cursor := start; cursor.Before(end); { + chunk := end.Sub(cursor) + next := end + if chunk == time.Duration(math.MaxInt64) { + next = cursor.Add(chunk) + } + + chunk %= interval + if remainder >= interval-chunk { + remainder -= interval - chunk + } else { + remainder += chunk + } + cursor = next + } + return remainder +} + +type cronRecurringSchedule struct { + cron.Schedule +} + +func (cronRecurringSchedule) coalescedAnchor(_ time.Time, now time.Time) time.Time { + return now +} + +func planRecurringRestart(schedule recurringSchedule, lastRestart, now time.Time) (scheduledAt, coalescedAnchor time.Time, due bool) { + scheduledAt = schedule.Next(lastRestart) + if now.Before(scheduledAt) { + return scheduledAt, lastRestart, false + } + return scheduledAt, schedule.coalescedAnchor(lastRestart, now), true +} diff --git a/test/e2e/catch_up_test.go b/test/e2e/catch_up_test.go new file mode 100644 index 0000000..6086c1e --- /dev/null +++ b/test/e2e/catch_up_test.go @@ -0,0 +1,133 @@ +//go:build e2e + +package e2e + +import ( + "encoding/json" + "fmt" + "os/exec" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/cstanislawski/lifecycle-controller/test/utils" +) + +func catchUpTests(namespaces *namespaceCleanup) { + Context("Catch-up policy", func() { + var namespace string + const name = "catch-up-target" + + BeforeEach(func() { + namespace = namespaces.create("lifecycle-e2e-catch-up") + }) + + DescribeTable("skips a stale one-time action", func(action, policy string) { + applyCatchUpDeployment(namespace, map[string]string{ + "lifecycle.cezary.dev/" + action + "-at": time.Now().UTC().Add(-time.Hour).Format(time.RFC3339), + "lifecycle.cezary.dev/catch-up": policy, + }) + + Eventually(func(g Gomega) { + dep := utils.GetDeployment(name, namespace, g) + g.Expect(dep.Annotations).NotTo(HaveKey("lifecycle.cezary.dev/" + action + "-at")) + g.Expect(dep.Annotations["lifecycle.cezary.dev/catch-up"]).To(Equal(policy)) + g.Expect(dep.Spec.Template.Annotations).NotTo(HaveKey("lifecycle.cezary.dev/restartedAt")) + g.Expect(utils.GetEvents(g, namespace, name, "Deployment")).To( + ContainElement(HaveField("Reason", "ActionSkipped"))) + }).WithTimeout(30 * time.Second).WithPolling(pollInterval).Should(Succeed()) + + Consistently(func(g Gomega) { + dep := utils.GetDeployment(name, namespace, g) + g.Expect(dep.Spec.Template.Annotations).NotTo(HaveKey("lifecycle.cezary.dev/restartedAt")) + }).WithTimeout(2 * time.Second).WithPolling(pollInterval).Should(Succeed()) + }, + Entry("delete with never", "delete", "never"), + Entry("restart with never", "restart", "never"), + Entry("delete after its catch-up deadline", "delete", "15m"), + Entry("restart after its catch-up deadline", "restart", "15m"), + ) + + DescribeTable("executes a recent missed action", func(action string) { + applyCatchUpDeployment(namespace, map[string]string{ + "lifecycle.cezary.dev/" + action + "-at": time.Now().UTC().Add(-time.Minute).Format(time.RFC3339), + "lifecycle.cezary.dev/catch-up": "15m", + }) + expectCatchUpAction(name, namespace, action) + }, + Entry("delete within its catch-up deadline", "delete"), + Entry("restart within its catch-up deadline", "restart"), + ) + + DescribeTable("executes an armed action with never", func(action string) { + applyCatchUpDeployment(namespace, map[string]string{ + "lifecycle.cezary.dev/" + action + "-at": time.Now().UTC().Add(6 * time.Second).Format(time.RFC3339), + "lifecycle.cezary.dev/catch-up": "never", + }) + expectCatchUpAction(name, namespace, action) + }, + Entry("delete at a future deadline", "delete"), + Entry("restart at a future deadline", "restart"), + ) + + It("skips missed recurring restarts and executes the next armed occurrence", func() { + anchor := time.Now().UTC().Add(-330 * time.Second).Format(time.RFC3339) + applyCatchUpDeployment(namespace, map[string]string{ + "lifecycle.cezary.dev/restart-every": "1m", + "lifecycle.cezary.dev/last-restart-timestamp": anchor, + "lifecycle.cezary.dev/catch-up": "never", + }) + + Eventually(func(g Gomega) { + dep := utils.GetDeployment(name, namespace, g) + g.Expect(dep.Annotations["lifecycle.cezary.dev/last-restart-timestamp"]).NotTo(Equal(anchor)) + g.Expect(dep.Annotations["lifecycle.cezary.dev/restart-every"]).To(Equal("1m")) + g.Expect(dep.Annotations["lifecycle.cezary.dev/catch-up"]).To(Equal("never")) + g.Expect(dep.Spec.Template.Annotations).NotTo(HaveKey("lifecycle.cezary.dev/restartedAt")) + g.Expect(utils.GetEvents(g, namespace, name, "Deployment")).To( + ContainElement(HaveField("Reason", "ActionSkipped"))) + }).WithTimeout(10 * time.Second).WithPolling(pollInterval).Should(Succeed()) + + Eventually(func(g Gomega) { + dep := utils.GetDeployment(name, namespace, g) + g.Expect(dep.Spec.Template.Annotations).To(HaveKey("lifecycle.cezary.dev/restartedAt")) + g.Expect(dep.Annotations["lifecycle.cezary.dev/restart-every"]).To(Equal("1m")) + g.Expect(dep.Annotations["lifecycle.cezary.dev/catch-up"]).To(Equal("never")) + }).WithTimeout(65 * time.Second).WithPolling(pollInterval).Should(Succeed()) + }) + }) +} + +func applyCatchUpDeployment(namespace string, annotations map[string]string) { + encoded, err := json.Marshal(annotations) + Expect(err).NotTo(HaveOccurred()) + utils.ApplyYAML(fmt.Sprintf(` +apiVersion: apps/v1 +kind: Deployment +metadata: + name: catch-up-target + namespace: %s + annotations: %s +spec: + replicas: 0 + selector: { matchLabels: { app: catch-up-target }} + template: + metadata: { labels: { app: catch-up-target }} + spec: { containers: [ { name: nginx, image: nginx:latest } ] } +`, namespace, encoded)) +} + +func expectCatchUpAction(name, namespace, action string) { + Eventually(func(g Gomega) { + if action == "delete" { + _, err := utils.Run(exec.Command("kubectl", "get", "deployment", name, "-n", namespace)) + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring("not found")) + return + } + dep := utils.GetDeployment(name, namespace, g) + g.Expect(dep.Annotations).NotTo(HaveKey("lifecycle.cezary.dev/restart-at")) + g.Expect(dep.Spec.Template.Annotations).To(HaveKey("lifecycle.cezary.dev/restartedAt")) + }).WithTimeout(30 * time.Second).WithPolling(pollInterval).Should(Succeed()) +} diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go index 51d4e3a..62d7e40 100644 --- a/test/e2e/e2e_test.go +++ b/test/e2e/e2e_test.go @@ -82,6 +82,7 @@ var _ = Describe("Lifecycle Controller E2E", Ordered, func() { }) lifecycleActionTests(namespaces) + catchUpTests(namespaces) Context("Recurring and Advanced Actions", func() { var testNamespace string