Repository navigation
Conversation
|
Thanks @jtatum, this addresses the main problem in #631: the pre-update backup now has an upper bound, and Two follow-ups:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-update backups currently stop the instance for an unbounded full upload. A slow or stuck Job can keep the old version offline indefinitely, and users cannot exclude caches or tune concurrency. Add a configurable backup deadline with safe recovery, plus shared upload filters and transfer controls.
Fixes #631.
Behavior
spec.autoUpdate.backupTimeoutdefaults to30m(clamped to5m-24h). The persisted start includes scale-down, pod termination, Job scheduling, and retries. Jobs also receiveactiveDeadlineSecondsfor the remaining budget.AbortingBackup, foreground-delete the Job, wait for its Pods, then restart the old version without applying the update. Cleanup and scale-up errors are retried across reconciles/restarts. A new attempt waits forcheckInterval; cleanup and application startup can extend beyond the backup budget.spec.backup.excludes,transfers,checkers, andfastListto pre-update, pre-delete, and periodic uploads/snapshot copies. Existing defaults remain 8 transfers, 16 checkers, no excludes, and fast-list disabled. Filter text is quoted for shell commands and escaped for kubelet variable expansion. Restore Jobs remain unfiltered./data/.Pre-update backups still use independent timestamped destinations and upload a full snapshot. Incremental uploads are a follow-up; this change does not share the periodic backup's mutable
latestpath. Excluded data is unavailable in new snapshots, and existing remote objects are not purged by these filters.Verification
replicas=0after repeated reconciles. The regression now passes.GOTOOLCHAIN=go1.25.0 make testpasses (unit/envtest and all 35 installer tests), as doesGOTOOLCHAIN=go1.25.0 make lint.rclone:1.68container with a loopback S3 server and synthetic files: both fresh destinations uploaded all 1,001 files; configured excludes reduced a fresh backup and periodic snapshot to one file; restoring the old full snapshot restored all 1,001 files. Run withOPENCLAW_BACKUP_CONTAINER_TEST=1 go test ./internal/controller -run '^TestBackupRcloneContainer$' -count=1 -v.DeadlineExceeded, real backup Pod deletion/garbage collection, unchanged image tag, and StatefulSet readiness recovery. The test drives the public reconciler directly and uses a BusyBox workload in place of OpenClaw. Run on a cluster with CRDs installed and no operator deployment usingOPENCLAW_BACKUP_E2E=1 KUBECONFIG=... go test ./test/e2e -run '^TestPreUpdateBackupRecoveryOnCluster$' -count=1 -v. The disposable cluster was removed afterward.The reporter's Cloudflare R2 throughput/90-minute wall-clock outage was not benchmarked; the regression reproduces its controller failure mode and the container test verifies the full-upload/filter behavior.