Skip to content

fix: bound pre-update backups and expose transfer controls - #632

Open
jtatum wants to merge 1 commit into
stubbi:mainfrom
jtatum:codex/fix-pre-update-backup
Open

jtatum wants to merge 1 commit into
stubbi:mainfrom
jtatum:codex/fix-pre-update-backup

Conversation

@jtatum

@jtatum jtatum commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Pre-update backups currently stop the instance for an unbounded full upload. A slow or stuck Job can keep the old version offline indefinitely, and users cannot exclude caches or tune concurrency. Add a configurable backup deadline with safe recovery, plus shared upload filters and transfer controls.

Fixes #631.

Behavior

  • spec.autoUpdate.backupTimeout defaults to 30m (clamped to 5m-24h). The persisted start includes scale-down, pod termination, Job scheduling, and retries. Jobs also receive activeDeadlineSeconds for the remaining budget.
  • On timeout or failure, persist AbortingBackup, foreground-delete the Job, wait for its Pods, then restart the old version without applying the update. Cleanup and scale-up errors are retried across reconciles/restarts. A new attempt waits for checkInterval; cleanup and application startup can extend beyond the backup budget.
  • Add spec.backup.excludes, transfers, checkers, and fastList to pre-update, pre-delete, and periodic uploads/snapshot copies. Existing defaults remain 8 transfers, 16 checkers, no excludes, and fast-list disabled. Filter text is quoted for shell commands and escaped for kubelet variable expansion. Restore Jobs remain unfiltered.
  • Retire a previous attempt's Job before creating a new backup, and correct rollback destination extraction to read the destination argument rather than /data/.
  • Regenerate deepcopy code, CRDs, Helm/OLM copies, and API docs; document the new settings and recovery behavior.

Pre-update backups still use independent timestamped destinations and upload a full snapshot. Incremental uploads are a follow-up; this change does not share the periodic backup's mutable latest path. Excluded data is unavailable in new snapshots, and existing remote objects are not purged by these filters.

Verification

  • Reproduced the unbounded wait on the original implementation: a 90-minute-old unfinished Job remained pending with replicas=0 after repeated reconciles. The regression now passes.
  • GOTOOLCHAIN=go1.25.0 make test passes (unit/envtest and all 35 installer tests), as does GOTOOLCHAIN=go1.25.0 make lint.
  • Helm/OLM CRD synchronization, Helm RBAC checks, and the chart image repository guard pass.
  • Real rclone:1.68 container with a loopback S3 server and synthetic files: both fresh destinations uploaded all 1,001 files; configured excludes reduced a fresh backup and periodic snapshot to one file; restoring the old full snapshot restored all 1,001 files. Run with OPENCLAW_BACKUP_CONTAINER_TEST=1 go test ./internal/controller -run '^TestBackupRcloneContainer$' -count=1 -v.
  • Disposable kind/Kubernetes 1.32.2 cluster: tested both controller timeout and Kubernetes DeadlineExceeded, real backup Pod deletion/garbage collection, unchanged image tag, and StatefulSet readiness recovery. The test drives the public reconciler directly and uses a BusyBox workload in place of OpenClaw. Run on a cluster with CRDs installed and no operator deployment using OPENCLAW_BACKUP_E2E=1 KUBECONFIG=... go test ./test/e2e -run '^TestPreUpdateBackupRecoveryOnCluster$' -count=1 -v. The disposable cluster was removed afterward.

The reporter's Cloudflare R2 throughput/90-minute wall-clock outage was not benchmarked; the regression reproduces its controller failure mode and the container test verifies the full-upload/filter behavior.

@ArthurBook

Copy link
Copy Markdown

Thanks @jtatum, this addresses the main problem in #631: the pre-update backup now has an upper bound, and excludes / transfers / checkers / fastList cover the tuning we asked for.

Two follow-ups:

  1. Incremental pre-update uploads. Each update still uploads a full snapshot. Could this be tracked in a separate issue so Pre-update backup: full non-incremental sync with fixed rclone flags, no excludes and no deadline keeps the instance down for long periods #631 can close on this PR?
  2. Repeated timeouts block updates without a visible error. If a backup always takes longer than backupTimeout (default 30m), every attempt aborts and the instance stays on the old version, retrying only after checkInterval. On our PVC (~29k files remaining after moving temp paths to emptyDir, ~820 files/min to R2) a full upload is estimated at ~35 min, so the default would block every update. Could the docs mention this, and could the abort be surfaced as a condition or Event (not only updatePhase) so it is easy to alert on?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pre-update backup: full non-incremental sync with fixed rclone flags, no excludes and no deadline keeps the instance down for long periods

2 participants