feat(deployments): per-project Docker image retention with nightly pruning

[bherila]

Summary

Built deployment images accumulated indefinitely because the existing nightly Docker cleanup only pruned dangling tags. This adds a per-project image_retention_hours policy (NULL = 48-hour system default) and removes expired Temps-managed local images during the nightly cleanup pass.

Behavior and safety

• Adds migration m20260629_000001_add_image_retention_hours and exposes the setting through the project response and PATCH /projects/{id}/settings.
• Accepts 1–8760 hours; an explicit JSON null clears the override and restores the system default, while an omitted field leaves it unchanged.
• Considers only local temps-* tags, leaving registry/external image references alone.
• Removes each eligible image once and only when every deployment reference is older than its owning project retention cutoff, so a newer rollback or promotion preserves the reused image.
• Uses non-forced Docker removal. Images referenced by running or stopped containers are retained and logged rather than untagged underneath them.
• Removes the misleading ~0 MB freed report because per-image reclaimed-byte accounting is not available from this Docker API call.
• Regenerates the tracked web SDK types for the nullable retention field without pulling in unrelated API changes from other open PRs.

Validation

• cargo fmt --all -- --check
• cargo check -p temps-deployments -p temps-projects --lib
• tsc --noEmit
• Added regression coverage for newer image references, external-image exclusion, and omitted/null/value PATCH semantics.
• Built and booted the combined release binary locally; /healthz and /readyz both pass.
• Full GitHub Actions matrix is running on the rebuilt head.

This remains a draft until the full CI matrix is complete.

[dviejokfs]

@bherila I reviewed this and pushed the fixes to your branch (28101f8d) rather than leaving you a long list — shout if you'd rather I'd only commented. The conflicts with main are also resolved (merge, not rebase, so your commits are untouched); PR is MERGEABLE again.

The design was sound — AND-ing eligibility across every reference, restricting to local temps- tags, non-forced removal, and the omitted/null/value PATCH semantics were all right. Three things would have caused real damage in production though.

Data loss (the important one)

is_temps_managed_image matched any temps--prefixed local tag. Uploaded images are tagged temps-{slug}-{env}:upload-{ts} by the upload endpoint and the tarball isn't kept, so after the retention window the image was deleted with no source to rebuild it from — rollback and promotion both hard-fail with "image no longer exists locally" (services.rs:1408) and there's no way back. Now matched on deployment provenance (context_vars.trigger, metadata.externalImageRef/Id, non-git source_type) rather than tag text, since the upload endpoint accepts a caller-supplied tag.

Also protected: whatever each environment is currently serving (environments.current_deployment_id), regardless of age. Non-forced removal only protects an image while a container object still exists, so it isn't sufficient on its own. And images whose containers are on a worker node — this pass only talks to the local Docker daemon, so on a multi-node cluster it was trying to delete tags it doesn't have while worker disks grew unchecked.

The protection queries now abort the pass on error rather than failing open. A skipped night costs disk; a wrong deletion costs someone their deployment.

The 48h default

Since rollback needs the image, retention is the rollback window. At 48h any project that didn't deploy over a long weekend silently lost the ability to roll back to anything. I moved the default to 336h (14 days). That's a judgement call on my side — easy to change, and it's now an operator setting rather than a constant.

Unbounded query

find_also_related(projects) with .all() materialised every deployment row ever created, including deployment_config, context_vars, commit_json and metadata, plus a duplicated project row each. Now a 4-column partial select with retention overrides fetched separately into a map.

Also in the push

• AppSettings.image_retention (enabled + default_hours, clamped to 1..=8760) — with_default_image_retention_hours was never called from plugin.rs, so the "system default" was a recompile-only constant. Settings row rather than env var, per CLAUDE.md.
• UI + CLI. The feature was API-only. Added a card in project settings and temps projects settings --image-retention-hours / --reset-image-retention, both warning below 48h. Argument validation runs before the network call.
• Removed vs retained counts. A run where all 400 removals were refused logged "No expired deployment images to remove" — reads as healthy when nothing is happening.
• Audit. image_retention_hours added to ProjectSettingsUpdatedFields; shortening retention is destructive and should be attributable.
• Migration redated 20260629 → 20260803. It sorted into the middle of already-applied history and collided with m20260629_000001_otel_metrics_full_fidelity. It now applies last (verified: 163/163).
• One Docker connection per batch instead of per image; skip_serializing_if on the double-Option field; needs_preview_update → needs_project_row_update.

Evidence

cargo check --workspace --all-targets clean, cargo clippy --workspace --all-targets -- -D warnings exit 0, cargo fmt --check clean, tsc --noEmit clean in web/. Tests: 13 in docker_cleanup_service (all pass), plus 528/283/74/58 in temps-deployments/temps-core/temps-config/temps-projects.

Migration against a real Postgres:

→ [163/163] m20260803_000001_add_image_retention_hours … ✓ (11ms)
✓ 163 migration(s) applied in 2.18s.
projects.image_retention_hours | integer |

The retention rule is only as good as the non-forced removal underneath it, and that lives in bollard, so there's now a Docker-backed test (skips gracefully, no #[ignore]) that builds two real images, holds one with a container, and asserts the unreferenced one is actually gone while the in-use one is refused:

test test_real_docker_removes_unused_and_retains_in_use_image ... ok

Worth noting what that test caught: my first version tagged both images off a shared busybox base and the "in use" assertion failed — remove_image on a tag sharing an image ID with another tag just untags it without consulting container references. Temps builds one unique temps-{slug}:{deployment_id} tag per deployment so the shipping path is fine, but it's a sharp edge worth knowing about.

Two things I could not verify and am not claiming:

• down() was reviewed but not executed — temps migrate has no down command.
• The SDK deltas in web/src/api/client/types.gen.ts and apps/temps-cli/src/api/types.gen.ts were applied by hand, not by a real regen (that needs a running server + minted key). They typecheck, but please confirm with a full bun run openapi-ts before merge.

On the "what else should be pruned" question

prune_builder_cache already exists (7d), as do dangling-image prune, stale static chunks, CAS blob GC, and TimescaleDB retention on proxy_logs/otel_*/events/status_checks. Real gaps, roughly by disk impact:

1. Stopped containers — no prune_containers anywhere in the workspace. Every failed deploy and replaced container leaves an exited container holding its writable layer and pinning its image. This is also why non-forced removals here will often be refused, so it's arguably a prerequisite for image retention reclaiming anything.
2. Orphaned volumes — ~20 remove_volume call sites, all on explicit-delete happy paths. Nothing reconciles volumes whose owning service/sandbox row is gone. Should reconcile against the DB, never prune_volumes.
3. <data_dir>/backups/tmp — staging dir for dumps, no reaper. A backup that dies mid-dump leaves a full database dump forever.
4. /tmp transfer artifacts — temps-image-{uuid}.tar, temps-static-*, temps-bundle-*, temps-sourcemaps-*. Cleaned on success, leaked on panic/restart; the image tarballs are gigabytes.
5. Pipeline/deployment logs — pipeline_logs_dir() has no age cap.
6. Unused networks — small on disk, but per-preview-env networks hit the bridge-network limit and then new deployments fail.

I'd suggest doing those as a follow-up that restructures perform_cleanup into a list of named reapers each returning (items, bytes, errors) with its own age setting, rather than bolting them on one at a time. Happy to open that issue if useful. I deliberately kept this PR to the image-retention scope.

[dviejokfs]

@bherila heads-up — main moved while CI was running and PR #546 landed m20260803_000001_add_template_slug_to_projects, colliding with my redated migration on the same 20260803_000001 prefix. Merged main again and renumbered mine to m20260803_000002_add_image_retention_hours so the ordering is unambiguous rather than resolved by module-name sort.

Re-verified against a real Postgres:

→ [163/164] m20260803_000001_add_template_slug_to_projects … ✓ (5ms)
→ [164/164] m20260803_000002_add_image_retention_hours … ✓ (6ms)
✓ 164 migration(s) applied in 2.71s.

cargo fmt --check clean, cargo clippy --workspace --all-targets -- -D warnings exit 0, 13/13 cleanup tests pass. PR is MERGEABLE again on d5f88c93; CI re-running.

The full check matrix was green on the previous head (28101f8d) — all unit shards, all integration suites including migrations and docker-deployments, MariaDB PITR E2E, Web TypeScript. The only delta since then is the merge and the migration renumber.