fix(cli): snapshot the blob volume unless blobs live in external S3

[larryro]

Data loss: tale backup never captured uploaded files

Before. SNAPSHOT_VOLUMES (tools/cli/src/lib/backup/constants.ts) was db-data, convex-data, caddy-data, caddy-config. The blob store's volume object-store-data — every uploaded document, chat attachment, audio file and generated media of a self-hosted deployment — was in no snapshot. An operator who ran tale backup, copied the backups volume off-host as the docs instruct, and later rebuilt a host from it got their database, config and certificates back and lost every blob, with nothing in the CLI output hinting at it.

After. object-store-data is one of the snapshot volumes and is restored symmetrically. When the deployment's blobs live in an external S3 instead, the local volume is skipped and tale backup prints a one-line notice naming the endpoint and bucket; organizations that bring their own bucket are named the same way. Either way the omission is visible, never silent.

Root cause

A regression by omission, not a design decision. The backup feature (#1863, 2026-06-11) predates the object store: blobs then lived in Convex _storage inside convex-data, which is snapshotted. #3107 (2026-08-30) moved blobs to MinIO on object-store-data and did not update the snapshot set; the constant's own doc-comment still promised "every project volume that holds non-rederivable state". Today's docs alignment (#3154) then documented the gap as "deliberately skips" — describing the status quo, not a rationale.

The LOCAL / BYO-S3 signal — reused, not invented

There is no deployment-level env switch in the CLI: the generated compose hardcodes OBJECT_STORE_ENDPOINT: http://object-store:9000 for both backend tiers and always ships the object-store service. Where blobs live is decided by files the backend resolves (backend/lib/object-store.ts): <slug>/object-storage/connection.json (an org's own bucket) → default/object-storage/connection.json (the deployment default, seeded by domains/object_storage/bootstrap.ts against the bundled store) → fail closed. The bootstrap tells the bundled store from an operator-repointed one by comparing the default connection's endpoint with the bundled address.

The CLI now reads exactly those files from the config volume (convex-data, mounted at TALE_CONFIG_DIR in the backend tier) through the same pinned alpine helper the backup already uses, and applies the same endpoint comparison. The bundled address the compose generators hardcoded twice is hoisted into BUNDLED_OBJECT_STORE_ENDPOINT (compose/generators/constants.ts) so the generators and the backup read one constant.

Default connection points at	Blob volume	Operator sees
bundled object-store	captured	volume in the snapshot list
external S3	skipped	Blobs live in external S3 (<endpoint>, bucket "<bucket>") — not in this snapshot: back that bucket up yourself; the local object-store-data volume is skipped.
unreadable / not seeded / config volume unreadable	captured (fail-safe)	debug line; a warn when the inspection itself failed
any, with orgs owning a bucket	as above	Blobs of N organization(s) with their own bucket (a, b) live in those buckets — not in this snapshot: back them up under your own contract.

Per seam

1. fix(cli): snapshot the blob volume unless blobs live in external S3 — constants.ts (BLOB_VOLUME, CONFIG_VOLUME, blob volume in SNAPSHOT_VOLUMES, archiveTimeoutSeconds()), new inspect-blob-store.ts, create-snapshot.ts (candidate set + notices), bundled-endpoint hoist.
   Tests (create-snapshot.test.ts, +6): full local stack → manifest carries object-store-data, the inspection mounts convex-data read-only, the blob tar gets the wider bound (red on main: the snapshot set lacked the volume); external default → volume skipped, exactly one notice with endpoint + bucket; own-bucket orgs → captured, orgs named; inspection failure, unreadable default file, never-seeded store → captured.
2. fix(cli): restore the blob archive and flag snapshots without one — restore.ts: extract uses the shared per-volume bound; a snapshot with no blob archive prints a notice before confirmation and leaves the volume untouched; the listing marks such snapshots · without blobs.
   Tests (restore.test.ts, +3): manifest with blob archive → three wipe+extract runs, blob volume re-created, wider bound (red on main: the allow-list dropped it); older manifest → two runs, notice; listing marker.
3. test(cli): cover listing and verifying snapshots with the blob archive — new list-snapshots.test.ts (+4) and verify-snapshot.test.ts (+3). Rotation deletes whole snapshot directories (rm -rf /backup/<id>) and is archive-agnostic — verified by reading, nothing for a test to express.
4. docs(cli): say which blobs a backup captures and which are yours — en/de/fr self-hosted/operate/backups-and-restore.md (volume table row, the blob paragraph, pause duration, restore of older snapshots), self-hosted/overview.md caveat, self-hosted/install/linux-server.md backup hook. DE/FR authored natively, UI label matches the shipped Datenresidenz / Résidence des données.

Trade-offs worth knowing

• The blob archive is as large as the store and the store is paused while it is tarred (uploads/downloads stall). Pre-deploy snapshots now include it. The blob volume gets a 4 h bound (others keep 30 min); both guard against a hung docker, not a slow but progressing tar. Documented on the backups page.
• tar czf gzips blobs that are already compressed — wasted CPU, roughly 30–50 MB/s. Switching the blob archive to an uncompressed tar is a format change (archive name, verify glob, restore extract) and priced as its own PR; it does not change what is captured.

Gates (observed)

• bun test in tools/cli: 327 pass / 0 fail (345 ran, pre-existing skips); baseline for the three touched files was 17 pass, now 33 (+16).
• bunx tsc --noEmit (after bun run generate for the gitignored embedded files): exit 0. bunx oxlint: exit 0.
• bun run --filter @tale/docs test: 30 files / 194 tests pass.
• The inline sh inspection script: sh -n clean; executed against a fake <slug>/object-storage/connection.json tree under dash and busybox sh (alpine's shell): rows parse, escaped \n inside JSON strings survive, empty tree exits 0.
• Not run (per brief): docker builds / live stacks. The end-to-end tale backup → tale restore round-trip on a real stack is the one thing left to observe.

Cross-class discoveries (not touched here)

• generateDevCompose emits the backend tier with depends_on: object-store but defines no object-store service — the tale dev stack cannot satisfy that dependency.
• llm-gateway-data (the sandbox LLM gateway's Bifrost state) is also outside the snapshot set; whether it is rederivable is unassessed.