feat(deletion): add durable whole-community deletion

[kalvinnchau]

Summary

Adds a durable, operator-controlled V1 for deleting an entire Buzz community without deleting another tenant's data.

The workflow is exposed through buzz-admin deletions:

• sweep records independent fleet storage-taxonomy observations
• submit, list, inspect, and approve manage a deletion request
• run and drain execute bounded work

Requests advance through a PostgreSQL-backed state machine and stop at retention_pending after logical deletion has been independently verified across PostgreSQL, object storage, and Redis.

This PR ships the engine and CLI, not a continuously running worker or Kubernetes packaging. For V1, a cluster/VM administrator invokes /usr/local/bin/buzz-admin from the existing relay image, for example with kubectl exec or an equivalent container/VM exec path.

What whole-community V1 removes

For the target community, V1 removes:

• rows from the allowlisted community-scoped PostgreSQL catalog, including members, profiles, authored events and bodies, DMs, reactions, mentions, memberships, tokens, workflows, moderation, audit, feedback, and rate-limit state
• media sidecars and upload-attribution records under _meta/<community>/ and _uploads/<community>/
• Git repository pointers under repos/<community>/
• Redis keys under buzz:<community>:*

The community row survives as a permanent tombstone, and deletion control-plane records remain as evidence of the request, approval, execution, and result.

Safety model

Deletion is not a broad DELETE CASCADE followed by optimistic cleanup. The destructive boundaries are durable and fail closed.

1. Inventory and approval

• submit resolves the target and freezes the schema plus summary-only storage inventory.
• Approval is bound to the exact request, community, and frozen inventory digest.
• Unsupported manifest versions, malformed keys inside the target's owned prefixes, schema/catalog drift, and approval mismatch block execution rather than guessing.
• Storage inventory is server-side prefix scoped to exactly:
  • _meta/<community>/
  • _uploads/<community>/
  • repos/<community>/
• The deletion path never lists the whole shared bucket and has no arbitrary per-community object cap. Its listing work is proportional to the target community's bindings, not total fleet storage.
• Fleet-wide taxonomy sweeps remain independent observability. They report unknown writer shapes but do not gate deletion submission, fencing, or destructive progress. Maintainers must add deletion taxonomy coverage whenever a new community-owned object-key class is introduced; writer-coverage tests bind the current media and Git writers to that contract.

2. Quiesce, fence, and destructive freeze

• Writes continue through submission, inventory, and approval. They stop when execution moves the target into quiescing and then establishes the durable fence.
• Already-admitted external effects finish under heartbeated serving-write leases; the executor drains those leases before destructive work.
• Database triggers enforce the community write fence across the complete catalog of community-scoped tables. Startup/readiness and destructive execution validate that catalog so a newly added but unfenced table cannot silently escape.
• After the fence closes writers, storage is re-enumerated into chunked side-table rows. Per-prefix counts and digests bind those concrete keys to the destructive manifest.
• Manifest chunk insertion, update, and deletion are protected after freeze. This closes the race where an unbound key could otherwise appear after the manifest was committed.

3. Checkpointed destruction

• Target-owned object bindings are deleted from the frozen destructive manifest in bounded batches with durable progress.
• The concrete key list lives in chunked side-table rows rather than one request-row JSON value. It supports large communities, resumable execution, and terminal cleanup.
• Missing objects are accepted as idempotent crash-window outcomes; malformed ownership, changed evidence, and unexplained target-prefix drift fail closed.
• PostgreSQL purging remains scoped by community_id, including the guarded NIP-RS hard-delete path discovered with real Desktop kind 30078 read-state data.
• Redis cleanup explicitly scans and UNLINKs only buzz:<community_id>:*. Natural expiry is insufficient because some keys, including tunnel generation counters used as fencing state, are deliberately persistent.

4. Independent verification

• PostgreSQL logical absence is checked after purge.
• The three target-owned storage prefixes are freshly inventoried again and must be empty.
• Redis requires two complete empty namespace scans.
• Only after all three stores pass does the request advance through logically_verified to retention_pending.

What V1 deliberately does not erase

Shared content-addressed storage

Per-community deletion removes bindings, metadata, attribution records, and Git pointers. It does not physically delete fleet-shared CAS bytes that another community may still reference:

• media blobs and thumbnails
• Git manifests, packs, and indexes (manifests/, packs/, and idx/)

Safe reclamation requires a separate fleet-wide reachability and retention GC. Unknown keys elsewhere in the shared bucket do not block one community's deletion; malformed or unrecognized keys inside that community's three owned prefixes still fail closed.

External retained copies

The online logical-deletion proof does not erase object versions/replicas, database backups/WAL, CDN copies, provider retention copies, or observability exports. Those require their own retention and purge controls.

Member-only erasure

This PR erases a whole community. It does not implement the different operation "erase one npub while preserving the community."

Removing membership or accepting NIP-09 is not member erasure. A member-only workflow would need to find and selectively remove or redact authored event content and pubkeys, profile data, DMs, reactions, mentions, memberships/roles, tokens, workflows/subscriptions, upload attribution, moderation/audit history, repository attribution, and identity embedded in tags or JSON. It would also need explicit rules for ownership transfer, surviving replies and thread metadata, audit-chain integrity, immutable Git history, and shared-CAS reachability. That requires a pubkey-level fence and selective graph rewrite; it is a separate deletion product, not a safe extension of this whole-tenant worker.

In scope

• migration 0027_community_deletion.sql: requests, approvals, leases, manifest chunks, checkpoints, tombstones, and the universal write-fence catalog
• durable executor leases, generations, heartbeats, retry/block state, and resumable stage transitions
• operator-driven sweep, submit, list, inspect, approve, run, and drain commands
• serving-path fences for database writes and external effects across event ingest, media, Git, workflow, push, invites, mesh/tunnel, and related paths
• target-prefix-only storage inventory, summary manifests, post-fence destructive chunks, and bounded batch deletion
• exact community Redis namespace purge and two-pass absence verification
• cross-community isolation, crash/resume, manifest-integrity, writer-taxonomy, and schema/migration regressions
• desired-state schema/schema.sql support without requiring a SQLx migration ledger

Deferred / not covered

• dedicated Helm/chart worker Deployment, service account, secrets, probes, resources, and network policy
• autonomous buzz-admin deletions worker poll loop and worker-only health server
• least-privilege separation among migration, relay-serving, and destructive execution roles
• fleet-wide shared-CAS physical GC
• backup/provider/CDN/observability retention completion
• member-only erasure
• provider-native conditional-delete improvements
• a general force-continue or operator repair escape hatch; permanent safety failures remain fail closed

The removed continuous-worker implementation is preserved on am/community-deletion-worker-followup for separate review.

Validation

Current PR head

Current head: c8c73018e03751de7da5f156eb1c6d61b0654626, rebased onto 985cdcc6eac33ccd77bc50c26e22c701d07eda4e.

At this head:

• cargo fmt --all -- --check passed
• affected package suites passed: buzz-db, buzz-media, buzz-deletion, and buzz-relay; the known relay telemetry test failed once in a combined run, then the complete relay rerun passed with 844 library and 11 binary tests
• the repository pre-push hook passed, including 2,145 Desktop tests and 158 Kubernetes backend tests
• branch, remote, and PR head were verified equal after the rebase
• CI is green for unit tests, Rust lint, security, server cross-compiles, Desktop core/build lanes, and completed image builds; remaining image/E2E platform jobs were still running when this description was updated
• PR diff: 34 files, 8,478 additions, 371 deletions, all scoped to deletion state, fencing, inventory, execution, cleanup, verification, or desired-state schema parity

Exact-head-before-final-prefix-change destructive matrix

The full destructive rerun used source bda58dc06a4f638077c60009eded5f796421a832 and immutable image buzz-e2e:sha-bda58dc06a4f (sha256:79a146bf341614854b0ca3c557ff679bcd1327f0f30e730ae2a82d733abc1575). That run covered:

• one Compose relay deployment serving target and control communities through different hostnames on the same listener
• PostgreSQL, Redis, and MinIO isolation between target and control
• guarded kind 30078 NIP-RS state
• Redis-down failure at fenced, durable retry metadata, and successful resume
• future-dated and stale taxonomy-sweep rejection at that revision
• rejection of direct post-freeze manifest-chunk insertion
• 10,001 target objects with production 10,000-key chunks, producing 10,000 + 1, deleting all target bindings, and cleaning the side table
• Docker Desktop Kubernetes with three relay replicas and manual in-pod sweep → submit → approve → run
• target tombstone/purge, control survival, shared-CAS survival, and all replicas Ready afterward

Evidence: /tmp/buzz-e2e/20260803T204244Z-bda58dc06a4f/ (CASE_LEDGER.md, 179-entry SHA256SUMS, and raw artifacts). Compose resources and the Kubernetes namespace were removed afterward.

The final commit changed only deletion inventory/gating semantics: it replaced fleet-wide deletion-path listing with the three target-prefix listings, removed the deletion-path cap and sweep dependency, retained fail-closed validation under owned prefixes, and added writer-coverage tests. The full destructive Compose/Kubernetes matrix has not been rerun at c8c73018e; this description does not relabel the older image evidence as exact-head evidence.

One boundary remains explicitly untested: deterministic post-S3/pre-checkpoint crash injection. The product exposes no deterministic fault point there, and a timing kill would manufacture confidence rather than evidence.

Rollout constraint

Do not approve a deletion during a mixed-version relay rollout. Apply migration 0027, roll every serving replica to this release or newer, verify old replicas are gone and deletion serving-lease metrics are healthy, then invoke deletion execution. Older replicas do not participate in the external-effect lease protocol required to make fencing safe.