Repository navigation
Storage lifecycle: close delete leaks, move sidecars into D1, authoritative sizes - #43
Merged
Merged
Conversation
objectKeysFor / screenshotObjectKeysFor define every R2 object a recording or screenshot owns, in one place, so no delete path has to maintain its own (drifting) list. Pure extraction plus tests; kept free of runtime imports so cron.ts can stay wrangler-bundleable.
finalize fetched R2's object size and threw it away, persisting the client's claimed byte count instead — the number every quota check ran on. headObject becomes objectSize; both finalize routes store what R2 reports and re-check the cap when the truth exceeds the claim, deleting the over-cap object instead of keeping it. Webcam bytes now count toward the storage total, which previously ignored them entirely.
Interlocking changes to how recording/screenshot data lives and dies: - All three recording delete paths (dashboard action, DELETE /api/r/[id], retention sweep) now clear the full object closure; each previously stranded a different subset (webcam stream, config/summary sidecars). - recording_activity gains a real FK with ON DELETE CASCADE (migration 0008, table rebuild) — the constraint replaces per-path manual cleanup, two of three paths having forgotten it. - The daily sweep gains the screenshot arm idx_screenshots_gc was built for: same 30-days-from-last-view policy as recordings, soft-deleted rows purged after 24h with their objects re-checked, and a not-found page disclosing the policy (recordings already had one). - Both sweeps and the multipart GC drain their backlog in bounded pages instead of taking one LIMIT-100 sample per run, which silently falls behind forever once expiries outpace one batch per day. - Recording config and AI summary/chapters move from R2 JSON sidecars into recordings.config_json / summary_chapters_json (migration 0007): reads come off the row the page already holds (two R2 GETs per view gone), writes are plain scoped UPDATEs, and the sidecar object class — the one the delete paths kept leaking — stops being created at all. Legacy sidecar keys stay in objectKeysFor so pre-migration recordings still delete clean; scripts/backfill-recording-sidecars.py copies existing sidecar payloads into the new columns. Migrations are not yet applied remotely; run npx wrangler d1 migrations apply captureflow --remote then the backfill script (and re-run it once after deploying).
sardorml
force-pushed
the
fix/storage-leaks
branch
from
October 1, 2026 21:17
2801c29 to
3fbaefc
Compare
sardorml
marked this pull request as ready for review
October 1, 2026 21:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes out the storage-lifecycle audit: every R2 object and D1 row a recording or screenshot owns now has exactly one definition, one deletion path, and one accounting entry — and the numbers in D1 finally match the bytes in R2.
What was wrong (measured against prod)
DELETE /api/r/[id], retention cron) each cleaned a different subset of a recording's objects: webcam streams,*.config.jsonand*.summary-chapters.jsonsidecars, andrecording_activityrows leaked depending on which path ran.idx_screenshots_gcexisted for a sweep that was never written, and soft-deleted rows lived forever (4 ghost rows in prod).recordings.size_byteswas the client's claimed value; finalize fetched R2's real size and discarded it. Webcam bytes were invisible to quota entirely.LIMIT 100sample per day — a silent backlog cliff.cron.tsreferenced an R2 lifecycle safety-net rule that does not exist on the bucket.Changes
objectKeysFor/screenshotObjectKeysFor— single source of truth for the object closure; all delete paths and sweeps use it.recording_activityFK withON DELETE CASCADE(migration0008, table rebuild) — the constraint replaces all per-path manual cleanup.0007:config_json,summary_chapters_json) — reads come off the row the page already holds (−2 R2 GETs per view), writes are scoped UPDATEs through thehydrate*validators, and the leak-prone object class is never created again. Legacy keys stay in the delete closure./s/[id]not-found page discloses the policy like the recording one does.objectSize(néeheadObject) persists R2's size at finalize; if the real size exceeds the claim and busts the cap, the object is deleted and finalize returns 413.totalStorageForUsernow meters webcam bytes.Rollout order
Verification
pnpm typecheck(9 projects),pnpm --filter @captureflow/web test(43/43, incl. newobject-keys.test.ts), web build — all green.@opennextjs/cloudflare/next/*reaches the wrangler-bundled worker.0008copy filters orphaned activity rows (prod count: 0) and preserves ids.