feat(backup): shared collector + streaming server-side archive (phase 0) - #133
Merged
Conversation
Groundwork for scheduled cloud backups. No user-visible change. The existing exporter runs entirely in the browser and assembles the whole archive in memory as a Blob. An unattended backup has no browser, and the Cloud Run instance it would run on has memoryMiB:1024 (apphosting.yaml) while a 300-page aircraft is ~600 MB of scans. So this adds a second, streaming path rather than reusing the first. - collect.ts: collectBackupData() — row collection + manifest, shared by BOTH the browser export and the server builder so the .zip a user downloads and the one pushed to their cloud cannot drift into different formats. Isomorphic on purpose: no server-only import, no storage access. Carries the query-order warning that PR #98 earned. - serverArchive.ts: streams via fflate's Zip (already a dependency — no new package). One blob resident at a time, flushed downstream before the next is read, with backpressure honoured between entries. Blobs are stored not deflated, matching the browser path's level:0 — they're already-compressed JPEG/PNG/PDF and deflating them would burn CPU on an instance shared with live traffic. - The blob reader is injected rather than imported, so the module stays free of `server-only` and is unit-testable under node:test. - An unreadable blob is skipped and counted, never fatal: one missing scan must not cost the user the other 299 pages and every record. Measured on a 600 MB archive under a 512 MB heap cap: RSS grows ~95 MB, i.e. bounded rather than proportional to archive size. That is the claim the whole design rests on, so it is verified rather than asserted. Tests include a format-equivalence check asserting the server archive and the browser export produce identical entry sets and identical data.json — nothing else would catch the two drifting apart. Claude-Session: https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for scheduled cloud backups (
docs/plan-cloud-backups.md, added here). No user-visible change.Why a second path instead of reusing the exporter
lib/backup/export.tsruns entirely in the browser and assembles the whole archive in memory as aBlob. An unattended backup has no browser — and the Cloud Run instance it would run on hasmemoryMiB: 1024while a 300-page aircraft is ~600 MB of scans. Buffering that would OOM the instance serving live traffic.What's here
collect.ts—collectBackupData(): row collection + manifest, shared by both the browser export and the server builder, so the.zipa user downloads and the one pushed to their cloud cannot drift into different formats. Isomorphic on purpose: noserver-onlyimport, no storage access. Carries forward the query-order warning that PR fix(backup): export swapped pages ↔ log_entries (restore failed) #98 earned.serverArchive.ts: streams via fflate'sZip— already a dependency, no new package. One blob resident at a time, flushed downstream before the next is read, with backpressure honoured between entries. Blobs are stored, not deflated, matching the browser path'slevel: 0: they're already-compressed JPEG/PNG/PDF, so deflating burns CPU on an instance shared with live traffic and saves nothing.server-onlyso it's unit-testable under the plainnode:testrunner — and so Phase 1 can passlib/storage.ts'sgetBlobunchanged.The claim this design rests on, verified
Memory has to be bounded rather than proportional to archive size, or none of this works. Measured on a 600 MB archive under a 512 MB heap cap:
Bounded. (My first attempt at this measurement reported
peak RSS: 0 MBbecause the sampler never fired — the number above is sampled synchronously per chunk instead.)Tests
6 new cases in
test/backup-server-archive.test.ts, including a format-equivalence test asserting the server archive and the browser export produce identical entry sets and byte-identicaldata.json. Nothing else would catch those two drifting apart — and if they drift, the archive sitting in someone's Dropbox quietly stops being restorable by the importer that handles the one they downloaded.368/368 unit tests, typecheck, lint and build clean.
e2e/backup-roundtrip.spec.tsis the guard that the refactor didn't change the format an actual restore depends on.Plan decisions confirmed
Dropbox → Google Drive; Box dropped. Cadence
off / monthly / quarterly.https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R