Skip to content

feat(backup): shared collector + streaming server-side archive (phase 0) - #133

Merged
iiamit merged 1 commit into
mainfrom
feat/backup-phase0-streaming-archive
Aug 2, 2026
Merged

feat(backup): shared collector + streaming server-side archive (phase 0)#133
iiamit merged 1 commit into
mainfrom
feat/backup-phase0-streaming-archive

Conversation

@iiamit

@iiamit iiamit commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Groundwork for scheduled cloud backups (docs/plan-cloud-backups.md, added here). No user-visible change.

Why a second path instead of reusing the exporter

lib/backup/export.ts runs entirely in the browser and assembles the whole archive in memory as a Blob. An unattended backup has no browser — and the Cloud Run instance it would run on has memoryMiB: 1024 while a 300-page aircraft is ~600 MB of scans. Buffering that would OOM the instance serving live traffic.

What's here

  • collect.tscollectBackupData(): row collection + manifest, shared by both the browser export and the server builder, so the .zip a user downloads and the one pushed to their cloud cannot drift into different formats. Isomorphic on purpose: no server-only import, no storage access. Carries forward the query-order warning that PR fix(backup): export swapped pages ↔ log_entries (restore failed) #98 earned.
  • serverArchive.ts: streams via fflate's Zip — already a dependency, no new package. One blob resident at a time, flushed downstream before the next is read, with backpressure honoured between entries. Blobs are stored, not deflated, matching the browser path's level: 0: they're already-compressed JPEG/PNG/PDF, so deflating burns CPU on an instance shared with live traffic and saves nothing.
  • The blob reader is injected rather than imported, keeping the module free of server-only so it's unit-testable under the plain node:test runner — and so Phase 1 can pass lib/storage.ts's getBlob unchanged.
  • An unreadable blob is skipped and counted, never fatal. One missing scan must not cost the user the other 299 pages and every record.

The claim this design rests on, verified

Memory has to be bounded rather than proportional to archive size, or none of this works. Measured on a 600 MB archive under a 512 MB heap cap:

archive:    600 MB in 200 blobs (600 MB drained, 202 chunks)
RSS start:   76 MB
RSS peak:   171 MB   (growth 95 MB)

Bounded. (My first attempt at this measurement reported peak RSS: 0 MB because the sampler never fired — the number above is sampled synchronously per chunk instead.)

Tests

6 new cases in test/backup-server-archive.test.ts, including a format-equivalence test asserting the server archive and the browser export produce identical entry sets and byte-identical data.json. Nothing else would catch those two drifting apart — and if they drift, the archive sitting in someone's Dropbox quietly stops being restorable by the importer that handles the one they downloaded.

368/368 unit tests, typecheck, lint and build clean. e2e/backup-roundtrip.spec.ts is the guard that the refactor didn't change the format an actual restore depends on.

Plan decisions confirmed

Dropbox → Google Drive; Box dropped. Cadence off / monthly / quarterly.

https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R

Groundwork for scheduled cloud backups. No user-visible change.

The existing exporter runs entirely in the browser and assembles the whole
archive in memory as a Blob. An unattended backup has no browser, and the
Cloud Run instance it would run on has memoryMiB:1024 (apphosting.yaml)
while a 300-page aircraft is ~600 MB of scans. So this adds a second,
streaming path rather than reusing the first.

- collect.ts: collectBackupData() — row collection + manifest, shared by
  BOTH the browser export and the server builder so the .zip a user
  downloads and the one pushed to their cloud cannot drift into different
  formats. Isomorphic on purpose: no server-only import, no storage
  access. Carries the query-order warning that PR #98 earned.
- serverArchive.ts: streams via fflate's Zip (already a dependency — no
  new package). One blob resident at a time, flushed downstream before
  the next is read, with backpressure honoured between entries. Blobs are
  stored not deflated, matching the browser path's level:0 — they're
  already-compressed JPEG/PNG/PDF and deflating them would burn CPU on an
  instance shared with live traffic.
- The blob reader is injected rather than imported, so the module stays
  free of `server-only` and is unit-testable under node:test.
- An unreadable blob is skipped and counted, never fatal: one missing
  scan must not cost the user the other 299 pages and every record.

Measured on a 600 MB archive under a 512 MB heap cap: RSS grows ~95 MB,
i.e. bounded rather than proportional to archive size. That is the claim
the whole design rests on, so it is verified rather than asserted.

Tests include a format-equivalence check asserting the server archive and
the browser export produce identical entry sets and identical data.json —
nothing else would catch the two drifting apart.

Claude-Session: https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R
@iiamit
iiamit merged commit d187365 into main Aug 2, 2026
6 of 7 checks passed
@iiamit
iiamit deleted the feat/backup-phase0-streaming-archive branch August 2, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant