Skip to content

2.3: turn cloud sync back on through a shared schema - #9

Open
EnesYilmazcode wants to merge 18 commits into
rebuild/p3-run-enginefrom
rebuild/p4-sync
Open

EnesYilmazcode wants to merge 18 commits into
rebuild/p3-run-enginefrom
rebuild/p4-sync

Conversation

@EnesYilmazcode

@EnesYilmazcode EnesYilmazcode commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

Stack, merge in order: #5 > #6 > #7 > #8 > #9 (this one). Based on #8, so the diff here is only this step.
The proscan-web stack: EnesYilmazcode/proscan-web#1, EnesYilmazcode/proscan-web#2.

2.3 turns cloud sync back on. The audit found that what the extension wrote and what the dashboard read had never matched (F-25, F-29f), so this adds a shared schema first and makes sync write through it.

  • packages/schema/index.js has the validators and deterministic ids. The dashboard vendors the same file, and its check fails if the copy drifts. It is unchanged by the fixes below.
  • Each run gets one run id (F-20). Pages are queued only for the signed-in account, into an IndexedDB outbox (F-26). Each entry is deleted only after its commit, so a crash should not lose or duplicate it (F-21, F-22, F-24). A run's end entry goes to the account its pages went to, so signing out mid-run no longer leaves its cloud header active.
  • A product document is one merge write that sends deletes for the keys latest, prev and delta lack. latest is still replaced whole, and sourceIds now keeps every source instead of only the last one (NEW-SYNC-1).
  • firstSeenAt is written only when the product doc is missing (F-29b). Image, rank, prev and catalog size are now written (F-25).
  • In the popup: sign-up and reset links (F-56), a session-expired notice instead of a silent sign-out (F-57), and a sync status line.
  • A rules refusal no longer signs the user out. An entry the rules refuse (permission-denied or invalid-argument) is marked failed and skipped, so the scans behind it still sync; the popup counts them and Export to ProScan retries them. If every entry of a flush is refused, nothing is marked, since the account or the rules are the problem. An entry that fails validation or one of the rules' length caps is dropped instead of retried forever.
  • Tab events flush only when they ended the run, and after a failed flush the automatic ones back off (30 s, doubling to an hour). Export ignores the wait.

Write cost (F-23): a page of k products is 1 + 2k writes, and the run header and source now go once per run per flush instead of on every page. A 20 page run of 48 products a page is about 1,960 writes. Spark's 20,000 writes a day are shared by the whole project, so about 10 such runs a day across all users use them up. That is still far from the audit's target of about pages + 3 writes per run, because every product gets its own document and history write. Shipping 2.3 with CLOUD_SYNC on is a decision to make with that number in mind.

npm run test:contract runs the real sync module against the Firestore emulator with the dashboard's rules. It now refuses to start if one of its ports is taken, and on exit stops only the emulator processes it started (it used to taskkill whatever listened on its ports).

Tested: npm test is 704 jest and 36 tool tests. The contract test passes 7/7 against the rules on proscan-web #2, including a product in two sources and an entry the rules refuse. Playwright: 25 passed, 1 skipped.

Needs a change on proscan-web #2: its sync harness (scripts/lib/extension-sync.mjs) has to pass deleteField in its Firestore map, and the NEW-SYNC-1 known-failure marks in scripts/qa-sync-rules.mjs and scripts/e2e.mjs come off. With those two changes its test:rules passes against this branch; without them the strict marks report FAIL.

The dashboard rules from the proscan-web stack need to be deployed before 2.3 goes to the store.

Does not change permissions:

[permission-lock] OK manifest.json
[permission-lock] OK dist\manifest.json
[version-gate] OK: 2.3.0 > live 2.0

A permission-denied from Firestore means the rules rejected a document, not
that the session is gone. Treating it as expired signed the user out on every
flush.
…ever

Planning is pure, so an entry that cannot be planned, such as the end of a
run from before 2.3, would fail on every flush and hold up the rest of the
queue.
The runner used to taskkill every process listening on its ports on
exit, even when the emulators never started because another process
held them. It now checks the ports first and exits without starting or
stopping anything if one is taken, and on exit kills only listeners that
started after it did.
…and write the header once per flush

Product documents are now one merge write with deletes for the keys
latest, prev and delta lack, so the sourceIds arrayUnion keeps earlier
sources (NEW-SYNC-1) without an extra write. An entry the rules refuse
is marked failed and skipped so the rest still sync; Export retries it
and the popup counts it. Automatic flushes back off after a failure, and
tab events flush only when they ended the run. The run header and source
are written once per run per flush instead of on every page, and the
rules' length caps are checked before a commit. A run's end entry goes
to the account its pages were queued for, also for runs ended from
IndexedDB.
Sign-in is only for sending scans to the dashboard, so it now sits in a
collapsed "Dashboard sync (optional)" row under the export buttons, like
the AI chat settings. It unfolds on its own only when a session expires.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant