2.3: turn cloud sync back on through a shared schema

CLAUDE.md

@@ -4,11 +4,11 @@
 
 Chrome extension (Manifest V3) that scrapes Amazon seller product listings, provides analytics for resellers/arbitrage, and includes a floating AI chatbot on Amazon pages powered by Gemini API. No server required — everything runs client-side.
 
-## Architecture (v2.2)
+## Architecture (v2.3)
 
 ```text
 AmazonSellerScraper/
-├── manifest.json              # Extension config (v2.2)
+├── manifest.json              # Extension config (v2.3)
 ├── popup/                     # UI Layer
 │   ├── popup.html            # Popup interface (dashboard + settings)
 │   ├── popup.css             # Popup styling
@@ -23,19 +23,23 @@ AmazonSellerScraper/
 │   │   ├── parsers.js        # Pure search/offer parsing (global Parsers)
 │   │   ├── messages.js       # Every message type and who may send it (global Msg)
 │   │   ├── run.js            # Run state machine, bound to one tab (global Run)
-│   │   ├── flags.js          # Build flags (global Flags); CLOUD_SYNC is off until 2.3
+│   │   ├── flags.js          # Build flags (global Flags); CLOUD_SYNC is on from 2.3
 │   │   ├── migrate.js        # Storage schema migrations, run by the SW
 │   │   └── chat.js           # Gemini request builder, run scoping, error text
 │   ├── background/
 │   │   ├── service-worker.js # Wires router, engine, chat, auth and migrations
 │   │   ├── router.js         # The one typed message router
 │   │   ├── engine.js         # The run engine; the only writer of run data
-│   │   └── db.js             # IndexedDB: runs, products, placements, lastValues, outbox, spread
+│   │   ├── db.js             # IndexedDB: runs, products, placements, lastValues, outbox, spread
+│   │   ├── sync-plan.js      # Outbox entry -> cloud writes (pure)
+│   │   └── sync.js           # Drains the outbox into Firestore, one entry at a time
 │   └── modules/
 │       ├── storage.js        # Chrome storage wrapper
 │       ├── analyzer.js       # Data analysis & insights
 │       ├── spread-analyzer.js # Price spread & arbitrage scoring
 │       └── exporter.js       # Excel/CSV/JSON export
+├── packages/
+│   └── schema/index.js       # Cloud schema: types, validators, ids (shared with the dashboard)
 ├── styles/
 │   └── chatbot.css           # Chatbot widget styles (loaded into Shadow DOM)
 ├── tests/                     # Test suite (Jest)
@@ -102,6 +106,37 @@ Run states: idle, starting, running, stopping, then stopped, blocked, failed or
 `reason` says why it ended: complete, stopped, blocked, selectors_broken, storage_full,
 storage_error, interrupted or updated.
 
+### Cloud sync (2.3)
+
+1. The popup signs in with email and password through the SW (`PROSCAN_SIGN_IN`); sign-up opens
+   the dashboard, and "Forgot password?" sends `PROSCAN_RESET_PASSWORD` (same answer either way)
+2. The SW keeps `account` {uid, email} in `chrome.storage.local` while signed in. The engine
+   queues `{kind:'page', pageIndex, uid}` per saved page and `{kind:'run', uid}` when a run ends,
+   only when an account is signed in
+3. `lastValues` snapshots carry the uid that took them; another account's snapshot gives no delta
+4. `scheduleFlush()` runs a few seconds after PAGE_RESULT, STOP_RUN, a tab change that ended the
+   run, popup open, sign-in, browser start and update. No alarms. After a failed flush the
+   automatic ones back off (30 s, doubling to 1 h, `lastSync.failures`); Export does not wait
+5. `sync.js` drains the outbox for the signed-in uid, oldest first. `sync-plan.js` turns an entry
+   into writes; most are `set` with `mergeFields`, each field replaced whole.
+   A product document is a `merge: true` write instead, so the `sourceIds` arrayUnion keeps earlier
+   sources; the keys its `latest`, `prev` and `delta` lack are sent as deletes.
+   The run header and source go on each run's last entry in a flush round.
+   `firstSeenAt` is written only when a read shows the product document does not exist
+6. An entry is deleted after its own commit. Entries of another uid are left alone. An entry
+   that fails schema validation or a rules cap (`RULE_CAPS` in sync-plan.js) is dropped, since
+   retrying it would fail the same way. The run's end entry goes to `run.syncUid`, the account its
+   pages were queued for
+7. An expired or invalid token signs out and sets `authNotice: 'expired'`; the popup shows
+   "Your session expired". A `permission-denied` or `invalid-argument` (the rules refusing a
+   write) is not a sign-out: the entry is marked `failed` and skipped so the rest go on, and
+   Export retries it. If every entry of a flush is refused, nothing is marked and the error is
+   thrown, since the account or the rules are the problem
+
+Cloud paths and shapes: `packages/schema/index.js` (keep the dashboard's copy identical, bump `SV`
+on a shape change). Run id `{sourceId}_{startMs}`, minted once in `engine.start`; page id `p0001`;
+`dayKey` is the local date the run started.
+
 ### AI Chatbot (client-side, no server)
 
 1. `chatbot.js` injects a floating widget (bottom-right) on Amazon pages with product listings
@@ -134,14 +169,17 @@ npm run test:coverage # With coverage report
 - Pure logic modules (`analyzer.js`, `spread-analyzer.js`) are tested via `require()` directly
 - Content scripts (`scraper.js`, `offer-fetcher.js`) have no `module.exports` — loaded via `vm.runInContext` into a JSDOM context with Chrome API mocks and an `innerText` polyfill
 - `tests/setup/engine-rig.js` runs the real router and engine over fake-indexeddb with the real `scraper.js` in a JSDOM page per tab; `tests/unit/engine.test.js` drives runs through it
+- `npm run test:contract` runs the real `sync.js` against the Firebase emulators with the dashboard's
+  `firestore.rules` (`tools/contract.mjs`, `tests/contract/`); ESM files under `packages/` and
+  `scripts/background/` load in Jest through `tests/setup/esm-to-cjs-transform.js`
 - `npm run test:e2e` runs the same scenarios in Chromium (`tests/e2e/`), including the worker stopped via CDP between pages and 2.0 and 2.1 builds updated mid-run
 - HTML fixtures in `tests/fixtures/` match the exact CSS selectors the code uses
 - Chrome APIs (`storage`, `runtime`, `tabs`, `downloads`) are mocked in `tests/setup/chrome-mock.js`
 - XLSX is mocked with jest.fn() stubs in exporter.test.js; export-xlsx.test.js uses the real libs/xlsx.full.min.js
 
 ## Chrome APIs Used
 
-- `chrome.storage.local`: settings, the Gemini key and `schemaVersion` only
+- `chrome.storage.local`: settings, the Gemini key, `schemaVersion`, and `account`, `authNotice`, `lastSync`
 - `chrome.storage.session`: the live run record (content scripts cannot read it)
 - IndexedDB: run data, in the extension origin; needs no permission
 - `chrome.runtime.sendMessage/onMessage`: messages, all in `scripts/lib/messages.js`
@@ -196,6 +234,8 @@ One record per ASIN per run, from `Parsers.parseSearchPage` and the scraper:
 - `url`: always `https://www.amazon.com/dp/{asin}`, never the sspa ad link
 - `sponsored`: true if any placement was an ad; `organicRank`: run-wide rank of the first organic card, or null
 - `placements`: every card the ASIN had, as `{page, position, sponsored, rank}`
+- `img`: the card image on Amazon's image host, or null
+- `prev`: the `lastValues` snapshot the delta was taken against, or null
 - `delta`: taken once, on the ASIN's first sighting in the run, against `lastValues` from earlier runs.
   A field that fails to parse keeps its last good value in `lastValues`, with the time in `carried`
 
@@ -210,16 +250,16 @@ One record per ASIN per run, from `Parsers.parseSearchPage` and the scraper:
   the IndexedDB writes commit before anything is removed.
 - IndexedDB `proscan` (`scripts/background/db.js`): `runs`, `products` (key `[runId, n]`,
   n is the order found), `placements` (one record per page), `lastValues` (by ASIN),
-  `outbox` (pages waiting for sync), `spread` and `meta` (`latestRunId`). The popup shows
+  `outbox` (`{kind, runId, uid, pageIndex}` entries waiting for sync), `spread` and `meta` (`latestRunId`). The popup shows
   the latest run's products.
   `tests/fixtures/v2.0-storage.json` is what the live 2.0 build stores, captured with
   `node tests/e2e/capture-v20-storage.mjs`.
 - `lastValues` is pruned after every page: 5,000 ASINs, none older than a year
   (`Delta.MAX_ENTRIES`, `Delta.MAX_AGE_DAYS`).
 - A page write that fails ends the run as `storage_full` (quota) or `storage_error`.
   The popup warns from 90% of `navigator.storage.estimate()`.
-- Cloud sync is off in 2.1 and 2.2 (`Flags.CLOUD_SYNC`): nothing goes into the outbox, the SW
-  refuses `PROSCAN_EXPORT`, and the popup hides sign-in and Export to ProScan.
+- Cloud sync was off in 2.1 and 2.2 and is on from 2.3 (`Flags.CLOUD_SYNC`). Signed out, nothing
+  goes into the outbox.
 
 ## Export
 

README.md

@@ -74,6 +74,30 @@ User clicks "Start Scraping"
   → User exports via exporter.js (Excel/CSV/JSON)
 ```
 
+### Cloud Sync
+
+```
+Signed in to a ProScan account in the popup (email and password)
+  → Each saved page goes into the IndexedDB outbox, tagged with the account's uid;
+    the end of the run goes in after its pages
+  → A few seconds after a page, a run end, a popup open or a browser start,
+    the worker flushes the outbox (no alarms). After a failed flush the next
+    automatic one waits 30 s, doubling up to an hour
+  → sync.js writes one entry at a time: the page chunk, and a product and a
+    history document per ASIN on the page. The run header and the source go
+    once per run per flush. latest, prev and delta are replaced whole, never
+    merged; sourceIds keeps every source
+  → An entry leaves the outbox only once its own writes commit; a page saved
+    during a flush goes out before the flush returns
+  → An entry the rules refuse is set aside, so the entries behind it still sync;
+    the popup counts them and Export to ProScan tries them again
+  → Export to ProScan flushes right away
+```
+
+Write cost: a page of k products is 1 + 2k writes, plus 2 per run per flush. A 20 page run of 48 products a page is about 1,960 writes. Spark allows 20,000 writes a day for the whole project, shared by every user, so about 10 such runs a day fill it (audit F-23).
+
+The document shapes, ids and validators are in `packages/schema/index.js`, which the dashboard copies. Money is integer cents, unknown values are null (left out of compact points), and every document carries `sv`. A signed-out user queues nothing. If Firebase drops the session, the popup says so and the queued pages wait for the same account to sign in again.
+
 ### AI Chatbot Flow
 
 ```
@@ -153,6 +177,8 @@ For local Firebase work, `npm run build:dev` points the build at the emulators u
 
 The Jest suite includes a golden corpus of saved Amazon pages (`tests/pages/`, see its README). `npm run test:e2e` loads the built extension into Chromium and runs scrape scenarios against those pages, with every request answered locally. Run `npx playwright install --no-shell chromium` once first. Known bugs run as expected failures tagged with their audit finding id; `PROSCAN_SHOW_KNOWN=1 npm run test:e2e` shows what they fail on.
 
+`npm run test:contract` runs the real sync module against the Firebase emulators with the dashboard's `firestore.rules` (from `PROSCAN_RULES`, or a `web` or `proscan-web` checkout next to this repo): queues of 1, 201 and 600 products, a page added mid-flush, replace semantics across runs, create-only `firstSeenAt`, the write count per run, a product in two sources and an entry the rules refuse. It refuses to start if any of its ports is taken, and only stops the emulator processes it started. It needs the Firebase CLI and Java, and uses the `demo-proscan` project only.
+
 The Jest suite includes a golden corpus of saved Amazon pages (`tests/pages/`, see its README). `npm run test:e2e` loads the built extension into Chromium and runs scrape scenarios against those pages, with every request answered locally. Run `npx playwright install --no-shell chromium` once first. Known bugs run as expected failures tagged with their audit finding id; `PROSCAN_SHOW_KNOWN=1 npm run test:e2e` shows what they fail on.
 
 ## Usage
@@ -176,7 +202,7 @@ The Jest suite includes a golden corpus of saved Amazon pages (`tests/pages/`, s
 
 ## Chrome APIs Used
 
-- `chrome.storage.local` -- Settings and the schema version only
+- `chrome.storage.local` -- Settings, the schema version, and the signed-in account (`account`, `authNotice`, `lastSync`)
 - `chrome.storage.session` -- The live run record, written by the service worker
 - IndexedDB (the extension's own origin, no permission) -- Runs, products, pages, lastValues and the sync outbox. Starting a run keeps the 10 newest runs and removes older ones, except runs still waiting to sync.
 - `chrome.runtime.sendMessage` / `onMessage` -- Messages, all listed in `scripts/lib/messages.js`
@@ -201,19 +227,23 @@ AmazonSellerScraper/
 │   │   ├── parsers.js            # Pure search and offer page parsing
 │   │   ├── messages.js           # Every message type and who may send it
 │   │   ├── run.js                # The run state machine and its end reasons
-│   │   ├── flags.js              # Build flags (cloud sync is off until 2.3)
+│   │   ├── flags.js              # Build flags (cloud sync is on from 2.3)
 │   │   ├── migrate.js            # Storage schema migrations (schemaVersion)
 │   │   └── chat.js               # Gemini request builder and run scoping
 │   ├── background/
 │   │   ├── service-worker.js     # Wires the router, the engine, the chat and migrations
 │   │   ├── router.js             # The one message router
 │   │   ├── engine.js             # Runs scrapes; the only writer of run data
-│   │   └── db.js                 # IndexedDB stores
+│   │   ├── db.js                 # IndexedDB stores
+│   │   ├── sync-plan.js          # What each outbox entry writes (pure)
+│   │   └── sync.js               # Drains the outbox into Firestore
 │   └── modules/
 │       ├── storage.js            # Chrome storage abstraction layer
 │       ├── analyzer.js           # Analytics engine + opportunity scoring
 │       ├── spread-analyzer.js    # Price spread statistics (CV, arbitrage score)
 │       └── exporter.js           # Multi-format export (Excel/CSV/JSON)
+├── packages/
+│   └── schema/index.js           # Cloud schema shared with the dashboard
 ├── styles/
 │   └── chatbot.css               # Chatbot widget styles (Shadow DOM)
 ├── libs/

jest.config.js

@@ -6,12 +6,13 @@ module.exports = {
   // Older builds checked out by the e2e upgrade harness have their own tests.
   testPathIgnorePatterns: ['/node_modules/', '<rootDir>/tests/e2e/\\.build/'],
   modulePathIgnorePatterns: ['<rootDir>/tests/e2e/\\.build/'],
-  // sync.js and service-worker.js are authored as ESM (esbuild bundles
-  // them). A tiny scoped transform rewrites their import/export to
-  // CommonJS so they can be unit-tested; every other file keeps the
-  // default babel-jest transform.
+  // sync.js, sync-plan.js, service-worker.js and the shared schema are
+  // authored as ESM (esbuild bundles them). A tiny scoped transform
+  // rewrites their import/export to CommonJS so they can be unit-tested;
+  // every other file keeps the default babel-jest transform.
   transform: {
-    '[\\\\/]scripts[\\\\/]background[\\\\/](sync|service-worker)\\.js$': '<rootDir>/tests/setup/esm-to-cjs-transform.js',
+    '[\\\\/]scripts[\\\\/]background[\\\\/](sync|sync-plan|service-worker)\\.js$': '<rootDir>/tests/setup/esm-to-cjs-transform.js',
+    '[\\\\/]packages[\\\\/]schema[\\\\/]index\\.js$': '<rootDir>/tests/setup/esm-to-cjs-transform.js',
     '\\.[jt]sx?$': 'babel-jest'
   },
   coverageDirectory: 'coverage',
@@ -20,6 +21,7 @@ module.exports = {
     'scripts/lib/*.js',
     'scripts/content/scraper.js',
     'scripts/content/offer-fetcher.js',
+    'packages/schema/index.js',
     '!**/node_modules/**'
   ]
 };

manifest.json

@@ -1,7 +1,7 @@
 {
     "manifest_version": 3,
     "name": "ProScan - Amazon Product Scraper",
-    "version": "2.2.0",
+    "version": "2.3.0",
     "description": "Scrape Amazon seller products with analytics and AI-powered product insights",
     "permissions": [
         "storage",

package.json

@@ -10,6 +10,7 @@
     "test:integration": "jest tests/integration --verbose",
     "test:tools": "node --test --test-concurrency=1 \"tools/tests/*.test.mjs\"",
     "test:e2e": "playwright test",
+    "test:contract": "node tools/contract.mjs",
     "test:coverage": "jest --coverage",
     "lock": "node tools/permission-lock.mjs && node tools/version-gate.mjs",
     "scan:secrets": "node tools/secret-scan.mjs",