2.1: bring-your-own Gemini key, runs bound to a tab, clean product data - #7
Open
EnesYilmazcode wants to merge 44 commits into
Open
EnesYilmazcode wants to merge 44 commits into
EnesYilmazcode wants to merge 44 commits into
Conversation
This was referenced Sep 24, 2026
Open
One constant for the model id (gemini-3.8-flash), the key in the x-goog-api-key header, rules in systemInstruction and scraped titles in a fenced JSON block marked as untrusted. Context is scoped to the current run, deduped by ASIN and capped at 50 products. Errors map by status and reason instead of calling every 400 a bad key, with a 30 second timeout.
The worker reads the key and the current run from storage itself, so the content script only sends the question and a short history. Adds a CHAT_STATUS message that says whether a key is set without returning it.
The key is stored in chrome.storage.local and only the service worker reads it. The field never shows a saved key back.
…tory The widget asks the worker for CHAT_STATUS instead of reading storage, greets with the run it covers, and points at AI chat settings when no key is set. An orphaned script after an update now says to reload the page. Follow-up questions carry the last few turns.
Thinking tokens count against maxOutputTokens, so a 512 cap could return an empty answer. Raise the cap, set thinkingLevel to low, and drop the temperature override Google advises against for Gemini 3 models.
A page with no result cards is now a captcha, a bot check, a sign-in wall, an empty search or unknown, instead of always empty. The next page is the Next link's own href, and a page with no live Next button is the last one.
WHO_AM_I lets a page check that the running run is its own. Closing the run's tab or restarting the browser ends the run as interrupted. The STOP relay here only answered messages from tabs, so the popup's Stop never used it; it is gone. New installs default to a 20 page cap.
The popup pings the tab before it touches storage. If the content script is missing, as in tabs left open across an update, it changes nothing and offers to reload the tab, then starts once the reload finishes. A search page is required to start, so a captcha tab is refused with a reason. The content script scrapes only when its own tab owns the running run, calls init once whatever the ready state, and saves each page together with the run's progress. Stop goes to the run's tab, which cancels the pending navigation and records the run as stopped. A run ends with a reason (complete, stopped, blocked, selectors_broken, storage_full, interrupted) and the popup shows anything but complete as a warning. The page cap in settings is honoured.
Captcha, Stop, second tab, product tab, full storage and the resumed v2.0 run now pass in Chromium. New scenarios cover the page cap, Start on a captcha page and Start on a tab orphaned by an update. The storage scenario now waits for the run to end instead of for a flag that is unset before Start.
…r N/A Only a dollar price gets priceCents; other currencies keep their symbol in currency and no amount. A card with "1 rating" now counts its review.
Card links on sponsored results are sspa click redirects that expire. The URL is now built from the ASIN, and the sponsored flag comes from the label, an AdHolder card or an sspa link.
Only s-search-result cards count when the page marks them, so carousel and video cards stay out. A product seen as an ad and as a result keeps both placements, the sponsored flag and the rank of its first organic card.
A repeat on a later page adds its placements to the stored record and its queued copy instead of diffing against the first sighting, which zeroed the real change. Organic ranks count across the run's pages.
The queue now holds one product per ASIN, so placements and sponsored come from each product's placements. Sync writes the /dp/ URL and leaves out a name it does not have.
… empty A search page with no cards is only empty when Amazon's No results marker is on it; otherwise it is unreadable and ends the run as a warning. Past page 1, no results never ends as complete. On page load the scraper now ends a stale run and a run whose tab shows a page other than the one it navigated to, instead of scraping it into the run. The storage full message now says a new scan clears the stored results.
EnesYilmazcode
force-pushed
the
rebuild/p2-v2.1
branch
from
September 24, 2026 21:05
40be432 to
0044e2c
Compare
EnesYilmazcode
force-pushed
the
rebuild/p1-tests
branch
from
September 24, 2026 21:05
a6aa58e to
f035b71
Compare
Amazon's newer storefront pages load more on scroll and show no pagination strip, so a run stopped after the first 16 of 59 products. When the header shows more results than the page reached, go to the next page number instead. Found on a live storefront.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack, merge in order: #5 > #6 > #7 (this one) > #8 > #9. Based on #6, so the diff here is only this step.
The proscan-web stack: EnesYilmazcode/proscan-web#1, EnesYilmazcode/proscan-web#2.
This is the 2.1 store build. It fixes what the corpus and harness in the previous PR showed was broken, and nothing more structural than that.
Chat (F-60 to F-65): the retired model id is gone (now gemini-3.8-flash, one constant). Each user pastes their own key under AI chat settings in the popup. Only the service worker reads it, and it is sent in a header. Chat only sees the current run, deduped and capped at 50 products. Titles go in as untrusted data, and errors are mapped to what actually went wrong.
Runs (F-10 to F-15): each run belongs to the tab it started in and ends with a reason (complete, blocked, stopped, selectors_broken, interrupted, storage_full). A captcha page no longer counts as a result page. The scraper follows the real Next link. A search page with no cards counts as empty only when Amazon's "No results for" marker is on it; a 503 error page or renamed cards end the run as a warning, and past page 1 no results never ends as complete (F-12). On page load the run tab scrapes only the page the run opened: a new search typed in that tab, or a run whose tab went quiet for over a minute, ends the run as interrupted instead of joining it.
Data (F-16, F-17, F-25, F-27, F-28, F-29): each ASIN is kept once per run with its placements, sponsored cards are flagged, links are /dp/{asin}, and a missing value is null instead of 0 or N/A. On the saved yoga mat page, 70 cards used to become 70 products; now they become 59 unique products, which matches the page.
Storage and export (F-26, F-70, F-74, F-75, F-78 partly, F-100 to F-102): failed writes end the run instead of being ignored, and the storage full message says a new scan clears the stored results, which is what the popup offers. Cloud sync is behind Flags.CLOUD_SYNC = false, so Export to ProScan and sign-in are hidden. CSV quotes every field and neutralises formulas. A 2 to 3 migration cleans up 2.0 storage. It was tested against a fixture captured from the real 2.0 build.
Tested: npm test 572 jest and 35 tool tests. Playwright: 21 passed, 1 skipped (F-20, it only runs with sync on), including the new run tab and 503 scenarios. tests/e2e/permissions.spec.mjs asks Chromium for install warnings with getPermissionWarningsByManifest, and the built and live manifests give the same two. No real Gemini call was made, so the new model id is untested live.
Store zip for this head (0044e2c), built with npm run zip from a clean tree: proscan-v2.1.0-0044e2c.zip, sha256 81883e045a89d96f211df75ba994a8d9d31625ac4ce87ad18db17fb432d6b782.
Does not change permissions: