2.2: run scrapes from the service worker and store them in IndexedDB - #8
Open
EnesYilmazcode wants to merge 25 commits into
Open
EnesYilmazcode wants to merge 25 commits into
EnesYilmazcode wants to merge 25 commits into
Conversation
This was referenced Sep 24, 2026
Open
EnesYilmazcode
force-pushed
the
rebuild/p2-v2.1
branch
from
September 24, 2026 21:05
40be432 to
0044e2c
Compare
EnesYilmazcode
force-pushed
the
rebuild/p3-run-engine
branch
from
September 24, 2026 21:05
ade153b to
4ffbb2b
Compare
The content script no longer writes storage or navigates. It says PAGE_READY on load, parses when the worker asks, sends PAGE_RESULT and keeps a heartbeat while the next page is pending. The run scenarios of scraper-run.test.js and every test of scraper-dedupe.test.js now run in engine.test.js, over this script and the engine, with the same names and checks. scraper-run.test.js keeps the content script's own contract. The characterization test drives the engine instead of seeding storage. Products, pages, next links and helpers are unchanged on every corpus page. The snapshot changes only in the messages sent and in non-search pages, which Start now refuses instead of being forced into a run.
…der the 3 to 4 migration With no session record (the extension was disabled and enabled, or its process crashed), GET_STATE now ends the durable run as interrupted and Stop ends it as stopped, so the popup is no longer stuck on a run that cannot be stopped. Starting a run removes runs past the newest 10, except ones still in the outbox, and a full disk at start drops the rest and tries once more, so the storage full message is true. The 3 to 4 migration no longer overwrites latestRunId or outbox rows the engine wrote first. The engine also checks that the page a tab reports is the one it opened.
EnesYilmazcode
force-pushed
the
rebuild/p3-run-engine
branch
from
September 25, 2026 03:12
4ffbb2b to
7ce2963
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack, merge in order: #5 > #6 > #7 > #8 (this one) > #9. Based on #7, so the diff here is only this step.
The proscan-web stack: EnesYilmazcode/proscan-web#1, EnesYilmazcode/proscan-web#2.
2.2. The content script used to own the run, so run state lived in three places and a stopped worker or a second tab could corrupt it. Now the service worker owns it (F-10, F-11, F-13, F-14, F-18).
Tested: npm test 655 jest and 35 tool tests. Playwright with one worker, before this restack: 21 passed, 1 skipped (the same F-20 sync case). That includes killing the worker over CDP between pages and swapping the extension mid-run from both 2.0 and 2.1. After the restack I ran the run tab and 503 scenarios here (all pass) and the full suite on #7 and #9. The page cap test had a race on slow runners, fixed in the last commit before this restack.
Not sure about: Chrome ignores a CDP quota override for extension origins, so storage_full is only covered in jest. I also have not checked whether a real Amazon replaceState can fire a loading event and end a run by mistake. The harness cannot show that.
Does not change permissions. The content script list loses run.js, flags.js and delta.js and gains messages.js: