feat: add Google Safe Browsing protection (Linux and Mac)#781
Draft
younglim wants to merge 19 commits into
Draft
feat: add Google Safe Browsing protection (Linux and Mac)#781younglim wants to merge 19 commits into
younglim wants to merge 19 commits into
Conversation
younglim
force-pushed
the
feat/safe-browsing-v2
branch
from
July 9, 2026 10:11
65bea77 to
9232b0b
Compare
younglim
marked this pull request as draft
July 9, 2026 14:21
…ction Modern Chrome (v128+) uses the Safe Browsing v5 hash-real-time protocol to check URLs against Google's threat database on every navigation. This commit enables that by: - Installing Google Chrome in Docker (amd64 only, arm64 skipped) - Adding ensureSafeBrowsingPreferences() to seed profile with SB enabled - Modifying getPlaywrightLaunchOptions() to not suppress SB flags when GOOGLE_SAFE_BROWSING env var is set - Adding launchPersistentSafeContext() wrapper used by all crawler entry points - Logging SB status on first browser launch No local DB warmup or pre-seeding is needed — Chrome handles everything via real-time OHTTP lookups. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The real-time OHTTP approach alone doesn't trigger Chrome's interstitial warning pages in Playwright automation mode. The local hash-prefix DB (UrlSoceng.store.*, UrlMalware.store.*) is required as a first-pass filter. Restores safeBrowsingProfile.ts with improvements: - File lock to prevent concurrent warmup corruption - Process group kill for Chrome cleanup (SIGKILL -pid) - Xvfb lifecycle management on Linux (verify alive, kill after) - Windows: prints unsupported message and skips Also adds allowChromeErrors to urlGuard so Safe Browsing interstitial pages (chrome-error://) are not redirected away. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Uses a proper bash script with exported DISPLAY to run Chrome + Xvfb during docker build. The DB downloads in ~145s and is baked into the image at /opt/oobee-safe-browsing/. At runtime, the DB is copied instantly from this location into browser profiles. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Avoids a wasted 120s warmup timeout on arm64 Linux or environments without Chrome installed. Prints a clear message directing users to build with --platform linux/amd64. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
getDefaultChromeDataDir() only checked Windows/macOS. On Linux with Chrome installed, it returned null causing fallback to Chromium. Now detects /usr/bin/google-chrome and returns ~/.config/google-chrome. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…port) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cloneChromeProfiles fails when no Local State file exists in the data dir, which is always the case in fresh Docker containers. Creating a minimal one prevents the warning and allows the scan to proceed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The function only handles win32/darwin cookie paths. On Linux, profileCookiesDir was undefined causing a TypeError on .length. Now returns true early on Linux (no cookies to clone). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
getPreLaunchHook now injects the Safe Browsing threat database into each browser pool instance's user data directory. Both crawlDomain and crawlSitemap handle chrome-error:// interstitials (blocked pages) by logging them as excluded with STATUS_CODE_METADATA[3]. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ot user The Safe Browsing DB at /opt/oobee-safe-browsing/ is seeded as root during docker build, but the app runs as the unprivileged 'purple' user. Without world-readable permissions, Node's fs.readdirSync fails with EACCES and Safe Browsing protection is silently disabled. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On fresh Docker containers, Chrome's real-time Safe Browsing v5 (OHTTP) needs time to connect to Google's relay servers after launch. Add a 10s warm-up delay before navigating to the target URL when GOOGLE_SAFE_BROWSING is enabled. Also switch to Enhanced Safe Browsing (enhanced: true) for more aggressive real-time URL checking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Chrome's Safe Browsing does not enforce interstitial blocking in headless mode. When GOOGLE_SAFE_BROWSING is enabled on Linux without a display, start Xvfb and force headful mode so Chrome renders to a virtual framebuffer and enforces real-time URL protection. Also removes the legacy DB seed script from Dockerfile (Chrome 128+ uses real-time v5 API, not local UrlSoceng.store files) and adds debug logging throughout the Safe Browsing flow. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
younglim
force-pushed
the
feat/safe-browsing-v2
branch
from
July 14, 2026 06:44
b7561f8 to
ac64e27
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GOOGLE_SAFE_BROWSING=1env varcrawlDomain), sitemap crawl, intelligent sitemap, and custom flowsHow it works
ensureAndInjectSafeBrowsing()writessafebrowsing.enabled: true, enhanced: trueinto the browser profile preferencesgetPlaywrightLaunchOptions()removes Playwright's default flags that suppress Safe Browsing (--safebrowsing-disable-auto-update,--disable-client-side-phishing-detection,--disable-background-networking)chrome-error://and are logged as "Blocked by Safe Browsing"Key changes
src/safeBrowsingProfile.tssrc/constants/common.tslaunchPersistentSafeContext()wrapper;getPlaywrightLaunchOptions()removes SB-suppressing flags, starts Xvfb + forces headful when Safe Browsing enabled; 10s warm-up incheckUrlsrc/crawlers/commonCrawlerFunc.tsgetPreLaunchHookinjects Safe Browsing preferences into each browser pool instancesrc/crawlers/crawlDomain.tschrome-error://interstitial pages as excluded withSTATUS_CODE_METADATA[3]src/crawlers/crawlSitemap.tschrome-error://handling for sitemap crawlssrc/constants/constants.tsSTATUS_CODE_METADATA[3]: "Blocked by Safe Browsing"DockerfileEnvironment variables
GOOGLE_SAFE_BROWSING=1GOOGLE_SAFE_BROWSING_DEBUG=1Test plan
ERR_BLOCKED_BY_CLIENTduringcheckUrlGOOGLE_SAFE_BROWSINGenv var, no Safe Browsing overhead (no Xvfb, no warm-up)Test on Docker