Skip to content

feat: add Google Safe Browsing protection (Linux and Mac)#781

Draft
younglim wants to merge 19 commits into
masterfrom
feat/safe-browsing-v2
Draft

feat: add Google Safe Browsing protection (Linux and Mac)#781
younglim wants to merge 19 commits into
masterfrom
feat/safe-browsing-v2

Conversation

@younglim

@younglim younglim commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds optional Google Safe Browsing URL protection, gated by GOOGLE_SAFE_BROWSING=1 env var
  • Uses Chrome's real-time Safe Browsing v5 API (OHTTP hash-prefix lookups) — no local threat database needed
  • Works across all scan types: website crawl (crawlDomain), sitemap crawl, intelligent sitemap, and custom flows
  • On macOS: works immediately with system Chrome (headful mode)
  • On Linux Docker (amd64): uses Xvfb virtual framebuffer to run Chrome headful, since Safe Browsing interstitials are not enforced in headless mode
  • Supported: macOS, Linux amd64. Not supported: Windows (prints warning), Linux arm64 (Chrome unavailable)

How it works

  1. Chrome is installed in the Docker image (amd64 only)
  2. At runtime, ensureAndInjectSafeBrowsing() writes safebrowsing.enabled: true, enhanced: true into the browser profile preferences
  3. getPlaywrightLaunchOptions() removes Playwright's default flags that suppress Safe Browsing (--safebrowsing-disable-auto-update, --disable-client-side-phishing-detection, --disable-background-networking)
  4. On Linux without a display, Xvfb is started and Chrome runs headful against the virtual framebuffer
  5. A 10s warm-up delay gives Chrome time to connect to Google's OHTTP relay before navigating
  6. Pages blocked by Safe Browsing land on chrome-error:// and are logged as "Blocked by Safe Browsing"

Key changes

File Change
src/safeBrowsingProfile.ts New — DB warmup (skips legacy download for Chrome 128+), preference injection, platform checks, debug logging
src/constants/common.ts launchPersistentSafeContext() wrapper; getPlaywrightLaunchOptions() removes SB-suppressing flags, starts Xvfb + forces headful when Safe Browsing enabled; 10s warm-up in checkUrl
src/crawlers/commonCrawlerFunc.ts getPreLaunchHook injects Safe Browsing preferences into each browser pool instance
src/crawlers/crawlDomain.ts Handles chrome-error:// interstitial pages as excluded with STATUS_CODE_METADATA[3]
src/crawlers/crawlSitemap.ts Same chrome-error:// handling for sitemap crawls
src/constants/constants.ts Adds STATUS_CODE_METADATA[3]: "Blocked by Safe Browsing"
Dockerfile Installs Chrome (amd64 only), installs Xvfb, removes legacy DB seed script

Environment variables

Variable Purpose
GOOGLE_SAFE_BROWSING=1 Enable Safe Browsing protection
GOOGLE_SAFE_BROWSING_DEBUG=1 Enable Chrome's internal Safe Browsing debug logging

Test plan

  • macOS: phishing page blocked with ERR_BLOCKED_BY_CLIENT during checkUrl
  • Linux Docker (amd64): Xvfb starts, Chrome runs headful, phishing page blocked
  • Non-phishing pages scan normally with Safe Browsing enabled
  • Without GOOGLE_SAFE_BROWSING env var, no Safe Browsing overhead (no Xvfb, no warm-up)
  • arm64 Docker: prints notice, scans proceed without Safe Browsing
  • Windows: prints warning, scans proceed without Safe Browsing
  • Safe Browsing blocked pages appear in report as "Blocked by Safe Browsing" (not scanned)

Test on Docker

# Build (amd64 for Chrome support)
docker build --platform linux/amd64 --progress=plain -t oobee .

# Run scan against phishing test page
docker run --platform linux/amd64 --rm -e GOOGLE_SAFE_BROWSING=1 -e OOBEE_VERBOSE=1 oobee node dist/cli.js -u https://testsafebrowsing.appspot.com/s/phishing.html -b chrome -c 2 -k "A11y Team:accessibility@tech.gov.sg" -p 2

# Or share the same dist directory as host
docker run --platform linux/amd64 --rm \
  -v $(pwd)/dist:/app/oobee/dist \
  -e GOOGLE_SAFE_BROWSING=1 -e OOBEE_VERBOSE=1 \
  oobee node dist/cli.js -u https://testsafebrowsing.appspot.com/s/phishing.html -b chrome -c 2 -k "A11y Team:accessibility@tech.gov.sg" -p 2

@younglim younglim changed the title feat: add Google Safe Browsing support via Linux Chrome real-time URL protection feat: add Google Safe Browsing protection (Linux and Mac) Jul 9, 2026
@younglim
younglim force-pushed the feat/safe-browsing-v2 branch from 65bea77 to 9232b0b Compare July 9, 2026 10:11
@younglim
younglim marked this pull request as draft July 9, 2026 14:21
younglim and others added 17 commits July 14, 2026 05:42
…ction

Modern Chrome (v128+) uses the Safe Browsing v5 hash-real-time protocol
to check URLs against Google's threat database on every navigation. This
commit enables that by:

- Installing Google Chrome in Docker (amd64 only, arm64 skipped)
- Adding ensureSafeBrowsingPreferences() to seed profile with SB enabled
- Modifying getPlaywrightLaunchOptions() to not suppress SB flags when
  GOOGLE_SAFE_BROWSING env var is set
- Adding launchPersistentSafeContext() wrapper used by all crawler entry points
- Logging SB status on first browser launch

No local DB warmup or pre-seeding is needed — Chrome handles everything
via real-time OHTTP lookups.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The real-time OHTTP approach alone doesn't trigger Chrome's interstitial
warning pages in Playwright automation mode. The local hash-prefix DB
(UrlSoceng.store.*, UrlMalware.store.*) is required as a first-pass filter.

Restores safeBrowsingProfile.ts with improvements:
- File lock to prevent concurrent warmup corruption
- Process group kill for Chrome cleanup (SIGKILL -pid)
- Xvfb lifecycle management on Linux (verify alive, kill after)
- Windows: prints unsupported message and skips

Also adds allowChromeErrors to urlGuard so Safe Browsing interstitial
pages (chrome-error://) are not redirected away.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Uses a proper bash script with exported DISPLAY to run Chrome + Xvfb
during docker build. The DB downloads in ~145s and is baked into the
image at /opt/oobee-safe-browsing/. At runtime, the DB is copied
instantly from this location into browser profiles.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Avoids a wasted 120s warmup timeout on arm64 Linux or environments
without Chrome installed. Prints a clear message directing users to
build with --platform linux/amd64.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
getDefaultChromeDataDir() only checked Windows/macOS. On Linux with
Chrome installed, it returned null causing fallback to Chromium.
Now detects /usr/bin/google-chrome and returns ~/.config/google-chrome.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…port)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
cloneChromeProfiles fails when no Local State file exists in the data
dir, which is always the case in fresh Docker containers. Creating a
minimal one prevents the warning and allows the scan to proceed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The function only handles win32/darwin cookie paths. On Linux,
profileCookiesDir was undefined causing a TypeError on .length.
Now returns true early on Linux (no cookies to clone).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
getPreLaunchHook now injects the Safe Browsing threat database into each
browser pool instance's user data directory. Both crawlDomain and
crawlSitemap handle chrome-error:// interstitials (blocked pages) by
logging them as excluded with STATUS_CODE_METADATA[3].

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ot user

The Safe Browsing DB at /opt/oobee-safe-browsing/ is seeded as root during
docker build, but the app runs as the unprivileged 'purple' user. Without
world-readable permissions, Node's fs.readdirSync fails with EACCES and
Safe Browsing protection is silently disabled.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On fresh Docker containers, Chrome's real-time Safe Browsing v5 (OHTTP)
needs time to connect to Google's relay servers after launch. Add a 10s
warm-up delay before navigating to the target URL when GOOGLE_SAFE_BROWSING
is enabled. Also switch to Enhanced Safe Browsing (enhanced: true) for
more aggressive real-time URL checking.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Chrome's Safe Browsing does not enforce interstitial blocking in headless
mode. When GOOGLE_SAFE_BROWSING is enabled on Linux without a display,
start Xvfb and force headful mode so Chrome renders to a virtual
framebuffer and enforces real-time URL protection.

Also removes the legacy DB seed script from Dockerfile (Chrome 128+ uses
real-time v5 API, not local UrlSoceng.store files) and adds debug logging
throughout the Safe Browsing flow.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@younglim
younglim force-pushed the feat/safe-browsing-v2 branch from b7561f8 to ac64e27 Compare July 14, 2026 06:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant