feat: add Google Safe Browsing protection (Linux and Mac)

AGENTS.md

@@ -144,8 +144,13 @@ The `constants` default export object holds runtime state:
 | `OOBEE_SCAN_PRODUCT` | Adds `scanProduct` tag to Sentry events |
 | `OOBEE_CONSECUTIVE_MAX_RETRIES` | Max consecutive HTTP failures before circuit breaker aborts crawl (default 100) |
 | `OOBEE_VALIDATE_URL` | If set, exit after URL validation without scanning |
+<<<<<<< HEAD
 | `OOBEE_SAVE_DOM` | `1` or `true` = save full-page DOM HTML to `pageDOMs/` in results directory. Supported scan types: Website, Sitemap, Intelligent, LocalFile, Custom |
 | `OOBEE_SAVE_PAGE_SCREENSHOT` | `1` or `true` = save full-page desktop + mobile viewport screenshots to `pageDOMs/desktopPageScreenshots/` and `pageDOMs/mobilePageScreenshots/`. Mobile viewport uses iPhone 11 width programmatically. Supported scan types: Website, Sitemap, Intelligent, LocalFile, Custom |
+=======
+| `GOOGLE_SAFE_BROWSING` | `1` = enable Google Safe Browsing (requires Chrome, not Chromium) |
+| `GOOGLE_SAFE_BROWSING_DEBUG` | `1` = enable verbose Safe Browsing debug logging |
+>>>>>>> 86d9da36 (feat: add Google Safe Browsing support via Chrome real-time URL protection)
 | `HTTP_PROXY` / `HTTPS_PROXY` / `ALL_PROXY` | Proxy configuration |
 | `NO_PROXY` / `INCLUDE_PROXY` | Proxy bypass/include lists |
 
@@ -180,6 +185,63 @@ The `constants` default export object holds runtime state:
 - Path separator differences in cookie profile regex
 - `CRAWLEE_SYSTEM_INFO_V2=1` needed (wmic deprecation)
 
+## Safe Browsing
+
+### Overview
+
+Google Safe Browsing protects users by blocking navigation to phishing/malware URLs. It requires the local hash-prefix threat database (`UrlSoceng.store.*`, `UrlMalware.store.*`) to be present in the browser profile. Chrome uses this local DB as a first-pass filter and shows interstitial warning pages when a match is found.
+
+### Requirements
+
+1. **Google Chrome** (not Chromium) — Safe Browsing requires Google's proprietary API keys. Chromium does not include them.
+2. **`GOOGLE_SAFE_BROWSING`** environment variable set to any value.
+3. **macOS or Linux only** — Windows is not yet supported (prints a warning and skips).
+
+### How It Works
+
+When `GOOGLE_SAFE_BROWSING` is set:
+
+1. `ensureAndInjectSafeBrowsing()` in `src/safeBrowsingProfile.ts` runs before each browser launch:
+   - **Fast path**: Copies the threat DB from your system Chrome profile (`~/Library/Application Support/Google/Chrome/Safe Browsing` on macOS, `~/.config/google-chrome/Safe Browsing` on Linux). This is instant.
+   - **Slow path**: If no system profile exists, spawns a real Chrome process for up to 120s to download the DB. On Linux without DISPLAY, tries Xvfb first, falls back to `--headless=old`.
+   - Writes `safebrowsing: { enabled: true }` into the profile's Preferences.
+   - Uses a file lock (`~/.oobee/safe-browsing-profile/.warmup-lock`) to prevent concurrent processes from corrupting the DB.
+
+2. `getPlaywrightLaunchOptions()` in `src/constants/common.ts` adds three Playwright default args to `ignoreDefaultArgs`:
+   - `--safebrowsing-disable-auto-update`
+   - `--disable-background-networking`
+   - `--disable-client-side-phishing-detection`
+
+3. `urlGuard.ts` allows `chrome-error://` protocol when Safe Browsing is active, so the interstitial warning page is not redirected away.
+
+### Docker / Architecture Constraints
+
+- **amd64 (x86_64)**: Google Chrome .deb is installed in Dockerfile. Safe Browsing works fully.
+- **arm64 (aarch64)**: Chrome is NOT available for ARM64 Linux. Safe Browsing is unavailable.
+- Build the image with `docker build --platform linux/amd64` to ensure Chrome is available.
+- On first run in Docker, the slow path (spawning Chrome) is used since there's no system profile.
+
+### Environment Variables
+
+| Variable | Purpose |
+|----------|---------|
+| `GOOGLE_SAFE_BROWSING` | Enable Safe Browsing (any value; requires Chrome) |
+| `GOOGLE_SAFE_BROWSING_DEBUG` | Enable verbose Chrome Safe Browsing logging |
+
+### Key Files
+
+- `src/safeBrowsingProfile.ts` — DB warmup, injection, and seeding logic
+- `src/constants/common.ts` — `getPlaywrightLaunchOptions()` (ignoreDefaultArgs), `launchPersistentSafeContext()` wrapper
+- `src/crawlers/guards/urlGuard.ts` — `allowChromeErrors` for interstitial pages
+- `src/crawlers/runCustom.ts` — passes `allowChromeErrors` to urlGuard
+- `Dockerfile` — Conditional Chrome installation for amd64
+
+### What Does NOT Work
+
+- Chromium (Playwright's bundled browser) — lacks Safe Browsing entirely
+- Windows — not yet supported (prints warning)
+- ARM64 Linux Docker — Chrome .deb not published for arm64
+
 ## Testing
 
 ```bash

Dockerfile

@@ -6,54 +6,105 @@ FROM mcr.microsoft.com/playwright:v1.61.1-noble
 RUN apt-get update && apt-get install -y --no-install-recommends \
     git \
     unzip \
-    zip && \
+    zip \
+    xvfb && \
     rm -rf /var/lib/apt/lists/*
-
-WORKDIR /app/oobee
 
-# Clone oobee repository
-# RUN git clone --branch master https://github.com/GovTechSG/oobee.git /app/oobee
+# Install Playwright browsers
+RUN npx playwright install chromium
 
-# OR Copy oobee files from local directory
-COPY . .
+# =============================================================================
+# Google Chrome installation for Safe Browsing support
+# =============================================================================
+# WHY: Chrome's Safe Browsing (v5, hash-real-time protocol) protects users by
+#      checking URLs against Google's threat database in real-time via OHTTP.
+#      This is a Chrome-only feature — Chromium does NOT include it because it
+#      requires Google's proprietary API keys baked into the Chrome build.
+#
+# HOW IT WORKS:
+#   Chrome needs a local hash-prefix database (UrlSoceng.store.*) to know which
+#   URLs require a real-time lookup via the v5 API. Without this DB, Chrome skips
+#   the check entirely. The DB is downloaded by Chrome when Safe Browsing is
+#   enabled and the browser runs with network access.
+#
+# ARCHITECTURE LIMITATION:
+#   Google Chrome .deb packages are only available for amd64 (x86_64).
+#   On arm64 builds, this step is skipped and Safe Browsing will not be available.
+#
+# TO ENABLE: Set env var GOOGLE_SAFE_BROWSING=1 when running the container.
+# =============================================================================
+RUN if [ "$(dpkg --print-architecture)" = "amd64" ]; then \
+      wget -q -O /tmp/chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb && \
+      apt-get update && apt-get install -y --no-install-recommends /tmp/chrome.deb && \
+      rm -f /tmp/chrome.deb && rm -rf /var/lib/apt/lists/*; \
+    else \
+      echo "NOTICE: Skipping Chrome install (Safe Browsing unavailable on $(dpkg --print-architecture))"; \
+    fi
+
+# Pre-seed Safe Browsing hash-prefix database into the image.
+# Chrome needs Xvfb (virtual display) to run non-headless and download the DB.
+# Under emulation this can take several minutes; timeout is set to 1800s.
+COPY <<'SEEDSCRIPT' /tmp/seed-safe-browsing.sh
+#!/bin/bash
+set -e
+mkdir -p /opt/oobee-safe-browsing/Default
+echo '{"safebrowsing":{"enabled":true,"enhanced":true}}' > /opt/oobee-safe-browsing/Default/Preferences
+export DISPLAY=:99
+Xvfb :99 -screen 0 1024x768x24 &
+XVFB_PID=$!
+sleep 2
+google-chrome \
+  --user-data-dir=/opt/oobee-safe-browsing \
+  --no-first-run --no-default-browser-check --disable-extensions \
+  --no-sandbox --disable-setuid-sandbox \
+  --enable-features=SafeBrowsingEnhancedProtection \
+  --window-position=-10000,-10000 --window-size=1,1 \
+  about:blank &
+CHROME_PID=$!
+echo "Waiting for Safe Browsing DB to download (up to 1800s)..."
+WAITED=0
+while [ $WAITED -lt 1800 ]; do
+  if ls /opt/oobee-safe-browsing/Safe\ Browsing/UrlSoceng.store.* >/dev/null 2>&1 || \
+     ls /opt/oobee-safe-browsing/Safe\ Browsing/UrlMalware.store.* >/dev/null 2>&1; then
+    echo "Safe Browsing DB downloaded successfully (${WAITED}s)"
+    break
+  fi
+  sleep 10
+  WAITED=$((WAITED + 10))
+  echo "[${WAITED}s] Still waiting... $(ls /opt/oobee-safe-browsing/Safe\ Browsing/ 2>/dev/null | wc -l) files in Safe Browsing dir"
+done
+kill $CHROME_PID 2>/dev/null || true
+kill $XVFB_PID 2>/dev/null || true
+if ls /opt/oobee-safe-browsing/Safe\ Browsing/UrlSoceng.store.* >/dev/null 2>&1; then
+  echo "Safe Browsing DB baked into image"
+else
+  echo "WARNING: Safe Browsing DB did not populate - will attempt at runtime"
+fi
+SEEDSCRIPT
+RUN if [ "$(dpkg --print-architecture)" = "amd64" ] && command -v google-chrome >/dev/null 2>&1; then \
+      bash /tmp/seed-safe-browsing.sh; \
+    fi && rm -f /tmp/seed-safe-browsing.sh && \
+    chmod -R a+rX /opt/oobee-safe-browsing 2>/dev/null || true
+
+# --- App code (changes here don't invalidate Chrome/seed layers above) ---
 
 # Environment variables for node and Playwright
 ENV NODE_ENV=production
 ENV PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD="true"
 
-# Install oobee dependencies
-# TODO: Move back to npm ci --omit=dev once module package-lock issue vs MacOS is resolved
-RUN npm install --omit=dev
-
-# Compile TypeScript for oobee
-RUN npm run build || true # true exits with code 0 - workaround for TS errors
-
-# Install Playwright browsers
-RUN npx playwright install chromium
-
-# Add non-privileged user
-# Create a group named "purple"
-RUN groupadd -r purple
+# Add non-privileged user before app copy so ownership can be set during COPY.
+RUN groupadd -r purple && useradd -r -g purple purple && \
+  mkdir -p /home/purple /app/oobee && chown purple:purple /home/purple /app /app/oobee
 
-# Create a user named "purple" and assign it to the group "purple"
-RUN useradd -r -g purple purple
-
-# Create a dedicated directory for the "purple" user and set permissions
-RUN mkdir -p /home/purple && chown -R purple:purple /home/purple
-
-WORKDIR /app
-
-# Set the ownership of the oobee directory to the user "purple"
-RUN chown -R purple:purple /app
-
-# Copy any application and support files
-# COPY . .
-
-# Install any app dependencies for your application
-# RUN npm ci --omit=dev
-
-# For oobee to be run from present working directory, comment out as necessary
 WORKDIR /app/oobee
 
-# Run everything after as non-privileged user.
+# Run app build steps as non-root to avoid a full recursive chown later.
 USER purple
+
+# Install dependencies first (cached unless package.json/package-lock.json change)
+COPY --chown=purple:purple package.json package-lock.json ./
+RUN npm install --omit=dev
+
+# Copy source and compile TypeScript
+COPY --chown=purple:purple . .
+RUN npm run build || true # true exits with code 0 - workaround for TS errors

README.md

@@ -107,6 +107,7 @@ verapdf --version
 | ALL_PROXY | URL of the proxy server to be used for all requests, typically used for SOCKS5 proxies (e.g. `socks5://proxy.example.com:1080`. Note: IPv6 direct connections may still continue even though socks5 proxy is specified due to a known issue with Chrome/Chromium. (Recommended workaround is to turn off IPv6 at host-level). | |
 | NO_PROXY | Comma-separated list of domains that should bypass the proxy (e.g. `localhost,127.0.0.1,.example.com`). | |
 | INCLUDE_PROXY | Comma-separated list of domains that should specifically be routed through the proxy. | |
+| GOOGLE_SAFE_BROWSING | When set, enables Google Safe Browsing URL protection. Requires Google Chrome (not Chromium) to be installed. Seeds the Safe Browsing threat database from your system Chrome profile on first run. macOS and Linux only (Windows not yet supported). | |
 
 #### Environment variables used internally (Do not set)
 Do not set these environment variables or behaviour might change unexpectedly.