fix: cookie clone from pristine profile, pool cleanup, intelligent sitemap optimization, download/403 handling

[younglim]

fix: cookie clone from pristine profile, pool cleanup, intelligent sitemap optimization, download/403 handling, unsupported document classification

Summary

• Always clone cookies from the pristine user profile clone — never let a running browser modify the base directory
• Reliable mid-scan deletion of retired browser pool directories (preLaunchHook + postPageCloseHook with retries)
• Cap browser disk cache at 10MB per instance to prevent storage exhaustion
• Fix Windows glob bug in cleanUp() that left pool directories behind
• Optimize intelligent sitemap: discover <a> links during sitemap phase, skip second-pass click-discovery in domain phase
• Handle "Download is starting" navigation errors as unsupported documents (or PDF downloads)
• Re-enqueue rate-limited 403 URLs for one retry after concurrency recovers
• Disable Eco QoS on Windows to prevent Chromium CPU throttling
• Correctly classify non-HTML URLs (images, PDFs, media) as "Unsupported Documents" instead of "Pages Not Scanned"

Problem

403 errors after ~3700 pages in intelligent crawl

The first browser (launchCount === 1) used userDataDirectory directly — Chrome modified the Cookies DB, added journal files, and tainted the session. All subsequent pool rotations copied from this corrupted base instead of the user's original authenticated cookies.

Out of storage on Docker / disk bloat locally

Each pool rotation created a Chrome profile with unbounded cache (100-500MB each). A 3700-page crawl with 7+ rotations could accumulate 700MB-3.5GB on disk, all held until scan end. This exhausts small Docker storage provisions mid-crawl.

Additionally, mid-scan cleanup was unreliable:

• postPageCloseHook fired too early: The hook fires on every page close where activePages === 0, but between pages the pool immediately assigns new pages to the same browser. The 2s setTimeout fired while Chrome still had file locks, rm failed silently, and directories survived.
• cleanUp() sweep broken on Windows: globSync interprets backslashes as escape characters, not path separators. The pattern C:\...\oobee-..._pool* matched nothing — pool dirs were never found for deletion. No retry logic existed for Chrome's async shutdown file locks.

Intelligent sitemap scan hangs for hours after completing page scans

After crawlDomain's first crawler.run() completes, a second-pass loop re-visits ALL scanned same-hostname pages for customEnqueueLinksByClickingElements. On a 3400-page site at rate-limited concurrency 1, this takes 10+ hours with no log output (appears hung). On Linux the circuit breaker fires (stricter WAF on datacenter IPs) which sets isAbortingScanNow = true and skips the loop. On Windows residential IPs, the adaptive concurrency successfully recovers, the circuit breaker never fires, and the loop runs unbounded.

Download URLs classified as crawler errors

URLs like Salesforce /download/ endpoints trigger a file download instead of loading a page. Playwright throws "Download is starting", Crawlee retries 3 times (all fail), and failedRequestHandler classifies them as "Web Crawler Errored". Should be "Not A Supported Document" (or downloaded as PDF if scanning enabled).

Rate-limited 403 URLs permanently lost

Crawlee retries 3 times in rapid succession (all during the same rate-limit window), exhausts retries, and records as error. The URL is never retried after concurrency recovers.

Non-HTML URLs misclassified as "Pages Not Scanned"

Sitemap URLs pointing to images (.png, .jpg) or URLs that redirect (302) to PDFs/images return HTTP 200 with non-HTML content-type. In crawlSitemap, when isWhitelistedContentType fails for these, they fall to the else block which pushes to urlsCrawled.invalid with STATUS_CODE_METADATA[200] ("Oobee was not able to scan the page due to access restrictions or compatibility issues") and httpStatusCode: 200. The report UI only classifies items as "Unsupported Documents" when httpStatusCode === 1, so these appeared under "Pages Not Scanned". In crawlDomain, shouldSkipDueToUnsupportedContent and isBlacklistedFileExtensions detected them correctly but silently returned without pushing to any array.

Fix

1. Cookie source (pristine clone)

Every browser gets its own _pool{N} directory. The base userDataDirectory is treated as read-only — cookies are always copied fresh from it. No running browser ever touches the original authenticated profile.

2. Eager cleanup via getPreLaunchHook

When a new browser launches (meaning the previous one was retired), schedule cleanup of the previous pool directory after 35-40s (accounts for closeInactiveBrowserAfterSecs: 30 + Windows file-lock grace). 3 retries with 5s backoff.

3. postPageCloseHook hardened (secondary cleanup)

Increased initial delay to 5s on Windows (2s elsewhere), added 3 retries with 2s backoff as secondary best-effort cleanup for when the preLaunchHook path doesn't fire (e.g., final browser before scan ends).

4. cleanUp() sweep fixed for Windows

Normalize backslashes to forward slashes in glob pattern. Added retry loop (up to 10 times at 5s intervals) for Chrome's async shutdown file locks.

5. Disk cache capped

--disk-cache-size=10485760 (10MB) added to Chrome launch args. Retains cache benefit for repeated CSS/JS assets while keeping each pool directory small (~15-25MB vs 100-500MB).

Peak disk usage: ~30-50MB (down from potentially 3.5GB).

6. Skip second-pass click-discovery for intelligent scans

Added !fromCrawlIntelligentSitemap guard on the second-pass loop in crawlDomain. The domain phase of intelligent crawl is only meant to discover new pages via <a> links, not exhaustively click thousands of already-scanned pages.

7. Link discovery during sitemap phase

crawlSitemap now always creates a RequestQueue alongside the RequestList. When fromCrawlIntelligentSitemap is true, it calls enqueueLinks after each successful scan to extract <a> links. Discovered URLs are scanned in the same pass (after all sitemap URLs complete). This eliminates most work the subsequent crawlDomain supplement phase would need to do.

8. Download error handling

In failedRequestHandler, detect "Download is starting" errors. If PDF scanning enabled: re-enqueue with skipNavigation: true for the PDF download path. If disabled: classify as "Not A Supported Document" with SKIPPED log.

9. 403 rate-limit retry

In failedRequestHandler, re-enqueue 403 URLs once (with rateLimitRetried flag) so they get a fresh attempt after adaptive concurrency recovers. No onFailure counted on the first pass to avoid double-counting toward the circuit breaker.

10. Windows Eco QoS disabled

--disable-features=UseEcoQoSForBackgroundProcess prevents Windows from throttling background Chromium processes to efficiency cores.

11. Non-HTML URL classification

Shared blackListedFileExtensions (in constants.ts): Expanded with additional non-scannable types (woff2, ico, bmp, tiff, tif, avi, mov, wmv, flv, ogg, wav, doc, docx, xls, xlsx, ppt, pptx). Both crawlers reference this single list.

Both crawlers' preNavigationHooks: Detect URLs with blacklisted extensions and skip navigation entirely — avoids wasting browser resources loading images/media. In crawlSitemap, also sets isNotSupportedDocument userData for immediate classification in the requestHandler.

crawlSitemap else block: When navigation succeeds but content-type is non-HTML (detected via !contentType.startsWith('text/html') && !contentType.includes('html')), classifies as STATUS_CODE_METADATA[1] with httpStatusCode: 1. Handles 302-to-PDF and image URLs served with 200. Empty content-type falls through to existing logic.

crawlDomain: shouldSkipDueToUnsupportedContent and isBlacklistedFileExtensions blocks now push to urlsCrawled.userExcluded with httpStatusCode: 1 instead of silently returning. The preNavigationHook catches blacklisted extensions before navigation; for URLs without obvious extensions (e.g. redirects to non-HTML), the post-navigation content-type check is the safety net.

No impact to HTML page scanning: HTML pages never reach these paths — shouldSkipDueToUnsupportedContent returns false for text/html, extensions don't match, and isWhitelistedContentType passes. The full lifecycle (waitForPageLoaded → DOM mutation observer → runAxeScript) is unchanged.

Standalone sitemap and website scans: Download handling and 403 retry now work in all modes (requestQueue always available). enqueueLinks discovery remains gated to intelligent mode only.

Test plan

• Run intelligent sitemap scan against an authenticated site — verify no 403 wall after 3000+ pages
• Verify userDataDirectory base clone remains unmodified throughout the scan
• Run intelligent sitemap scan on a large site (3000+ sitemap URLs) — verify no hang after sitemap phase
• Verify discovered <a> links are scanned after sitemap URLs complete (check report for non-sitemap pages)
• Run standalone sitemap scan — verify no enqueueLinks behavior, but download/403 handling works
• Run standalone website scan — verify second-pass click-discovery still runs
• Run a large crawl on a Docker container with limited storage — verify no ENOSPC
• Monitor disk usage during crawl — confirm only 1-2 pool directories exist at any time
• On Windows, verify no residual pool directories after scan completes (SIGINT and normal exit)
• Verify "Download is starting" URLs → "Not A Supported Document" (PDF disabled) or PDF scan (enabled)
• Verify 403 URLs during rate-limit burst are retried and scanned after concurrency recovers
• Verify image URLs (.png, .jpg) from sitemap → "Unsupported Documents" tab (not "Pages Not Scanned")
• Verify URLs that 302-redirect to PDF/image → "Unsupported Documents" tab
• Verify normal HTML pages without file extensions are scanned normally (not classified as unsupported)
• Verify scan-end cleanup glob matches _pool1, _pool2, etc. on both platforms