Skip to content

Recover DNS outages and monitor browser readiness - #9

Merged
Horcag merged 15 commits into
mainfrom
fix/crawl-recovery
Oct 26, 2026
Merged

Horcag merged 15 commits into
mainfrom
fix/crawl-recovery

Conversation

@Horcag

@Horcag Horcag commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Reads failed with 502 when Docker inherited a WSL resolver with no usable external DNS upstream. The service still reported healthy, and crawler transport failures skipped browser recovery.

Configure explicit DNS for outbound services and verify DNS, a complete proxy CONNECT exchange, an uncached owned browser canary, and the MCP catalog. A host watchdog recovers unhealthy services using verified Compose identities, persistent cooldowns and restart budgets; proxy failures recover the proxy itself. GET reads retry transient failures in a fresh browser and can use verified CDP fallback. POST requests, configuration failures and robots denials remain terminal across inappropriate recovery routes. Recovery receipts remain available through MCP.

This also brings the shared browser/CAPTCHA recovery and bounded search components used by those paths into main. Deterministic tests close their owned connections on supported Python versions and bound parameter IDs independently of oversized fixture payloads.

Validation:

  • GitHub quality gates: 919 executed cases each on Python 3.12 and 3.13; 94.02% coverage. Formatting, lint, types, source budgets and Compose routes passed.
  • GitHub dashboard/browser workflow and packaged-image smoke with real bundled Chromium passed.
  • Local complete workspace gate: 1236 tests passed. An independently isolated Python 3.12 run confirmed all 919 delivery cases executed without skips.
  • Live fault injection: stopped the verified ByeDPI process while its TCP listener remained available. The watchdog detected the proxy failure, restarted ByeDPI and restored API readiness automatically in 117.3 seconds. Final runtime layers match the fault-tested image.
  • Both original URLs returned HTTP 200 and assessed readable content through MCP after recovery: ShadowHint questions and the No Data No Growth Magnit task. Existing private CDP/CAPTCHA configuration is preserved and CDP open/close was verified.
  • Live MCP health reports DNS, proxy and browser healthy. Search returned three results; an independent Yandex parsing error remains reported while another engine serves results.

@Horcag
Horcag merged commit b730379 into main Oct 26, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant