Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 25 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,15 @@ handles pages requiring interaction. A small dashboard keeps runs, source status
## Start

```sh
docker compose up -d --build
bash scripts/up.sh
```

Open **http://127.0.0.1:11235/** for the research workspace. The same service exposes
[OpenAPI](http://127.0.0.1:11235/docs); the MCP endpoint is `http://127.0.0.1:8765/mcp`
(Streamable HTTP). Windows helpers remain available in `scripts/up.ps1` and `scripts/verify.ps1`.
Docker needs an amd64 Linux environment; the image installs Google Chrome, Chromium and Xvfb.
Linux/WSL startup also enables the host watchdog. See [health and recovery](docs/RECOVERY.md)
for probes, restart limits, DNS configuration and deployments without systemd.

## Research workflow

Expand Down Expand Up @@ -44,6 +46,11 @@ subtopics; an external agent supplies semantic reformulations and decides when r
| `research_import_browser_result`, `research_export` | Record actual browser material and export evidence. |
| `browser_open`, `browser_snapshot`, `browser_action`, `browser_close` | Optional CDP tabs and interactions owned by local-web. |

Search tools and durable discovery use the same [bounded search backend](docs/SEARCH.md).
The default language is `auto`: Cyrillic queries select Russian; other queries select all languages.
Pass `domains` and `required_entities` for strict constraints. Results are filtered individually,
ranked before truncation, and retain provider diagnostics and acquisition evidence.

The repository includes a focused [research skill](.agents/skills/local-web-research/SKILL.md)
and [agent workflow](docs/AGENT_WORKFLOW.md). Tools do not treat retrieved page instructions as authority.

Expand All @@ -62,20 +69,35 @@ to an explicitly chosen reachable Chrome CDP endpoint. The adapter creates and c
uses the selected browser context, and disconnects on shutdown without closing the browser or existing
tabs. Docker cannot reach a host browser through container `127.0.0.1`; configure the route deliberately.
Do not expose a debugging endpoint publicly. A configured endpoint does not prove connectivity.
The optional `browser-gateway` Compose profile connects a private host Unix socket to the CDP
browser route. See [CAPTCHA recovery](docs/CAPTCHA.md) for its socket and Host-header configuration.

With that route configured, `read_url` automatically tries the existing browser profile when crawling
returns a blocked, partial or empty failed page. This fallback has a 30-second deadline and makes no
additional CAPTCHA model calls. It accepts only a verified target URL, a known successful HTTP status
and complete readable content. Responses preserve the initial gate receipt alongside the current
browser result and identify the acquisition method. A rejected fallback stays pending for browser work.

The crawler's persistent Docker profile is separate from a user's desktop Chrome profile.
Patchright/headed Chrome is not a guarantee against CAPTCHA or access restrictions.

Optional [CAPTCHA recovery](docs/CAPTCHA.md) sends cropped text-image or ordered-click challenges
to one explicitly configured multimodal API. Browser actions stay local, attempts are bounded,
and successful page recovery is verified before reporting a solve. Unknown puzzles remain blocked.

## Configuration and data

| Setting | Default / behavior |
| --- | --- |
| `LOCAL_WEB_ALLOWED_HOSTS` | Localhost and named Compose routes; configure explicit reverse-proxy hosts. |
| `LOCAL_WEB_RESEARCH_DB` | `work/research.sqlite3` locally; `/app/work/research.sqlite3` in Compose. |
| `LOCAL_WEB_BROWSER_CDP_URL` | Empty: explicit external-browser handoff/import. |
| `LOCAL_WEB_SEARCH_DELAY_SECONDS` | Minimum spacing between research search requests. |
| `LOCAL_WEB_BROWSER_CDP_HOST_HEADER` | Empty: preserve the normal Host header; opt in for a private CDP relay. |
| `LOCAL_WEB_SEARCH_DELAY_SECONDS` | Minimum spacing between all provider dispatches; default 1 second. |
| `LOCAL_WEB_RESEARCH_ALLOW_PRIVATE` | Off; enable only for explicitly scoped internal targets/test fixtures. |
| `LOCAL_WEB_SEARCH_ENGINES` | Existing MCP pool: `mojeek,mwmbl,yandex`. |
| `LOCAL_WEB_SEARCH_ENGINES` | Shared general search pool: `mojeek,mwmbl,yandex`. |
| `LOCAL_WEB_SEARCH_CACHE_TTL_SECONDS` | Successful non-degraded searches cached for 900 seconds; zero disables. |
| `LOCAL_WEB_BROWSER_TARGETED_READS` | True: automatic reads attach only to their own hidden CDP target. |
| `CRAWL4AI_DOMAIN_DELAY_SECONDS` | Existing per-domain crawl spacing. |
| `CRAWL4AI_PERSISTENT_CONTEXT` | True; browser profile retained in the data volume. |

Expand Down
114 changes: 109 additions & 5 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,29 +1,108 @@
name: local-web-stack

# Do not inherit a WSL-only resolver that Docker cannot forward to after reboot.
# Docker's embedded resolver still handles service names on the Compose network.
x-external-dns: &external-dns
- "${LOCAL_WEB_DNS_PRIMARY:-1.1.1.1}"
- "${LOCAL_WEB_DNS_SECONDARY:-8.8.8.8}"

services:
browser-relay:
image: "${LOCAL_WEB_GATEWAY_RELAY_IMAGE:-local-web-stack-crawl4ai}"
profiles: [browser-gateway]
container_name: local-web-browser-relay
user: "${LOCAL_WEB_BROWSER_RELAY_UID:-1000}:${LOCAL_WEB_BROWSER_RELAY_GID:-1000}"
init: true
restart: unless-stopped
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
pids_limit: 64
environment:
PYTHONDONTWRITEBYTECODE: "1"
LOCAL_WEB_GATEWAY_RELAY_BIND_HOST: container
LOCAL_WEB_GATEWAY_RELAY_BIND_PORT: "19226"
LOCAL_WEB_GATEWAY_RELAY_TARGET_UNIX_PATH: /run/browser-gateway.sock
volumes:
- type: bind
source: "${LOCAL_WEB_BROWSER_RELAY_SOCKET:-/run/local-web/browser-gateway.sock}"
target: /run/browser-gateway.sock
read_only: true
bind:
create_host_path: false
command: .venv/bin/python -m service.gateway_relay

gateway-relay:
image: "${LOCAL_WEB_GATEWAY_RELAY_IMAGE:-local-web-stack-crawl4ai}"
profiles: [captcha-gateway]
container_name: local-web-gateway-relay
user: "${LOCAL_WEB_GATEWAY_RELAY_UID:-1000}:${LOCAL_WEB_GATEWAY_RELAY_GID:-1000}"
init: true
restart: unless-stopped
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
pids_limit: 64
environment:
PYTHONDONTWRITEBYTECODE: "1"
LOCAL_WEB_GATEWAY_RELAY_BIND_HOST: "${LOCAL_WEB_GATEWAY_RELAY_BIND_HOST:-}"
LOCAL_WEB_GATEWAY_RELAY_BIND_PORT: "${LOCAL_WEB_GATEWAY_RELAY_BIND_PORT:-18045}"
LOCAL_WEB_GATEWAY_RELAY_TARGET_HOST: "${LOCAL_WEB_GATEWAY_RELAY_TARGET_HOST:-127.0.0.1}"
LOCAL_WEB_GATEWAY_RELAY_TARGET_PORT: "${LOCAL_WEB_GATEWAY_RELAY_TARGET_PORT:-8045}"
LOCAL_WEB_GATEWAY_RELAY_TARGET_UNIX_PATH: /run/model-gateway.sock
volumes:
- type: bind
source: "${LOCAL_WEB_GATEWAY_RELAY_SOCKET:-/run/local-web/model-gateway.sock}"
target: /run/model-gateway.sock
read_only: true
bind:
create_host_path: false
command: .venv/bin/python -m service.gateway_relay

redis:
image: valkey/valkey:8-alpine
container_name: local-web-valkey
command: valkey-server --save 30 1 --loglevel warning
restart: unless-stopped
healthcheck:
test: [CMD, valkey-cli, ping]
interval: 30s
timeout: 5s
retries: 3
volumes:
- valkey-data:/data

byedpi:
dns: *external-dns
build:
context: .
dockerfile: Dockerfile.byedpi
container_name: local-web-byedpi
restart: unless-stopped
healthcheck:
test: [CMD-SHELL, 'nslookup example.com >/dev/null && nc -z -w 3 127.0.0.1 1080']
interval: 30s
timeout: 8s
retries: 3
start_period: 15s
command: --ip 0.0.0.0 --port 1080 --auto=torst -s 1+s

searxng:
dns: *external-dns
image: searxng/searxng:2026.8.4-c63835bd2
container_name: local-web-searxng
restart: unless-stopped
depends_on:
- redis
- byedpi
redis:
condition: service_healthy
byedpi:
condition: service_healthy
healthcheck:
test: [CMD, /usr/local/searxng/.venv/bin/python, -c, 'import socket, urllib.request; socket.getaddrinfo("example.com", 443); urllib.request.urlopen("http://127.0.0.1:8080/healthz", timeout=4).close()']
interval: 30s
timeout: 8s
retries: 3
start_period: 30s
ports:
# On Windows hosts WinNAT re-rolls its dynamic TCP exclusion ranges at
# boot and can swallow this port, after which Docker fails to bind with
Expand All @@ -42,25 +121,37 @@ services:
- ./searxng:/etc/searxng:rw

crawl4ai:
dns: *external-dns
build: .
container_name: local-web-crawl4ai
# tini as PID 1: uvicorn does not reap orphaned Chrome children, and the
# zombies pile up until the container runs out of PID slots.
init: true
restart: unless-stopped
depends_on:
- byedpi
byedpi:
condition: service_healthy
healthcheck:
test: [CMD, .venv/bin/python, -m, service.healthcheck, crawl4ai]
interval: 30s
timeout: 10s
retries: 3
start_period: 90s
ports:
- "127.0.0.1:11235:11235"
environment:
SEARXNG_URL: "http://local-web-searxng:8080"
LOCAL_WEB_RESEARCH_DB: "/app/work/research.sqlite3"
LOCAL_WEB_BROWSER_CDP_URL: "${LOCAL_WEB_BROWSER_CDP_URL:-}"
LOCAL_WEB_BROWSER_CDP_HOST_HEADER: "${LOCAL_WEB_BROWSER_CDP_HOST_HEADER:-}"
CRAWL4AI_DEFAULT_PROXY: "socks5://byedpi:1080"
CRAWL4AI_CACHE_MODE: "enabled"
CRAWL4AI_PERSISTENT_CONTEXT: "true"
CRAWL4AI_USER_DATA_DIR: "/app/work/browser-profile"
CRAWL4AI_SERIALIZE_REQUESTS: "true"
LOCAL_WEB_CRAWL_CANARY_ENABLED: "true"
LOCAL_WEB_CRAWL_CANARY_URL: "http://crawl4ai:11235/health/canary"
LOCAL_WEB_CRAWL_TIMEOUT_SECONDS: "75"
CRAWL4AI_PAGE_TIMEOUT_MS: "60000"
CRAWL4AI_WAIT_UNTIL: "domcontentloaded"
# crawl4ai's overlay heuristic strips any element it reads as a modal.
Expand All @@ -86,6 +177,11 @@ services:
CRAWL4AI_MAGIC: "false"
CRAWL4AI_SIMULATE_USER: "false"
CRAWL4AI_OVERRIDE_NAVIGATOR: "false"
LOCAL_WEB_CAPTCHA_ENABLED: "${LOCAL_WEB_CAPTCHA_ENABLED:-false}"
LOCAL_WEB_CAPTCHA_BASE_URL: "${LOCAL_WEB_CAPTCHA_BASE_URL:-}"
LOCAL_WEB_CAPTCHA_API_KEY: "${LOCAL_WEB_CAPTCHA_API_KEY:-}"
LOCAL_WEB_CAPTCHA_MODEL: "${LOCAL_WEB_CAPTCHA_MODEL:-gemini-3.8-flash-high}"
LOCAL_WEB_CAPTCHA_TIMEOUT_SECONDS: "${LOCAL_WEB_CAPTCHA_TIMEOUT_SECONDS:-15}"
CRAWL4AI_DELAY_BEFORE_RETURN_HTML: "2.5"
CRAWL4AI_MEAN_DELAY: "1.0"
CRAWL4AI_MAX_DELAY_RANGE: "1.5"
Expand All @@ -98,8 +194,16 @@ services:
container_name: local-web-mcp
restart: unless-stopped
depends_on:
- searxng
- crawl4ai
searxng:
condition: service_healthy
crawl4ai:
condition: service_healthy
healthcheck:
test: [CMD, .venv/bin/python, -m, service.healthcheck, mcp]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
ports:
- "127.0.0.1:8765:8765"
environment:
Expand Down
Loading