diff --git a/CHANGELOG.md b/CHANGELOG.md index 0be6d0c..8bb524b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,42 @@ # Changelog + +## 1.4.0 — 2026-09-11 + +### Added +- **A bot wall now says so.** When a site's protection refuses the fetch, the + error names the wall instead of a bare `403 Client Error: Forbidden` — e.g. + *"Blocked by Cloudflare — tapology.com uses bot protection and refused this + request (HTTP 403). Automated access isn't allowed."* Cloudflare, DataDome, + PerimeterX/HUMAN, Akamai and Imperva + Incapsula are recognised by their response headers and challenge-page markers. + A new `bytecrawl.BlockedError` (a subclass of `requests.HTTPError`) carries it, + so `except requests.HTTPError` still catches it and you can catch the block + specifically. Every surface benefits at once — the library, the crawlers, the + hosted API and the playground — since the detection lives at the fetch layer. + Detection is gated on a block-looking status, so a normal page served from + behind Cloudflare is never mislabelled. +- **A rate limit now says so, too.** A genuine `HTTP 429 Too Many Requests` + (no bot-wall signature) no longer reads as "couldn't reach the site" — it gets + its own calm message: *"The site is rate-limiting us… it's up and reachable, + just asking for fewer requests. Wait a moment and try again."* A new + `bytecrawl.RateLimitError` (also a `requests.HTTPError` subclass) carries the + server's `Retry-After` when present — both forms, a delta in seconds and an + HTTP date. The hosted API answers `429` with a `Retry-After` header and a + `retry_after` field, and the playground shows a wait hint. A 429 that *does* + carry a bot-wall signature still reports as blocked — the more specific answer + wins. +- **One error card everywhere.** Unreachable seeds (bad DNS, refused + connection, timeout) now raise a single `bytecrawl.UnreachableError` across the + library, crawlers, hosted API and playground, so every playground mode — + Markdown, links, JSON API, extract, auto and compare — renders the same card + for the same failure instead of three different-looking errors. +- **MCP tools now tell the agent what went wrong.** A blocked, rate-limited or + unreachable seed used to reach an MCP client as a masked *"Error executing + tool …"* — the SDK treats a stray `requests` exception as a crash and withholds + its text. Every tool (local and hosted) now re-raises these as a `ToolError`, + so the agent receives the real message — the wall's name, or the wait hint for + a 429 — the MCP counterpart of the API's typed 403/429 body. + ## 1.3.0 — 2026-09-11 ### Changed diff --git a/README.es.md b/README.es.md index ec8097d..78d336a 100644 --- a/README.es.md +++ b/README.es.md @@ -10,8 +10,8 @@ Shark-Search y OPIC en Python puro. - **Web**: https://bytecrawl.vercel.app/ - **Endpoint MCP hosteado**: https://bytecrawl.vercel.app/mcp - **Skill para agentes**: https://bytecrawl.vercel.app/agent-onboarding/SKILL.md -- **Última versión**: **1.3.0** — publicación automática a PyPI y checks de CI - separados (lint, formato, tipos, seguridad) ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md)) +- **Última versión**: **1.4.0** — errores más claros cuando la protección + anti-bot de un sitio (Cloudflare, DataDome, …) bloquea el fetch ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md)) - **PyPI**: https://pypi.org/project/bytecrawl/ - **GitHub**: https://github.com/abrahamperz/ByteCrawl diff --git a/README.md b/README.md index 702fd92..b82fe6e 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,8 @@ and OPIC in pure Python. - **Webpage**: https://bytecrawl.vercel.app/ - **Hosted MCP endpoint**: https://bytecrawl.vercel.app/mcp - **Agent skill**: https://bytecrawl.vercel.app/agent-onboarding/SKILL.md -- **Latest release**: **1.3.0** — automated PyPI publishing and separated CI - gates (lint, format, types, security) ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md)) +- **Latest release**: **1.4.0** — clearer errors when a site's bot protection + (Cloudflare, DataDome, …) blocks a fetch ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md)) ## Why focused crawling? diff --git a/pyproject.toml b/pyproject.toml index 79f2862..2d5aa91 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -29,7 +29,10 @@ browser = ["playwright>=1.40"] llm = ["markdownify>=0.11", "trafilatura>=1.6"] mcp = ["mcp>=2; python_version >= '3.10'", "markdownify>=0.11", "trafilatura>=1.6"] all = ["playwright>=1.40", "markdownify>=0.11", "trafilatura>=1.6", "mcp>=2; python_version >= '3.10'"] -dev = ["pytest>=7", "pytest-cov", "ruff", "mypy", "bandit"] +# flask/python-dotenv/posthog are here (not runtime deps of the library) so the +# test suite can import the web landing app and exercise the /api seed-error +# surface. Kept in sync with web/landing/requirements.txt. +dev = ["pytest>=7", "pytest-cov", "ruff", "mypy", "bandit", "flask", "python-dotenv", "posthog"] [project.scripts] bytecrawl-mcp = "bytecrawl.mcp_server:main" diff --git a/src/bytecrawl/__init__.py b/src/bytecrawl/__init__.py index beca973..2ca8588 100644 --- a/src/bytecrawl/__init__.py +++ b/src/bytecrawl/__init__.py @@ -22,7 +22,17 @@ LLM: page.markdown() · page.tokens() """ -from .core import Page, Scraper, Session +from .core import ( + RATE_LIMIT_MESSAGE, + UNREACHABLE_MESSAGE, + BlockedError, + Page, + RateLimitError, + Scraper, + Session, + UnreachableError, + open_seed, +) from .crawler import ( BFS, OPIC, @@ -39,6 +49,12 @@ "Scraper", "Page", "Session", + "BlockedError", + "RateLimitError", + "UnreachableError", + "RATE_LIMIT_MESSAGE", + "UNREACHABLE_MESSAGE", + "open_seed", "Crawler", "BFS", "SharkSearch", @@ -49,4 +65,4 @@ "cosine", "compare", ] -__version__ = "1.3.0" +__version__ = "1.4.0" diff --git a/src/bytecrawl/core.py b/src/bytecrawl/core.py index 3b06f1d..9d911cc 100644 --- a/src/bytecrawl/core.py +++ b/src/bytecrawl/core.py @@ -14,12 +14,29 @@ from .scraping.scraper import Scraper from .scraping.selectors import _split_selector, _value_from from .scraping.session import Session -from .scraping.url import _SKIP_EXT, DEFAULT_UA, _decoded_html, normalize +from .scraping.url import ( + _SKIP_EXT, + DEFAULT_UA, + RATE_LIMIT_MESSAGE, + UNREACHABLE_MESSAGE, + BlockedError, + RateLimitError, + UnreachableError, + _decoded_html, + normalize, + open_seed, +) __all__ = [ "Page", "Session", "Scraper", + "BlockedError", + "RateLimitError", + "UnreachableError", + "RATE_LIMIT_MESSAGE", + "UNREACHABLE_MESSAGE", + "open_seed", "normalize", "DEFAULT_UA", "_SKIP_EXT", diff --git a/src/bytecrawl/crawl/base.py b/src/bytecrawl/crawl/base.py index 090399d..a87c7ed 100644 --- a/src/bytecrawl/crawl/base.py +++ b/src/bytecrawl/crawl/base.py @@ -15,7 +15,7 @@ # crawlers agree on what counts as a link; the crawler module re-exports it # because that is where the tests and docs import it from. from ..scraping.scraper import Scraper -from ..scraping.url import normalize +from ..scraping.url import BlockedError, RateLimitError, normalize from .frontier import Frontier, _root_domain from .result import CrawlResult from .text import relevance @@ -81,6 +81,21 @@ def crawl(self, start: str, max_pages: int = 50, max_depth: int = 10) -> CrawlRe try: page = self.scraper.static(url) + except BlockedError as e: + # A single walled page shouldn't abort the crawl, but remember + # the first wall: if the crawl comes back empty, the caller can + # say "blocked" instead of the misleading "nothing relevant". + errors += 1 + if result.blocked is None: + result.blocked = e + continue + except RateLimitError as e: + # Likewise for a 429: keep crawling other pages, but remember the + # first throttle so an empty crawl reports "slow down and retry". + errors += 1 + if result.rate_limited is None: + result.rate_limited = e + continue except Exception: errors += 1 continue diff --git a/src/bytecrawl/crawl/compare.py b/src/bytecrawl/crawl/compare.py index 7aca8d9..31bc716 100644 --- a/src/bytecrawl/crawl/compare.py +++ b/src/bytecrawl/crawl/compare.py @@ -4,18 +4,25 @@ from concurrent.futures import ThreadPoolExecutor +from ..scraping.url import BlockedError, RateLimitError, UnreachableError from .strategies import STRATEGIES def _leg( name: str, start: str, query: str, max_pages: int, max_depth: int, delay: float, timeout: int ) -> dict: - """One strategy's leg of a comparison. Never raises: a strategy that fails - reports inside its own entry so the other two still come back.""" + """One strategy's leg of a comparison. A strategy that fails on its own + reports inside its own entry so the other two still come back — but a + seed-level failure (a walled or unreachable start URL) is not that: it + fails every leg identically, so re-raise it and let :func:`compare` surface + it once instead of burying three copies in three error strings.""" try: result = STRATEGIES[name](query=query, delay=delay, timeout=timeout).crawl( start, max_pages=max_pages, max_depth=max_depth ) + result.raise_for_seed() + except (BlockedError, RateLimitError, UnreachableError): + raise except Exception as e: return {"error": f"{type(e).__name__}: {e}"} return { diff --git a/src/bytecrawl/crawl/result.py b/src/bytecrawl/crawl/result.py index 9b400e8..33edcce 100644 --- a/src/bytecrawl/crawl/result.py +++ b/src/bytecrawl/crawl/result.py @@ -4,6 +4,8 @@ from dataclasses import dataclass, field +from ..scraping.url import BlockedError, RateLimitError, UnreachableError + @dataclass class CrawlResult: @@ -13,9 +15,53 @@ class CrawlResult: pages: list[dict] = field(default_factory=list) # url, title, score, relevance, depth, order graph: dict[str, list[str]] = field(default_factory=dict) stats: dict = field(default_factory=dict) + # The bot wall that turned the crawl away, if any. A crawl swallows a + # per-page block and keeps going, but when the seed is walled it comes + # back empty — callers read this to tell "walled" apart from "nothing here". + blocked: BlockedError | None = None + # Same idea for a throttled seed: the crawl swallows a per-page 429 and + # keeps going, but a rate-limited *seed* comes back empty — remembered here + # so the caller can say "slow down and retry" instead of "nothing here". + rate_limited: RateLimitError | None = None def relevant(self, threshold: float = 0.1) -> list[dict]: return [p for p in self.pages if p["relevance"] >= threshold] def top(self, n: int = 10) -> list[dict]: return sorted(self.pages, key=lambda p: p["relevance"], reverse=True)[:n] + + @property + def unreachable(self) -> bool: + """Every fetch failed: the crawl never read a single page. + + A crawl swallows per-page errors and keeps going, so an empty result + is ambiguous — the seed was unreachable (a domain that doesn't + resolve, a refused connection, a 404, or two URLs pasted into one), or + it read fine and simply matched nothing. When every request failed + (``requests - errors <= 0``) it is the former. A walled seed also + lands here, but is reported through :attr:`blocked` first. + """ + reqs = self.stats.get("requests", 0) + return reqs >= 1 and reqs - self.stats.get("errors", 0) <= 0 + + def raise_for_seed(self) -> None: + """Raise if the crawl never got off the ground; return quietly otherwise. + + Turns a dead-on-arrival crawl into the same exception a single fetch + would have raised, so every caller (the HTTP API, the MCP tools) can + report a walled or unreachable seed uniformly instead of dressing an + empty crawl up as "found nothing relevant". A crawl that read at least + one page returns quietly — even one that matched nothing, which is a + real answer, not a failure. + + Precedence, most specific first: a bot wall (blocked), then a rate limit + (throttled but reachable), then a dead seed (unreachable). A walled seed + can set both blocked and unreachable; "a bot wall refused you" is the + more actionable answer, and a 429 sits between the two. + """ + if self.blocked is not None and not self.pages: + raise self.blocked + if self.rate_limited is not None and not self.pages: + raise self.rate_limited + if self.unreachable: + raise UnreachableError() # default message: the one shared wording diff --git a/src/bytecrawl/mcp_server.py b/src/bytecrawl/mcp_server.py index 12931a2..e7a349b 100644 --- a/src/bytecrawl/mcp_server.py +++ b/src/bytecrawl/mcp_server.py @@ -15,18 +15,62 @@ try: from mcp.server.mcpserver import MCPServer + from mcp.server.mcpserver.exceptions import ToolError except ImportError as e: # pragma: no cover - exercised only without the extra raise ImportError("The MCP server needs the 'mcp' extra: pip install bytecrawl[mcp]") from e -from .core import Scraper +from contextlib import contextmanager + +from .core import ( + BlockedError, + RateLimitError, + Scraper, + UnreachableError, + open_seed, +) from .crawler import STRATEGIES as _STRATEGIES from .crawler import compare, pagerank DELAY = 0.2 # seconds between requests, per crawler + +@contextmanager +def _seed_errors_as_tool_errors(): + """Report a seed failure as an anticipated :class:`ToolError`. + + Our seed errors are ``requests`` exceptions. To the MCP SDK an escaping + ``requests`` exception is a *crash*: it withholds the message and hands the + agent a bare ``Error executing tool ``. Re-raised as ``ToolError`` they + come back as an ``is_error`` result carrying our text — the MCP counterpart + of the HTTP API's typed body. The specifics fold into the message (the wall's + name, the wait hint) because an MCP client reads the sentence, not a JSON + field, so this is the same information the API returns as ``retry_after``. + """ + try: + yield + except RateLimitError as e: + wait = f" (retry after ~{e.retry_after}s)" if e.retry_after is not None else "" + raise ToolError(f"{e}{wait}") from e + except (BlockedError, UnreachableError) as e: + raise ToolError(str(e)) from e + + # One polite scraper shared by every tool call. _scraper = Scraper(delay=DELAY) + +def _open(url: str, *, api: bool = False, **kw): + """Fetch the seed through the shared :func:`bytecrawl.open_seed` guard. + + Keeps the technique choice here (``api`` -> ``Scraper.api``, otherwise the + auto ``Scraper.fetch``) while the block/unreachable translation lives in the + core, so a walled or unopenable URL reports the same way here, in the HTTP + API, and in the crawl tools' ``CrawlResult.raise_for_seed()``. + """ + with _seed_errors_as_tool_errors(): + return open_seed(lambda: _scraper.api(url, **kw) if api else _scraper.fetch(url)) + + server = MCPServer( "bytecrawl", instructions=( @@ -47,7 +91,7 @@ ) ) def fetch_markdown(url: str) -> dict: - page = _scraper.fetch(url) + page = _open(url) md = page.markdown() # tokens_html alongside it, the same pair the playground shows. On its own # tokens_estimate is a number with nothing to compare against; next to the @@ -95,7 +139,7 @@ def extract( ) # Checked before the fetch: a malformed call shouldn't cost the site a # request. - page = _scraper.fetch(url) + page = _open(url) if select: values = page.css_all(select) return {"url": url, "select": select, "count": len(values), "values": values} @@ -124,7 +168,7 @@ def extract( ) ) def list_links(url: str, raw: bool = False) -> dict: - page = _scraper.fetch(url) + page = _open(url) links = page.links(raw=raw) return {"url": url, "count": len(links), "raw": raw, "links": links} @@ -143,6 +187,12 @@ def focused_crawl(url: str, query: str = "", strategy: str = "shark", max_pages: max_pages = min(max_pages, 50) # keep agent calls bounded and polite crawler = _STRATEGIES[strategy](query=query, delay=DELAY) result = crawler.crawl(url, max_pages=max_pages) + # A walled, throttled or unreachable seed comes back as an empty crawl; raise + # it as the same blocked/rate-limited/unreachable error a single fetch would + # (as a ToolError, so the agent sees it), rather than returning an empty page + # list dressed up as "nothing relevant". + with _seed_errors_as_tool_errors(): + result.raise_for_seed() ranks = pagerank(result.graph) return { "strategy": strategy, @@ -164,13 +214,14 @@ def focused_crawl(url: str, query: str = "", strategy: str = "shark", max_pages: ) def compare_strategies(url: str, query: str, max_pages: int = 20) -> dict: max_pages = min(max_pages, 50) # same bound as focused_crawl, per strategy - out = compare(url, query, max_pages=max_pages, delay=DELAY) + with _seed_errors_as_tool_errors(): + out = compare(url, query, max_pages=max_pages, delay=DELAY) return {"url": url, "query": query, "max_pages": max_pages, **out} @server.tool(description="Fetch a JSON API endpoint (the 'hidden API' scraping technique).") def fetch_json_api(url: str, params: dict | None = None) -> dict: - page = _scraper.api(url, params=params) + page = _open(url, api=True, params=params) return {"url": url, "status": page.status, "data": page.json()} diff --git a/src/bytecrawl/scraping/__init__.py b/src/bytecrawl/scraping/__init__.py index 65f3ec0..bf12b90 100644 --- a/src/bytecrawl/scraping/__init__.py +++ b/src/bytecrawl/scraping/__init__.py @@ -10,6 +10,6 @@ from .page import Page from .scraper import Scraper from .session import Session -from .url import normalize +from .url import BlockedError, RateLimitError, normalize -__all__ = ["Page", "Session", "Scraper", "normalize"] +__all__ = ["Page", "Session", "Scraper", "normalize", "BlockedError", "RateLimitError"] diff --git a/src/bytecrawl/scraping/scraper.py b/src/bytecrawl/scraping/scraper.py index fd7129b..f5e7d89 100644 --- a/src/bytecrawl/scraping/scraper.py +++ b/src/bytecrawl/scraping/scraper.py @@ -14,7 +14,7 @@ from .page import Page from .selectors import _split_selector, _value_from from .session import Session -from .url import DEFAULT_UA, _decoded_html +from .url import DEFAULT_UA, _decoded_html, _raise_for_status class Scraper: @@ -32,7 +32,7 @@ def static(self, url: str) -> Page: """Technique 1: static HTML with requests.""" t0 = time.perf_counter() r = self._session.get(url, timeout=self.timeout) - r.raise_for_status() + _raise_for_status(r) self._wait() return Page( url=url, @@ -46,7 +46,7 @@ def api(self, url: str, params: dict | None = None) -> Page: """Technique 3: requests an API and stores the JSON.""" t0 = time.perf_counter() r = self._session.get(url, params=params, timeout=self.timeout) - r.raise_for_status() + _raise_for_status(r) self._wait() try: data = r.json() diff --git a/src/bytecrawl/scraping/session.py b/src/bytecrawl/scraping/session.py index 981a15f..4b9138f 100644 --- a/src/bytecrawl/scraping/session.py +++ b/src/bytecrawl/scraping/session.py @@ -14,7 +14,7 @@ from bs4 import BeautifulSoup from .page import Page -from .url import _decoded_html +from .url import _decoded_html, _raise_for_status if TYPE_CHECKING: from .scraper import Scraper @@ -32,13 +32,13 @@ def login(self, url: str, data: dict, csrf_field: str | None = None) -> Session: """Logs in. If csrf_field is given, reads it from the form first.""" if csrf_field: r = self._s.get(url, timeout=self._scraper.timeout) - r.raise_for_status() + _raise_for_status(r) form = BeautifulSoup(_decoded_html(r), "lxml") token = form.select_one(f'input[name="{csrf_field}"]') if token: data = {**data, csrf_field: token.get("value", "")} r = self._s.post(url, data=data, timeout=self._scraper.timeout) - r.raise_for_status() + _raise_for_status(r) return self def bearer(self, token: str) -> Session: @@ -48,7 +48,7 @@ def bearer(self, token: str) -> Session: def fetch(self, url: str) -> Page: t0 = time.perf_counter() r = self._s.get(url, timeout=self._scraper.timeout) - r.raise_for_status() + _raise_for_status(r) return Page( url=url, html=_decoded_html(r), diff --git a/src/bytecrawl/scraping/url.py b/src/bytecrawl/scraping/url.py index 4123886..ff6d7fc 100644 --- a/src/bytecrawl/scraping/url.py +++ b/src/bytecrawl/scraping/url.py @@ -7,6 +7,8 @@ from __future__ import annotations import re +from datetime import datetime, timezone +from email.utils import parsedate_to_datetime from urllib.parse import urldefrag, urljoin, urlparse import requests @@ -51,3 +53,221 @@ def _decoded_html(r: requests.Response) -> str: if r.encoding is None or r.encoding.lower() == "iso-8859-1": r.encoding = r.apparent_encoding return r.text + + +class BlockedError(requests.HTTPError): + """A site's bot-protection / WAF refused the request. + + Subclasses :class:`requests.HTTPError`, so code that already does + ``except requests.HTTPError`` keeps catching it. Catch ``BlockedError`` + specifically to tell a bot wall (Cloudflare, DataDome, …) apart from an + ordinary 404/500. + + Carries the ``wall`` name and ``status`` so a caller can render a + structured message without re-parsing the string. + """ + + def __init__(self, *args: object, wall: str | None = None, status: int | None = None) -> None: + super().__init__(*args) + self.wall = wall + self.status = status + + +# The one wording for "the seed never loaded", so the API, the MCP tools and +# the crawl path (via CrawlResult.raise_for_seed) all say the same thing and a +# reword lands everywhere at once. UnreachableError() with no message uses it. +UNREACHABLE_MESSAGE = ( + "Couldn't open that page — the site didn't respond. Check the URL is real and complete." +) + + +# The one wording for "the site is throttling us", shared like UNREACHABLE_MESSAGE +# so a 429 reads the same on every surface. RateLimitError() with no message uses +# it; the actual seconds-to-wait ride alongside on `.retry_after`. +RATE_LIMIT_MESSAGE = ( + "The site is rate-limiting us (HTTP 429 Too Many Requests). It's up and " + "reachable — just asking for fewer requests. Wait a moment and try again." +) + + +def _retry_after_seconds(value: str | None) -> int | None: + """Seconds to wait from a ``Retry-After`` header, or ``None``. + + RFC 7231 allows either a delta in seconds ("120") or an HTTP date. We read + both: the numeric form directly, and the date form as the delta from now + (clamped at zero, so a past date reads as "retry now" rather than negative). + """ + if not value: + return None + value = value.strip() + if value.isdigit(): + return int(value) + try: + when = parsedate_to_datetime(value) + except (TypeError, ValueError): + return None + if when is None: + return None + if when.tzinfo is None: + when = when.replace(tzinfo=timezone.utc) + delta = (when - datetime.now(timezone.utc)).total_seconds() + return max(0, int(delta)) + + +class RateLimitError(requests.HTTPError): + """The seed answered 429 Too Many Requests — reachable, just throttled. + + Distinct from :class:`BlockedError` (a bot wall refusing automated access + outright) and :class:`UnreachableError` (the page never loaded at all): a + rate limit is temporary and often carries a ``Retry-After`` saying how long + to wait. Subclasses :class:`requests.HTTPError`, so ``except + requests.HTTPError`` keeps catching it; catch ``RateLimitError`` specifically + to tell "slow down and retry" apart from a hard block or a dead seed. + + Raised with no message it carries :data:`RATE_LIMIT_MESSAGE`, the single + wording every surface shares. ``retry_after`` is the seconds to wait when + the site said so (else ``None``); ``status`` is 429. + """ + + def __init__( + self, + *args: object, + retry_after: int | None = None, + status: int = 429, + url: str | None = None, + ) -> None: + if not args: + args = (RATE_LIMIT_MESSAGE,) + super().__init__(*args) + self.retry_after = retry_after + self.status = status + self.url = url + + +class UnreachableError(requests.RequestException): + """The seed URL couldn't be opened at all — the page never loaded. + + A domain that doesn't resolve, a refused connection, a 404, or two URLs + pasted into one. Distinct from :class:`BlockedError` (a bot wall said no) + and from a crawl that read the page fine but matched nothing (a real + answer, ``pages`` non-empty). Subclasses :class:`requests.RequestException` + so ``except requests.RequestException`` keeps catching it; catch + ``UnreachableError`` specifically to tell "couldn't open it" apart from an + ordinary request failure. + + Raised with no message it carries :data:`UNREACHABLE_MESSAGE`, the single + wording every surface shares. Carries the offending ``url`` when a caller + has it, for a structured message without re-parsing the string. + """ + + def __init__(self, *args: object, url: str | None = None) -> None: + if not args: + args = (UNREACHABLE_MESSAGE,) + super().__init__(*args) + self.url = url + + +def open_seed(fetch): + """Run one seed fetch, turning a non-block failure into UnreachableError. + + A bot wall stays a :class:`BlockedError` (the caller still gets the named + wall); anything else that stopped the page loading — a domain that doesn't + resolve, a refused connection, a 404, two URLs pasted into one — becomes a + single "couldn't open that page" answer instead of a raw traceback. The one + place the single-fetch surfaces (the HTTP API, the MCP tools) share, so the + fix lands in all of them at once; the crawl surfaces reach the same + :class:`UnreachableError` through :meth:`CrawlResult.raise_for_seed`. + + ``fetch`` is a zero-arg callable returning a Page — e.g. + ``lambda: scraper.static(url)`` or ``lambda: scraper.api(url)`` — so the + caller keeps choosing the technique. + """ + try: + return fetch() + except (BlockedError, RateLimitError): + # A named wall and a 429 are both reachable-but-refused answers, more + # specific than "couldn't open it" — let them through unchanged. + raise + except requests.RequestException as e: + raise UnreachableError() from e + + +# WAF / bot-protection fingerprints: (label, response-header keys, body markers). +# Only consulted on a block-looking status, so serving a normal page from behind +# Cloudflare never trips it — just a challenge/deny does. +_BOT_WALLS = ( + ( + "Cloudflare", + ("cf-mitigated", "cf-ray"), + ("just a moment", "attention required", "cf-browser-verification", "cf_chl", "_cf_chl"), + ), + ("DataDome", ("x-datadome", "x-dd-b"), ("datadome",)), + ("PerimeterX / HUMAN", ("x-px",), ("px-captcha", "perimeterx")), + # "access denied" / "reference #" alone are too generic (a plain auth + # failure says them too); "akamaighost" is Akamai's own edge signature. + ("Akamai", (), ("akamaighost",)), + ("Imperva Incapsula", ("x-iinfo",), ("incapsula", "_incap_", "incap_ses")), +) + +# Statuses a bot wall actually uses to turn a request away. +_BLOCK_STATUS = frozenset({401, 403, 406, 429, 503}) + + +def _detect_bot_wall(r: requests.Response) -> str | None: + """Return the name of the bot wall that blocked ``r``, or ``None``. + + Gated on a block-looking status first, then sniffs response headers, the + ``Server`` header and a slice of the body for known WAF signatures. + """ + if r.status_code not in _BLOCK_STATUS: + return None + headers = {k.lower(): str(v).lower() for k, v in r.headers.items()} + server = headers.get("server", "") + try: + body = r.text[:4096].lower() + except Exception: # a challenge page may not decode cleanly + body = "" + for label, header_keys, markers in _BOT_WALLS: + vendor = label.split()[0].lower() # "cloudflare", "datadome", "imperva", … + if any(k in headers for k in header_keys): + return label + if vendor in server: + return label + if any(m in body for m in markers): + return label + return None + + +def _raise_for_status(r: requests.Response) -> None: + """Like ``Response.raise_for_status()`` but clearer about bot walls. + + On an error status, if the response carries a bot-protection signature, + raise :class:`BlockedError` with a human message naming the wall instead + of a bare ``403 Client Error: Forbidden``. Everything else raises the + ordinary :class:`requests.HTTPError`, unchanged. + """ + try: + r.raise_for_status() + except requests.HTTPError as e: + wall = _detect_bot_wall(r) + if wall: + host = urlparse(getattr(r, "url", "") or "").hostname or "the site" + host = host.removeprefix("www.") + raise BlockedError( + f"Blocked by {wall} — {host} uses bot protection and refused " + f"this request (HTTP {r.status_code}). Automated access isn't " + f"allowed.", + wall=wall, + status=r.status_code, + ) from e + if r.status_code == 429: + # A bare 429 (no WAF signature) is an honest "too many requests": + # the site is up and answered, it just wants fewer of them. That is + # not "couldn't open the page" — surface it as its own retryable + # state, carrying the Retry-After wait when the site gave one. + raise RateLimitError( + retry_after=_retry_after_seconds(r.headers.get("Retry-After")), + status=r.status_code, + url=getattr(r, "url", None), + ) from e + raise diff --git a/src/bytecrawl/security.py b/src/bytecrawl/security.py index 437bd3d..fd1130a 100644 --- a/src/bytecrawl/security.py +++ b/src/bytecrawl/security.py @@ -11,6 +11,8 @@ import socket from urllib.parse import urlparse +from .scraping.url import UnreachableError + def assert_public_url(url: str) -> None: """Reject URLs that could reach private infrastructure. @@ -18,6 +20,15 @@ def assert_public_url(url: str) -> None: Resolves the hostname and requires every returned address to be globally routable. Blocks localhost, RFC1918 ranges, link-local (cloud metadata endpoints like 169.254.169.254) and other reserved space. + + A host that doesn't resolve isn't a security refusal — it just can't be + reached — so it raises :class:`UnreachableError` (the same exception a + failed fetch would), not ``ValueError``. That way an unresolvable or + mistyped domain gets the one shared "couldn't open that page" answer + across every surface, instead of a raw "Cannot resolve host" on the one + path that happens to pre-check. The genuine refusals — a non-http(s) + scheme, a missing hostname, or an address that resolves into private + space — stay ``ValueError``. """ parsed = urlparse(url) if parsed.scheme not in ("http", "https"): @@ -28,7 +39,7 @@ def assert_public_url(url: str) -> None: try: infos = socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP) except socket.gaierror as e: - raise ValueError(f"Cannot resolve host: {host}") from e + raise UnreachableError(url=url) from e for info in infos: ip = ipaddress.ip_address(info[4][0]) if not ip.is_global or ip.is_multicast: diff --git a/tests/conftest.py b/tests/conftest.py index d6ca5d4..5e35c0a 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -30,7 +30,11 @@ def __init__( self.encoding = encoding self.apparent_encoding = apparent_encoding self._json = json_data - self.headers = headers or {"content-type": "text/html"} + # A real Response.headers is case-insensitive; mirror that so code that + # reads e.g. "Retry-After" finds a header sent as "retry-after". + self.headers = requests.structures.CaseInsensitiveDict( + headers or {"content-type": "text/html"} + ) @property def text(self): diff --git a/tests/test_blocked.py b/tests/test_blocked.py new file mode 100644 index 0000000..2f394b7 --- /dev/null +++ b/tests/test_blocked.py @@ -0,0 +1,141 @@ +"""Bot-wall detection: a WAF block becomes a clear BlockedError, not a bare 403. + +Covers the two things that matter: known walls (Cloudflare, DataDome, …) are +named in a human message, and ordinary errors (a plain 404, an unfingerprinted +403, a normal page served from behind Cloudflare) are left exactly as they were. +""" + +from __future__ import annotations + +import pytest +import requests + +import bytecrawl +from bytecrawl import BlockedError, Scraper +from bytecrawl.scraping.url import _detect_bot_wall, _raise_for_status + +from .conftest import FakeResponse + + +def _resp(status, headers=None, text="", url="https://blocked.test/page"): + r = FakeResponse(status=status, headers=headers or {"content-type": "text/html"}, text=text) + r.url = url + return r + + +# --- detection: each vendor, by header and by body --------------------------- +@pytest.mark.parametrize( + "headers, body, expected", + [ + ({"server": "cloudflare", "cf-ray": "8a2b"}, "", "Cloudflare"), + ({"cf-mitigated": "challenge"}, "", "Cloudflare"), + ({}, "Just a moment...", "Cloudflare"), + ({}, "Attention Required! | Cloudflare", "Cloudflare"), + ({"x-datadome": "protected"}, "", "DataDome"), + ({}, "please enable JS — datadome", "DataDome"), + ({"x-px": "1"}, "", "PerimeterX / HUMAN"), + ({}, "px-captcha challenge", "PerimeterX / HUMAN"), + ({}, "Access Denied — AkamaiGHost", "Akamai"), + ({"x-iinfo": "9-1"}, "", "Imperva Incapsula"), + ], +) +def test_detects_named_walls(headers, body, expected): + assert _detect_bot_wall(_resp(403, headers, body)) == expected + + +def test_ignores_walls_on_a_success_status(): + # A normal page served from behind Cloudflare must not look like a block. + ok = _resp(200, {"server": "cloudflare", "cf-ray": "8a2b"}, "real content") + assert _detect_bot_wall(ok) is None + + +def test_unfingerprinted_403_is_not_a_wall(): + assert _detect_bot_wall(_resp(403, {"server": "nginx"}, "Forbidden")) is None + + +def test_plain_access_denied_is_not_a_wall(): + # A credential/permission failure says "Access Denied" too — must NOT be + # mislabelled as a bot wall without a real vendor signature. + r = _resp(403, {"server": "Apache"}, "Access Denied. You lack permission.") + assert _detect_bot_wall(r) is None + + +# --- _raise_for_status: what actually gets raised ---------------------------- +def test_blocked_status_raises_blockederror_with_a_clear_message(): + r = _resp(403, {"server": "cloudflare", "cf-ray": "8a2b"}, "Just a moment...") + with pytest.raises(BlockedError) as exc: + _raise_for_status(r) + msg = str(exc.value) + assert "Cloudflare" in msg + assert "HTTP 403" in msg + assert "blocked.test" in msg # host pulled from the response URL + + +def test_blockederror_carries_wall_and_status(): + # The frontend renders a structured notice from these, not a re-parsed string. + r = _resp(429, {"x-datadome": "protected"}, "") + err = _capture(r) + assert err.wall == "DataDome" + assert err.status == 429 + + +def test_message_strips_www_prefix(): + r = _resp(403, {"cf-mitigated": "block"}, "", "https://www.example.com/x") + with pytest.raises(BlockedError, match=r"example\.com"): + _raise_for_status(r) + assert "www." not in str(_capture(r)) + + +def _capture(r): + try: + _raise_for_status(r) + except BlockedError as e: + return e + return None + + +def test_blockederror_is_an_httperror(): + # Existing `except requests.HTTPError` handlers keep catching it. + assert issubclass(BlockedError, requests.HTTPError) + assert bytecrawl.BlockedError is BlockedError + + +def test_plain_404_stays_a_plain_httperror(): + r = _resp(404, {"server": "nginx"}, "Not Found") + with pytest.raises(requests.HTTPError) as exc: + _raise_for_status(r) + assert not isinstance(exc.value, BlockedError) + + +def test_success_passes_through(): + _raise_for_status(_resp(200)) # does not raise + + +# --- through a crawl --------------------------------------------------------- +def test_crawl_records_the_wall_and_comes_back_empty(http): + # A walled seed yields no pages, but the crawl remembers the wall so the + # caller can say "blocked" instead of the misleading "nothing relevant". + from bytecrawl.crawl import BFS + + url = "https://walled.test/" + http.routes[url] = _resp(403, {"server": "cloudflare", "cf-ray": "1"}, "Just a moment...", url) + result = BFS(query="anything").crawl(url, max_pages=5) + assert result.pages == [] + assert isinstance(result.blocked, BlockedError) + assert result.blocked.wall == "Cloudflare" + + +# --- through the public Scraper surface -------------------------------------- +def test_scraper_static_surfaces_the_block(http): + url = "https://walled.test/" + http.routes[url] = _resp(403, {"server": "cloudflare", "cf-ray": "1"}, "Just a moment...", url) + with pytest.raises(BlockedError, match="Cloudflare"): + Scraper().static(url) + + +def test_scraper_static_still_raises_plain_404(http): + url = "https://gone.test/" + http.routes[url] = _resp(404, {"server": "nginx"}, "nope", url) + with pytest.raises(requests.HTTPError) as exc: + Scraper().static(url) + assert not isinstance(exc.value, BlockedError) diff --git a/tests/test_seed_failures.py b/tests/test_seed_failures.py new file mode 100644 index 0000000..60eac4e --- /dev/null +++ b/tests/test_seed_failures.py @@ -0,0 +1,544 @@ +"""The two seed-failure states, reported identically on every surface. + +A crawl swallows per-page failures and keeps going, so an empty result is +ambiguous: the seed was *walled* (a bot wall said no), *unreachable* (the page +never loaded — bad domain, refused connection, 404, two URLs pasted into one), +or read fine and simply matched nothing. The first two are failures the caller +should hear about plainly; the third is a real answer. + +The point of this file is that the *same* two failures come back the *same* +way whether you go through the core primitives, the MCP tools, or the HTTP +API — so the tests are written as parallel triples: one core assertion, the MCP +tool, the /api method, against the same fake walled and dead seeds. If a +surface ever drifts (an empty crawl dressed up as "nothing relevant" again, a +raw traceback for a dead domain), exactly one of these fails. + +The single seam is ``requests.Session.get`` (the ``http`` fixture): every +surface — ``Scraper.static``/``.api``/``.fetch``, and the crawlers on top of +them — funnels through it, so one routing table drives all three. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest +import requests + +import bytecrawl +from bytecrawl import BlockedError, RateLimitError, UnreachableError +from bytecrawl.crawl import BFS +from bytecrawl.crawl.result import CrawlResult +from bytecrawl.crawler import compare + +from .conftest import FakeResponse + +# The Flask landing app lives under web/, outside the importable package tree. +# create_app() puts web/ and src/ on sys.path at import, but importing the +# module to *reach* create_app needs web/ on the path first. +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "web")) + + +SEED = "https://seed.test/" + + +def _walled(url=SEED): + """A Cloudflare wall on `url` — a real vendor signature, not a bare 403.""" + r = FakeResponse( + status=403, + headers={"server": "cloudflare", "cf-ray": "1a2b"}, + text="Just a moment...", + ) + r.url = url + return r + + +def _dead(url=SEED): + """A seed that 404s: the page never loaded. Distinct from a wall.""" + r = FakeResponse(status=404, headers={"server": "nginx"}, text="Not Found") + r.url = url + return r + + +def _throttled(url=SEED, retry_after="30"): + """A bare 429 with a Retry-After: reachable, just asking us to slow down. + + No WAF signature — a plain "Too Many Requests" from a live site, which must + read as a rate limit, not a dead seed and not a bot wall. + """ + r = FakeResponse( + status=429, + headers={"server": "nginx", "retry-after": retry_after}, + text="Too Many Requests", + ) + r.url = url + return r + + +def _walled_429(url=SEED): + """A 429 that also carries a Cloudflare signature — a wall, not a rate limit. + + Locks the precedence: a bot wall answer is more specific than "slow down". + """ + r = FakeResponse( + status=429, + headers={"server": "cloudflare", "cf-ray": "9z9z"}, + text="Just a moment...", + ) + r.url = url + return r + + +# --------------------------------------------------------------------------- # +# Core: the shared primitives every surface is built on. +# --------------------------------------------------------------------------- # +class TestCorePrimitives: + def test_unreachable_is_true_when_every_fetch_failed(self): + r = CrawlResult(strategy="bfs", stats={"requests": 3, "errors": 3}) + assert r.unreachable is True + + def test_unreachable_is_false_when_a_page_was_read(self): + # One good read among failures is a real (if thin) answer, not a + # dead seed. + r = CrawlResult(strategy="bfs", stats={"requests": 3, "errors": 2}) + assert r.unreachable is False + + def test_unreachable_is_false_before_any_request(self): + # An empty stats dict (nothing ran) must not read as "unreachable". + assert CrawlResult(strategy="bfs", stats={}).unreachable is False + + def test_raise_for_seed_raises_unreachable_on_a_dead_seed(self): + r = CrawlResult(strategy="bfs", stats={"requests": 1, "errors": 1}) + with pytest.raises(UnreachableError): + r.raise_for_seed() + + def test_raise_for_seed_is_quiet_when_a_page_was_read(self): + r = CrawlResult( + strategy="bfs", + pages=[{"url": SEED, "relevance": 0.0}], + stats={"requests": 1, "errors": 0}, + ) + r.raise_for_seed() # does not raise + + def test_block_wins_over_unreachable(self): + # A walled seed sets both flags; "a bot wall refused you" is the more + # specific, more actionable answer, so it takes precedence. + wall = BlockedError("Cloudflare wall", wall="Cloudflare", status=403) + r = CrawlResult(strategy="bfs", stats={"requests": 1, "errors": 1}, blocked=wall) + with pytest.raises(BlockedError): + r.raise_for_seed() + + def test_unreachable_error_is_a_requestexception(self): + # `except requests.RequestException` handlers keep catching it. + assert issubclass(UnreachableError, requests.RequestException) + assert bytecrawl.UnreachableError is UnreachableError + + def test_rate_limit_error_is_an_httperror(self): + # `except requests.HTTPError` keeps catching a 429, and it is exported + # at the top level like the other two seed errors. + assert issubclass(RateLimitError, requests.HTTPError) + assert bytecrawl.RateLimitError is RateLimitError + + def test_raise_for_seed_raises_rate_limit_on_a_throttled_seed(self): + throttle = RateLimitError(retry_after=30) + r = CrawlResult(strategy="bfs", stats={"requests": 1, "errors": 1}, rate_limited=throttle) + with pytest.raises(RateLimitError) as exc: + r.raise_for_seed() + assert exc.value.retry_after == 30 + + def test_block_wins_over_rate_limit(self): + # A seed can look both walled and throttled; the bot-wall answer is the + # more specific, more actionable one, so it takes precedence. + wall = BlockedError("Cloudflare wall", wall="Cloudflare", status=429) + r = CrawlResult( + strategy="bfs", + stats={"requests": 1, "errors": 1}, + blocked=wall, + rate_limited=RateLimitError(), + ) + with pytest.raises(BlockedError): + r.raise_for_seed() + + def test_rate_limit_wins_over_unreachable(self): + # A throttled seed also has every fetch failing (so `unreachable` is + # true), but "slow down and retry" is the more specific answer than + # "couldn't open it". + r = CrawlResult( + strategy="bfs", stats={"requests": 1, "errors": 1}, rate_limited=RateLimitError() + ) + assert r.unreachable is True # every fetch failed… + with pytest.raises(RateLimitError): # …but the rate limit is reported + r.raise_for_seed() + + def test_crawl_over_a_throttled_seed_reports_rate_limited(self, http): + http.routes[SEED] = _throttled() + result = BFS(query="anything").crawl(SEED, max_pages=5) + assert result.pages == [] + assert result.rate_limited is not None + assert result.rate_limited.retry_after == 30 + with pytest.raises(RateLimitError): + result.raise_for_seed() + + def test_compare_surfaces_a_throttled_seed_once(self, http): + http.routes[SEED] = _throttled() + with pytest.raises(RateLimitError): + compare(SEED, query="machine learning", max_pages=3, delay=0.0) + + def test_retry_after_parses_numeric_and_http_date(self): + # Both RFC 7231 forms: a delta in seconds, and an absolute HTTP date + # read as the delta from now (clamped at zero for a past date). + from datetime import datetime, timedelta, timezone + from email.utils import format_datetime + + from bytecrawl.scraping.url import _retry_after_seconds + + assert _retry_after_seconds("30") == 30 + assert _retry_after_seconds(None) is None + assert _retry_after_seconds("soon") is None + + future = datetime.now(timezone.utc) + timedelta(seconds=120) + secs = _retry_after_seconds(format_datetime(future)) + assert 110 <= secs <= 120 # ~120s out, allowing for clock drift + + past = datetime.now(timezone.utc) - timedelta(hours=1) + assert _retry_after_seconds(format_datetime(past)) == 0 + + def test_throttle_with_http_date_retry_after_is_reported(self, http): + # A 429 whose Retry-After is an HTTP date still yields a positive wait. + from datetime import datetime, timedelta, timezone + from email.utils import format_datetime + + when = format_datetime(datetime.now(timezone.utc) + timedelta(seconds=60)) + http.routes[SEED] = _throttled(retry_after=when) + result = BFS(query="anything").crawl(SEED, max_pages=5) + assert result.rate_limited is not None + assert 50 <= result.rate_limited.retry_after <= 60 + + def test_crawl_over_a_dead_seed_reports_unreachable(self, http): + http.routes[SEED] = _dead() + result = BFS(query="anything").crawl(SEED, max_pages=5) + assert result.pages == [] + assert result.unreachable is True + with pytest.raises(UnreachableError): + result.raise_for_seed() + + def test_compare_surfaces_a_dead_seed_once(self, http): + # A dead seed fails all three legs identically; compare() re-raises it + # once instead of burying three fake-empty "winners". + http.routes[SEED] = _dead() + with pytest.raises(UnreachableError): + compare(SEED, query="machine learning", max_pages=3, delay=0.0) + + def test_compare_surfaces_a_walled_seed_once(self, http): + http.routes[SEED] = _walled() + with pytest.raises(BlockedError): + compare(SEED, query="machine learning", max_pages=3, delay=0.0) + + +# --------------------------------------------------------------------------- # +# MCP tools: every tool raises the right seed error. +# --------------------------------------------------------------------------- # +# Only this block needs the `mcp` SDK (the mcp extra, Python 3.10+); the core +# tests above and the /api tests below need Flask, not mcp. Import the SDK +# softly and gate just the MCP tests with `requires_mcp`, so a run without the +# mcp extra (e.g. Python 3.9 CI) still collects and runs everything else instead +# of the whole module aborting on a module-level importorskip. +try: + from mcp.server.mcpserver.exceptions import ToolError, UnexpectedToolError + + from bytecrawl import mcp_server + + _HAS_MCP = True +except ImportError: + _HAS_MCP = False + +requires_mcp = pytest.mark.skipif(not _HAS_MCP, reason="mcp extra not installed (Python 3.10+)") + + +@pytest.fixture(autouse=True) +def _no_delay(): + """The module-level MCP scraper shouldn't sleep in tests.""" + if _HAS_MCP: + mcp_server._scraper.delay = 0.0 + yield + + +# (tool name, kwargs) for each of the six tools, resolved against mcp_server +# inside the test so this list imports without the mcp extra. Single-fetch tools +# and crawl tools reach the seed by different code paths (_open vs +# raise_for_seed); both must land on the same two exceptions. +MCP_TOOLS = [ + ("fetch_markdown", {}), + ("extract", {}), + ("list_links", {}), + ("fetch_json_api", {}), + ("focused_crawl", {"query": "x", "max_pages": 3}), + ("compare_strategies", {"query": "x", "max_pages": 3}), +] + + +# A seed error escaping a tool is re-raised as a ToolError so the SDK returns +# it to the agent as an is_error result carrying our message (a bare requests +# exception would be a crash, masked to "Error executing tool "). The +# typed seed error is preserved as __cause__, so the classification still holds. +@requires_mcp +@pytest.mark.parametrize("tool_name, kw", MCP_TOOLS) +def test_mcp_tool_reports_blocked_on_a_walled_seed(http, tool_name, kw): + http.routes[SEED] = _walled() + with pytest.raises(ToolError) as exc: + getattr(mcp_server, tool_name)(SEED, **kw) + assert isinstance(exc.value.__cause__, BlockedError) + + +@requires_mcp +@pytest.mark.parametrize("tool_name, kw", MCP_TOOLS) +def test_mcp_tool_reports_unreachable_on_a_dead_seed(http, tool_name, kw): + http.routes[SEED] = _dead() + with pytest.raises(ToolError) as exc: + getattr(mcp_server, tool_name)(SEED, **kw) + assert isinstance(exc.value.__cause__, UnreachableError) + + +@requires_mcp +@pytest.mark.parametrize("tool_name, kw", MCP_TOOLS) +def test_mcp_tool_reports_rate_limit_on_a_throttled_seed(http, tool_name, kw): + http.routes[SEED] = _throttled() + with pytest.raises(ToolError) as exc: + getattr(mcp_server, tool_name)(SEED, **kw) + assert isinstance(exc.value.__cause__, RateLimitError) + assert "retry after ~30s" in str(exc.value) # the wait folds into the text + + +@requires_mcp +@pytest.mark.parametrize( + "route, needle", + [ + (_throttled(), "retry after ~30s"), + (_walled(), "Blocked"), + (_dead(), "Couldn't open"), + ], +) +def test_mcp_client_receives_the_seed_message(http, route, needle): + # The real SDK path: call_tool raises an *anticipated* ToolError (not the + # UnexpectedToolError a crash would raise), whose str carries our message. + # The RPC handler turns exactly this into an is_error result with the text + # intact — whereas a crash is masked to a bare "Error executing tool ". + import asyncio + + http.routes[SEED] = route + with pytest.raises(ToolError) as exc: + asyncio.run(mcp_server.server.call_tool("fetch_markdown", {"url": SEED})) + assert not isinstance(exc.value, UnexpectedToolError) # not a masked crash + assert needle in str(exc.value) # our message survives to the client + + +# --------------------------------------------------------------------------- # +# HTTP API: /api returns 403 {blocked} and 502 {unreachable} for every method. +# --------------------------------------------------------------------------- # +@pytest.fixture +def public_dns(monkeypatch): + """Let the SSRF guard resolve the fake seed host, offline. + + ``assert_public_url`` calls ``socket.getaddrinfo`` and requires a globally + routable address; a made-up ``.test`` host would fail to resolve (a real + network call, which the suite forbids) and 400 before the fetch we're + testing ever runs. Pin resolution to a fixed public IP so the guard passes + and control reaches the walled/dead response. + """ + import socket + + from bytecrawl import security + + def fake_getaddrinfo(host, *a, **kw): + return [(socket.AF_INET, socket.SOCK_STREAM, socket.IPPROTO_TCP, "", ("93.184.216.34", 0))] + + monkeypatch.setattr(security.socket, "getaddrinfo", fake_getaddrinfo) + + +@pytest.fixture +def client(public_dns): + from landing.app import create_app + + app = create_app() + app.config.update(TESTING=True) + return app.test_client() + + +# Every /api method that reaches the seed. `select` gives extract a real +# selector so it takes the fetch path rather than the (fetch-first) discovery +# path — both fetch, but this keeps the call unambiguous. +API_METHODS = [ + {"method": "markdown"}, + {"method": "text"}, + {"method": "html"}, + {"method": "links"}, + {"method": "extract", "select": "h3::text"}, + {"method": "json"}, + {"method": "crawl", "query": "x", "strategy": "bfs"}, + {"method": "compare", "query": "x"}, +] + + +def _api_ids(params): + return params["method"] + + +@pytest.mark.parametrize("params", API_METHODS, ids=_api_ids) +def test_api_reports_a_walled_seed_as_403_blocked(client, http, params): + http.routes[SEED] = _walled() + resp = client.get("/api", query_string={"url": SEED, **params}) + assert resp.status_code == 403 + body = resp.get_json() + assert body["blocked"] is True + assert body["wall"] == "Cloudflare" + + +@pytest.mark.parametrize("params", API_METHODS, ids=_api_ids) +def test_api_reports_a_dead_seed_as_502_unreachable(client, http, params): + http.routes[SEED] = _dead() + resp = client.get("/api", query_string={"url": SEED, **params}) + assert resp.status_code == 502 + body = resp.get_json() + assert body["unreachable"] is True + + +@pytest.mark.parametrize("params", API_METHODS, ids=_api_ids) +def test_api_reports_a_throttled_seed_as_429_rate_limited(client, http, params): + http.routes[SEED] = _throttled() + resp = client.get("/api", query_string={"url": SEED, **params}) + assert resp.status_code == 429 + body = resp.get_json() + assert body["rate_limited"] is True + # The site's Retry-After is echoed in the body and the HTTP header so a + # client can back off by the amount the site actually asked for. + assert body["retry_after"] == 30 + assert resp.headers.get("Retry-After") == "30" + + +def test_api_walled_429_is_blocked_not_rate_limited(client, http): + """A 429 that carries a WAF signature is a wall, not a rate limit. + + Precedence in one place: the bot-wall answer (403 {blocked}) is more + specific than "slow down", so a walled 429 must not slip through as a 429 + {rate_limited}. + """ + http.routes[SEED] = _walled_429() + resp = client.get("/api", query_string={"url": SEED, "method": "markdown"}) + assert resp.status_code == 403 + body = resp.get_json() + assert body["blocked"] is True + assert body.get("rate_limited") is None + assert body["wall"] == "Cloudflare" + + +@requires_mcp +def test_api_walled_and_mcp_walled_agree(client, http): + """The contract in one place: the same walled seed, the tool and the + endpoint, the same named wall — one raised (as a ToolError wrapping the + typed BlockedError), one flagged in a 403 body.""" + http.routes[SEED] = _walled() + + with pytest.raises(ToolError) as exc: + mcp_server.fetch_markdown(SEED) + blocked = exc.value.__cause__ + assert isinstance(blocked, BlockedError) + + resp = client.get("/api", query_string={"url": SEED, "method": "markdown"}) + assert resp.status_code == 403 + assert resp.get_json()["wall"] == blocked.wall == "Cloudflare" + + +def test_auto_analyze_reports_a_dead_seed_as_unreachable(http): + """The 'auto' surface (/analyze) classifies a dead seed like the rest. + + Markdown in the playground runs /analyze, not /api?method=markdown. That + path used to swallow every fetch failure into a raw "Could not download the + page: " with no flag — so a mistyped domain showed red text there + while /api?method=links showed the friendly card on the same page. Now it + funnels the seed fetch through open_seed like every other surface, so it + raises UnreachableError and the response carries the `unreachable` flag and + the one shared wording the calm card renders from. + """ + http.routes[SEED] = _dead() + + from landing.app import create_app + + app = create_app() + app.config.update(TESTING=True) + resp = app.test_client().post("/analyze", json={"url": SEED, "lang": "en"}) + + body = resp.get_json() + assert body["ok"] is False + assert body["unreachable"] is True + assert body["error"] == bytecrawl.UNREACHABLE_MESSAGE + + +def test_auto_analyze_reports_a_walled_seed_as_blocked(http): + """And a bot wall on the 'auto' surface stays a named block, not unreachable.""" + http.routes[SEED] = _walled() + + from landing.app import create_app + + app = create_app() + app.config.update(TESTING=True) + resp = app.test_client().post("/analyze", json={"url": SEED, "lang": "en"}) + + body = resp.get_json() + assert body["ok"] is False + assert body["blocked"] is True + assert body["wall"] == "Cloudflare" + + +def test_auto_analyze_reports_a_throttled_seed_as_rate_limited(http): + """A 429 on the 'auto' surface (/analyze) reads as a rate limit, not a dead + seed — the same `rate_limited` flag, Retry-After and 429 status the rest give.""" + http.routes[SEED] = _throttled() + + from landing.app import create_app + + app = create_app() + app.config.update(TESTING=True) + resp = app.test_client().post("/analyze", json={"url": SEED, "lang": "en"}) + + assert resp.status_code == 429 + body = resp.get_json() + assert body["ok"] is False + assert body["rate_limited"] is True + assert body["retry_after"] == 30 + assert body["error"] == bytecrawl.RATE_LIMIT_MESSAGE + + +def test_api_unresolvable_host_is_502_unreachable_not_a_raw_400(monkeypatch): + """A host that doesn't resolve must get the friendly unreachable card too. + + The SSRF guard resolves DNS before the fetch. A mistyped domain fails that + resolution, and it used to escape as a raw ``400 {"error": "Cannot resolve + host: …"}`` — the one path that pre-checks, so the landing rendered red + error text while the playground (which never pre-checks) showed the card. + Now the guard reports an unresolvable host as UnreachableError, so /api + answers 502 {unreachable} with the one shared wording, matching every + other surface. This is the exact regression the landing showed. + """ + import socket + + from bytecrawl import UNREACHABLE_MESSAGE, security + + def boom(host, *a, **kw): + raise socket.gaierror("Name or service not known") + + monkeypatch.setattr(security.socket, "getaddrinfo", boom) + + from landing.app import create_app + + app = create_app() + app.config.update(TESTING=True) + resp = app.test_client().get("/api", query_string={"url": "https://en.wikipedi2a.org/wiki/x"}) + + assert resp.status_code == 502 + body = resp.get_json() + assert body["unreachable"] is True + # The friendly shared wording, not a raw "Cannot resolve host". + assert body["error"] == UNREACHABLE_MESSAGE + assert "Cannot resolve host" not in body["error"] diff --git a/web/landing/content.py b/web/landing/content.py index f4d8a05..2eb8717 100644 --- a/web/landing/content.py +++ b/web/landing/content.py @@ -35,7 +35,7 @@ "site": "quotes.toscrape.com", "description": "Hundreds of pages: walk a 'next' chain, or let BFS / Shark-Search / OPIC " "order a whole site under a request budget.", - "api": "?method=crawl&query=san+francisco", + "api": "?method=crawl&query=books", }, "login": { "title": "5 · API with login", @@ -54,6 +54,27 @@ } +# Worked examples behind the two live demos: the hero bar on "/" and the +# playground on "/playground". Both pages read from this one dict (injected as +# JSON into each template) so an example URL or query is edited in exactly one +# place and the two pages can never drift apart again. +# +# Keys are the playground's mode names. The hero bar only has Crawl and +# Markdown, so it uses "crawl" and "markdown"; the other three are +# playground-only. "query" is the focused-crawl topic (crawl) or the CSS +# selector (extract); "hint" is the i18n key for the hint under the playground's +# extra field. URLs carry their scheme; the hero strips it for display. +DEMO_EXAMPLES = { + "crawl": {"url": "https://en.wikipedia.org/wiki/Python_programming_language", + "query": "web framework", "hint": "pm_hint_crawl"}, + "markdown": {"url": "https://docs.python.org/3/tutorial/introduction.html"}, + "links": {"url": "https://news.ycombinator.com/"}, + "json": {"url": "https://data.sfgov.org/resource/wg3w-h783.json?$limit=5"}, + "extract": {"url": "https://books.toscrape.com/", + "query": "h3 a::attr(title)", "hint": "pm_hint_extract"}, +} + + ANALYZE_I18N = { "en": { "bad_url": "Enter a URL that starts with http:// or https://", diff --git a/web/landing/routes.py b/web/landing/routes.py index d7497c3..3b6ca80 100644 --- a/web/landing/routes.py +++ b/web/landing/routes.py @@ -21,7 +21,7 @@ import bytecrawl from landing import analytics, services -from landing.content import TECHNIQUES +from landing.content import DEMO_EXAMPLES, TECHNIQUES pages_bp = Blueprint("pages", __name__) api_bp = Blueprint("api", __name__) @@ -32,7 +32,7 @@ # --------------------------------------------------------------------------- # @pages_bp.route("/") def landing(): - return render_template("landing.html") + return render_template("landing.html", demos=DEMO_EXAMPLES) @pages_bp.route("/agent-onboarding/skill.json") @@ -86,7 +86,7 @@ def methods(): @pages_bp.route("/playground") def playground(): - return render_template("try.html", techniques=TECHNIQUES) + return render_template("try.html", techniques=TECHNIQUES, demos=DEMO_EXAMPLES) @pages_bp.route("/try") @@ -158,10 +158,6 @@ def api(): url = services.clean_url(src.get("url") or "") if not url: return jsonify({"error": "the 'url' parameter is required"}), 400 - try: - services.assert_public_url(url) - except ValueError as e: - return jsonify({"error": str(e)}), 400 method = (src.get("method") or "markdown").lower() cache_key = "|".join([url, method, src.get("query", ""), src.get("select", ""), @@ -171,9 +167,31 @@ def api(): return jsonify(cached) try: + # The SSRF guard runs in the same funnel as the fetch so its failures + # get the same shaped answers: a non-public URL is a 400 ValueError, + # but a host that simply doesn't resolve is UnreachableError — the same + # 502 "couldn't open that page" a mistyped domain gets everywhere else, + # instead of a raw "Cannot resolve host" only on this pre-checked path. + services.assert_public_url(url) payload = services.run_api_method(method, url, src) except ValueError as e: return jsonify({"error": str(e)}), 400 + except bytecrawl.BlockedError as e: + return jsonify({"error": str(e), "blocked": True, "wall": e.wall}), 403 + except bytecrawl.RateLimitError as e: + # The seed is up but throttling us (429). An honest 429 back, with the + # site's Retry-After echoed both in the body and the header so a client + # can back off, instead of dressing a live-but-busy site as unreachable. + resp = jsonify({"error": str(e), "rate_limited": True, + "retry_after": e.retry_after}) + if e.retry_after is not None: + resp.headers["Retry-After"] = str(e.retry_after) + return resp, 429 + except bytecrawl.UnreachableError as e: + # The seed URL couldn't be opened at all (bad domain, refused, 404, + # two links pasted into one) — an honest 502 with a clear flag, the + # same answer every method and the MCP tools give. + return jsonify({"error": str(e), "unreachable": True}), 502 except Exception as e: return jsonify({"error": f"request failed: {e}"}), 502 diff --git a/web/landing/services.py b/web/landing/services.py index 4cdaac3..8ecadb4 100644 --- a/web/landing/services.py +++ b/web/landing/services.py @@ -14,7 +14,7 @@ import re import time -from bytecrawl import Scraper +from bytecrawl import BlockedError, RateLimitError, Scraper, UnreachableError, open_seed from bytecrawl.crawler import STRATEGIES as CRAWL_STRATEGIES from bytecrawl.crawler import compare, pagerank from bytecrawl.security import assert_public_url # re-exported for the routes @@ -80,7 +80,25 @@ def analyze_url(url: str, lang: str) -> tuple[dict, int, dict | None]: bot = Scraper(timeout=20) steps = [] try: - page = bot.static(url) + # open_seed is the shared classifier: a bot wall stays a BlockedError, + # and anything else that stopped the page loading (a domain that doesn't + # resolve, a refused connection, a 404) becomes UnreachableError. Without + # it, 'auto' was the one surface that dressed a dead seed as a raw + # requests traceback while /api?method=links gave the friendly card. + page = open_seed(lambda: bot.static(url)) + except BlockedError as e: + return {"ok": False, "error": str(e), "blocked": True, "wall": e.wall}, 403, None + except RateLimitError as e: + # Reachable but throttled (429): the same `rate_limited` flag and + # Retry-After every surface gives, so the frontend renders the calm + # "slow down and retry" card off the flag, not the raw error string. + return {"ok": False, "error": str(e), "rate_limited": True, + "retry_after": e.retry_after}, 429, None + except UnreachableError as e: + # The seed never opened — the same "couldn't open that page" answer, and + # the same `unreachable` flag, every other surface gives, so the frontend + # renders the calm card off the flag instead of the raw error string. + return {"ok": False, "error": str(e), "unreachable": True}, 200, None except Exception as e: return {"ok": False, "error": tr["download_err"].format(e=e)}, 200, None @@ -165,6 +183,22 @@ def run_crawl(url: str, strategy: str, query: str) -> tuple[dict, int, dict | No except Exception as e: return {"ok": False, "error": f"crawl failed: {e}"}, 200, None + if result.blocked is not None and not result.pages: + return {"ok": False, "error": str(result.blocked), + "blocked": True, "wall": result.blocked.wall}, 403, None + + if result.rate_limited is not None and not result.pages: + # Seed throttled: say "slow down and retry", carrying the wait, instead + # of a hollow "0 relevant". + return {"ok": False, "error": str(result.rate_limited), + "rate_limited": True, + "retry_after": result.rate_limited.retry_after}, 429, None + + if result.unreachable: + # Seed unreachable: say so plainly instead of a hollow "0 relevant". + return {"ok": False, "unreachable": True, + "error": "Couldn't open that page."}, 200, None + pr = pagerank(result.graph) props = {"strategy": strategy, **result.stats} payload = { @@ -181,31 +215,33 @@ def run_api_method(method: str, url: str, src) -> dict: """Dispatch one /api call to ByteCrawl and return its payload dict. ``src`` is the request's arg/form mapping (anything with ``.get``). Bad - input raises ValueError (the route maps it to 400); everything else - propagates for the route to turn into a 502. + input raises ValueError (the route maps it to 400); a walled seed raises + BlockedError (403) and an unopenable one UnreachableError (502), the same + across every method; anything else propagates for the route to turn into a + 502. """ bot = Scraper(timeout=15) if method in ("markdown", "md"): - page = bot.static(url) + page = open_seed(lambda: bot.static(url)) return {"url": url, "method": "markdown", "markdown": page.markdown(), "tokens": page.tokens(page.markdown())} if method == "text": - page = bot.static(url) + page = open_seed(lambda: bot.static(url)) return {"url": url, "method": "text", "text": page.soup.get_text(" ", strip=True)} if method == "html": - page = bot.static(url) + page = open_seed(lambda: bot.static(url)) return {"url": url, "method": "html", "html": page.html} if method == "links": - page = bot.static(url) + page = open_seed(lambda: bot.static(url)) return {"url": url, "method": "links", "links": page.links()} if method == "json": - page = bot.api(url) + page = open_seed(lambda: bot.api(url)) return {"url": url, "method": "json", "data": page.json()} if method == "extract": select = (src.get("select") or "").strip() - page = bot.static(url) + page = open_seed(lambda: bot.static(url)) if not select: # No selector is not an error, it is the question before it: # "what can I pull off this page?" Answering with the repeated @@ -225,6 +261,10 @@ def run_api_method(method: str, url: str, src) -> dict: pages = API_MAX_PAGES crawler = CRAWL_STRATEGIES[strategy](query=query, delay=0.0, timeout=8) result = crawler.crawl(url, max_pages=pages, max_depth=4) + # A walled seed raises the block (403); an unreachable one raises + # UnreachableError (502). Either way the route reports it plainly + # rather than letting an empty crawl read as "nothing relevant". + result.raise_for_seed() ranked = result.top(pages) if query else result.pages return {"url": url, "method": "crawl", "strategy": strategy, "query": query, "stats": result.stats, "pages": ranked} diff --git a/web/landing/static/i18n.js b/web/landing/static/i18n.js index 2b226b3..7da75d9 100644 --- a/web/landing/static/i18n.js +++ b/web/landing/static/i18n.js @@ -91,6 +91,10 @@ azr_title: "Analyze any URL", azr_sub: "Paste a page: ByteCrawl picks the strategy (static or browser) and explains what it did and why.", azr_hint: "Change the URL and try yours.", + unreach_t: "Couldn’t open that page", + unreach_d: "The site didn’t respond — check the URL is real or complete.", + ratelimit_t: "The site asked us to slow down", + ratelimit_d: "It’s up and reachable — just rate-limiting requests (HTTP 429). Wait a moment and try again.", analyze_btn: "Run", pm_markdown: "Markdown", pm_extract: "Extract", pm_json: "JSON API", pm_crawl: "Graph crawl", pm_links: "Links", @@ -103,6 +107,7 @@ badge_tokens: "Markdown saves {pct}% of LLM tokens", st_enter_url: "✗ Enter a URL", st_analyzing: "Analyzing the page...", st_strategy: "Strategy used:", st_three: "BFS, Shark-Search and OPIC on the same budget:", + pm_stop_confirm: "This run is still going. Stop it and switch?", pm_cmp_running: "running…", pm_cmp_head: "Same budget, three orderings — query:", pm_cmp_best: "Full ranking from the winner:", @@ -260,6 +265,10 @@ azr_title: "Analiza cualquier URL", azr_sub: "Pega una página: ByteCrawl elige la estrategia (estático o navegador) y te explica qué hizo y por qué.", azr_hint: "Cambia la URL y prueba la tuya.", + unreach_t: "No pude abrir esa página", + unreach_d: "El sitio no respondió — revisa que la URL exista o esté completa.", + ratelimit_t: "El sitio nos pidió bajar el ritmo", + ratelimit_d: "Está en línea y responde — solo está limitando las solicitudes (HTTP 429). Espera un momento y vuelve a intentar.", analyze_btn: "Ejecutar", pm_markdown: "Markdown", pm_extract: "Extract", pm_json: "API JSON", pm_crawl: "Crawl de grafo", pm_links: "Links", @@ -272,6 +281,7 @@ badge_tokens: "El Markdown ahorra {pct}% de tokens de LLM", st_enter_url: "✗ Escribe una URL", st_analyzing: "Analizando la página...", st_strategy: "Estrategia usada:", st_three: "BFS, Shark-Search y OPIC con el mismo presupuesto:", + pm_stop_confirm: "Este análisis sigue corriendo. ¿Detenerlo y cambiar?", pm_cmp_running: "corriendo…", pm_cmp_head: "Mismo presupuesto, tres órdenes — consulta:", pm_cmp_best: "Ranking completo del ganador:", @@ -429,6 +439,10 @@ azr_title: "Analise qualquer URL", azr_sub: "Cole uma página: o ByteCrawl escolhe a estratégia (estático ou navegador) e explica o que fez e por quê.", azr_hint: "Troque a URL e teste a sua.", + unreach_t: "Não consegui abrir essa página", + unreach_d: "O site não respondeu — confira se a URL existe ou está completa.", + ratelimit_t: "O site pediu para diminuirmos o ritmo", + ratelimit_d: "Está no ar e respondendo — só está limitando as requisições (HTTP 429). Aguarde um momento e tente de novo.", analyze_btn: "Executar", pm_markdown: "Markdown", pm_extract: "Extract", pm_json: "API JSON", pm_crawl: "Crawl de grafo", pm_links: "Links", @@ -441,6 +455,7 @@ badge_tokens: "O Markdown economiza {pct}% dos tokens de LLM", st_enter_url: "✗ Digite uma URL", st_analyzing: "Analisando a página...", st_strategy: "Estratégia usada:", st_three: "BFS, Shark-Search e OPIC com o mesmo orçamento:", + pm_stop_confirm: "Esta análise ainda está rodando. Parar e trocar?", pm_cmp_running: "rodando…", pm_cmp_head: "Mesmo orçamento, três ordens — consulta:", pm_cmp_best: "Ranking completo do vencedor:", diff --git a/web/landing/templates/landing.html b/web/landing/templates/landing.html index 8572c66..5652509 100644 --- a/web/landing/templates/landing.html +++ b/web/landing/templates/landing.html @@ -207,8 +207,13 @@ /* live results under the bar */ .hout { max-width: 620px; margin: 0 auto 22px; text-align: left; display: none; } - .hout.show { display: block; animation: ns-in .28s ease both; } - .hout-box { border: 1px solid var(--border); border-radius: 14px; background: var(--panel); overflow: hidden; } + /* The fade lives on the box, not the container: innerHTML swaps replace the + box with a fresh node, so the animation re-runs on every render. Put it on + .hout.show instead and it fires once (empty -> loading) but not on the + loading -> result swap, so the result hard-pops in — the flicker. */ + .hout.show { display: block; } + .hout-box { border: 1px solid var(--border); border-radius: 14px; background: var(--panel); overflow: hidden; + animation: ns-in .22s ease both; } .hout-head { display: flex; align-items: center; gap: 8px; padding: 10px 16px; border-bottom: 1px solid var(--border); font-size: 12px; color: var(--muted); background: var(--panel-2); } .hout-head .ok { color: var(--text); font-weight: 600; } @@ -233,7 +238,14 @@ .hout pre { padding: 14px 16px; margin: 0; white-space: pre-wrap; word-break: break-word; font-family: var(--mono); font-size: 12px; line-height: 1.6; color: var(--code); } .hout .err { padding: 14px 16px; color: #b91c1c; font-size: 13px; } - + /* bot-wall block: a calm notice, not an all-red shout */ + .hout .block { display: flex; gap: 13px; padding: 16px; align-items: flex-start; } + .hout .block .ic { flex: 0 0 auto; width: 30px; height: 30px; border-radius: 9px; + display: flex; align-items: center; justify-content: center; + background: #fef2f2; color: #dc2626; } + .hout .block .ic svg { width: 17px; height: 17px; } + .hout .block .title { font-weight: 600; font-size: 13.5px; color: var(--text); margin-bottom: 3px; } + .hout .block .detail { font-size: 12.5px; line-height: 1.55; color: var(--muted-2); } .install { display: inline-flex; align-items: center; gap: 12px; font-family: var(--mono); font-size: 14px; background: var(--panel); border: 1px solid var(--border); border-radius: 9px; padding: 11px 18px; @@ -447,16 +459,19 @@

Crawling for AI
- +
- + (query to search)
+ + +
@@ -611,6 +626,8 @@

Build with ByteCrawl today