feat: name rate limits and unreachable seeds, surface them on every surface

CHANGELOG.md

@@ -1,4 +1,42 @@
 # Changelog
+
+## 1.4.0 — 2026-09-11
+
+### Added
+- **A bot wall now says so.** When a site's protection refuses the fetch, the
+  error names the wall instead of a bare `403 Client Error: Forbidden` — e.g.
+  *"Blocked by Cloudflare — tapology.com uses bot protection and refused this
+  request (HTTP 403). Automated access isn't allowed."* Cloudflare, DataDome,
+  PerimeterX/HUMAN, Akamai and Imperva
+  Incapsula are recognised by their response headers and challenge-page markers.
+  A new `bytecrawl.BlockedError` (a subclass of `requests.HTTPError`) carries it,
+  so `except requests.HTTPError` still catches it and you can catch the block
+  specifically. Every surface benefits at once — the library, the crawlers, the
+  hosted API and the playground — since the detection lives at the fetch layer.
+  Detection is gated on a block-looking status, so a normal page served from
+  behind Cloudflare is never mislabelled.
+- **A rate limit now says so, too.** A genuine `HTTP 429 Too Many Requests`
+  (no bot-wall signature) no longer reads as "couldn't reach the site" — it gets
+  its own calm message: *"The site is rate-limiting us… it's up and reachable,
+  just asking for fewer requests. Wait a moment and try again."* A new
+  `bytecrawl.RateLimitError` (also a `requests.HTTPError` subclass) carries the
+  server's `Retry-After` when present — both forms, a delta in seconds and an
+  HTTP date. The hosted API answers `429` with a `Retry-After` header and a
+  `retry_after` field, and the playground shows a wait hint. A 429 that *does*
+  carry a bot-wall signature still reports as blocked — the more specific answer
+  wins.
+- **One error card everywhere.** Unreachable seeds (bad DNS, refused
+  connection, timeout) now raise a single `bytecrawl.UnreachableError` across the
+  library, crawlers, hosted API and playground, so every playground mode —
+  Markdown, links, JSON API, extract, auto and compare — renders the same card
+  for the same failure instead of three different-looking errors.
+- **MCP tools now tell the agent what went wrong.** A blocked, rate-limited or
+  unreachable seed used to reach an MCP client as a masked *"Error executing
+  tool …"* — the SDK treats a stray `requests` exception as a crash and withholds
+  its text. Every tool (local and hosted) now re-raises these as a `ToolError`,
+  so the agent receives the real message — the wall's name, or the wait hint for
+  a 429 — the MCP counterpart of the API's typed 403/429 body.
+
 ## 1.3.0 — 2026-09-11
 
 ### Changed

README.es.md

@@ -10,8 +10,8 @@ Shark-Search y OPIC en Python puro.
 - **Web**: https://bytecrawl.vercel.app/
 - **Endpoint MCP hosteado**: https://bytecrawl.vercel.app/mcp
 - **Skill para agentes**: https://bytecrawl.vercel.app/agent-onboarding/SKILL.md
-- **Última versión**: **1.3.0** — publicación automática a PyPI y checks de CI
-  separados (lint, formato, tipos, seguridad) ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md))
+- **Última versión**: **1.4.0** — errores más claros cuando la protección
+  anti-bot de un sitio (Cloudflare, DataDome, …) bloquea el fetch ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md))
 - **PyPI**: https://pypi.org/project/bytecrawl/
 - **GitHub**: https://github.com/abrahamperz/ByteCrawl
 

README.md

@@ -16,8 +16,8 @@ and OPIC in pure Python.
 - **Webpage**: https://bytecrawl.vercel.app/
 - **Hosted MCP endpoint**: https://bytecrawl.vercel.app/mcp
 - **Agent skill**: https://bytecrawl.vercel.app/agent-onboarding/SKILL.md
-- **Latest release**: **1.3.0** — automated PyPI publishing and separated CI
-  gates (lint, format, types, security) ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md))
+- **Latest release**: **1.4.0** — clearer errors when a site's bot protection
+  (Cloudflare, DataDome, …) blocks a fetch ([changelog](https://github.com/abrahamperz/ByteCrawl/blob/main/CHANGELOG.md))
 
 ## Why focused crawling?
 

pyproject.toml

@@ -29,7 +29,10 @@ browser = ["playwright>=1.40"]
 llm = ["markdownify>=0.11", "trafilatura>=1.6"]
 mcp = ["mcp>=2; python_version >= '3.10'", "markdownify>=0.11", "trafilatura>=1.6"]
 all = ["playwright>=1.40", "markdownify>=0.11", "trafilatura>=1.6", "mcp>=2; python_version >= '3.10'"]
-dev = ["pytest>=7", "pytest-cov", "ruff", "mypy", "bandit"]
+# flask/python-dotenv/posthog are here (not runtime deps of the library) so the
+# test suite can import the web landing app and exercise the /api seed-error
+# surface. Kept in sync with web/landing/requirements.txt.
+dev = ["pytest>=7", "pytest-cov", "ruff", "mypy", "bandit", "flask", "python-dotenv", "posthog"]
 
 [project.scripts]
 bytecrawl-mcp = "bytecrawl.mcp_server:main"

src/bytecrawl/__init__.py

@@ -22,7 +22,17 @@
 LLM:                  page.markdown() · page.tokens()
 """
 
-from .core import Page, Scraper, Session
+from .core import (
+    RATE_LIMIT_MESSAGE,
+    UNREACHABLE_MESSAGE,
+    BlockedError,
+    Page,
+    RateLimitError,
+    Scraper,
+    Session,
+    UnreachableError,
+    open_seed,
+)
 from .crawler import (
     BFS,
     OPIC,
@@ -39,6 +49,12 @@
     "Scraper",
     "Page",
     "Session",
+    "BlockedError",
+    "RateLimitError",
+    "UnreachableError",
+    "RATE_LIMIT_MESSAGE",
+    "UNREACHABLE_MESSAGE",
+    "open_seed",
     "Crawler",
     "BFS",
     "SharkSearch",
@@ -49,4 +65,4 @@
     "cosine",
     "compare",
 ]
-__version__ = "1.3.0"
+__version__ = "1.4.0"

src/bytecrawl/core.py

@@ -14,12 +14,29 @@
 from .scraping.scraper import Scraper
 from .scraping.selectors import _split_selector, _value_from
 from .scraping.session import Session
-from .scraping.url import _SKIP_EXT, DEFAULT_UA, _decoded_html, normalize
+from .scraping.url import (
+    _SKIP_EXT,
+    DEFAULT_UA,
+    RATE_LIMIT_MESSAGE,
+    UNREACHABLE_MESSAGE,
+    BlockedError,
+    RateLimitError,
+    UnreachableError,
+    _decoded_html,
+    normalize,
+    open_seed,
+)
 
 __all__ = [
     "Page",
     "Session",
     "Scraper",
+    "BlockedError",
+    "RateLimitError",
+    "UnreachableError",
+    "RATE_LIMIT_MESSAGE",
+    "UNREACHABLE_MESSAGE",
+    "open_seed",
     "normalize",
     "DEFAULT_UA",
     "_SKIP_EXT",

src/bytecrawl/crawl/base.py

@@ -15,7 +15,7 @@
 # crawlers agree on what counts as a link; the crawler module re-exports it
 # because that is where the tests and docs import it from.
 from ..scraping.scraper import Scraper
-from ..scraping.url import normalize
+from ..scraping.url import BlockedError, RateLimitError, normalize
 from .frontier import Frontier, _root_domain
 from .result import CrawlResult
 from .text import relevance
@@ -81,6 +81,21 @@ def crawl(self, start: str, max_pages: int = 50, max_depth: int = 10) -> CrawlRe
 
             try:
                 page = self.scraper.static(url)
+            except BlockedError as e:
+                # A single walled page shouldn't abort the crawl, but remember
+                # the first wall: if the crawl comes back empty, the caller can
+                # say "blocked" instead of the misleading "nothing relevant".
+                errors += 1
+                if result.blocked is None:
+                    result.blocked = e
+                continue
+            except RateLimitError as e:
+                # Likewise for a 429: keep crawling other pages, but remember the
+                # first throttle so an empty crawl reports "slow down and retry".
+                errors += 1
+                if result.rate_limited is None:
+                    result.rate_limited = e
+                continue
             except Exception:
                 errors += 1
                 continue

src/bytecrawl/crawl/compare.py

@@ -4,18 +4,25 @@
 
 from concurrent.futures import ThreadPoolExecutor
 
+from ..scraping.url import BlockedError, RateLimitError, UnreachableError
 from .strategies import STRATEGIES
 
 
 def _leg(
     name: str, start: str, query: str, max_pages: int, max_depth: int, delay: float, timeout: int
 ) -> dict:
-    """One strategy's leg of a comparison. Never raises: a strategy that fails
-    reports inside its own entry so the other two still come back."""
+    """One strategy's leg of a comparison. A strategy that fails on its own
+    reports inside its own entry so the other two still come back — but a
+    seed-level failure (a walled or unreachable start URL) is not that: it
+    fails every leg identically, so re-raise it and let :func:`compare` surface
+    it once instead of burying three copies in three error strings."""
     try:
         result = STRATEGIES[name](query=query, delay=delay, timeout=timeout).crawl(
             start, max_pages=max_pages, max_depth=max_depth
         )
+        result.raise_for_seed()
+    except (BlockedError, RateLimitError, UnreachableError):
+        raise
     except Exception as e:
         return {"error": f"{type(e).__name__}: {e}"}
     return {

src/bytecrawl/crawl/result.py

@@ -4,6 +4,8 @@
 
 from dataclasses import dataclass, field
 
+from ..scraping.url import BlockedError, RateLimitError, UnreachableError
+
 
 @dataclass
 class CrawlResult:
@@ -13,9 +15,53 @@ class CrawlResult:
     pages: list[dict] = field(default_factory=list)  # url, title, score, relevance, depth, order
     graph: dict[str, list[str]] = field(default_factory=dict)
     stats: dict = field(default_factory=dict)
+    # The bot wall that turned the crawl away, if any. A crawl swallows a
+    # per-page block and keeps going, but when the seed is walled it comes
+    # back empty — callers read this to tell "walled" apart from "nothing here".
+    blocked: BlockedError | None = None
+    # Same idea for a throttled seed: the crawl swallows a per-page 429 and
+    # keeps going, but a rate-limited *seed* comes back empty — remembered here
+    # so the caller can say "slow down and retry" instead of "nothing here".
+    rate_limited: RateLimitError | None = None
 
     def relevant(self, threshold: float = 0.1) -> list[dict]:
         return [p for p in self.pages if p["relevance"] >= threshold]
 
     def top(self, n: int = 10) -> list[dict]:
         return sorted(self.pages, key=lambda p: p["relevance"], reverse=True)[:n]
+
+    @property
+    def unreachable(self) -> bool:
+        """Every fetch failed: the crawl never read a single page.
+
+        A crawl swallows per-page errors and keeps going, so an empty result
+        is ambiguous — the seed was unreachable (a domain that doesn't
+        resolve, a refused connection, a 404, or two URLs pasted into one), or
+        it read fine and simply matched nothing. When every request failed
+        (``requests - errors <= 0``) it is the former. A walled seed also
+        lands here, but is reported through :attr:`blocked` first.
+        """
+        reqs = self.stats.get("requests", 0)
+        return reqs >= 1 and reqs - self.stats.get("errors", 0) <= 0
+
+    def raise_for_seed(self) -> None:
+        """Raise if the crawl never got off the ground; return quietly otherwise.
+
+        Turns a dead-on-arrival crawl into the same exception a single fetch
+        would have raised, so every caller (the HTTP API, the MCP tools) can
+        report a walled or unreachable seed uniformly instead of dressing an
+        empty crawl up as "found nothing relevant". A crawl that read at least
+        one page returns quietly — even one that matched nothing, which is a
+        real answer, not a failure.
+
+        Precedence, most specific first: a bot wall (blocked), then a rate limit
+        (throttled but reachable), then a dead seed (unreachable). A walled seed
+        can set both blocked and unreachable; "a bot wall refused you" is the
+        more actionable answer, and a 429 sits between the two.
+        """
+        if self.blocked is not None and not self.pages:
+            raise self.blocked
+        if self.rate_limited is not None and not self.pages:
+            raise self.rate_limited
+        if self.unreachable:
+            raise UnreachableError()  # default message: the one shared wording

src/bytecrawl/mcp_server.py

@@ -15,18 +15,62 @@
 
 try:
     from mcp.server.mcpserver import MCPServer
+    from mcp.server.mcpserver.exceptions import ToolError
 except ImportError as e:  # pragma: no cover - exercised only without the extra
     raise ImportError("The MCP server needs the 'mcp' extra: pip install bytecrawl[mcp]") from e
 
-from .core import Scraper
+from contextlib import contextmanager
+
+from .core import (
+    BlockedError,
+    RateLimitError,
+    Scraper,
+    UnreachableError,
+    open_seed,
+)
 from .crawler import STRATEGIES as _STRATEGIES
 from .crawler import compare, pagerank
 
 DELAY = 0.2  # seconds between requests, per crawler
 
+
+@contextmanager
+def _seed_errors_as_tool_errors():
+    """Report a seed failure as an anticipated :class:`ToolError`.
+
+    Our seed errors are ``requests`` exceptions. To the MCP SDK an escaping
+    ``requests`` exception is a *crash*: it withholds the message and hands the
+    agent a bare ``Error executing tool <name>``. Re-raised as ``ToolError`` they
+    come back as an ``is_error`` result carrying our text — the MCP counterpart
+    of the HTTP API's typed body. The specifics fold into the message (the wall's
+    name, the wait hint) because an MCP client reads the sentence, not a JSON
+    field, so this is the same information the API returns as ``retry_after``.
+    """
+    try:
+        yield
+    except RateLimitError as e:
+        wait = f" (retry after ~{e.retry_after}s)" if e.retry_after is not None else ""
+        raise ToolError(f"{e}{wait}") from e
+    except (BlockedError, UnreachableError) as e:
+        raise ToolError(str(e)) from e
+
+
 # One polite scraper shared by every tool call.
 _scraper = Scraper(delay=DELAY)
 
+
+def _open(url: str, *, api: bool = False, **kw):
+    """Fetch the seed through the shared :func:`bytecrawl.open_seed` guard.
+
+    Keeps the technique choice here (``api`` -> ``Scraper.api``, otherwise the
+    auto ``Scraper.fetch``) while the block/unreachable translation lives in the
+    core, so a walled or unopenable URL reports the same way here, in the HTTP
+    API, and in the crawl tools' ``CrawlResult.raise_for_seed()``.
+    """
+    with _seed_errors_as_tool_errors():
+        return open_seed(lambda: _scraper.api(url, **kw) if api else _scraper.fetch(url))
+
+
 server = MCPServer(
     "bytecrawl",
     instructions=(
@@ -47,7 +91,7 @@
     )
 )
 def fetch_markdown(url: str) -> dict:
-    page = _scraper.fetch(url)
+    page = _open(url)
     md = page.markdown()
     # tokens_html alongside it, the same pair the playground shows. On its own
     # tokens_estimate is a number with nothing to compare against; next to the
@@ -95,7 +139,7 @@ def extract(
         )
     # Checked before the fetch: a malformed call shouldn't cost the site a
     # request.
-    page = _scraper.fetch(url)
+    page = _open(url)
     if select:
         values = page.css_all(select)
         return {"url": url, "select": select, "count": len(values), "values": values}
@@ -124,7 +168,7 @@ def extract(
     )
 )
 def list_links(url: str, raw: bool = False) -> dict:
-    page = _scraper.fetch(url)
+    page = _open(url)
     links = page.links(raw=raw)
     return {"url": url, "count": len(links), "raw": raw, "links": links}
 
@@ -143,6 +187,12 @@ def focused_crawl(url: str, query: str = "", strategy: str = "shark", max_pages:
     max_pages = min(max_pages, 50)  # keep agent calls bounded and polite
     crawler = _STRATEGIES[strategy](query=query, delay=DELAY)
     result = crawler.crawl(url, max_pages=max_pages)
+    # A walled, throttled or unreachable seed comes back as an empty crawl; raise
+    # it as the same blocked/rate-limited/unreachable error a single fetch would
+    # (as a ToolError, so the agent sees it), rather than returning an empty page
+    # list dressed up as "nothing relevant".
+    with _seed_errors_as_tool_errors():
+        result.raise_for_seed()
     ranks = pagerank(result.graph)
     return {
         "strategy": strategy,
@@ -164,13 +214,14 @@ def focused_crawl(url: str, query: str = "", strategy: str = "shark", max_pages:
 )
 def compare_strategies(url: str, query: str, max_pages: int = 20) -> dict:
     max_pages = min(max_pages, 50)  # same bound as focused_crawl, per strategy
-    out = compare(url, query, max_pages=max_pages, delay=DELAY)
+    with _seed_errors_as_tool_errors():
+        out = compare(url, query, max_pages=max_pages, delay=DELAY)
     return {"url": url, "query": query, "max_pages": max_pages, **out}
 
 
 @server.tool(description="Fetch a JSON API endpoint (the 'hidden API' scraping technique).")
 def fetch_json_api(url: str, params: dict | None = None) -> dict:
-    page = _scraper.api(url, params=params)
+    page = _open(url, api=True, params=params)
     return {"url": url, "status": page.status, "data": page.json()}
 
 

src/bytecrawl/scraping/__init__.py

@@ -10,6 +10,6 @@
 from .page import Page
 from .scraper import Scraper
 from .session import Session
-from .url import normalize
+from .url import BlockedError, RateLimitError, normalize
 
-__all__ = ["Page", "Session", "Scraper", "normalize"]
+__all__ = ["Page", "Session", "Scraper", "normalize", "BlockedError", "RateLimitError"]