Repository navigation
feat: name rate limits and unreachable seeds, surface them on every surface - #16
Merged
Merged
Conversation
Split the single CI "test" job into separate top-level checks — lint (ruff check), format (ruff format --check), type check (mypy) and security (bandit) — alongside the unit-test matrix, so a red check points straight at the cause instead of hiding inside one job. Releases now publish to PyPI automatically via GitHub Actions Trusted Publishing (OIDC): a push to main with a new __version__ runs the tests, tags the release, and uploads to PyPI — nothing publishes without a version bump. Also apply ruff format across the codebase, single-source the version from __version__ (guard tests pin the CHANGELOG top and both READMEs), mypy fixes across scraping/crawl, and one audited # nosec on the MCP HTTP bind. No behaviour change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The type-check job installs .[llm,dev,mcp] but not the optional [browser] extra, so mypy could not resolve playwright.sync_api (imported lazily in Scraper.browser() behind a try/except) and failed with import-not-found. Add a [tool.mypy] override so playwright.* is exempt from missing-import errors, keeping the heavy browser stack out of CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…urface A genuine HTTP 429 now raises bytecrawl.RateLimitError (a requests.HTTPError subclass) instead of folding into "unreachable" — carrying Retry-After in both RFC 7231 forms (delta-seconds and HTTP-date). Unreachable seeds raise a single UnreachableError. A 429 with a bot-wall signature still reports as BlockedError. Every surface reports these the same way: the hosted API answers 429 with a Retry-After header and retry_after field (403 for blocked); the playground renders one calm card per failure across all modes; and the MCP tools (local and hosted) re-raise as ToolError so the agent sees the real message instead of a masked "Error executing tool", the MCP counterpart of the API's typed body. All rolled into the 1.4.0 changelog entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
abrahamperz
added
bug
# Conflicts: # CHANGELOG.md # README.es.md # README.md # src/bytecrawl/__init__.py # src/bytecrawl/crawl/compare.py # src/bytecrawl/scraping/session.py
The guard used pytest.importorskip("mcp") but the block below imports the
mcp.server.mcpserver.exceptions submodule. On 3.9 CI (no mcp extra) a shallow
mcp package can import while the submodule is absent, so the guard passed and
collection then failed with ModuleNotFoundError. Point importorskip at the
exact submodule these tests use, matching the sibling MCP test files, so the
whole module skips instead of erroring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…CI its deps Two problems surfaced by the 3.9/3.12 split: - The MCP block used a module-level importorskip, so a run without the mcp extra (3.9 CI) aborted the whole module and silently skipped the core and /api tests too. Import the SDK softly and gate only the MCP tests with a skipif marker, resolving each tool by name so the parametrize list imports without the extra. Core and /api tests now run on 3.9 and 3.12; the MCP tests skip cleanly where the extra is absent. - The /api tests import the Flask landing app, which needs flask, python-dotenv and posthog. Those live only in web/landing/requirements.txt, so 3.12 failed with ModuleNotFoundError: No module named 'dotenv'. Add them to the dev extra (test-only, not runtime deps of the library) so CI exercises the /api seed-error surface, kept in sync with the web requirements. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce a DEMO_EXAMPLES dict in content.py as the one source of truth for the worked examples, injected as JSON into both the "/" hero bar and the "/playground" so the two pages can never drift apart. Bring the hero to 5-mode parity (Crawl, Markdown, Links, JSON, Extract). Set the crawl demo to Wikipedia's Python page focused on "web framework": Shark-Search finds 5/6 relevant (top 0.679) while BFS and OPIC find 0 — the cleanest contrast of the candidates, on a clean seed URL with no parens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A genuine HTTP 429 now raises
bytecrawl.RateLimitError(arequests.HTTPErrorsubclass) instead of folding into "unreachable" — carryingRetry-Afterin both RFC 7231 forms (delta-seconds and HTTP-date). Unreachable seeds raise a singleUnreachableError. A 429 that also carries a bot-wall signature still reports asBlockedError.Bumps the version to 1.4.0 (minor per merge; base is 1.3.0).
One core, every surface
The classification lives in the core (
open_seed/_raise_for_status/CrawlResult.raise_for_seed), and every surface reports it the same way:429with aRetry-Afterheader and aretry_afterfield (and403for blocked), so the consumer (code) reads a typed body.ToolErrorso the agent sees the real message (the wall's name, the wait hint) instead of the SDK's maskedError executing tool <name>. The MCP counterpart of the API's typed body: the specifics fold into the message text because an MCP client reads the sentence, not a JSON field.Retry-After parsing
_retry_after_secondsreads both forms: numeric seconds directly, and an HTTP date as the delta from now (clamped at 0, so a past date reads as "retry now" rather than a negative wait). Malformed values returnNone.Tests
New coverage for both
Retry-Afterforms, the three seed states across all six MCP tools (assertingToolErrorwith the typed seed error preserved as__cause__), and cross-surface agreement (API-walled == MCP-walled). Suite: 251 passed, 1 skipped; ruff and the packaging/version coherence test green.Note for the reviewer
The Commits tab lists two extra commits (
release 1.3.0,fix ci) that are already inmainby content under different SHAs — a rebase to drop them was intentionally not force-pushed. The net diff is only the 22 files of this feature; "Files changed" reflects that.