Skip to content

Crawl-delay compliance - #1

Merged
gilnobrega merged 12 commits into
mainfrom
dev-crawl-delay
Jan 14, 2026
Merged

gilnobrega merged 12 commits into
mainfrom
dev-crawl-delay

Conversation

@gilnobrega

@gilnobrega gilnobrega commented Jan 14, 2026 •

Copy link
Copy Markdown
Contributor

Implements compliance with Crawl-delay, supporting 3 modes:

  • Await (default) - awaits necessary time before continuing with request
  • Failure - throws exception if request called within delay time of previous request
  • Ignore - ignores Crawl-delay (many crawlers do this)

Other changes:
Makes codebase more scalable
Introduces unit tests
Bumps version to 0.2.0

@gilnobrega
gilnobrega merged commit c060fca into main Jan 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant