docs: crawler persistence + stealth design (ADR-0005..0011)#61
Merged
Conversation
…ossary) Grilling session (/start) reviewing crawlex against two system-design references (web-crawler + bot-detection) with RedDB persistence as the shared substrate. - ADR-0005 seen-set TTL / last-crawled scope - ADR-0006 two-plane storage + content-addressed BlobStore (fs/s3/r2) - ADR-0007 Crawl as a first-class resumable entity - ADR-0008 coherence-aware durable StealthProfile (SessionState burn policy) - ADR-0009 behavioral CadenceGovernor + time-series self-monitoring - ADR-0010 host-eligibility frontier scheduler (+ robots crawl-delay) - ADR-0011 vector Challenge memory (RedDB SEARCH SIMILAR) - CONTEXT.md: Frontier, Seen-set, Content identity, BlobStore, Crawl, StealthProfile, CadenceGovernor, Host-eligibility scheduling, Challenge memory Claude-Session: https://claude.ai/code/session_018pq1VBZFMmMbjJzQg2cKUN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design decisions from a
/startgrilling session reviewing crawlex against two system-design references (web-crawler + bot-detection), with RedDB persistence as the shared substrate. Docs only — no code changes.ADRs
BlobStore(fs/s3/r2); kills the ~1 MiB truncation ceiling; content-hash dedup (non-destructive).Crawlas a first-class resumable entity (crawl list/resume/refresh; one-shot == full crawl at different scope).StealthProfileper host (reuse/rotate mapped onto theSessionStateladder).CadenceGovernor+crawlex_crawl_eventstime-series self-monitoring.crawl-delay); ends same-domain thrash.Challenge memory(RedDBSEARCH SIMILAR) for semantic antibot recognition.Glossary (
CONTEXT.md)New terms: Frontier, Seen-set, Content identity, BlobStore, Crawl, StealthProfile, CadenceGovernor, Host-eligibility scheduling, Challenge memory. Resolved the
crawl/runambiguity.