Host-eligibility frontier scheduler + robots crawl-delay (ADR-0010)

[filipeforattini]

What to build

The Frontier pops the highest-priority Job among hosts whose next-allowed time has arrived, where next-allowed = max(rate-limiter floor, robots crawl-delay, cadence envelope), interleaving eligible hosts to avoid same-domain thrash. Per-host schedule state is persisted so a resumed Crawl preserves pacing; robots crawl-delay is parsed, persisted, and enforced. The same-host bias becomes a tiebreak within the eligible set. See ADR-0010.

Acceptance criteria

• The next Job is chosen among eligible hosts (next-allowed reached), interleaving hosts.
• robots crawl-delay is parsed, persisted, and enforced as an input to next-allowed.
• Per-host schedule state persists so a resumed Crawl does not re-hammer hosts.
• Tests cover eligibility ordering, crawl-delay enforcement, and absence of same-domain thrash.

Blocked by

• Seen-set TTL admission + conditional-GET refresh (ADR-0005) #62
• CadenceGovernor + time-series self-monitoring (ADR-0009) #74