LLM provider failover, failover notification, and an assessment-coverage alarm

[mircealungu]

Follow-up to #717, from two questions: "does the fallback work now — if we run out of DeepSeek credit will we be back on Anthropic?" and "we need a check for crawled-but-unassessed too".

The fallback did not work

_select_provider_and_key switched providers only when the API key was missing. A provider that has a key and refuses to serve — out of credit, over a spend cap, rate limited — raised a plain Exception that nothing caught, and the article was left unassessed.

That is precisely the Anthropic-cap failure that starved Danish for a week, and moving the crawl to DeepSeek re-armed it on the other side: running out of DeepSeek credit would look identical from the outside (articles crawled, none assessed, nothing in the logs but a per-article error).

1. Failover

ProviderUnavailable marks the provider cannot serve us, and one call is retried on the other provider. Content failures — paywall, advertorial, wrong language — are facts about the article and are deliberately not retried elsewhere; that would spend twice for the same answer.

Status codes alone are insufficient. Anthropic reports its monthly cap as a 400 invalid_request_error, not a quota status, so the body is matched as well. The exact production strings for both providers are pinned in the tests.

2. Notification, once per process

A dead provider fails over on every article, and the crawl handles ~2,000/day. Per-event mail would be 2,000 messages and would train us to filter the alert away. The crawler is a fresh process per hourly run, so a sustained outage sends roughly one mail an hour — enough to notice, few enough to read.

Two details that matter:

• Sent after the fallback call succeeds. A failover that rescued nothing is not a rescue; that case raises and is caught by the check below.
• Wrapped so a broken mailer can never turn a successfully assessed article into a failed one.

3. Assessment-coverage alarm

Failover degrades two providers to one. This is what says so when both are gone.

The existing delivery checks are a lagging indicator — the August outage ran a week before anyone noticed — so this watches the upstream step directly: of the articles crawled in the last 24h, how many got a CEFR level?

The threshold is measured, not guessed:

	coverage
healthy, Aug 15–19	97, 97, 98, 98, 98 %
capped, Aug 12, 13, 20	15, 38, 37 %

80% sits far below every healthy day and far above every broken one. Every one of those days is replayed through the real classifier in the tests, including the Aug 14 partial-recovery day (83%) which must not alert.

Below 50 eligible articles it reports without alerting: "almost nothing was crawled" is crawler_liveness_check's job, and firing on a quiet window would teach us to ignore this one.

A failing report is actionable rather than just red:

❌ ASSESSMENT COVERAGE (last 24h): 562/1522 = 37% (alert below 80%)
    da: 4/120 = 3%
    de: 300/700 = 43%

    Articles are being crawled but not assessed: no CEFR level, no
    summary, no per-level text. Feed cards fall back to the publisher's
    blurb, identical at every CEFR level.
    Most likely BOTH LLM providers are unavailable (provider failover
    covers one being down). Check, in order:
      - DeepSeek balance    https://platform.deepseek.com/usage
      - Anthropic spend cap (Console -> Billing/Limits)
      - /var/log/zeeguu/crawler/ for the actual API error

Notes

• No cron change needed — the check runs inside the existing feed-delivery entry, whose crontab line lives in the ops repo.
• Anthropic is capped until Sept 1, so failover to Anthropic could not be exercised against the live API; the failover logic is covered by unit tests against the recorded error bodies instead.
• Worth knowing: the new field-based assess prompt from Per-level card text: fix the collapse, restore per-level titles, un-break deepseek #717 is still validated on DeepSeek only. A failover to Anthropic would send Haiku a prompt shape it has not been tested against. That is a graceful-degradation attempt, not a guarantee — and it is the thing to check first after Sept 1.

490 tests pass.

[github-actions]

ArchLens detected architectural changes in the following views: