fix(email): retry transient send failures with per-provider error classification

[clau1902]

Depends on #296

Stacked on top of #296 (suppression list) — the send path reuses the same suppression-filtered to/cc/bcc. Until #296 merges, GitHub will show a combined diff; once it merges into main, this PR's diff will shrink to just its own commits.

Summary

Previously any send failure went straight to "captured" — a transient SES throttle or a Scaleway blip permanently dropped the email instead of retrying.

• Error classification (EmailError::SendFailed { retryable, .. }) per provider: HTTP status for Scaleway (5xx/429 retryable), typed SendEmailError variants for SES (TooManyRequestsException/LimitExceededException retryable, message-level rejections like MessageRejected/SendingPausedException not — classified via the SDK's typed variants + ProvideErrorMetadata::code(), not string-matching), SMTP reply code via lettre's is_transient/is_timeout for SMTP.
• Bounded retry (2 attempts, 500ms delay) — only for errors classified as transient; a permanent rejection fails identically on retry so a second attempt is skipped. The whole attempt sequence is bounded by an overall 45s deadline (tokio::time::timeout) so a slow/hanging provider can't stall the request indefinitely.
• Inactive-provider fix: is_active previously only hid a provider from selection UI while the send path kept using it regardless — now checked explicitly before a provider is used, so disabling one actually takes it out of the send path.
• emails.provider_id/emails.attempt_count record which provider attempted the send and how many attempts it took (a clean first-try success records 1).

Not in this PR (dropped after review — enterprise-shaped for OSS core, and unsound without a real deferred-send queue): multi-provider failover chain, per-provider circuit breaker, per-provider rate limiter. See PR discussion for the full reasoning; may return as a follow-up once a deferred-send primitive exists.

Test plan

• cargo check --workspace --lib — clean
• cargo test -p temps-email --lib — 173 tests pass, including a new regression test for the inactive-provider fix

🤖 Generated with Claude Code

[dviejokfs]

Thanks — the code quality here is genuinely good (permission guards + audit on every new endpoint, batch-fetched send chain, honest doc comments on the half-open simplification, additive migration). But I'd like to rescope before this merges.

Rescope: trim to retry + error classification

Failover is enterprise-shaped, and this is OSS core. The typical temps self-hoster configures exactly one provider — often none (Mailhog capture mode is a first-class path in this same function). An ordered per-domain failover chain with its own table, CRUD surface, and priority management is for someone who has a second SES/Scaleway account — that's a vanishingly small slice of users, and it's permanent core surface + UI to maintain. If demand shows up, it can be revisited later (possibly not in core at all). Fits the "core stays thin" philosophy.

And once failover goes, the circuit breaker and rate limiter have to go with it. Their only semantic in this PR is "skip to the next provider in the chain." With one provider and no re-send queue, "skip" means capture the email terminally without attempting the send:

• Circuit breaker: 5 consecutive failures → 60s where every email is captured with zero send attempts. Worse: record_send_failure fires for non-retryable errors too, so 5 message-level rejections (bad recipients) open the circuit and drop healthy mail for everyone.
• Rate limiter: over-cap email → captured. Real MTAs defer on throttle; this drops. A burst of password resets over the cap silently doesn't deliver.

Both are strictly worse than what's on main for single-provider setups. The primitive actually missing is a deferred-send queue (throttle = delay, not drop). Once that exists in a follow-up, rate limiting becomes useful even for one provider, and CB/failover can be reconsidered on top of it.

Keep

• EmailError::SendFailed { retryable } + per-provider classification (SES/Scaleway/SMTP) — this is the actual bug fix from the PR description (transient throttle → permanent drop)
• Bounded 2-attempt retry with delay
• emails.retry_count / provider_id columns (observability)
• The inactive-provider fix — previously is_active only hid a provider from selection UI while the send path kept using it. That fix shouldn't die with the chain: keep a "provider inactive → capture, don't send" check

Drop

• email_domain_fallback_providers table + the 3 fallback endpoints + audit type + get_send_chain
• resilience.rs (circuit breaker + rate limiter)
• email_providers.rate_limit_per_minute

Findings on the kept portion

1. SES retryability is classified by string-matching the error message (contains("throttl"), contains("internal"), …) — fragile against SDK message wording. aws_sdk_sesv2 exposes typed variants (TooManyRequestsException, LimitExceededException, SendingPausedException) and ProvideErrorMetadata::code(); match on those.
2. Worst-case in-request latency: retries (and today, the whole chain) run inside POST /emails — attempts × provider timeout + sleeps. Even trimmed to one provider × 2 attempts, consider a total deadline across the send.
3. retry_count stores total attempts (clean first-try send records 1) — rename to attempt_count or store attempts - 1.

If any of the dropped pieces come back later: UpdateEmailProviderRequest.rate_limit_per_minute: Option<i32> + .map(Some) reintroduces the known double-Option PATCH gotcha (present null collapses to "unchanged", so the limit can never be cleared back to unlimited) — temps-ai-gateway has the deserialize_optional_field helper for this. And get_send_chain swallows DB errors on the primary lookup (if let Ok(primary)), silently degrading a domain to fallbacks-only on a transient DB blip.

Also noting: no CI checks have run on this branch, and since #296 hasn't merged the diff here is still the combined stack.

[clau1902]

Rescoped per review — pushed in a new commit.

Dropped entirely: email_domain_fallback_providers table + its 3 CRUD endpoints + audit type + get_send_chain, resilience.rs (circuit breaker + rate limiter), email_providers.rate_limit_per_minute. The migration file was replaced (not just edited) since it hadn't been merged yet — m20260712_000001_add_email_retry_tracking now only adds emails.provider_id/emails.attempt_count.

Kept, and fixed:

• EmailError::SendFailed { retryable } + per-provider classification, bounded 2-attempt retry — the actual bug fix from the original description.
• The inactive-provider fix survived the chain's removal: email_service.rs now explicitly checks provider.is_active before using it (previously that flag only hid a provider from selection UI while the send path used it regardless). Added a regression test for this specifically (test_send_email_inactive_provider_is_captured_not_sent).
• Renamed retry_count → attempt_count (a clean first-try send now unambiguously records 1).
• SES retryability no longer string-matches the error message — it now matches on SendEmailError's typed variants (TooManyRequestsException, LimitExceededException retryable; MessageRejected, SendingPausedException, etc. not) with ProvideErrorMetadata::code() as a fallback for unhandled/future variants.
• The whole attempt sequence (both tries) is now bounded by an overall 45s deadline via tokio::time::timeout, so a slow/hanging provider can't stall the request indefinitely regardless of attempt count.

Verified: cargo check --workspace --lib clean, cargo test -p temps-email --lib 173/173 passing.

🤖 Generated with Claude Code