fix(workflow): stop a provider spend limit classifying as transient_infra

[thewoolleyman]

Summary

An ACP provider spend-limit error is currently classified as
transient_infra. The classifier examines the rendered message and full cause
chain, and the embedded spawned_at Cargo registry source path contains
index.crates.io, one of the transient-infrastructure hints. Provenance text
therefore wins over the actual permanent, human-actionable billing failure.

A second observed provider message exposes the companion failure after that
provenance is removed: You've hit your limit · resets Jul 31, 5am (UTC)
matches none of the existing budget hints and would fall through to
Deterministic. A resetting usage quota is a resource limit, not a code fault,
and Deterministic would incorrectly feed it to failure-signature tracking.
This PR covers these two observed phrasings without claiming to recognize every
provider-limit message.

This does not change per-stage retry policy: Handler, Engine, and Io
errors remain retryable regardless of FailureCategory. It does correct the
operator-facing category and failure-class edge conditions, and it prevents
these failures from taking loop_restart edges, which require
transient_infra. Signature tracking also remains disabled: neither the
observed pre-fix TransientInfra nor the corrected BudgetExhausted category
is tracked.

Observed payloads

The monthly spend-cap payload is permanent within its billing period and asks
for human action:

message:    "ACP turn failed"
causes[0]:  "ACP protocol error"
causes[1]:  "Internal error: You've hit your org's monthly spend limit · ask your
             admin to raise it at claude.ai/settings/usage: {
               \"spawned_at\": \"/home/ubuntu/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/agent-client-protocol-0.11.1/src/session.rs:567:14\",
               \"data\": { \"errorKind\": \"rate_limit\" }
             }"
category:   "transient_infra"

In this payload, the registry source path is the sole transient hint match:
errorKind uses rate_limit with an underscore, while the classifier looks
for rate limit.

The second provider message reports a resetting usage window:

Internal error: You've hit your limit · resets Jul 31, 5am (UTC)

Provenance sanitization prevents a Cargo source path from deciding its class,
but sanitization alone would leave this phrase on the Deterministic fallback.

Fix

Before matching classifier hints, blank a spawned_at value when it contains a
local Cargo registry/src/<registry>-<hash>/… source path. spawned_at is
provenance: it identifies where the provider-side process raised the error,
not what failed. Restricting the sanitizer to that structured field avoids
interpreting arbitrary prose or consuming fault text outside that field value.

The matcher handles absolute, relative, non-default-registry, Windows, and
mixed-separator paths. It accepts both 7–64-character hexadecimal registry
hashes and, as a compatibility guard for callers that may have already applied
the classifier's hex normalization, the resulting <hex> placeholder. It
decodes the JSON key and value before matching, so escaped key spellings,
Unicode-escaped local paths, and partially escaped hashes behave like their
plain equivalents. The current classifier runs the sanitizer before case
normalization and global hex masking. JSON escapes are case-sensitive, so
validating the original spelling prevents malformed \U, \N, and similar
escapes from becoming valid after lowercasing; running before hex masking
ensures hex-like registry identifiers and URI schemes retain their meaning.

URL detection uses that same decoded value. Raw, escaped-solidus,
Unicode-escaped, protocol-relative, file:, and other multi-character URI
schemes remain untouched; one-character drive prefixes remain local Windows
paths. Invalid registry hashes, registry paths in unrelated fields, and
ordinary registry fetch/download errors also remain untouched.
Malformed JSON key/value strings are outside the structured boundary and are
preserved verbatim, including case-changing invalid escapes, so fault text in
them remains available to the classifier.

A separate permanent-provider-limit hint maps spend limit to
BudgetExhausted before transient hints are checked. The measured
you've hit your limit phrase joins the ordinary budget hints after transient
matching. That ordering is deliberate: the monthly cap overrides
transient-looking provider prose, while a genuine adjacent network fault still
wins over the resetting quota phrase.

Fabro's public failure documentation defines budget_exhausted as a resource
limit and lists quota exhaustion as an example, so the resetting usage window
uses an existing category rather than inventing a new one. The embedded
errorKind: rate_limit is intentionally not used as the discriminator:
ordinary throttling and the permanent monthly spend cap can carry the same
value, while Fabro's structured SDK RateLimit category is normally
TransientInfra.

Together, provenance sanitization and the two evidence-backed message hints
prevent these observed provider limits from landing on the signature-tracked
Deterministic fallback. This remains heuristic fallback classification for
rendered errors, not a claim that all provider-limit prose is recognized.

Verification

• The representative payload classifies as BudgetExhausted through both the
  direct classifier and the
  Error::handler_with_source → FailureDetail path.
• The exact resetting-window message and a representative full ACP cause-chain
  form classify as BudgetExhausted.
• A genuine adjacent connection reset still wins over the ordinary
  resetting-window budget hint and remains TransientInfra.
• Structured Cargo source provenance without fault text falls back to
  Deterministic, including absolute, relative, hex-masked, non-default
  registry, Windows, and mixed-separator paths.
• A real network failure elsewhere in the same payload remains
  TransientInfra.
• Registry fetch/download failures, covered URL-valued spawned_at forms, and
  registry paths in unrelated fields remain TransientInfra, including
  escaped URLs and hex-like multi-character URI schemes.
• Short, non-hex, and overlong registry suffixes are not sanitized.
• Malformed spawned_at keys and values with invalid lowercase or
  case-changing uppercase escapes, or a literal newline, are preserved,
  including their real transient fault text.
• Existing transient and structural hint-count guards remain unchanged; the
  ordinary budget-hint guard increases from 10 to 11, and the new one-entry
  permanent list has its own guard.
• The diff contains 527 insertions and four replaced lines; no existing test or
  guard is removed, weakened, or ignored.
• cargo test -p fabro-workflow --lib error:: — 175 passed
• cargo test -p fabro-workflow --lib — 1,146 passed
• cargo nextest run -p fabro-workflow — the unchanged
  handler::parallel::tests::for_each_accepts_an_array_at_the_item_limit
  exceeds the repository's 6-second nextest timeout; it passes under
  cargo test, including in the full library run above
• cargo +nightly-2026-04-14 fmt --check --all — passed
• cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings
  — passed

[Copilot]

Pull request overview

This PR refines fabro-workflow failure classification so permanent provider billing ceilings (specifically ACP “spend limit” failures) are no longer misclassified as transient_infra due to Cargo registry source-path provenance appearing inside structured spawned_at payloads. This improves operator-facing categorization and prevents loop_restart edges (which require transient_infra) from being taken for spend-limit failures.

Changes:

• Add a permanent provider spend-limit hint (spend limit) that maps to FailureCategory::BudgetExhausted before transient-infra hints are evaluated.
• Sanitize structured spawned_at values that match Cargo registry/src/<registry>-<hash>/... source provenance so they don’t trigger transient-infra heuristics (while preserving unrelated fields and malformed JSON verbatim).
• Add comprehensive unit tests covering spend-limit classification, structured provenance discounting, URL detection exceptions, malformed JSON boundaries, and non-weakening guards.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[brynary]

Hi @thewoolleyman -- Thanks for this.

Just to zoom out for a moment... If I understand the PR properly it is saying that a spending limit was categorized as transient_infra. At fist blush, that sounds potentially reasonable. (However, I have some pre-existing concern the error categories are ambiguous.)

How would you expect a spending error to be categorized?

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.