payments: a down facilitator must cost one chain, not every paid route

[MikeyPetrillo]

Brings main up to what production needs. Two commits.

1b7b986 — per-rail configured-vs-offered visibility

A rail whose facilitator is down is dropped from the offer so the other chains keep settling. That drop used to be silent: Celo vanished, the 402 quietly advertised 11 networks while config asked for 12, /health read ok, and the only trace was a boot log. It surfaced days later via a canary WARN.

GET /api/rails now reports, per chain, whether it is offered and why not:

{ "configured": 12, "offered": 11, "degraded": 1,
  "rails": [ { "network": "celo", "offered": false,
               "reason": "no reachable facilitator advertises exact for it" } ] }

Public deliberately — the 402 already advertises the offered set, so this adds no secret, it adds the difference, which was the invisible part.

29c7517 — a DOWN facilitator must cost one chain, not every paid route

The existing drop-don't-break guards only cover a missing url/key. A facilitator that is configured and simply down takes the opposite path: the client is added, its /supported handshake fails, the network stays in the offer with nothing able to settle it, and @x402/core refuses to BUILD the 402 — so every paid route on every chain answers 500 while /health stays 200.

Reproduced with Base healthy and Celo's facilitator always-500 (its real state on 2026-08-02):

before: POST /api/hash -> 500  RouteConfigurationError
after:  POST /api/hash -> 402  accepts: [exact eip155:8453]

Control-tested against the old code on identical inputs: 500.

The fix applies the discipline the upto gate always used and that never covered exact: verify the offer against what the facilitators we actually route through advertise, and drop what nobody can settle. Two deliberate safety properties:

• a network is dropped only on a positive answer (some facilitator responded and did not list it). If none respond, the configured offer is left alone rather than emptied on a boot-time blip.
• each client gets a second attempt, so a slow facilitator is not mistaken for a dead one and a transient blip cannot cost a rail.

One probe now serves both the exact and upto gates instead of two round trips.

Tests

scripts/test-rail-outage-isolation.js (new, wired into CI, 7 assertions) reproduces the outage end to end and asserts the 402 is payable, not merely a 402 — an empty accepts array would pass a status check while being just as unusable to a buyer. Mutation-checked: disabling the gate fails 5 of 7. scripts/test-rails-isolation.js pins the status shape. Full battery green: 210 strict / 296 lenient / shape clean.

Note

This lagging behind prod is itself the recurring problem: a Railway variable upsert rebuilds from main, so anything only on the dev branch is silently reverted on the next variable change. Merging is what makes the fix real.

[MikeyPetrillo]

Closing as superseded. main already carries the class-level fix for this in 502c1d5 ("one dead facilitator must cost one rail, not every paid route"), merged via #650, which landed before this branch's version was written. That fix is the reason production tolerated a configured Celo with a 500ing facilitator while this branch's local reproduction still showed the outage: the repro was against the dev branch, which lacked it.

502c1d5 is also the better implementation - it memoizes getSupported per instance with a shared in-flight promise, which this branch's version does not.

Re-submitting only the ADDITIVE part on top of it: GET /api/rails, the configured-vs-offered surface with per-rail reasons, so "we configured 12, we serve 11, here is which and why" is queryable rather than something you read out of boot logs.