diagnostics: make network failures, degraded rails and metered spend all name themselves - #656
Merged
Merged
Conversation
Same defect as the facilitator path, in the busier place. Naming the lane was
only half the fix: Node flattens every network-level failure to the bare string
"fetch failed", so
all 2 RPCs failed - relay.example: fetch failed | publicnode: fetch failed
is still unactionable. It cannot separate a DNS failure from a refused
connection from an unroutable IPv6 address, and those are three different fixes
(bad hostname / dead host / needs a relay). That indistinguishability is
precisely what gets written off as a transient blip.
Now:
all 2 RPCs failed - nonexistent.invalid: fetch failed [ENOTFOUND syscall=getaddrinfo]
| 127.0.0.1:45999: fetch failed [ECONNREFUSED syscall=connect address=127.0.0.1 port=45999]
The dual-stack branch matters most here. Most of these RPCs sit behind
Cloudflare with both A and AAAA records, so Happy Eyeballs failing every address
returns an AggregateError whose WRAPPER carries no code at all - the per-address
reasons live in `.errors`. Without that branch a dual-stack failure still logs
bare "fetch failed"; with it the log says which family failed and at which
address, which is the difference between "unknowable" and "our egress cannot
reach Cloudflare over v6, add a relay like Sei and Algorand already needed".
describeError() is exported and tested rather than inlined, because this exact
flattening exists in ~45 other places and the next one should reuse it instead
of rediscovering the problem.
An error with no cause is returned unchanged - asserted, so this cannot add
noise to the ordinary case.
scripts/test-revenue-scan.js 21 -> 26, mutation-checked: dropping the cause
collapses the output back to bare "fetch failed".
… of only surviving it Celo was dropped from the offer for DAYS and the only reason anyone found out is that you asked why the revenue page looked short. Every piece of that was working as designed, which is the problem. payments.js drops a rail nobody can settle so the other chains keep earning. Paid revenue kept flowing. /health stayed 200. The single trace was one boot log line. The system survived it perfectly and told nobody, and "survivable" quietly became "invisible". The heartbeat now reads /api/rails every cycle and opens ONE tracked issue naming each configured-but-unoffered rail and its reason. It closes itself when every rail is offered again, which now happens on its own: rail-selfheal.yml restarts the service once the rail's own facilitator answers, and the next heartbeat closes the issue. Deliberately an ISSUE, not a page. A degraded rail is not an outage - the other eleven chains are still settling - so it sits at the same severity as "burner LOW": visible, tracked, auto-closing, nobody woken at 3am for someone else's facilitator. The trap this class of check keeps falling into, guarded explicitly: an unreadable /api/rails is NOT treated as healthy. A failed curl would otherwise look identical to "degraded=0" and CLOSE a real issue mid-outage - which is exactly how the charged-failure alarm managed to never fire once in months. Dry-run against live production: degraded=1, celo named with its reason, and the recovered and unreadable branches both behave correctly. scripts/test-rail-selfheal.js 13 -> 20, mutation-checked: making an unreadable response count as healthy fails the guard. One test bug caught while writing it: the "must not page" assertion split the file on the step name and read everything after, which swallowed later steps that legitimately page on production being down. It failed for the wrong reason until scoped to the step itself.
…already known broken Found while auditing why Brave billed ~56 requests against 6 sold search calls this month. Keyed self-check successes were cached for 6h, but FAILURES were deliberately not cached so a real key problem would re-test on the next poll. Right instinct, missing bound: /api/selfcheck is polled every 30 minutes and a failing check retries once, so a broken Brave key or a Brave outage costs ~96 billed requests a day. The failure mode bills hardest exactly when the thing is already broken - the outage pays for itself twice. Failures are now cached for 30 minutes (successes keep the 6h TTL). The property that must not change, and is asserted: caching a failure CANNOT hide it. The cached result is still ok:false, so it stays in `failing`, the endpoint still reports it, and tool-alert.yml still pages and keeps its issue open. The only thing that changes is how often we PAY to re-confirm something we already know. Measured on a stub: an always-failing metered tool now costs 2 upstream calls (attempt + retry) and then nothing until the TTL expires, instead of 2 per poll forever. The success path is untouched - still one call, then cached 6h. scripts/test-selfcheck.js +4, mutation-checked: un-caching negatives takes the stub from 2 upstream calls to 6 across three polls and fails the guard. NOT the whole story on the Brave spend, and worth recording honestly: the biggest single contributor this month was DEPLOY FREQUENCY. The keyed cache is in-memory, so every restart clears it and the next poll re-checks; there have been 13 deploys since Aug 1. That is a cost of shipping often, not a defect, and it is not something to "fix" by deploying less.
Two fixes and one dismissal, from the "why is Brave billing more than search earns" thread. 1. multi-search deduped. The price is flat for 2-5 queries but EVERY query was a separate billed Brave request, so ["x","x","x","x","x"] cost five upstream calls for one $0.08 sale. A margin leak on honest duplicates and a free 5x multiplier for anyone who noticed. Fan-out is now over unique queries; the response is still built per INPUT query so the caller's shape and ordering are byte-identical. Six inputs with two distinct queries: 5 billed calls before, 2 after. 2. /__operator/ledger-sync.json cached for 15s (CodeQL js/missing-rate-limiting, alert #81 - a fair hit on code I added last week). Each request fired one live eth_blockNumber PER CHAIN, so holding refresh hammered a dozen shared public RPCs. Operator auth bounds WHO can do that, not how often. Cursor positions do not move faster than the window, so the cache costs no accuracy. 3. CodeQL #80 dismissed as a false positive with evidence: rate limiting IS present on that path, CodeQL just cannot trace it through the middleware - powHttpLimiter.check(req.ip) on PoW redemption and trialToolLimiter/ trialIpLimiter on the trial path, both the shared limiter in rate-limit.js. On multi-search PRICING, which is what was actually asked: I am not changing it. At a measured ~$0.007/Brave request the tool runs 56% margin at five queries and 82% at two, so it was never underwater. The month's shortfall was unpaid calls, not underpricing, and the dedupe removes the worst of what remained. Repricing a profitable tool to fix a spend leak would have been the wrong lever. scripts/test-brave-leak.js 27 -> 32.
MikeyPetrillo
temporarily deployed
to
agent402 / production
August 2, 2026 16:14 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
MikeyPetrillo
marked this pull request as ready for review
August 2, 2026 16:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four commits from an investigation that started with "why does the revenue page look short" and ended in the metered-upstream billing.
a865839— rpcCall names WHY each lane failedNaming the lane was half a fix. Node flattens every network-level failure to the bare string
"fetch failed"and hangs the reason onerr.cause, sorelay: fetch failed | publicnode: fetch failedstill cannot separate DNS from a refused connection from an unroutable IPv6 address — three different fixes. That indistinguishability is exactly what gets written off as "transient".Now:
fetch failed [ECONNREFUSED syscall=connect address=1.2.3.4 port=443]. Dual-stack hosts (most of these RPCs sit behind Cloudflare with A and AAAA) fail viaAggregateErrorwhose wrapper carries no code at all, so the per-address branch is what makes "IPv6 is unroutable from here" distinguishable from "the host is down".describeError()is exported because this flattening exists in ~45 other places.60ef3d1— the heartbeat tells someone when a rail goes downCelo was dropped for days and surfaced only because someone asked. Every component worked as designed: the rail was dropped so others kept earning, revenue flowed,
/healthstayed 200, and the only trace was one boot log. Survivable had quietly become invisible.The heartbeat now opens one tracked issue naming each configured-but-unoffered rail and its reason, and closes it when they return. Deliberately an issue, not a page — a degraded rail is not an outage.
Guarded explicitly: an unreadable
/api/railsis NOT treated as healthy. A failed curl would otherwise look identical todegraded=0and close a real issue mid-outage, which is how the charged-failure alarm went months without firing.cb4e5ff— stop re-billing a metered upstream that is already brokenKeyed self-check successes cached for 6h; failures deliberately did not, so a real key problem re-tested every poll. With a 30-minute poll and a retry that is ~96 billed Brave requests a day — the failure mode bills hardest exactly when the thing is already broken.
Failures now cache 30 min. The property asserted: caching a failure cannot hide it. The result is still
ok:false, stays infailing, andtool-alert.ymlstill pages. Only the paying-to-re-confirm changes.Tests
test-revenue-scan21→26,test-rail-selfheal13→20,test-selfcheck10→14, all mutation-checked. Full battery green: 210 strict / 296 lenient,test-brave-leak27,test-rails114.