fix(airc): bound the drop-nudge spin + never report an install that cannot run - #1317
Merged
Conversation
…iscovery to death
Measured live on the M5, 2026-08-04, in ONE daemon log:
44,704 relay self-elections
4,951 "gh request budget exceeded for Registry (30/30 of 30 in 60s)"
52 enrolled peers, ZERO reachable
~4.5s refresh re-entry against a 60s REFRESH_INTERVAL
The mechanism is a closed cycle. `set_disconnect_observer` nudges `route_wake`
on every session drop (#240, so a real reconnect doesn't wait out 60s). A
refresh DIALS every stored peer. On a node whose registry is mostly dead peers,
those dials drop, each drop nudges again, and the next refresh starts ~4.5s
later — the length of the dial sweep. Nothing bounded it.
The damage isn't the CPU, it's the starvation: every pass spends GitHub
requests until the governor cuts the node off at 30/60s, and the account
registry is HOW PEERS FIND EACH OTHER. So the loop consumed the budget that
discovery needs, then kept hammering the closed door 44,704 times. It cannot
climb out on its own, which is why a restart "fixes" it for a minute and the
ghost returns a few times a day.
`MIN_REFRESH_SPACING` (15s) floors re-entry on the wake path only. The nudge is
kept — a genuine reconnect still beats the 60s interval by 4× — it simply cannot
re-enter faster than the floor, bounding the cycle at 4/min instead of
unbounded. The timer path is untouched (it already waits 60s).
Test pins the storm shape directly: 600 nudges across 60 virtual seconds must
produce at most `1 + 60/MIN_REFRESH_SPACING + 1` refreshes. Unfloored this test
counts hundreds — it fails on the exact defect measured above.
This bounds the spin. It does NOT yet make the gh governor's "retry in 60s"
actual backpressure (the loop still re-attempts inside the refused window) —
that is the sibling fix and it is next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…ry that cannot run Lived 2026-08-04: a locally-built airc landed on PATH and was SIGKILLed by macOS on every invocation — exit 137, zero output. `airc --version`, `airc status`, `airc join` and the daemon each silently did nothing. Twenty minutes of a completely dead mesh that read, from the outside, exactly like a transport bug. One `codesign -s -` and it ran immediately. (The corrupt signature was a hand-run `cp` straight over the installed path, not this script — install.sh already does the correct cp-to-temp + atomic mv. The gap this closes is different: the script printed "Installed airc" without ever executing what it installed.) "Installed" was a claim about a file existing. It is now a claim about a binary that answered `--version`, and the reported line carries the version it actually printed. The check is deliberately cause-agnostic — bad signature, missing dylib, wrong-arch build all fail identically and all get caught. On macOS it attempts one ad-hoc re-sign and re-verifies before giving up, so the common case self-heals instead of stranding the operator. A silently-dead airc is indistinguishable from a broken mesh, which is the most expensive failure this project has: it sends you hunting the network for a defect that is a file permission bit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
…t backpressure
The starvation half of the M5 ghost. One daemon log, 2026-08-04: 4,951
"gh request budget exceeded for Registry (30/30 of 30 in 60s); retry in 60s"
while 52 enrolled peers stayed unreachable. The account registry is HOW PEERS
FIND EACH OTHER, so a permanently-refused registry is a permanently blind node.
The governor had already computed the answer — "retry in 60s" — and the code
threw it away. `gh_run` formatted the denial into an `Adapter(String)`, the
refresh loop logged it, and the ticker brought it straight back to be refused
again inside the window it had just been told was closed. The node could not
climb out on its own, which is why a restart "fixed" it for a minute.
Three changes, one idea — make the wait DATA the caller can honor:
- `AccountRegistryError::RateLimited { retry_after_secs }` — typed, not prose.
`gh_run` returns it (clamping the governor's i64 rather than panicking on a
clock oddity); `reason` still reaches the operator via Display + a debug probe.
- `TickOutcome { Imported, NoChange, RateLimited { .. } }` replaces `gated_tick`
returning a bare bool, which could only say "nudge or don't" and had nowhere
to put "the door is shut for N seconds".
- `honor()` acts on it: nudge route-refresh on an import, and actually SLEEP a
denial before the loop comes back around.
Tests pin both directions: a denial costs exactly its window (60s virtual), and
a normal tick costs zero wall-clock, so the backoff can't silently become a
stall. Needed tokio `test-util` in dev-deps for paused time.
Pairs with the wake-path floor in this branch. That one bounded how often we
DIAL; this one stops us spending the discovery budget we'd already been refused.
Neither is yet proven to end the disconnects — the floor measured a 5× spin
reduction on the live M5 but budget denials continued at ~1.2/min, which is the
gap this commit closes. Verification is a real observation window, not this
message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two low-risk, tested fixes split off PR #1316 so they can land without waiting on the identity work (which is unproven and CI-red). Nothing here touches identity, transport, or the wire.
1. The ghost — measured live on the M5, 2026-08-04
One daemon log:
set_disconnect_observernudgesroute_wakeon every session drop (#240, so a real reconnect doesn't wait out 60s). A refresh dials every stored peer. On a node whose registry is mostly dead peers, those dials drop, each drop nudges again, and the next refresh starts ~4.5s later — the dial sweep's length. Nothing bounded it.The damage isn't CPU, it's starvation: every pass spends GitHub requests until the governor cuts the node off at 30/60s, and the account registry is how peers find each other. The loop eats the budget discovery needs, then hammers the closed door 44,704 times. It can't climb out, which is why a restart clears it for a minute and it returns a few times a day.
MIN_REFRESH_SPACING(15s) floors re-entry on the wake path only. The nudge is kept — a genuine reconnect still beats the 60s interval by 4× — it just can't re-enter faster than the floor, bounding the cycle at 4/min. The timer path is untouched.Test pins the storm directly: 600 nudges across 60 virtual seconds must yield at most
1 + 60/MIN_REFRESH_SPACING + 1refreshes. Unfloored it counts hundreds.Not yet proven: that this ends the disconnects. Early post-deploy counters look right but the window was ~20s, which is not evidence. Needs a real observation window.
2. Verify the deploy
A locally-built airc landed on PATH and was SIGKILLed by macOS on every invocation — exit 137, zero output.
--version,status,join, and the daemon each silently did nothing. Twenty minutes of a completely dead mesh that read exactly like a transport bug. Onecodesign -s -and it ran.(The corrupt signature came from a hand-run
cpover the installed path, not from install.sh, which already does cp-to-temp + atomic mv. The gap closed here is different: the script printed "Installed airc" without ever executing what it installed.)"Installed" is now a claim about a binary that answered
--version, and the success line carries the version it printed. Cause-agnostic — bad signature, missing dylib, wrong-arch build all fail identically and all get caught. On macOS it attempts one ad-hoc re-sign and re-verifies before giving up.A silently-dead airc is indistinguishable from a broken mesh, which sends you hunting the network for a defect that is a signature bit.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo