fix(buzz-acp): retry startup channel discovery on transient relay failure - #4775
Open
justanotherkevin wants to merge 1 commit into
Open
fix(buzz-acp): retry startup channel discovery on transient relay failure#4775justanotherkevin wants to merge 1 commit into
justanotherkevin wants to merge 1 commit into
Conversation
…lure Related to block#4538. HarnessRelay::discover_channels() made a plain REST POST /query call with no retry, so a single transient relay failure crashed the entire harness process with no self-recovery — unlike the WebSocket-connect path, which already retries via retry_initial_connect and STARTUP_CONNECT_BACKOFFS. Add retry_channel_discovery(), reusing the same backoff ladder but without retry_initial_connect's terminal-error short-circuit, since is_terminal_connect_error classifies RelayError::Http as terminal (correct for the WS-connect call site, wrong for REST query failures like the 404 observed in practice). Signed-off-by: Kevin Hu <hu.kevin.w@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Related to #4538. Startup channel discovery (
HarnessRelay::discover_channels())makes plain REST
POST /querycalls with no retry — a single transientfailure crashes the entire harness process immediately, with no self-recovery,
unlike the WebSocket-connect path which already has bounded retry
(
retry_initial_connect+STARTUP_CONNECT_BACKOFFS).Observed in practice: a relay-side blip returned
404onPOST /queryalongside
database erroron two live subscriptions. The harness diedoutright (no graceful shutdown sequence) and required a manual restart, even
though the relay had recovered within moments.
This does not address the separate silent-"0 channels" symptom being
investigated in #4538 (possible relay-side accessible-channel scope bug in
bridge.rs) — this only stops a transient discovery failure from beingunrecoverable. Posting for the maintainers to weigh in on whether it's the
right shape given the ongoing #4538 investigation.
Changes
crates/buzz-acp/src/relay.rs: newretry_channel_discovery(), reusingSTARTUP_CONNECT_BACKOFFS/jittered_durationfromretry_initial_connectbut without its terminal-error short-circuit —
is_terminal_connect_errorclassifies
RelayError::Httpas terminal, which is correct for theWS-connect call site but wrong here, since
RelayError::Httpis exactlythe failure mode observed above.
crates/buzz-acp/src/lib.rs: startup call site now goes through the newretry wrapper before the existing
anyhowerror wrapping/?propagation(unchanged on final failure).
retry_initial_connecttestcoverage.
Test plan
cargo build -p buzz-acpcargo test -p buzz-acp --lib relay::— 82 passed (includes 2 new tests)cargo fmt --check -p buzz-acpcargo clippy -p buzz-acp --all-targets -- -D warnings