Bound a single attempt with attempt_timeout - #5
Merged
Conversation
The README promised "give up after ~30s total" and we did not deliver it. max_elapsed is only consulted between attempts, inside should_retry_after, which is reached only when an attempt returns. An operation that hangs -- a dead TCP peer, a lost response -- never returns, so the budget never got a chance to apply. Measured before the fix: a 500ms budget still running at 3 seconds, killed only by an external tokio::time::timeout. attempt_timeout(duration, on_timeout) arms a deadline alongside the in-flight future. When it fires the future is dropped, which in Rust is real cancellation, and the attempt becomes a failure that feeds the normal backoff and the normal .when predicate. on_timeout supplies the error, because the operation never produced one; mapping it into the caller's own type keeps everything downstream working on one type rather than forcing RetryError::error() to become an Option. The deadline runs on the injected Clock, so unlike tokio::time::timeout it is testable: the tests assert the exact sleep sequence on a mock clock and finish instantly. This also makes max_elapsed mean something. A timed-out attempt returns control to the loop, so the budget is finally consulted -- the end-to-end check now reports reason=max_elapsed where before it hung. One thing the work exposed: the mock clock advanced virtual time when a sleep was created rather than when it was polled. That was invisible while retry only ever built a sleep it intended to await. Arming a deadline a fast operation never polls made it visible, and the mock now matches real timer semantics. Breaking only for code that names Retry or RetryFuture, which gain a type parameter for the handler.
ADR007 covers the four decisions the last commit made without writing them down: why the error comes from a caller-supplied closure rather than making RetryError::error() an Option (don't tax every user for a case most never hit), why there is no blocking equivalent (interrupting a blocking closure needs another thread, which forces Send + 'static, and two tests exist to forbid that), why Retry gains a type parameter and why a defaulted NoTimeout does not work, and the mock-clock lesson. docs/ROADMAP.md was written on the circuit-breaker branch and stranded there, so the file recording what we refuse has never been on main. It lands updated: attempt_timeout shipped, standalone timeout refused with the reason, and the circuit breaker recorded as built, tested and not shipped, with the download numbers and the "does it answer 'a call failed, now what'" test that ruled it out. It also states what mettle is now, which is narrower than it started: retry solved completely, not a toolkit of shallow tools.
The docs claimed "timeout and circuit breaking are planned". Timeout shipped in the previous commit, and the circuit breaker was built and deliberately not shipped, so both halves were false on the docs.rs landing page. The tagline and a one-item "Tools" list also still sold a toolkit the roadmap had already decided against. The poll loop reached the timeout handler through an Option it had just proved was Some, and funnelled the error through a second Option for the same reason. Matching the deadline and its handler as a pair makes the arm that needs both the only one that can run, so neither expect() is needed. examples/retry.rs gains the case that motivated the feature: a call that never returns, which before hung forever and now gives up in 310ms with reason=max_elapsed. The blocking example says why it has no equivalent.
cargo-semver-checks runs on the pull request, not at release time, so a breaking change left at the published version fails CI regardless of merit. It flagged exactly what ADR007 predicted: Retry 4 -> 5 and RetryFuture 6 -> 7 required generic parameters, "semver requires new major version". 0.x makes that a minor bump, so 0.4.0 -> 0.5.0, with the CHANGELOG heading dated and an Upgrading section for the only people affected: those who name Retry or RetryFuture rather than using them inline. This follows ae3a5d2, where the last breaking change carried its own bump for the same reason. CONTRIBUTING said to run the semver check locally without saying how, and its feature-matrix list was missing the default and async-only configurations that CI actually runs. Both are now spelled out, along with the rule that a breaking change bumps the version in its own branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bound a single attempt with
attempt_timeoutmax_elapsedlives insideshould_retry_after, which is only reached when an attempt returns.An operation that hangs never returns, so the budget never applied. A dead TCP peer, a lost
response, a service that accepts your connection and then goes quiet: all produce a future that is
simply never ready, and the retry sat there forever.
That was not a missing feature so much as a broken promise —
README.mdsaid "give up after ~30stotal" and the crate did not do it. Measured before the fix: a 500 ms budget still running at 3
seconds, stopped only by an external
tokio::time::timeout. After: gives up in 310 ms withreason=max_elapsed.The per-attempt bound is what makes the total budget enforceable at all: a timed-out attempt
returns control to the loop, so
max_elapsedis finally consulted.Why
on_timeoutis a closureA timed-out attempt has no
E, because the operation never returned one. Three ways out, and onlyone is cheap for people who never use this:
RetryError::error() -> Option<&E>Optionmid-match, forever, for a case most never hitE: From<SomeTimeoutType>E = Stringandanyhow::Error, exactly what ADR006 fought to keepon_timeout: impl Fn() -> EVerified from outside the crate that the third stays a one-liner for every common error type —
io::Erroris\|\| ErrorKind::TimedOut.into(). The builder is order-independent in bothdirections, and the one-arg mistake a
tokio::time::timeoutuser would make produceserror[E0061]: this method takes 2 arguments, not a generics wall.Full reasoning, including why the deadline runs on the injected
Clockand why there is noblocking equivalent: ADR007.
Breaking
RetryandRetryFuturegain one type parameter, for the handler.retry(..)and every buildermethod are unchanged, so this only affects code that names those types — storing one in a struct
field, say. A defaulted
Q = NoTimeoutwas tried and does not work:NoTimeoutcannot implementFn() -> E, so the no-timeout case would need a second, overlappingIntoFutureimpl.Also in this branch
lib.rsclaimed "timeout and circuit breaking are planned" — timeoutships here, and the circuit breaker was built and deliberately not shipped. The "resilience
toolkit" tagline and a one-item
## Toolslist also still sold a toolkit the roadmap had alreadydecided against.
docs/ROADMAP.mdlands on main. It was written on thecircuit-breakerbranch and strandedthere, so the file recording what we refuse had never actually been on main.
expect()s removed from the poll loop. It reached the timeout handler throughan
Optionit had just proved wasSome. Matching the deadline and its handler as a pair makesthe arm that needs both the only one that can run.
examples/retry.rsgains the case that motivated the feature: a call that never returns.Verification
15 gates across all three feature configs (
--all-features, default,blocking-only): tests,clippy
-D warnings, rustdoc-D warnings, and MSRV 1.85. 65 unit + 14 doctests. Both examplesrun.