add jitter and decorrelated backoff

CHANGELOG.md

@@ -5,6 +5,41 @@ All notable changes to this project are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.3.0] - 2026-08-09
+
+Adds jitter. Purely additive apart from one collision noted under Upgrading.
+
+### Added
+- `Backoff::jittered()` and `Backoff::jittered_with_seed(seed)`. Wraps any strategy, including one
+  you wrote, so each delay becomes a uniform random value in `0 ..= delay` ("full jitter"). Opt-in:
+  the default schedule stays deterministic. The seeded form is for tests.
+- `DecorrelatedBackoff`, from a validated `DecorrelatedBackoffConfig`. Draws each delay from
+  `base ..= prev * 3`, capped at `max_delay`, and never below `base`. Seedable with `with_seed`.
+- `Clock` for `&C` and `Arc<C>`, on both the async and blocking traits, so a mock clock can be
+  passed as `.clock(&mock)` and still be read afterwards.
+- `ExponentialBackoffConfig { factor: 1, .. }` documented as the way to get a constant delay; there
+  is no separate `ConstantBackoff` type.
+
+### Changed
+- New required dependency: `fastrand` (no transitive dependencies, not feature-gated).
+
+### Upgrading
+
+0.2.0 code compiles unchanged, with one exception. If you wrote `impl Clock for &YourClock`
+yourself, it now collides with the impl this release adds and the build fails with
+`E0119: conflicting implementations`. Delete yours; this release makes it redundant.
+`cargo-semver-checks` does not flag added impls, so it would not have warned you.
+[ADR005](https://github.com/azeemshaik025/mettle/blob/main/docs/adr/ADR005.md) explains why the
+impls are there and why we shipped them anyway.
+
+`Jittered` and `DecorrelatedBackoff` are deliberately not `Clone`, because a copy carries the RNG
+state and replays the same delays. Keep the config and build a fresh strategy from it.
+
+Why the jitter side is shaped the way it is, including why there is no *mode* to choose:
+[ADR004](https://github.com/azeemshaik025/mettle/blob/main/docs/adr/ADR004.md).
+
 ## [0.2.0] - 2026-07-26
 
 Supersedes the yanked 0.1.1: that release removed public items in a patch, which was a breaking

Cargo.toml

@@ -1,9 +1,9 @@
 [package]
 name = "mettle"
-version = "0.2.0"
+version = "0.3.0"
 edition = "2024"
 rust-version = "1.85"
-description = "A resilience toolkit for Rust: async and blocking retries with configurable backoff."
+description = "A resilience toolkit for Rust: async and blocking retries with configurable backoff and jitter."
 readme = "README.md"
 license = "MIT OR Apache-2.0"
 keywords = ["retry", "backoff", "resilience", "async", "tokio"]
@@ -20,6 +20,7 @@ rustdoc-args = ["--cfg", "docsrs"]
 tokio = { version = "1", features = ["time"], optional = true }
 pin-project-lite = { version = "0.2", optional = true }
 tracing = { version = "0.1", default-features = false, features = ["std"] }
+fastrand = "2"
 
 [features]
 default = ["async", "blocking"]

README.md

@@ -37,6 +37,27 @@ let body = retry(|| async { fetch(&url).await })
 
 No async runtime? The blocking twin is identical but ends in `.call()` instead of `.await`.
 
+Retrying on a fixed schedule means every client that failed together retries together, so a service
+that is coming back up gets a synchronized wave. Jitter spreads them out:
+
+```rust
+use mettle::{Backoff, DecorrelatedBackoff, DecorrelatedBackoffConfig, ExponentialBackoff};
+
+// Randomize any strategy's delays into 0 ..= delay ("full jitter")...
+let backoff = ExponentialBackoff::default().jittered();
+
+// ...or use decorrelated jitter, where each delay is drawn from the previous one.
+let backoff = DecorrelatedBackoff::new(DecorrelatedBackoffConfig::default())?;
+```
+
+Which one: `.jittered()` works on any strategy, including one you wrote, and spreads delays as
+widely as possible. `DecorrelatedBackoff` is its own strategy and never draws below its `base`, so
+reach for it when you want a floor under every wait. The trade is that a floor also means never
+retrying sooner than `base`, so a dependency that frees up early isn't picked up until then.
+
+Both seed from entropy by default and take a fixed seed (`with_seed`) when you want a test to
+replay the same delays.
+
 ## Tools
 
 Each tool comes with a runnable example. Start there:

docs/adr/ADR004.md

@@ -0,0 +1,110 @@
+# ADR004: Jitter, a fourth dependency, and why randomized strategies aren't `Clone`
+
+**Status:** Accepted
+
+## Context
+
+Retrying with a fixed backoff means every client that failed at the same moment retries at the
+same moment. The dependency that just fell over gets a synchronized wave of traffic the instant
+it comes back, which is the thundering herd. Jitter is the fix, and it's the first thing in
+mettle that needs randomness, so it forces three decisions we hadn't had to make yet: what to
+depend on, how many knobs to expose, and what `Clone` means once a type carries RNG state.
+
+Shipping it also exposed two gaps in how testable the crate actually was. Those are ADR005.
+
+## Decisions
+
+**1. `fastrand` is a required dependency, not a feature.**
+Jitter needs a random number generator. `fastrand` gives us three things (`Rng::new()`,
+`Rng::with_seed(u64)`, and `rng.u64(range)`) and nothing else comes with it.
+
+ADR003 moved Tokio behind a feature so sync-only users didn't pay for a runtime they never
+touched, so the obvious move was to do the same here: `fastrand` optional, behind a `jitter`
+feature. We measured instead of guessing. `fastrand` is 1198 lines with zero transitive
+dependencies, and rebuilding it costs about the same as rebuilding the whole `tracing` tree
+(`tracing` plus `tracing-core` plus `once_cell`), which ADR003 already accepts unconditionally.
+Gating it would have put `#[cfg]` on roughly a dozen public items including a trait method,
+doubled the feature matrix CI has to cover, and given anyone on a minimal build a missing-type
+error instead of a working `Jitter`. That's a lot of friction to save less than what a dependency
+we already took costs.
+
+The weight class is what matters, not the count. Tokio is a runtime; `fastrand` is a PRNG. We'd
+make the same call again for something this size and revisit it for anything larger.
+
+We also considered inlining a small PRNG to keep the dependency count at two. We didn't, because
+seeding from entropy portably is the part that's actually easy to get wrong, and getting it wrong
+means a fleet seeded identically, which is the exact failure jitter exists to prevent.
+
+**1b. The RNG is a concrete `fastrand::Rng`, not a trait.**
+`Clock` is injected through a trait, so it's fair to ask why randomness isn't. The two look
+symmetrical and aren't.
+
+You cannot seed the system clock. The only way a test controls time is to replace the source
+outright, which is why `Clock` has to be a trait. Randomness has a cheaper answer: seeding *is* the
+determinism mechanism, and `with_seed` already gives complete reproducibility without replacing
+anything. A trait would buy the ability to swap the *algorithm*, which is a different capability and
+one nobody has asked for. Jitter does not need to be unpredictable to an adversary.
+
+The coupling is also small enough that the abstraction isn't needed to keep our options open. We use
+exactly three things (`Rng::new`, `Rng::with_seed`, `rng.u64(range)`), `fastrand` appears in no
+public signature, and swapping it would be a change inside one file. Against that, a trait means a
+second type parameter on `Jittered` and `DecorrelatedBackoff` forever.
+
+What we do owe users is honesty about what a seed guarantees: the same delays within a build, not
+across `fastrand` versions. Every seeded entry point says so, and this crate's own tests assert
+properties rather than golden values for that reason.
+
+**2. One kind of jitter, with no mode to choose.**
+`Jittered<B>` applies full jitter: every delay becomes a uniform random value in `0 ..= delay`.
+There is no `Jitter` enum and no argument to `jittered()`.
+
+We shipped `Full` and `Equal` first, mirroring the AWS post that names them. Then we read the post's
+own conclusion, which is that equal jitter "does slightly more work than Full Jitter, and takes much
+longer", and simulated both against mettle's actual implementations under a contended resource.
+Equal lost to full on both total calls and completion time in every configuration we tried, at
+several client counts and base delays. AWS's own SDK agrees: full jitter is the default for its
+STANDARD and ADAPTIVE retry modes, and equal survives only on the LEGACY throttling path.
+
+An option nobody should pick is not an option, it's a trap, and the name "equal" reads as the safe
+middle choice, which is exactly backwards. So it's gone, and with one mode left the enum was pure
+ceremony.
+
+The cost is the extension point. `Jitter` was `#[non_exhaustive]`, so adding a fourth mode later
+would have been free; now it would need a separate `jittered_with(mode)` method. We accept that:
+the literature defines three modes, we ship the one that measures best, and decorrelated cannot be
+a mode at all (see below).
+
+**2b. Decorrelated jitter is its own `Backoff`, not a mode.**
+A mode is a per-delay function: hand it a delay, get a randomized delay. Decorrelated doesn't fit
+that shape. Its next delay is drawn from a range set by the delay it actually drew last time
+(`next = min(cap, rand(base, prev * 3))`), so the randomness is inside the recurrence and there's
+no deterministic sequence underneath to wrap. It ships as `DecorrelatedBackoff`.
+
+It is also the answer for anyone who wants a floor, which is what `Equal` was reaching for and
+getting wrong: decorrelated never draws below `base`. The trade is real and documented, since a
+floor also means never retrying sooner than `base`.
+
+**3. Randomized strategies are not `Clone`.**
+`ExponentialBackoff` is `Clone` and cloning it is the right thing to do: you get the same
+deterministic sequence, which is what you asked for. Cloning `Jittered` or `DecorrelatedBackoff`
+copies the RNG state, so every copy replays identical delays. A user who builds one and clones it
+per request puts every concurrent request on that host into lockstep, and nothing warns them.
+That's the herd again, one host at a time.
+
+We considered a manual `Clone` that reseeds, and rejected it: a `Clone` that doesn't clone breaks
+the trait's contract and would silently break seeded tests. So the randomized types just aren't
+`Clone`. Keep the config, which is `Clone`, and build a fresh strategy from it.
+
+This direction is also the reversible one. Adding `Clone` later is a minor version; taking it away
+is a breaking change.
+
+## Consequences
+
+Jitter is available to everyone with no feature flag to discover, at the cost of one small
+dependency. The `Backoff` trait now has strategies that behave differently under `Clone`, which
+the trait docs have to explain rather than leaving to be discovered. Anyone reusing a randomized
+policy across calls has to hold the config rather than the strategy, which is one extra line and
+the only shape that's actually correct.
+
+Every randomized path is seedable, so none of this costs us deterministic tests. Making that
+seeding pleasant to reach is ADR005.

docs/adr/ADR005.md

@@ -0,0 +1,71 @@
+# ADR005: Making the tested path as usable as the production path
+
+**Status:** Accepted
+
+## Context
+
+mettle's headline claim is that you can test a resilience policy exactly, with no sleeping and no
+real clock. Shipping jitter put that claim under load for the first time, because a randomized
+policy is only testable if you can pin the randomness *and* still read back what happened.
+
+Two things turned out to be wrong, and neither was a bug in the sense a test would catch. Both were
+found the same way: by writing the API from outside the crate, in a separate crate with a path
+dependency, doing what someone who had only read the README would do. That took about ten minutes
+and found two problems that had been there since 0.1.0.
+
+## Decisions
+
+**1. The readable path and the deterministic path are the same shape.**
+`jittered()` seeds from entropy, which is what production wants. `jittered_with_seed(seed)` fixes
+the seed, which is what a test wants.
+
+The seeded constructor already existed as `Jittered::with_seed(inner, seed)`, so the combinator
+looks redundant. It isn't. Without it, turning on jitter meant a production line reading
+`ExponentialBackoff::default().jittered()` and a test line reading
+`Jittered::with_seed(ExponentialBackoff::default(), 42)`. Those are different constructions, so a
+test no longer visibly exercised the thing it was testing. For a crate that sells deterministic
+testing, the deterministic path being the awkward one is the wrong way round.
+
+Seeding in production is a mistake, since a fleet seeded identically retries in lockstep, so the
+method's docs say so outright rather than leaving it implied.
+
+**2. `Clock` and `Now` are implemented for `&C` and `Arc<C>`.**
+Both traits are taken by value. `retry(..).clock(c)` owns its clock, and a `CircuitBreaker` owns its
+time source for its whole life. Without a reference impl a test can hand over its mock and then
+never read it back, which defeats the entire point of injecting time. Passing `&mock` was a compile
+error; the only way through was to give the mock interior sharing so it could be cloned.
+
+`&C` covers the case where the retry doesn't outlive the clock. `Arc<C>` covers the case where it
+does, which is the usual shape for a breaker and for anything spawned.
+
+**3. Before a release, use the crate from outside it.**
+This is the decision that generalises, so it is written down as one.
+
+We missed the reference-impl gap for two releases because our own tests worked around it on day
+one. Every mock in this repo wraps its state in `Arc<Mutex<..>>` and gets passed by `.clone()`, even
+in single-threaded blocking tests where nothing is shared. Written once, copied everywhere, and from
+the inside it looked like "how you write a mock clock here" rather than like a tax. Nobody notices a
+papercut they have already bandaged.
+
+A test suite that lives inside the crate inherits its own solutions. It can be thorough and
+mutation-checked and still tell you nothing about whether the front door opens. So the release
+checklist includes building a throwaway crate against a path dependency and writing the code a
+first-time reader would write.
+
+## Consequences
+
+A mock clock is now an ordinary struct with a `Cell` or a `RefCell`, not something that needs
+interior sharing before it can be passed at all. `examples/breaker.rs` lost its `Rc<Cell<..>>` for
+exactly this reason, which is the clearest evidence the friction was real: our own example was
+working around it.
+
+Decision 2 is the one part of 0.3.0 that can break an existing build. Someone who hit the same
+papercut and wrote `impl Clock for &TheirClock` themselves now collides with ours, and the build
+fails with `E0119: conflicting implementations`. We shipped it anyway: the affected population is
+small, the failure is a compile error rather than silent misbehaviour, and the fix is deleting a
+line this release makes redundant. `cargo-semver-checks` does not flag added impls, so nothing in CI
+would have caught it and the CHANGELOG says it out loud instead.
+
+Decision 3 costs a few minutes per release and has no automation behind it. The cheap way to make it
+structural later is a `tests/` directory, since integration tests compile against the crate's public
+face and would have caught this without anyone thinking to look.

docs/adr/README.md

@@ -9,3 +9,5 @@ changes, edit its file and say what changed and why.
 | [001](ADR001.md) | Sans-IO core, hand-written async, one crate | Accepted |
 | [002](ADR002.md) | Public API, open traits, one validated `Backoff` | Accepted |
 | [003](ADR003.md) | Dependencies and Cargo features | Accepted |
+| [004](ADR004.md) | Jitter, a fourth dependency, and `Clone` | Accepted |
+| [005](ADR005.md) | Making the tested path as usable as the production path | Accepted |

examples/retry.rs

@@ -11,9 +11,15 @@
 //! - `.max_elapsed(d)` gives up once the next wait would push total time past `d`
 //! - `.clock(clock)` supplies the time source (default: Tokio)
 //!
+//! The default schedule is deterministic, which means a fleet of clients that failed together
+//! retries together. `.jittered()` spreads them out; see step 4.
+//!
 //! Run with: `cargo run --example retry`
 
-use mettle::{ExponentialBackoff, ExponentialBackoffConfig, retry};
+use mettle::{
+    Backoff, DecorrelatedBackoff, DecorrelatedBackoffConfig, ExponentialBackoff,
+    ExponentialBackoffConfig, retry,
+};
 use std::sync::atomic::{AtomicUsize, Ordering::SeqCst};
 use std::time::Duration;
 
@@ -24,7 +30,7 @@ enum FetchError {
 }
 
 #[tokio::main]
-async fn main() {
+async fn main() -> Result<(), mettle::BackoffConfigError> {
     // 1) Simplest form: pass the operation, take every default.
     let result = retry(flaky_fetch).await;
     println!("1. defaults:  {result:?}"); // Ok("user data"), succeeds on attempt 3
@@ -42,6 +48,27 @@ async fn main() {
         .when(|e| matches!(e, FetchError::Timeout))
         .await;
     println!("3. full:      {result:?}"); // Ok("user data"), succeeds on attempt 3
+
+    // 4) Jitter, so a fleet doesn't retry in lockstep. `.jittered()` randomizes each delay into
+    //    `0 ..= delay`; `DecorrelatedBackoff` instead draws each delay from the previous one and
+    //    never goes below `base`. Both are opt-in: the default above stays deterministic.
+    let result = retry(flaky_fetch)
+        .backoff(fast_backoff().jittered())
+        .when(|e| matches!(e, FetchError::Timeout))
+        .await;
+    println!("4. jittered:  {result:?}"); // same outcome, unpredictable delays
+
+    let result = retry(flaky_fetch)
+        .backoff(DecorrelatedBackoff::new(DecorrelatedBackoffConfig {
+            base: Duration::from_millis(20),
+            max_retries: 5,
+            max_delay: Duration::from_secs(1),
+        })?)
+        .when(|e| matches!(e, FetchError::Timeout))
+        .await;
+    println!("5. decorrel.: {result:?}"); // every delay at least 20ms
+
+    Ok(())
 }
 
 /// A flaky call: fails with `Timeout` twice, then succeeds. It resets after each success, so it