Skip to content

flaky: a_genuine_wrong_p_restricted_stays_removed races the CLOSED under suite load #901

Description

@orveth

Summary

seller_node::run::tests::a_genuine_wrong_p_restricted_stays_removed goes red intermittently with the tree unchanged. It is a second intermittent in the money-path suite, distinct from #894.

Found while verifying #897 (PR #900). Not caused by that change: it also passes and fails on trees that do not contain it.

The test says so itself

crates/maxplayer-core/src/seller_node/run.rs:12085, verbatim, already in the tree:

// The SDK deleted it, and nothing in the client puts it back. Waited for rather than read
// once: the relay recording the REQ and the SDK processing the CLOSED are different sides of
// the socket, so a bare read races the removal under load — which is a flaky test, not a
// finding.

The author saw this class and mitigated it with a poll loop. The mitigation is not sufficient under full-suite load.

The failure shape

The assertion is on a timeout expiring, not on a wrong value:

let removed = tokio::time::timeout(FIXTURE_WAIT, async {
    loop {
        if !relay.subscriptions().await.keys().any(|id| id.to_string() == foreign_id) {
            return;
        }
        tokio::time::sleep(Duration::from_millis(25)).await;
    }
})
.await
.is_ok();
assert!(removed, "`restricted:` must remain permanent-class: the subscription stays removed");

So a red says the removal had not been observed within FIXTURE_WAIT. It does not say the subscription came back, and it does not distinguish "not yet" from "never" — which is why the red is not, on its own, a finding about restricted: handling.

Observations, with the tree named for each

All rows are cargo test -p maxplayer-core --lib --features acp,gateway,git-delivery,wallet -j 6 unless stated.

Red and green on the same tree, and green on a tree predating the change entirely. Isolation never reproduced it; only full-suite parallel load did.

Why this matters beyond one red

A test that can go red without a code change can go green without one. That cuts both ways, and the second direction is the dangerous one: a genuine regression in restricted: handling would be indistinguishable from this noise, and the natural response — rerun it — produces a green that certifies nothing.

Retry is not a fix. What is owed before stable promotion is a deterministic readiness signal: wait on the SDK having processed the CLOSED, rather than polling the relay's subscription map and hoping the other side of the socket got there first. Until then every green on this test is weaker than it looks.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions