Skip to content

fix(test): poll the subtree re-resolution instead of sleeping once and hoping - #67

Merged
wshallwshall merged 1 commit into
mainfrom
claude/connscale-probe-poll
Jul 30, 2026
Merged

fix(test): poll the subtree re-resolution instead of sleeping once and hoping#67
wshallwshall merged 1 commit into
mainfrom
claude/connscale-probe-poll

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

test_subtree_re_resolution_picks_up_a_late_spawned_child failed on windows-2025 with AssertionError: assert 8204 in []twice in the same job (pytest-rerunfailures already retried), while windows-2022 and ubuntu passed the identical commit.

The empty list is the tell

It isn't "the walk ran and missed the child". It's "no walk succeeded at all".

On Windows _descendants_windows shells out to Get-CimInstance Win32_Process with a 5 s timeout. Under load that times out and it returns None — deliberately, to signal ERRORED rather than "no descendants". _resolve_pids then does exactly what its docstring promises: returns root-only for that tick, does not cache, and leaves _pids unresolved "so the next tick retries".

The test gave it one tick. So it asserted a stricter property than production requires — that a single enumeration always succeeds — on a probe explicitly written to tolerate one that doesn't. time.sleep(1.0) then one sample_proc() is the same sleep-and-hope shape df607719 removed from the /ws/stats tests hours earlier.

The fix

Poll sample_proc() until the child appears or a deadline expires — which is what the production caller does, tick after tick.

The deadline is derived, not hardcoded: max(30.0, 6 * _PROBE_TIMEOUT_S), imported from the probe. Raising the probe's per-walk timeout therefore can't silently leave this deadline too short to fit even one attempt.

The bigger fix is the diagnosis

The old assertion collapsed two unrelated failures into one useless message. They now report separately, and they mean opposite things:

  • "the process-table walk never succeeded in 30s (N attempts)" — the probe couldn't measure at all. An environment problem; the test could not assess re-resolution either way.
  • "the walk SUCCEEDED but did not include the late-spawned child <pid>" — the A3 regression this test exists to catch: a subtree resolved once pins the sampler to an idle parent, so a sharded engine's serve --shard workers are never counted.

Verification

Each regression injected into probe.py, checking both the failure and its message:

injected caught message
enumeration always errors (None) "never succeeded"
enumeration succeeds, no descendants "did not include the late-spawned child"

probe.py verified byte-identical to HEAD afterwards — no production code changed.

Stable rather than merely passing: 5 consecutive runs, ~2.1 s each — it exits as soon as the child appears, so it's no slower than the fixed 1 s sleep it replaces. Full module 12 passed; with test_connscale_smoke, 14.

The 24 mypy errors in this file are pre-existing (main's unmodified copy reports the same 24) and outside CI's mypy scope, which covers messagefoundry + messagefoundry_webconsole, not tests/.

…s contract is to retry

test_subtree_re_resolution_picks_up_a_late_spawned_child failed on windows-2025 with
`AssertionError: assert 8204 in []` -- twice in the same job, while windows-2022 and
ubuntu passed the identical commit.

The empty list is the tell. It is not "the walk ran and missed the child"; it is "no
walk succeeded at all". On Windows `_descendants_windows` shells out to
`Get-CimInstance Win32_Process` with a 5 s timeout; under load that times out and it
returns None -- deliberately, to signal ERRORED rather than "no descendants". The
caller then does exactly what its docstring says: returns root-only for that tick,
does NOT cache, and leaves `_pids` unresolved "so the next tick retries".

The test gave it one tick. So it asserted a STRICTER property than production requires:
that a single enumeration always succeeds, on a machine where the probe is explicitly
written to tolerate one that does not. `time.sleep(1.0)` then one `sample_proc()` is
the same sleep-and-hope shape df60771 removed from the /ws/stats tests hours earlier.

Now it polls `sample_proc()` until the child appears or a deadline expires -- which is
what the production caller does, tick after tick.

The deadline is DERIVED, not hardcoded: `max(30.0, 6 * _PROBE_TIMEOUT_S)`, imported
from the probe. Raising the probe's per-walk timeout therefore cannot silently leave
this deadline too short to fit even one attempt. Six walks' worth means a couple of
enumerations can time out entirely and the test still reaches a verdict, which is
precisely the transient-failure tolerance `_resolve_pids` provides.

The bigger fix is the diagnosis. The old assertion collapsed two unrelated failures
into one useless message. They are now separate, and they mean opposite things:

  * "the process-table walk never succeeded in 30s (N attempts)" -- the probe could not
    measure at all. An environment/probe problem; the test could not assess
    re-resolution either way.
  * "the walk SUCCEEDED but did not include the late-spawned child <pid>" -- the A3
    regression this test exists to catch: a subtree resolved once pins the sampler to
    an idle parent, so a sharded engine's `serve --shard` workers are never counted.

Proven by injecting each regression into probe.py and checking BOTH the failure and its
message:

  enumeration always errors (returns None)        -> caught, "never succeeded"
  enumeration succeeds, finds no descendants      -> caught, "did not include the
                                                     late-spawned child"

probe.py verified byte-identical to HEAD afterwards. No production code changed.

Also stable rather than merely passing: 5 consecutive runs of the fixed test, ~2.1 s
each -- it exits as soon as the child appears, so it is no slower than the fixed 1 s
sleep it replaces. Full module 12 passed; with test_connscale_smoke, 14.

The 24 mypy errors in this file are pre-existing (main's unmodified copy reports the
same 24) and outside CI's mypy scope, which covers messagefoundry +
messagefoundry_webconsole, not tests/.
@wshallwshall
wshallwshall merged commit a616220 into main Jul 30, 2026
32 checks passed
@wshallwshall
wshallwshall deleted the claude/connscale-probe-poll branch July 30, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant