fix(test): poll the subtree re-resolution instead of sleeping once and hoping - #67
Merged
Merged
Conversation
…s contract is to retry test_subtree_re_resolution_picks_up_a_late_spawned_child failed on windows-2025 with `AssertionError: assert 8204 in []` -- twice in the same job, while windows-2022 and ubuntu passed the identical commit. The empty list is the tell. It is not "the walk ran and missed the child"; it is "no walk succeeded at all". On Windows `_descendants_windows` shells out to `Get-CimInstance Win32_Process` with a 5 s timeout; under load that times out and it returns None -- deliberately, to signal ERRORED rather than "no descendants". The caller then does exactly what its docstring says: returns root-only for that tick, does NOT cache, and leaves `_pids` unresolved "so the next tick retries". The test gave it one tick. So it asserted a STRICTER property than production requires: that a single enumeration always succeeds, on a machine where the probe is explicitly written to tolerate one that does not. `time.sleep(1.0)` then one `sample_proc()` is the same sleep-and-hope shape df60771 removed from the /ws/stats tests hours earlier. Now it polls `sample_proc()` until the child appears or a deadline expires -- which is what the production caller does, tick after tick. The deadline is DERIVED, not hardcoded: `max(30.0, 6 * _PROBE_TIMEOUT_S)`, imported from the probe. Raising the probe's per-walk timeout therefore cannot silently leave this deadline too short to fit even one attempt. Six walks' worth means a couple of enumerations can time out entirely and the test still reaches a verdict, which is precisely the transient-failure tolerance `_resolve_pids` provides. The bigger fix is the diagnosis. The old assertion collapsed two unrelated failures into one useless message. They are now separate, and they mean opposite things: * "the process-table walk never succeeded in 30s (N attempts)" -- the probe could not measure at all. An environment/probe problem; the test could not assess re-resolution either way. * "the walk SUCCEEDED but did not include the late-spawned child <pid>" -- the A3 regression this test exists to catch: a subtree resolved once pins the sampler to an idle parent, so a sharded engine's `serve --shard` workers are never counted. Proven by injecting each regression into probe.py and checking BOTH the failure and its message: enumeration always errors (returns None) -> caught, "never succeeded" enumeration succeeds, finds no descendants -> caught, "did not include the late-spawned child" probe.py verified byte-identical to HEAD afterwards. No production code changed. Also stable rather than merely passing: 5 consecutive runs of the fixed test, ~2.1 s each -- it exits as soon as the child appears, so it is no slower than the fixed 1 s sleep it replaces. Full module 12 passed; with test_connscale_smoke, 14. The 24 mypy errors in this file are pre-existing (main's unmodified copy reports the same 24) and outside CI's mypy scope, which covers messagefoundry + messagefoundry_webconsole, not tests/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_subtree_re_resolution_picks_up_a_late_spawned_childfailed on windows-2025 withAssertionError: assert 8204 in []— twice in the same job (pytest-rerunfailures already retried), while windows-2022 and ubuntu passed the identical commit.The empty list is the tell
It isn't "the walk ran and missed the child". It's "no walk succeeded at all".
On Windows
_descendants_windowsshells out toGet-CimInstance Win32_Processwith a 5 s timeout. Under load that times out and it returnsNone— deliberately, to signal ERRORED rather than "no descendants"._resolve_pidsthen does exactly what its docstring promises: returns root-only for that tick, does not cache, and leaves_pidsunresolved "so the next tick retries".The test gave it one tick. So it asserted a stricter property than production requires — that a single enumeration always succeeds — on a probe explicitly written to tolerate one that doesn't.
time.sleep(1.0)then onesample_proc()is the same sleep-and-hope shape df607719 removed from the/ws/statstests hours earlier.The fix
Poll
sample_proc()until the child appears or a deadline expires — which is what the production caller does, tick after tick.The deadline is derived, not hardcoded:
max(30.0, 6 * _PROBE_TIMEOUT_S), imported from the probe. Raising the probe's per-walk timeout therefore can't silently leave this deadline too short to fit even one attempt.The bigger fix is the diagnosis
The old assertion collapsed two unrelated failures into one useless message. They now report separately, and they mean opposite things:
<pid>" — the A3 regression this test exists to catch: a subtree resolved once pins the sampler to an idle parent, so a sharded engine'sserve --shardworkers are never counted.Verification
Each regression injected into
probe.py, checking both the failure and its message:None)probe.pyverified byte-identical to HEAD afterwards — no production code changed.Stable rather than merely passing: 5 consecutive runs, ~2.1 s each — it exits as soon as the child appears, so it's no slower than the fixed 1 s sleep it replaces. Full module 12 passed; with
test_connscale_smoke, 14.The 24 mypy errors in this file are pre-existing (main's unmodified copy reports the same 24) and outside CI's mypy scope, which covers
messagefoundry+messagefoundry_webconsole, nottests/.