Repository navigation
mesh-worker-reads.integration timed out once under heavy load #393
Description
Observed once during the 2026-10-02 delivery session and not since: agent-comms mesh-worker-reads.integration timed out at 40s in one pre-push full run under heavy load, and passed on the retry and every later run. The machine was running six agent-comms MCP servers that leaked their per-session frame log (wire-mesh-core before 4.0.1, fixed in ExaDev/wire-mesh#335), with load average 240 to 400 and major GC pinning cores, so the likely cause is a load-induced wall-clock timeout, not a code defect. Filed so the observation is not forgotten; reopen it if it recurs on 4.0.1 or later, where this explanation would no longer hold.
The web-console e2e failure from the same session is tracked in ExaDev/wire-mesh#338, which is reported to have failed again after #335 merged, so the load explanation does not cover it.
Activity
Mearman commentedon Oct 2, 2026
Update: the wire-mesh notices-offline e2e failed two more times on 2026-10-02 (on the licence PR ExaDev/wire-mesh#337, whose diff cannot affect it), so the load-only explanation for that one no longer holds. It now has its own tracking issue with the failure timestamps and the analysis: ExaDev/wire-mesh#338. The mesh-worker-reads timeout remains a one-off consistent with load.
Mearman commentedon Oct 2, 2026
Add another occurrence, different test: coordinator-failover-hub.integration timed out on PR #395's first Test run (2026-10-02 16:03 UTC), the same test that timed out once on PR #317 back on 2026-09-28 and then passed. PR #395 bumps wire-mesh-core 4.0.1 to 4.0.2, which is licence-only commits, so no behaviour change can explain it. Rerunning the failed job; if it passes again, this test joins notices-offline as a recurring intermittent with a load-shaped signature.
Mearman commentedon Oct 2, 2026
Mechanism found for the coordinator-failover-hub recurrences (PR #317 in September, PR #395 twice today, 16:03 and 16:21 UTC): the test asserts a survivor rebinds the real well-known coordinator port 19876, and under vitest's default parallel worker pool in CI, any other worker holding or racing that port at that moment makes the rebind lose. Three consecutive local runs of the same test on PR #395's exact head (wire-mesh-core 4.0.2) pass, and 4.0.2 is licence-only commits, so the diff is exonerated; it is port contention inside the suite, the same family issues #364 and #371 fixed for other files by taking freeLocalPort. The remaining fix is making this test's coordinator port injectable the same way; tracked here rather than half-done on an unrelated dependency PR.
Mearman commentedon Oct 2, 2026
Correction to my previous comment: the port-contention mechanism is wrong. coordinator-failover-hub already takes every port from freeLocalPort() (lines 58 and 63); nothing binds 19876. The actual failing wait is 'the survivor rebound the vacated coordinator port; the survivor reported: []' with the survivor's report EMPTY, so the flake is in hub-mediated mesh visibility after a coordinator loss: the survivor rebinds but reports no peers. Same CI-only, timing-dependent signature as the others; local runs on the exact PR head pass 3/3. Leaving the mechanism open on this tracker rather than guessing again.
[-]Record: two one-off test failures during 2026-10-02, most likely the frame-log leak's GC pressure[/-][+]CI-only intermittent failures, including coordinator-failover-hub where the survivor reports no peers after a coordinator loss[/+][-]CI-only intermittent failures, including coordinator-failover-hub where the survivor reports no peers after a coordinator loss[/-][+]mesh-worker-reads.integration timed out once under heavy load[/+]