fix(test): waitForCondition can't enforce its budget, so slow polls fail opaquely

[EricAndrechek]

waitForCondition in tests/e2e/sdk/helpers.ts cannot enforce the budget it advertises. One slow fn() overruns it arbitrarily, so a latency spike surfaces as an opaque vitest Test timed out in 20000ms instead of a legible Condition not met after 6000ms.

This is a diagnosability defect, not a correctness one. The suite still passes on a machine with spare CPU, and on CI. What it costs is debugging time when something is slow — the failure names nothing, so the reader's first instinct is to suspect their own diff.

Re-scoped 2026-08-11. This issue was originally filed as "local make ci blocked" during #446. That framing was wrong: the frequency of those failures was environmental (CPU contention on one developer's machine — see "What this is not" below), and no repo change would have prevented them. The residue is the budget bug described here, which is machine-independent and worth fixing on its own merits. Priority dropped P2 → P3 to match.

The bug

export async function waitForCondition(fn, timeoutMs = 10_000, intervalMs = 250) {
  const start = Date.now();
  while (Date.now() - start < timeoutMs) {
    if (await fn()) return;          // ← unbounded: the deadline is checked
                                     //   BEFORE the call, never during it
    await new Promise((r) => setTimeout(r, intervalMs));
  }
  throw new Error(`Condition not met after ${timeoutMs}ms`);
}

The loop condition guards entry, not the call. Nothing interrupts fn() once it starts, so the effective ceiling is timeoutMs + (duration of the last fn()), unbounded.

Instrumented evidence from a real run:

poll#8 -> false (fn took   246ms, elapsed  4506ms)
poll#9 -> true  (fn took 12716ms, elapsed 17472ms)   # 6s budget, returned at 17.5s
poll#8 -> true  (fn took 23259ms, elapsed 28019ms)   # 10s budget, returned at 28.0s

A 10s budget ran 28s. Because that overshoots the caller's vitest testTimeout, the throw never happens — vitest kills the test first, and the resulting message says nothing about which condition failed or how long the poll actually waited.

chQuery (same file) has no timeout either, so it can be the slow call indefinitely.

Suggested fix

1. Race fn() against the remaining budget rather than checking the clock only between polls, so the documented Condition not met after <n>ms actually fires at <n>.
2. Give chQuery an AbortSignal.timeout(...) so a single stalled request fails as itself rather than as a mystery.

Neither makes anything faster. Both convert a silent wall into a failure that names its own cause.

Why it's worth doing anyway

The e2e suite's timing assumptions are tight enough that a merely busy machine can brush them, independent of any pathological process:

• The 500-row test's own comment records 5104ms observed on APFS for the JetStream publish-fsync, against a 2500ms ack assertion elsewhere in the same file.
• A healthy control run measured that fsync at 3763ms — real but not generous headroom.
• CI infrastructure isn't immune to spikes either; fix(ci): cache Go modules once, not once per compile flavor #446's runs alone hit an artifact-service 403 and a coverage-directory collision.

When any of those fire, the current code turns a five-second diagnosis into a long one.

What this is not

SUPERSEDED 2026-08-11 — this section's conclusion was wrong. The failures were not
environmental. Root cause: undici 8.8.0-8.9.0 stalls for seconds before writing a request onto an
idle pooled socket (nodejs/undici#5600, fixed in 8.10.0); Node 26 bundles 8.9.0 while .nvmrc
pins Node 22 for CI. Local make test-e2e went 3-fail-in-5 to 0-fail-in-5 with a one-line change.
The measurements and reasoning below are left intact for the record, but see
this comment first.

Recorded so the next reader doesn't repeat the investigation. The original burst of failures was CPU contention on one developer's machine, not a repo defect:

Evidence	Reading
Second developer, same hardware/OrbStack, fresh clone: suite green in 111.6s, the victim test at 5111ms	Code fine
GitHub CI green — including 4 of 5 times the same tree failed locally	Code fine
Idle ClickHouse: 1500 sequential queries p99 2ms; 600 fresh TCP connections p99 3ms	Docker/network fine
Same queries during a loaded run: 2.5–17.8s, 14 stalls totalling 89.4s	Contention
Load average 8.54 mean on a 14-core machine, one process averaging 82% of a core (203% peak)	Cause

Ruled out with measurements, not reasoning: stale/reused containers (fresh testcontainer per run), Docker/OrbStack health, host disk pressure (295 GB free), the ingest path (single-row insert acked in 10ms; server logged it 519ms after the preceding batch), connection churn, and TCP SYN drops.

The moving target — a different test failing each run — is the signature of contention against the tightest-bounded assertion, not of a specific broken test. batching.test.ts was the most frequent victim only because its 20s ceiling is the tightest.

Reproduction

Not reliably reproducible on an idle machine, by design — that's the point. To see the overrun directly, instrument waitForCondition to log fn() duration per poll and run the suite under artificial CPU load.

[coderabbitai]

🔗 Related PRs

#125 - fix(boot): non-fatal schema discovery, /health 503 with diagnostic [merged]
#174 - refactor(table names): handle unsafe table names [closed]
#177 - feat: caching for local table/scope invalidation [closed]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[EricAndrechek]

Correction: this was not environmental. Root cause found and fixed.

The "What this is not" section above concluded the local failures were CPU contention on one
developer's machine, and that no repo change would have prevented them. That was wrong, and this
comment supersedes it.

The budget bug this issue describes is real and is fixed. But it was also the reason the real cause
stayed hidden: waitForCondition couldn't report how many times it polled or how slow each
poll was, so a stalled connection and a busy machine produced identical output. Once the fixed
helper started printing N poll(s), slowest Xms, the two separated immediately.

Root cause

undici 8.8.0–8.9.0 stalls for seconds before writing a request onto a socket that has been idle a
few seconds. Upstream bug, already reported and already fixed: nodejs/undici#5600 — a scheduling
regression in scheduleIdleSocketValidation() (lib/dispatcher/client-h1.js), which changed from an
unref'd setTimeout(…, 0) to an unref'd setImmediate().

Bisected with Node held constant at 22, varying only the undici version:

undici	max latency, 6s idle gaps
7.29.0 – 8.7.0	clean (13–30 ms)
8.8.0	broken — 2708 ms
8.9.0	broken — 7164 ms
8.10.0	clean (13 ms) — fixed

diagnostics_channel shows exactly where the time goes — the socket is reused (no
beforeConnect/connected) and the request simply isn't written:

round 1: total=23989ms  request:create+1ms  client:sendHeaders+23987ms  SERVER-received+23988ms

The server received it 1 ms after sendHeaders. node:http over the same origin, same gaps, same
process is 1 ms every time.

Why it looked local-only

	node	undici	affected
CI	22 (.nvmrc via setup-env)	6.28.0	no
This developer	26.7.0 (Homebrew)	8.9.0	yes
Second developer	not 26	—	no

.nvmrc pins 22 and CI reads it via node-version-file, so CI has never run an affected
runtime. Locally .nvmrc is inert without a version manager, and engines: ">=22" is a floor that
26.7.0 satisfies — so the drift was silent. The second developer's control run was also on a
different tree (their output shows vitest 4.1.8; main pins 4.1.10), so it never was the clean
comparison it appeared to be.

Why this suite in particular

The suite has multi-second idle gaps by construction: defaultMaxWait = 5s
(internal/ingest/worker.go) sits between every write and the first poll of its visibility wait. So
every visibility wait in the suite lands in the triggering window.

Ruled out by measurement, not reasoning

ClickHouse (p99 ≤ 2.5 ms measured during live failing runs, on the real tables with the real query
shapes), Docker/OrbStack (reproduces against a bare host-loopback server with no container in the
path), DNS, the libuv threadpool, GC, event-loop blocking, machine load, and CPU contention.

Result

connection: close on chQuery, which stops it inheriting an idle pooled socket:

	before	after
local make test-e2e	2 pass / 3 fail	5 pass / 0 fail
vitest duration	128.7 – 137.1 s	115.7 – 124.5 s

Faster as well as green — removing the stalls shortens every run, which is the mechanism's own
prediction rather than just an absence of failures.

Two process notes

• An orphaned wavehouse-cov (left by any killed run) shares tmp/data and tmp/wavehouse-cov.log
  with the next run and corrupts it — it presents as a dozen unrelated tests failing to see their
  rows, in a log that blames a ClickHouse container that no longer exists. I hit this myself and
  briefly believed a false root cause from the contaminated log. The orchestrator now refuses to
  start and names the PID.
• The e2e setup banner now prints the Node/undici version and warns when the local major differs
  from .nvmrc, so the next version-specific failure is attributable in seconds rather than days.