test: run_warm_soak reports a client timeout as an #8163 residual — it counts verifier exits instead of classifying the failure #8286

My bug, in the arm I added in #8215. Filing it before fixing it because it is on main now and #8163 is closed, so the next person whose warm soak trips a timeout will see the words #8163 residual and reasonably conclude it has regressed.

What it does wrong

run_warm_soak in tests/release/packages/next-app-route/fixture.sh treats any non-zero verify.mjs exit as a failure and reports:

FAIL next-app-route — warm soak: N failing pass(es) [...], M host TypeError(s) — #8163 residual

It counts exits; it does not classify them. Two very different things produce a non-zero exit:

• The residual — the server returns an empty body. Signature: Unexpected end of JSON input in the verifier, plus TypeError: value is not a function and E180 failed to pipe response in the host log.
• A client timeout — UND_ERR_HEADERS_TIMEOUT. The server was merely slow; the host log is clean.

In the saturated regime the second happens on its own (see #8213: one pass ran 644 copying minors against a steady 84, and 8 of its 21 concurrent requests timed out). So the arm manufactures false residual reports precisely where it is most likely to be run.

Observed

In the #8163 close-condition run at 211/300 passes: 8 failing passes, all UND_ERR_HEADERS_TIMEOUT, and across 6,267 copying minors the three residual discriminators were 0 / 0 / 0. The arm would have printed #8163 residual for a run with no residual evidence whatsoever.

It is worth being precise about the direction: this arm was written to stop a green run implying more than it proves, and its actual first failure was the mirror image — a red verdict that isn't true. A gate that cannot be trusted when it goes red is as useless as one that cannot go red at all, and I did not think about that direction when I wrote it.

Fix (mine, in progress)

Classify rather than count:

• residual → empty body and a host-side TypeError: value is not a function in the same pass. Fail with #8163 residual.
• timeout with a clean host log → report separately as a latency/stall observation, with the pass's copying-minor count next to it, since that is the interesting quantity ([Next.js/dylib] Warm App Route server retains ~1,100 async box cells per request: 113 KB/req live Rust heap, minor-GC pause grows 59 ms -> 0.5 s #8213). Whether it should fail the arm is a policy question — it is a real user-visible failure, but not the thing this arm exists to detect — so it should at minimum be named differently.
• anything else → report verbatim rather than assuming a cause.

Also worth adding while there: emit the per-pass minor counts, since the run above showed a 7.7× single-pass storm that a total would have hidden entirely.

Do not fix this by widening the grep. The lesson is that the arm asserted a cause from an exit code, and the fix is to make it read the evidence it already has in the host log.