Repository navigation
test: run the serial backend group beside the other suites - #170
Conversation
The backend files that take Setup coordination ports are about two thirds of the portable Linux step and mostly wait. Run them in one lane next to a second lane holding both typechecks, the parallel backend group and the browser-model and automation tests, so the run takes as long as the longer lane. The serial group stays a single --test-concurrency=1 invocation that nothing joins. Add test:fast and test:all scripts that share the same plan, print each step's duration, and stop every lane at the first failure.
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Incremental static review of Files Reviewed (1 file)
Previous Review Summaries (10 snapshots, latest commit 05f89c6)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit 05f89c6)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (1 file)
Previous review (commit 985805e)Status: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (4 files)
Previous review (commit 4ada204)Status: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (4 files)
Previous review (commit 3c73f89)Status: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (4 files)
Previous review (commit 11a9a55)Status: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (12 files)
Previous review (commit a14b472)Status: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (7 files)
Previous review (commit 81953ce)Status: 3 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (5 files)
Previous review (commit fa38feb)Status: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (5 files)
Previous review (commit 6bc4230)Status: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (7 files)
Previous review (commit 68b055b)Status: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (11 files)
Reviewed by gpt-6.2-sol · Input: 21 · Output: 1.5K · Cached: 216.4K |
Each test file and each managed-step request worker compiles the same modules again. The compile cache keys compiled code by source content, so it changes no test input; it roughly halves the largest serial-group file and shortens the typechecks.
A signal to the runner did not reach the steps, which run in their own process groups, so Ctrl-C left them holding Setup ports. Stop every step on SIGINT or SIGTERM and print what the running steps had written. Compile cache entries are keyed by file name and the fixtures run from fresh temporary folders, so a kept cache grew by about 30 MB a run. Empty it at the start of each run; the speed-up comes from sharing within a run, which is unchanged. The earlier note that the cache shortens the typechecks was wrong: their times did not move. Describe the serial group as latency-bound rather than as waiting.
Runner: record every process group and kill them again once the steps settle and from an exit hook, so a crash or a closed pipe leaves none behind; handle SIGHUP and a closed output pipe like SIGINT and SIGTERM; treat a repeated signal, or a step that has not stopped within ten seconds, as a kill and abandon it even if a leftover process still holds its pipes; exit 128 plus the signal number when interrupted; do not detach steps on Windows and stop them with taskkill /T /F; settle each step once; report a step the runner did not stop as failed with its output; name the running steps about every minute; keep only the tail of a step's output; print one line when the compile cache is off. Plan: take a core count, so one to three cores are tested, and find the serial step by its lane. Fence plan-cleanup-service.test.ts when CP_BUILT_SERVICE=1: its built service takes Setup writer slots then, as a listen probe on the bundled service showed. Windows job: start each native file under Start-Process -Wait inside a thread job, so it is waited for with everything it started.
The secondary logon service creates one process at a time and refused the second Start-Process -Credential when both thread jobs started at once. Retry only that refusal, up to thirty times a second apart.
A leader that exits 0 while a same-group descendant keeps a pipe open let the grace period abandon the pipes and forget the group, so the descendant (and any Setup port it held) outlived the run, and an interrupt during the grace period did the same. Keep every group recorded whatever the leader's exit status, stop it (SIGTERM, then SIGKILL after a short bound) before runLanes returns, and signal the group of a leader that has already exited when the run is stopped. Skip the process-group and signal cases of the runner tests on Windows, and make every case stop the fixture processes it recorded, not only delete its files.
… or its tests signal
The runner kept the id of a passing step's group until both lanes finished, so a
reused id could be signalled by the closing sweep (and on Windows taskkill could
hit an exited PID). Sweep each step's group right after its leader exits and its
pipes close, retire a group the moment a probe finds it empty, re-probe before
every signal, never signal a retired group, and leave only unconfirmed groups to
the closing sweep and the exit hook. On Windows run taskkill only against a child
whose exit has not fired. Route every probe and signal through an injectable kill
function and refuse a pid that is not a whole number above 1, itself or its parent.
In the tests, write pid files atomically and have the cleanup accept only a safe
pid, so an empty file (Number('') is 0, the whole process group) is ignored.
When ps cannot run, spawnSync has no stdout and trimming it threw while the module loaded, so no test registered. Read the group with optional chaining and treat a failed lookup as unknown; cleanup then sends no group signal and signals only the validated pid itself.
…kill Name the platform on every mocked killTree call and give killTree an injectable command runner for the Windows path, so no call can shell out to a real taskkill. Add a mocked win32 call asserting the taskkill command and arguments.
Summary
Makes the portable test run finish sooner, locally and in CI, without removing or relaxing a check. In CI the Linux step goes from about 305 s to about 150 to 230 s, and the Windows step from about 320 s to about 220 to 280 s (full Windows set: about 1020 s to about 780 s). No test is removed, and every pass and skip count is unchanged.
Measured causes, from recent CI runs and local per-file timings:
--test-concurrency=1invocation (about 207 s of about 305 s), before the typechecks, the parallel backend group and the browser-model and automation tests. One file in it,managed-step-http.test.ts, is about 80 percent of it. It is latency-bound: one request after another, each starting a worker thread, plus a few deliberate deadline waits. It uses roughly one to two cores.apt-get updateand an install (7 to 29 s) for tools the image already has.Changes:
scripts/ci.tsruns the serial Setup-port group in one lane beside a second lane (both typechecks, the parallel backend group, the browser-model and automation tests). The serial group stays one--test-concurrency=1invocation that nothing else joins. The other lane has its worker count capped at cores minus two and runs undernice, so the serial group's timing assertions keep their headroom. The second build comparison, the SDK check and the package checks still run after all tests.NODE_OPTIONSthat wrapsnet.Server.prototype.listen(it follows worker threads and child processes, including ones given a hand-built environment, so the bundledbuild/service/main.jsis covered). Every non-serial step was run under it, with and withoutCP_BUILT_SERVICE=1afterbuild:all. Summary: no listen on a Setup port (25000 to 28999) from any of the 74 files in the parallel backend group or the browser and automation tests in the CI configuration; withCP_BUILT_SERVICE=1, one file,tests/backend/plan-cleanup-service.test.ts(its built-service restart test), takes writer slots (three service processes, two ports each), and the browser and automation tests still take none. That file is skipped in CI, but the plan now moves it into the serial group wheneverCP_BUILT_SERVICE=1is set (fencedWhenSetinscripts/ci-backend.ts), so no configuration puts a port-taking file beside the serial group. The probe was temporary and is not in the diff. My first probe, a source patch on the acquisition call, missed the bundled service; this one does not.bun run test:fast(typechecks, parallel backend group, browser-model and automation tests; skips the serial port-bound group) andbun run test:all(everything portable CI tests except packaging and the rebuild comparison). Both share one plan (scripts/test-plan.ts, which takes a core count so one to three cores are tested) with CI. The runner (scripts/test-lanes.ts) is POSIX-first, with a best-effort Windows stop path I could not run: it prints each step's output whole (the last 2 MB if noisier), names running steps about every minute, and lists durations. The first failure, SIGINT, SIGTERM, SIGHUP or a closed output pipe stops every lane; a repeated signal or a step that has not stopped in 10 s is killed and abandoned even if a leftover process holds its pipes; every group it started is killed again when the steps settle and when the runner exits or crashes; an interrupted run exits 128 plus the signal number. A step it stopped is reported as stopped, one that fails by itself as failed with its output. On Windows steps are not detached and are stopped withtaskkill /T /F(unit-tested for the command chosen, never run on Windows; the typecheck step's./node_modules/.bin/tscis a POSIX path). Documented in CONTRIBUTING.md.NODE_COMPILE_CACHE, under the ignorednode_modules/.cache/) for every step. It is checked against each source's hash, so a changed file is recompiled. Each run starts from an empty cache, because entries are keyed by file name and the fixtures run from fresh temporary folders, so a kept cache grew about 30 MB a run. Alone, the heaviest managed-step test went from 9.0 s to 4.8 s. It did not change the typecheck times.windows-platformandwindows-serviceas two processes at once, each in its own--test-concurrency=1runner and as the same unprivileged principal. Each runs asStart-Process -Waitinside a PowerShell thread job, so it is waited for together with everything it started, as when they ran one after the other (a bareWaitForExit()would wait only for the node process). The secondary logon service refused two simultaneous starts once, so the start is retried on that refusal only. Only the service file starts a server and request workers, which take the ports; the platform file drives the controller directly. I could not run PowerShell locally, so the evidence is the CI runs below.Job and check names, the secrets scan, the second build and the packaging are unchanged.
tasks.mdhas a ticked group 9 for this work.Removed or merged tests
None. The serial group's heavy tests assert real latency (for example a 4.5 s delayed security query that must take more than 5 s), so they cannot be shortened, and I found no duplicate whose coverage I could name.
Considered and not done
managed-step-http.test.tsinto concurrent files: ports arehash(target) % 4000, so concurrent fixtures can collide and be refused. Giving each fixture targets in a per-file port partition, or a network namespace per file, would work but changes many fixtures (or needs privileges the hosted runner restricts). Follow-up.bun installon Linux (about 1 s) or Node setup (about 5 s): not worth the cache surface. Windowsbun installis about 29 s; left alone.Follow-ups (not in this PR)
tests/backend/delegation-view-service.test.tsfailed once withECONNRESETin a local run at load average 41 (niced parallel group, other work on the machine); it passed three times alone and in every later full run. Cause not established. It may be a keep-alive socket reused after a long synchronous loop starved by load.ui/shared/{app,cleanup,search}.tswere left alone as agreed with the other PR.Verification
Local, on a 24-core machine shared with other work (load average 5 to 40, so wall-clock is indicative only):
bun run build:all,bun run typecheck,bun run tsc --noEmit -p ui/tsconfig.json(before and after)bun run test:allnode --experimental-strip-types scripts/test.ts --fast, SIGINT after 8 snode --testprocess lefttaskset -c 0,1 node --experimental-strip-types --test --test-name-pattern='portable lanes run every' tests/automation/ci.test.tsTest counts: backend parallel group 877 tests, 859 pass, 18 skipped; serial group 198 tests, 190 pass, 8 skipped (both unchanged); browser plus automation 1108, all pass (1059 before this PR; the rest are tests merged from
devsince, the 16 in the newtests/automation/test-lanes.test.ts, and the plan, built-service and cache tests inci.test.ts). The five native-only runtime tests are excluded as in CI and were not run; noopencodebinary was used.New automation tests, written first and run against the previous runner (12 failed there; a started-once check for a missing command and the stopped-step-prints-nothing check passed there and stay as guards): each of SIGINT, SIGTERM and SIGHUP stops every step, keeps their output and exits 130, 143 or 129; a grandchild that ignores SIGTERM, and a descendant in its own session holding the pipes, do not hold the runner; a repeated SIGINT is not swallowed; a closed output pipe and a crash of the runner leave no group behind; a step that fails by itself after the first failure is a failure with its output; the heartbeat, the output tail limit and the abandoned leftover pipe holder; the stop command and detaching per platform; the plan on 1, 2, 3, 4 and 24 cores; the
CP_BUILT_SERVICE=1fence; the compile cache folder.node --experimental-strip-types --test tests/automation/*.test.tsandopenspec validate release-1-5-2 --strictpass.CI on this PR (step "Run Linux tests, builds and package checks" and the Windows test step), against the ten recent runs before it:
AP_WINDOWS_FULL=1)mainpush)3c73f89)Runner variance is large (the same code took 149 s and 233 s), so read ranges, not single runs.
Pass and skip counts in CI are unchanged: Linux 877/859/18 and 198/190/8 backend; Windows fast set 26 tests, 21 pass, 5 skipped (18/16/2 platform, 8/5/3 service); Windows full set 26 tests, 24 pass, 2 skipped (18/16/2 and 8/8/0), as on
main. The longest service-file tests did not move (337 s against 335 s).Not verified: the Windows stop path of the runner (
taskkill /T /F) and the runner on Windows at all; macOS.Checklist
maincontains all nine and passes strict rebuild parity.