Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changeset/stall-guard-self-test.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
---

ci: the stall guard now proves it still fires (`pnpm check:stall-guard`), and a stalled run's SIGKILL escalation actually runs (#4250). CI-only — releases nothing.

`scripts/run-with-stall-guard.mjs` gains a `--self-test` that drives it against synthetic stalls — an idle hang, a sync-spinning hang, a hang that never prints a first line, and a descendant that traps SIGTERM — asserting the exit-75 verdict, the idle/ON-CPU classification, the SIGUSR2 stack harvest including the "no report = blocked event loop" inference, full process-group teardown, and the negative direction (a healthy run keeps its own exit status; steady output is never called a stall). Six jobs across five workflows depend on this guard, and until now nothing exercised its firing path between real stalls.

Writing that harness surfaced a real defect, since fixed: the SIGKILL escalation was armed as an unref'd timer and the guard exited from the direct child's `exit` handler, so the timer never fired. The direct child (`pnpm` → `turbo`, or `sh`) dies on SIGTERM immediately, so any **descendant** that traps SIGTERM outlived the guard — the shape `ObjectKernelConfig.gracefulShutdown` installs in every kernel a test boots. The guard now waits for the process group to actually empty and SIGKILLs the holdouts, naming them in the log.
19 changes: 19 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,25 @@ jobs:
- name: Check every driver runs the shared conformance cases
run: pnpm check:driver-conformance

# Stall-guard self-test (#4250). scripts/run-with-stall-guard.mjs is what
# turns a frozen Test Core into a labeled red; six jobs across five
# workflows now route their test steps through it. But it only executes its
# interesting path during an event that is rare (~5-10% of days) and not
# reproducible on demand — so between real stalls there was nothing at all
# asserting it still works, and a guard that has never fired is
# indistinguishable from a guard that does not. A refactor could have
# disarmed CI's only stall detector and every run would have stayed green.
#
# This runs the guard against SYNTHETIC stalls — an idle hang, a
# sync-spinning hang, a hang with no output at all, a SIGTERM-trapping
# descendant — and asserts the verdict, the idle/ON-CPU classification, the
# SIGUSR2 stack harvest (including the "no report = blocked event loop"
# inference), full process-group teardown, and the negative direction: a
# healthy run still propagates its own exit status and steady output is
# never called a stall. ~50s, no build, no network.
- name: Stall-guard self-test
run: pnpm check:stall-guard

- name: Type check (@objectstack/spec)
run: pnpm --filter @objectstack/spec exec tsc --noEmit

Expand Down
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@
"check:published-files": "node scripts/check-published-files.mjs --self-test && node scripts/check-published-files.mjs",
"check:type-check-coverage": "node scripts/check-type-check-coverage.mjs --self-test && node scripts/check-type-check-coverage.mjs",
"check:driver-conformance": "node scripts/check-driver-conformance.mjs --self-test && node scripts/check-driver-conformance.mjs",
"check:engine-double-contract": "node scripts/check-engine-double-contract.mjs --self-test && node scripts/check-engine-double-contract.mjs"
"check:engine-double-contract": "node scripts/check-engine-double-contract.mjs --self-test && node scripts/check-engine-double-contract.mjs",
"check:stall-guard": "node scripts/run-with-stall-guard.mjs --self-test"
},
"keywords": [
"objectstack",
Expand Down
Loading
Loading