Skip to content

fix(server): keep the server up when a stray signal reaches it - #580

Merged
ArgaFairuz merged 1 commit into
mainfrom
fix/supervise-server-processes
Nov 17, 2026
Merged

ArgaFairuz merged 1 commit into
mainfrom
fix/supervise-server-processes

Conversation

@ArgaFairuz

Copy link
Copy Markdown
Collaborator

Summary

Supervise the Clopen server (production launcher) and the dev backend/frontend so a stray signal no longer takes the app down: each runs in its own process group and is restarted if it stops without being asked to.

Why

A dev session ended while an agent ran browser automation: vite dev received a SIGTERM from an unknown sender, and concurrently (killOthersOn: failure) treated the 143 as a failure and stopped the backend. In production, bun clopen, the server and every engine process shared one process group, so a signal to that group would end the whole app. The fix is in Clopen alone, so it covers every engine.

Changes

  • shared/utils/process-supervisor.ts:
    • own process group per child (POSIX), restart on any unrequested exit (backoff, gives up after 5 in 60 s);
    • SIGINT/SIGTERM forwarded to the child's group with a 15 s graceful window, second signal forces;
    • SIGHUP ignored (matching the server), terminal write errors swallowed;
    • unrequested exits logged with recently started processes to <data-dir>/logs/supervisor.log.
  • bin/clopen.ts: the server runs under the supervisor; stdin is ignore (the server never reads it).
  • scripts/dev.ts: backend and frontend supervised independently with prefixed, colored output; concurrently removed.

Test plan

  • bun run check, bun run lint
  • bun run test: 1936 pass; the 2 renderScopeConfig failures also fail on main
  • Supervisor tests with real processes: own process group, restart after a stray group SIGTERM (report written), requested stop reaches grandchildren and is not restarted, gives up after the limit
  • Launcher integration test: SIGHUP leaves the server running, a stray SIGTERM brings it back, SIGINT shuts it down gracefully and exits 0
  • bun run dev: Ctrl+C stops both; kill <vite pid> restarts vite only and the backend stays up
  • clopen: Ctrl+C stops cleanly; kill <server pid> restarts the server; closing the terminal leaves it running

Notes

  • The original sender of the SIGTERM is still unknown; the supervisor log is meant to identify it if it happens again.
  • On Windows children keep sharing the console (no process groups); restart-on-exit still applies.

A dev session stopped while an agent was using browser automation: the
Vite process received a SIGTERM from outside, and the dev runner
(concurrently, killOthersOn: failure) took that 143 as a failure and
shut the backend down with it. The sender could not be identified (the
OS does not record it). In production the launcher, the server and
every engine process shared one process group, so a group signal would
end the whole app the same way.

Add one supervisor for the production launcher and the dev runner. It
runs each child in its own process group (POSIX) and restarts a child
that exits without being asked to, whatever its exit code, giving up
after 5 restarts in 60 s. It forwards SIGINT/SIGTERM to the child's
group with 15 s to shut down, and ignores SIGHUP as the server does.
Each unrequested exit is logged with the processes started just before
it, to the console and <data-dir>/logs/supervisor.log. The dev runner
supervises backend and frontend separately and no longer uses
concurrently.
@ArgaFairuz
ArgaFairuz merged commit 9d68a8c into main Nov 17, 2026
1 check passed
@ArgaFairuz
ArgaFairuz deleted the fix/supervise-server-processes branch November 17, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant