Skip to content

fix(server): crash diagnostics, event-loop stall visibility, and dead-pipe guards - #1736

Merged
chuks-qua merged 8 commits into
mainfrom
fix/server-crash-diagnostics
Sep 21, 2026
Merged

chuks-qua merged 8 commits into
mainfrom
fix/server-crash-diagnostics

Conversation

@chuks-qua

@chuks-qua chuks-qua commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

What

The Bun server had no uncaughtException or unhandledRejection handlers, so a stray throw exited the process with only Bun's stderr dump as evidence, and winston's buffered transport lost the final entries. That is why production incidents showed a healthy last log line and an empty server-stderr.log.

A new diagnostics module, installed before startServer(), synchronously appends a structured fatal record (timestamp, kind, pid, uptime, error name/message/stack, errno fields) to the daily mcode.log and a durable server-fatal.log, then preserves Bun's exit-1 semantics. It also logs every process exit and warn-logs event-loop stalls over 500ms.

Alongside the instrumentation, this fixes the concrete crash candidates found during investigation:

  • sendBroadcastPayload called ws.send unguarded while both sibling send paths caught failures. A socket dying mid-broadcast threw uncaught.
  • The Codex warm-up and Cursor runner wrote to child.stdin with no error listener. A provider exiting early closes the pipe and the unhandled stream error kills the process. This matches the observed EPIPE: broken pipe, write fatality.
  • Warm-idle ran the unmasked PRAGMA optimize, a full ANALYZE that can block Bun's single thread for seconds on a large database. Now bounded to 0x10002 everywhere, and warm/background maintenance log their durations.
  • Server stdout was spawned to NUL, discarding all console output. It now shares the server-stderr.log file handle.
  • The renderer's ready promise stayed resolved after onclose until connect() reset it, so rpc() sent on a dead socket and the terminal send callback had no readyState guard at all. ready now re-arms on close without stranding parked calls, close() resolves it, and terminal frames drop while the socket is not open.

Why

Production backend exits were undiagnosable: the crash never reached a log file, the stderr file was truncated on respawn, and stdout was discarded. Every future incident now leaves a full stack trace on disk in two locations, and event-loop stalls get a timestamped duration so they can be correlated with health-probe failures.

UI Changes

None. The renderer transport change alters reconnect timing behavior only.

Config Changes

None.

Review Notes

  • New file: apps/server/src/runtime/diagnostics/server-diagnostics.ts deliberately avoids project imports so it cannot fail with the component that crashed.
  • docs/agents/runtime.md updated with the new log paths.
  • Tests: 25/25 pass in the touched server suites (push.test.ts, server-diagnostics.test.ts, database-connection-policy.test.ts), including a new broadcast-throw regression test. tsc --noEmit clean on server, web, desktop, and providers; oxlint clean on changed files.
  • Known gap: which of the unguarded write paths caused the observed production exit is unproven; the instrumentation is designed to capture it on recurrence.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

The server had no uncaughtException or unhandledRejection handlers, so any
stray throw exited the process with only Bun's stderr dump as evidence,
and winston's buffered transport lost the final entries. Add a diagnostics
module that synchronously appends a structured fatal record to the daily
mcode.log and a durable server-fatal.log, records every process exit, and
warn-logs event-loop stalls over 500ms.
sendBroadcastPayload called ws.send unguarded while both sibling send
paths caught failures. A socket dying between the readyState check and
the send threw an uncaught exception that killed the process.
The Codex warm-up wrote to child.stdin with no error listener and the
Cursor runner wrote the prompt the same way. A provider exiting early
closed the pipe and the unhandled stream error killed the whole server.
Warm-idle ran the unmasked PRAGMA optimize, a full ANALYZE that can block
Bun's single thread for seconds on a large database. Use the bounded
0x10002 form everywhere and log how long warm and background maintenance
take so stalls correlate with health-probe failures.
The ready promise stayed resolved after onclose until connect() reset it,
so rpc() calls in the gap sent on a dead socket, and the terminal send
callback had no readyState guard at all. Re-arm ready on close without
stranding parked calls, resolve it on close(), and skip terminal frames
while the socket is not open.
The child was spawned with stdout ignored, discarding every console
write and leaving Bun's exit-time stdout flush to fail on a NUL handle.
Route stdout into the same server-stderr.log file.
@chuks-qua
chuks-qua merged commit ab1c728 into main Sep 21, 2026
9 checks passed
@chuks-qua
chuks-qua deleted the fix/server-crash-diagnostics branch September 21, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant