fix(server): crash diagnostics, event-loop stall visibility, and dead-pipe guards - #1736
Merged
Merged
Conversation
The server had no uncaughtException or unhandledRejection handlers, so any stray throw exited the process with only Bun's stderr dump as evidence, and winston's buffered transport lost the final entries. Add a diagnostics module that synchronously appends a structured fatal record to the daily mcode.log and a durable server-fatal.log, records every process exit, and warn-logs event-loop stalls over 500ms.
sendBroadcastPayload called ws.send unguarded while both sibling send paths caught failures. A socket dying between the readyState check and the send threw an uncaught exception that killed the process.
The Codex warm-up wrote to child.stdin with no error listener and the Cursor runner wrote the prompt the same way. A provider exiting early closed the pipe and the unhandled stream error killed the whole server.
Warm-idle ran the unmasked PRAGMA optimize, a full ANALYZE that can block Bun's single thread for seconds on a large database. Use the bounded 0x10002 form everywhere and log how long warm and background maintenance take so stalls correlate with health-probe failures.
The ready promise stayed resolved after onclose until connect() reset it, so rpc() calls in the gap sent on a dead socket, and the terminal send callback had no readyState guard at all. Re-arm ready on close without stranding parked calls, resolve it on close(), and skip terminal frames while the socket is not open.
The child was spawned with stdout ignored, discarding every console write and leaving Bun's exit-time stdout flush to fail on a NUL handle. Route stdout into the same server-stderr.log file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Bun server had no
uncaughtExceptionorunhandledRejectionhandlers, so a stray throw exited the process with only Bun's stderr dump as evidence, and winston's buffered transport lost the final entries. That is why production incidents showed a healthy last log line and an emptyserver-stderr.log.A new diagnostics module, installed before
startServer(), synchronously appends a structured fatal record (timestamp, kind, pid, uptime, error name/message/stack, errno fields) to the dailymcode.logand a durableserver-fatal.log, then preserves Bun's exit-1 semantics. It also logs every process exit and warn-logs event-loop stalls over 500ms.Alongside the instrumentation, this fixes the concrete crash candidates found during investigation:
sendBroadcastPayloadcalledws.sendunguarded while both sibling send paths caught failures. A socket dying mid-broadcast threw uncaught.child.stdinwith noerrorlistener. A provider exiting early closes the pipe and the unhandled stream error kills the process. This matches the observedEPIPE: broken pipe, writefatality.PRAGMA optimize, a fullANALYZEthat can block Bun's single thread for seconds on a large database. Now bounded to0x10002everywhere, and warm/background maintenance log their durations.server-stderr.logfile handle.readypromise stayed resolved afteroncloseuntilconnect()reset it, sorpc()sent on a dead socket and the terminal send callback had no readyState guard at all.readynow re-arms on close without stranding parked calls,close()resolves it, and terminal frames drop while the socket is not open.Why
Production backend exits were undiagnosable: the crash never reached a log file, the stderr file was truncated on respawn, and stdout was discarded. Every future incident now leaves a full stack trace on disk in two locations, and event-loop stalls get a timestamped duration so they can be correlated with health-probe failures.
UI Changes
None. The renderer transport change alters reconnect timing behavior only.
Config Changes
None.
Review Notes
apps/server/src/runtime/diagnostics/server-diagnostics.tsdeliberately avoids project imports so it cannot fail with the component that crashed.docs/agents/runtime.mdupdated with the new log paths.push.test.ts,server-diagnostics.test.ts,database-connection-policy.test.ts), including a new broadcast-throw regression test.tsc --noEmitclean on server, web, desktop, and providers; oxlint clean on changed files.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.