Repository navigation
Write a Node diagnostic report when the backend dies of a fatal V8 error - #1072
Merged
Merged
Conversation
On 13 Oct the Cloud backend died with "Zone Allocation failed - process out of memory" with the heap at a third of its cap, and the log had no stack. start.sh now passes --report-on-fatalerror so the next such crash leaves a report (native stack, heap spaces, resource usage, JS stack when available) in HEAP_SNAPSHOT_DIR, which is a volume on Cloud. Environment variables are excluded; on a Node without --report-exclude-env the report stays off rather than writing secrets to disk or failing to start.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On 13 Oct at 12:16 UTC the Cloud backend died with
FATAL ERROR: Zone Allocation failed - process out of memory. The heap was at about 1.4 GB of its 4.2 GB cap and the container was far from its 6 GB limit, so this was not the usual heap exhaustion, and the log had no stack to say what allocated it. Docker restarted it in about 35 seconds.This makes the next fatal V8 crash leave a Node diagnostic report, so we can see what was running.
Changes
start.sh: the backend command gets--report-on-fatalerror --report-exclude-env --report-compact --report-directory=$HEAP_SNAPSHOT_DIR(/app/diagnosticsin the image, already a volume on Cloud for heap snapshots).node --report-exclude-envworks (Node >= 22.13). The report would otherwise contain every environment variable, secrets included, and an older Node would refuse to start on the unknown flag.docs/operations/observability.md: new section on the crash reports.Cloud and self-hosted behave the same: same image, same script. Self-hosters keep the reports across container recreation by mounting a volume at
/app/diagnostics.Type
Testing
Added/updated unit tests
Tested manually (describe steps below)
No tests needed (explain why)
sh -n start.sh: OK.Extracted the flag logic and printed the resulting command: with Node 25 locally and
HEAP_SNAPSHOT_DIR=/app/diagnosticsit yields the flags above.Forced a heap OOM with the same flags locally: a compact JSON report was written, without
environmentVariables.In the running Cloud container: Node v26.11.1 accepts
--report-exclude-env, and/app/diagnosticsis writable byappuser.Checklist
npm test)Related Issues
None.