Skip to content

Write a Node diagnostic report when the backend dies of a fatal V8 error - #1072

Merged
helpcodeai merged 1 commit into
mainfrom
fix/node-fatal-report
Oct 13, 2026
Merged

helpcodeai merged 1 commit into
mainfrom
fix/node-fatal-report

Conversation

@helpcodeai

Copy link
Copy Markdown
Contributor

Summary

On 13 Oct at 12:16 UTC the Cloud backend died with FATAL ERROR: Zone Allocation failed - process out of memory. The heap was at about 1.4 GB of its 4.2 GB cap and the container was far from its 6 GB limit, so this was not the usual heap exhaustion, and the log had no stack to say what allocated it. Docker restarted it in about 35 seconds.

This makes the next fatal V8 crash leave a Node diagnostic report, so we can see what was running.

Changes

  • start.sh: the backend command gets --report-on-fatalerror --report-exclude-env --report-compact --report-directory=$HEAP_SNAPSHOT_DIR (/app/diagnostics in the image, already a volume on Cloud for heap snapshots).
  • The flags are added only when node --report-exclude-env works (Node >= 22.13). The report would otherwise contain every environment variable, secrets included, and an older Node would refuse to start on the unknown flag.
  • docs/operations/observability.md: new section on the crash reports.

Cloud and self-hosted behave the same: same image, same script. Self-hosters keep the reports across container recreation by mounting a volume at /app/diagnostics.

Type

  • Bug fix
  • New feature
  • Documentation
  • Refactoring
  • Performance improvement
  • Other (describe below)

Testing

  • Added/updated unit tests

  • Tested manually (describe steps below)

  • No tests needed (explain why)

  • sh -n start.sh: OK.

  • Extracted the flag logic and printed the resulting command: with Node 25 locally and HEAP_SNAPSHOT_DIR=/app/diagnostics it yields the flags above.

  • Forced a heap OOM with the same flags locally: a compact JSON report was written, without environmentVariables.

  • In the running Cloud container: Node v26.11.1 accepts --report-exclude-env, and /app/diagnostics is writable by appuser.

Checklist

  • My code follows the existing code style
  • I have tested my changes locally
  • I have updated documentation (if applicable)
  • All existing tests pass (npm test)
  • This PR has a descriptive title

Related Issues

None.

On 13 Oct the Cloud backend died with "Zone Allocation failed - process out of memory" with the heap at a third of its cap, and the log had no stack. start.sh now passes --report-on-fatalerror so the next such crash leaves a report (native stack, heap spaces, resource usage, JS stack when available) in HEAP_SNAPSHOT_DIR, which is a volume on Cloud. Environment variables are excluded; on a Node without --report-exclude-env the report stays off rather than writing secrets to disk or failing to start.
@helpcodeai
helpcodeai enabled auto-merge (squash) October 13, 2026 12:50
@helpcodeai
helpcodeai merged commit 1ab7979 into main Oct 13, 2026
14 checks passed
@helpcodeai
helpcodeai deleted the fix/node-fatal-report branch October 13, 2026 12:56
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 13, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant