Repository navigation
waze-on-cluster: keep the journal in RAM, write it to storage only around failures and at shutdown - #226
Merged
Conversation
…round failures and at shutdown The journal keeper appends to a rolling log in /run/waze-on-cluster-journal (tmpfs, same caps). It writes the newest lines to /var/opt/waze-on-cluster/journal only when there is something to keep: a few seconds after a failure line (INCIDENT, a crash or failure of carplay, aapd or the tools, a segfault or OOM kill, a boot recovery step, a patch removed) and a minute later, when the service stops, and at each head-unit shutdown from var-opt-ro.service's drop-in, before /var/opt goes read-only. Each write is one file of at most 256 KiB holding the last 10 minutes not written yet, lines repeated with other numbers kept 3 times, made under a temporary name, fsynced and linked into place. 16 files and 2 MiB at most; a shutdown's go before a failure's. The root step opens nothing through a link. Save logs takes the RAM journal and the kept files; incidents.txt lists the boots of both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Maps on cluster's journal keeper appended to a log on
/var/optevery couple of seconds while the head unit was on. Each ext4 commit wrote a block plus the file system's journal, so the keeper was the steadiest writer to the head unit's storage of all our tools. This keeps the journal in RAM and writes to/var/optonly when there is something to keep.RAM journal. The keeper writes the same filtered lines, with the same rate limits and caps (10 files of 384 KiB), to
/run/waze-on-cluster-journal/(tmpfs).Written to
/var/opt/waze-on-cluster/journal/only in these cases:INCIDENTline of the tools; systemd reporting a crash, failure or watchdog of carplay, aapd, recmgr, the crash handler or one of our units; a kernel segfault or OOM kill; a boot recovery step (restore, crash, kill, stays down); a patch removed. At most one write a minute.var-opt-ro.serviceruns our drop-in before it remounts/var/optread-only, so the drop-in runsjournal_keep.js keep shutdownas root aftershutdown_mark.sh. A failure there never holds up the shutdown.Each write is one file,
journal-<UTC>-<reason>.log, of at most 256 KiB. It holds the lines not written before, from the last 10 minutes. Lines that differ only in their numbers are kept 3 times, the newest. The file is made under a temporary name, fsynced, linked into place and the folder fsynced, so a power cut leaves either all of it or nothing. At most 16 files and 2 MiB are kept; a stop's or shutdown's files are removed before a failure's. The root step opens nothing through a symbolic link and creates only new files. A power cut loses only what is in RAM.Save logs collects the RAM journal and the kept files from
/var/opt.incidents.txtlists the boots seen in both. The rollingjournal.log*files of earlier releases are removed from/var/optwhen the service starts.AI: Claude Code, Claude Opus 5.5 (implementation, review).
Tested:
make test-nodepasses. New tests cover:node release/release.js versions origin/mainpasses.On a test unit with iD7 software, Maps on cluster showing a stream, and the keeper's folder on its own ext4 device mounted with the car's
/var/optoptions. Each run lasted 10 minutes:An aapd crash (SIGSEGV):
aapd.service: Main process exited, code=dumped, the crash handler's and recmgr's lines, and they were on/var/optafter the restart.Save logs after that restart held:
incidents.txt;aapd-status.txt,aapd-probe.txt,carplay-probe.txt;aapd-crashes.txt,recmgr-crashes.txt,core-dumps.txt,boot-recovery-shutdown.Not tested: a real car.
Human review: none before opening.
Screenshots: n/a (nothing on screen changes).
🤖 Generated with Claude Code