Skip to content

waze-on-cluster: keep the journal in RAM, write it to storage only around failures and at shutdown - #226

Merged
robertnisipeanu merged 1 commit into
mainfrom
woc-journal-ram
Nov 19, 2026
Merged

robertnisipeanu merged 1 commit into
mainfrom
woc-journal-ram

Conversation

@robertnisipeanu

Copy link
Copy Markdown
Owner

Maps on cluster's journal keeper appended to a log on /var/opt every couple of seconds while the head unit was on. Each ext4 commit wrote a block plus the file system's journal, so the keeper was the steadiest writer to the head unit's storage of all our tools. This keeps the journal in RAM and writes to /var/opt only when there is something to keep.

RAM journal. The keeper writes the same filtered lines, with the same rate limits and caps (10 files of 384 KiB), to /run/waze-on-cluster-journal/ (tmpfs).

Written to /var/opt/waze-on-cluster/journal/ only in these cases:

  • 5 s after a line that reports a failure (to include the burst around it), and once more a minute later for what followed. The failure lines are: an INCIDENT line of the tools; systemd reporting a crash, failure or watchdog of carplay, aapd, recmgr, the crash handler or one of our units; a kernel segfault or OOM kill; a boot recovery step (restore, crash, kill, stays down); a patch removed. At most one write a minute.
  • When the journal service stops.
  • At each head-unit shutdown. A restart by recmgr or the crash handler, and a switch-off, are NSM shutdowns that stop no service. var-opt-ro.service runs our drop-in before it remounts /var/opt read-only, so the drop-in runs journal_keep.js keep shutdown as root after shutdown_mark.sh. A failure there never holds up the shutdown.

Each write is one file, journal-<UTC>-<reason>.log, of at most 256 KiB. It holds the lines not written before, from the last 10 minutes. Lines that differ only in their numbers are kept 3 times, the newest. The file is made under a temporary name, fsynced, linked into place and the folder fsynced, so a power cut leaves either all of it or nothing. At most 16 files and 2 MiB are kept; a stop's or shutdown's files are removed before a failure's. The root step opens nothing through a symbolic link and creates only new files. A power cut loses only what is in RAM.

Save logs collects the RAM journal and the kept files from /var/opt. incidents.txt lists the boots seen in both. The rolling journal.log* files of earlier releases are removed from /var/opt when the service starts.

AI: Claude Code, Claude Opus 5.5 (implementation, review).

Tested:

  • make test-node passes. New tests cover:

    • which lines trigger a write;
    • the condensing;
    • the window, cap and naming of each write, and the protection of the existing file;
    • removal order and the size caps;
    • leftover temporary files;
    • symbolic links refused when run as root;
    • the service writing after a failure, a minute later and on stop, and the shutdown command;
    • the Save logs archive with kept files.
      node release/release.js versions origin/main passes.
  • On a test unit with iD7 software, Maps on cluster showing a stream, and the keeper's folder on its own ext4 device mounted with the car's /var/opt options. Each run lasted 10 minutes:

    Before After
    Map shown 1.9 MB (11.4 MB/h) 0 B
    Map not shown 2.6 MB (15.9 MB/h) 0 B
  • An aapd crash (SIGSEGV):

    • The keeper wrote the 10-minute window around the crash 6 s later (84 KB).
    • In a second run recmgr restarted the head unit within 4 s. The shutdown write kept systemd's aapd.service: Main process exited, code=dumped, the crash handler's and recmgr's lines, and they were on /var/opt after the restart.
    • The shutdown command took 0.4 s.
  • Save logs after that restart held:

    • the RAM journal and the kept file with the crash;
    • the boots in incidents.txt;
    • aapd-status.txt, aapd-probe.txt, carplay-probe.txt;
    • aapd-crashes.txt, recmgr-crashes.txt, core-dumps.txt, boot-recovery-shutdown.
  • Not tested: a real car.

Human review: none before opening.

Screenshots: n/a (nothing on screen changes).

🤖 Generated with Claude Code

…round failures and at shutdown

The journal keeper appends to a rolling log in /run/waze-on-cluster-journal
(tmpfs, same caps). It writes the newest lines to
/var/opt/waze-on-cluster/journal only when there is something to keep: a few
seconds after a failure line (INCIDENT, a crash or failure of carplay, aapd
or the tools, a segfault or OOM kill, a boot recovery step, a patch removed)
and a minute later, when the service stops, and at each head-unit shutdown
from var-opt-ro.service's drop-in, before /var/opt goes read-only. Each write
is one file of at most 256 KiB holding the last 10 minutes not written yet,
lines repeated with other numbers kept 3 times, made under a temporary name,
fsynced and linked into place. 16 files and 2 MiB at most; a shutdown's go
before a failure's. The root step opens nothing through a link.

Save logs takes the RAM journal and the kept files; incidents.txt lists the
boots of both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@robertnisipeanu
robertnisipeanu merged commit c5fe4c8 into main Nov 19, 2026
16 of 18 checks passed
@robertnisipeanu
robertnisipeanu deleted the woc-journal-ram branch November 19, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant