Skip to content

omamobile-audio 0.5.9: the watchdog tests no longer race the runner - #327

Merged
derekross merged 2 commits into
mainfrom
audio-watchdog-test-races
Nov 2, 2026
Merged

derekross merged 2 commits into
mainfrom
audio-watchdog-test-races

Conversation

@derekross

@derekross derekross commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

The watchdog tests in packages/omamobile-audio/test/all kept failing on GitHub's runners and passing locally. #315, #321 and #325 each patched one case. This PR fixes the whole class.

Root causes

  1. A lost update in the fake mixer. The JSON control backend's heartbeat read (JsonCtl.read, tests only) loads the whole state file, bumps HALO_HEARTBEAT and saves the file. test/fake-amixer cset does the same load-modify-save. If a test's cset landed between the watchdog's load and its save, the save wrote the old coil temperature back, so the watchdog never saw the jump. That is the "no jump" failure. In a tight loop, 38 of 40 writes were lost. Both sides now hold a flock on <state>.lock around load-modify-save, and that loop loses 0 of 40.
  2. Events timed by sleeps. Background helpers waited for watchdog: watching and then slept a fixed time. The kmsg, heartbeat-freeze, feedback and earpiece cases slept from the very start. Meanwhile the watchdog's grace, heartbeat timeout and settle window ran on the real clock. On a slow runner the change landed before the baseline, after the last round, or, for kmsg, before the log was opened. The start line was never late: log() already uses flush=True.

Fix

  • watch_loop reads a tests-only OMAMOBILE_WATCH_HOOK, a command it runs with the round number before each round. While it is set, the watchdog's clock is round × interval. The hook is honoured only with the tests' JSON control backend (OMAMOBILE_CTL_BACKEND=json:…). Anywhere else it is ignored and the watchdog logs that, so a stray line in speaker.conf can't change the phone's watchdog clock or run a command as root. A unit-level test covers that check (second commit, from review). Unset (on the phone), the loop is unchanged: real clock, nothing run, same thresholds and timing.
  • test/all gets at ROUND COMMAND, which watchrun hands to the hook. Every case that changed the tree mid-watch now lands at a known round relative to the stream start and the baseline: jump (both), c9 jump-off, c9settle (step, rise and cold), heartbeat freeze, both kmsg cases, feedback, and the earpiece run. None of them waits on watching plus a sleep any more. c9settle's steps are given as round numbers now, with the time each one stands for noted in a comment.
  • The unacknowledged-pause case now also checks that the note was logged, so it can't pass just because the record was missed. The kmsg and heartbeat-freeze cases now check their mute reason. The stuck-heartbeat message now prints tenths of a second ("for 0.3 s", not "for 0 s").
  • Version 0.5.9, because lib/speaker_dsp.py ships. The JSON backend and the hook are used only by the tests.

Some real-time dependencies remain, in the earpiece section:

  • the two negative checks ("not muted after 0.3 s"), which a slower machine can't make fail;
  • bgwatch's readiness poll, which gives up silently after 5 s;
  • OMAMOBILE_SPK_ACK_WAIT=2;
  • the background watchdog's 600 rounds, which bound how long that section can take.

The section passed under a 5× load. A machine much slower than that could still fail it.

Test evidence (laptop, 14 CPUs)

  • test/all: 4 plain runs, all passing (38 ok each, about 170 s per run).
  • test/all with 28 yes >/dev/null jobs on 14 CPUs: passed, 38 ok. It took 820 s, about 5× slower.
  • tools/pr-check origin/main: ok.

No phone was involved.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BeEcdVeSfyLCbijF8XQF1d

derekross and others added 2 commits November 2, 2026 01:27
The watchdog tests kept failing on GitHub's runners and passing locally,
and #315, #321 and #325 each patched one case. Two causes, both in the
test setup, neither in what the phone enforces:

- A lost update in the fake mixer. The JSON backend's heartbeat read
  loads the whole state file, bumps the heartbeat and saves the file;
  test/fake-amixer's cset does the same. When a test's cset landed
  between the watchdog's load and save, the save put the old coil
  temperature back, so the watchdog never saw the jump. A tight loop
  lost 38 of 40 writes. Both now hold a flock on <state>.lock around
  load-modify-save (0 of 40 lost).
- Events timed by sleeps. Helpers waited for "watchdog: watching" and
  then slept a fixed time (or slept from the start, for the kmsg,
  heartbeat-freeze, feedback and earpiece cases) before changing the
  tree, while the watchdog's grace, heartbeat timeout and settle window
  ran on the real clock. On a slow runner the change landed before the
  baseline, after the last round, or (kmsg) before the log was opened.
  The start line itself was never late: log() already flushes.

The watchdog loop takes a test-only OMAMOBILE_WATCH_HOOK: a command run
with the round number before each round, and with it set the clock is
round x interval. test/all's at ROUND COMMAND queues an event, so every
case that changed the tree mid-watch (jump, c9 jump-off, c9settle's
step/rise/cold, heartbeat freeze, both kmsg cases, feedback, earpiece
run) now lands at a known round relative to the baseline, however
slow the machine. The unacknowledged-pause case also checks the note
is logged now, so it can't pass by missing the record.

Unset on the phone, the loop runs exactly as before: the real clock,
nothing run, the same thresholds and timing. The JSON backend is
tests-only. The version moves because speaker_dsp.py ships.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeEcdVeSfyLCbijF8XQF1d
Review of #327: OMAMOBILE_WATCH_HOOK was honoured whenever it was set,
so a stray line in /etc/omamobile/speaker.conf could have changed the
real watchdog's clock or run a command every round as root. watch_hook()
now honours it only with the tests' JSON control backend
(OMAMOBILE_CTL_BACKEND=json:...), and otherwise ignores it and logs
that. A unit-level test checks the selection with no backend, alsa and
json, and that the probe command never ran (the real backend needs a
card, so the watchdog itself can't run there).

The kmsg and heartbeat-freeze cases now check their mute reason, as the
others do, so an unrelated earlier mute can't pass them. The stuck
heartbeat message prints tenths ("for 0.3 s", not "for 0 s").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeEcdVeSfyLCbijF8XQF1d
@derekross
derekross merged commit 00f0f5d into main Nov 2, 2026
2 checks passed
@derekross
derekross deleted the audio-watchdog-test-races branch November 2, 2026 13:20
derekross added a commit that referenced this pull request Nov 2, 2026
Thirteen lessons from the release PRs' reviews (#320, #324, #326,
#327), approved by the maintainer: booted is a fact about the
partitions, counters need their switch on, where a refusal runs, packing
is a boot change, inert test switches, end-state checks, mutation-testing
re-checks, per-package skips, allowlists of another tool's output,
consumers for new signatures, cross-PR interactions, shared tools gaining
inputs, and non-reproducible staging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeEcdVeSfyLCbijF8XQF1d
robertzas pushed a commit to robertzas/omarchy-mobile that referenced this pull request Nov 14, 2026
…erekross#327)

The fake control backend and fake-amixer did unlocked load-change-save, so a test's coil-temperature write could be lost under the watchdog's heartbeat read (37 of 40 in a tight loop); both now take a flock. Tests fired events after fixed sleeps on the real clock; a tests-only OMAMOBILE_WATCH_HOOK, honoured only with the JSON backend, runs the watchdog in lockstep so every event lands at a known round. Production behaviour is unchanged: with the hook unset (and always with the ALSA backend) watch_loop is the same code. The kmsg and heartbeat-freeze cases now assert their mute reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeEcdVeSfyLCbijF8XQF1d
robertzas pushed a commit to robertzas/omarchy-mobile that referenced this pull request Nov 14, 2026
Thirteen lessons from the release PRs' reviews (derekross#320, derekross#324, derekross#326,
derekross#327), approved by the maintainer: booted is a fact about the
partitions, counters need their switch on, where a refusal runs, packing
is a boot change, inert test switches, end-state checks, mutation-testing
re-checks, per-package skips, allowlists of another tool's output,
consumers for new signatures, cross-PR interactions, shared tools gaining
inputs, and non-reproducible staging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeEcdVeSfyLCbijF8XQF1d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant