Skip to content

Add an engine heartbeat and health monitor - #24

Open
Colton127 wants to merge 1 commit into
mainfrom
feat/engine-heartbeat
Open

Add an engine heartbeat and health monitor#24
Colton127 wants to merge 1 commit into
mainfrom
feat/engine-heartbeat

Conversation

@Colton127

Copy link
Copy Markdown
Owner

Summary

Nothing could answer "is the engine actually alive?". isInitialized stays true for a deadlocked engine, play() keeps returning valid handles, getAudioDeviceState() keeps reporting started, and getStreamTime() takes the very audio mutex a wedged engine has stranded — so the health check hangs on the thing it is checking.

Three separate bugs in this codebase have produced the same user-visible failure: handles valid, Dart reporting playback, no audio, sometimes a frozen UI. Two were found only after users hit them. This adds the signal that turns that class of failure into something detectable in seconds rather than a frozen app.

Native

  • gRenderedFrames, a process-global atomic incremented by the miniaudio device data callback after soloud->mix() returns — so a mixer that blocks stops the counter rather than appearing to make progress. Deliberately not a member of Soloud or Player: reading it must touch no mutex and no pointer teardown can swap.
  • getAudioFramesRendered(), a lock-free FFI export mirroring getAudioDeviceState(). Survives deinit()/init(); only its change over time is meaningful.

Dart

  • SoLoud.getAudioFramesRendered() — the raw counter. Returns -1 on web, whose wasm is a prebuilt asset with no such export, and whose single-threaded AudioContext build cannot exhibit the failure this detects.
  • SoLoud.monitorEngineHealth() — a stream emitting AudioEngineHealth on change, judging health from outside the engine by comparing the device's own claim against the heartbeat.
  • AudioEngineHealthTracker holds the decision rules, split out so they are testable without a running engine.
_healthSub = SoLoud.instance.monitorEngineHealth().listen((health) {
  if (health == AudioEngineHealth.stalled) {
    unawaited(_recoverEngine()); // report, then deinit + init
  }
});

Avoiding false positives

A stopped device is always idle, never stalled, and time spent stopped is not charged against the device when it restarts.

That matters where the device is stopped aggressively while idle — on iOS a silent stream breaks the Control Center transport controls, so a short idle timeout is required there. A false stall in that configuration would be worse than no monitoring at all.

Verification

  • 9 unit tests over the decision rules, covering the aggressive idle-stop pattern (60 s stopped, then restart), transitional device states, the first observation of an already-running engine, a counter reset across deinit/init, and the no-heartbeat path.
  • End to end against a live miniaudio device thread: the counter advanced 0 → 16384 frames over 300 ms, was read successfully while SoLoud's audio mutex was held (the exact lock a wedged engine strands), and froze after deinit().

flutter analyze unchanged; flutter test goes from 2 to 11 tests.

Caveat on recovery

Detection is the easy half. Once stalled fires, deinit() may itself block, since it joins the scheduler and takes every lock. Getting out cleanly needs a hard-reset path — abandon rather than join, leak the wedged Player, build a fresh one. Worth designing once this reports how often it actually fires in the field, rather than guessing now.

Independence

Mergeable in any order relative to the other PRs in this series. If #23 merges first, expect a trivial conflict in bindings.cpp — the two add code in different places near getAudioDeviceState().


Generated by Claude Code

Nothing could answer "is the engine actually alive?". isInitialized stays
true for a deadlocked engine, play() keeps returning valid handles,
getAudioDeviceState() keeps reporting started, and getStreamTime() takes the
very audio mutex a wedged engine has stranded -- so the health check hangs on
the thing it is checking.

Native:

- gRenderedFrames, a process-global atomic incremented by the miniaudio
  device data callback *after* soloud->mix() returns, so a mixer that blocks
  stops the counter rather than appearing to progress. Deliberately not a
  member of Soloud or Player: reading it must touch no mutex and no pointer
  teardown can swap.
- getAudioFramesRendered(), a lock-free FFI export mirroring
  getAudioDeviceState(). Survives deinit()/init(); only its change over time
  is meaningful.

Dart:

- SoLoud.getAudioFramesRendered(), the raw counter (-1 on web, whose wasm is
  a prebuilt asset with no such export, and whose single-threaded
  AudioContext build cannot exhibit the failure this detects).
- SoLoud.monitorEngineHealth(), a stream emitting AudioEngineHealth on
  change, judging health from outside the engine by comparing the device's
  own claim against the heartbeat.
- AudioEngineHealthTracker holds the decision rules, split out so they are
  testable without a running engine.

A stopped device is always idle, never stalled, and time spent stopped is not
charged against the device when it restarts. That matters where the device is
stopped aggressively while idle -- on iOS a silent stream breaks the Control
Center transport controls -- since a false stall there would be worse than no
monitoring.

Verified: 9 unit tests over the decision rules, covering the aggressive
idle-stop pattern, transitional device states, the first observation of an
already-running engine, and the no-heartbeat path. End to end against a live
miniaudio device thread the counter advanced 0 -> 16384 frames over 300ms,
was read successfully while SoLoud's audio mutex was held (the lock a wedged
engine strands), and froze after deinit().

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants