20261007 - Acknowledge RSPduo overloads outside the SDRplay event callback

[Purple10101]

Summary

Nodes that overload at start-up wedge the RSPduo. blah2 crash-loops on NumDevs=0 until the SDRplay service is force-reset on a start that happens not to overload, which has taken from 5 minutes to about 7 hours. It has hit 6 nodes across the last three releases (27 September, 4 October and 5 October).

The cause is the overload ack. event_callback() acked each PowerOverloadChange by calling sdrplay_api_Update() itself. When another thread had an Update in flight, that deadlocked the SDRplay API. Overloads arrive at exactly those moments: the gain re-stage straight after Init(), and retune() jumping onto a strong signal. The API serialises Updates, so the ack queued behind the in-flight Update, which could not complete while the callback was stuck. Streaming stopped, every Update failed after 5 s with ServiceNotResponding, blah2 exited, and the service was left hung.

The exposure dates from v0.3.13 (2026-06-16), which added the post-Init gain Updates. The mutex added in ec84ee8 (July) is not the root cause: acking from the callback with no lock at all wedges just the same.

ClickUp: 123zgec74cj

Change

One file plus its header, in src/capture/rspduo/:

• event_callback() now records the overload state and a per-tuner pending-ack bit, then returns. It never calls sdrplay_api_Update() and never takes a lock.
• New send_overload_acks() sends the pending acks under sdrplay_update_mutex, so an ack never overlaps a retune() or the start-up re-stage. process()'s control loop calls it on a 50 ms tick (it used to sleep 1 s). Stats still print once a second.
• Overload onsets are counted on state transitions, not per Overload_Detected event, because an unacknowledged event repeats about every 125 ms. PowerOverloadChange is logged on a change of state only, for the same reason.
• An event for sdrplay_api_Tuner_Both (seen on Uninit) now updates both tuners. Previously it was attributed to tuner B only.

The post-Init gain/AGC re-stage and its Updates are unchanged.

Evidence

Bench work on a test node (RSPduo, API 3.15, 213 MHz illuminator) with a standalone harness that mirrors blah2's device setup:

Variant	Result
blah2's current pattern (callback waits on the mutex, then acks)	Wedged 3/3
Ack from the callback with no mutex	Wedged 8/8
Deferred ack (this PR's design)	Clean 13/13 start-ups, 5/5 frequency jumps

Neither simpler explanation holds on its own. A blocked event thread alone does not stop Updates, and an unacknowledged overload alone does not stop Updates. The deadlock needs the callback to call back into the API while another Update is in flight.

Verification

The real blah2 image from this branch (CI build-blah2, publish=false, commit 4319b30) against ghcr.io/offworldlabs/blah2:v0.6.3 on the same test node. Test containers used a staged copy of the node's config with the DAB notch off, so tuner A overloads at start-up.

Test	v0.6.3	This PR
Start-up with an overload	Wedged 3/3 (overload ~280 ms after NumDevs=1, then Failed to update Tuner B gain: ServiceNotResponding 5.4 s later)	Clean 15/15 (10 back to back, 5 after a forced service reset)
Live retune via blah2-api, 400 MHz to 213 MHz at 20/20/LNA 1	Retune Tuner A gain failed: ServiceNotResponding on all 3 attempts	10/10 jumps acked
Overload onset count	n/a	Exactly 1 per overload

On every fixed run the overload still fired, blah2 kept streaming with a map 0-1 s old, both tuners reached their configured gains, and nothing logged ServiceNotResponding or a failed ack. CI unit tests pass.

Later the same day, with this build pinned as the node's blah2 and retina-gui #110 deployed, driving the real Configuration page:

• Three reboots with the DAB notch off at 20/20/LNA 1, so the node overloads at start-up. Each time the radio came up streaming with the overload detected, and no ServiceNotResponding.
• A full Auto-Calibrate run: 3 towers, 278 retunes, 3 frequency changes between towers. No device errors, every retune applied, then saved and applied.
• A Quick Calibrate run started from an overloaded state: it backed off on overload during its soak, then saved a tuning that cleared it.
• AGC on (5 Hz) with the overloading config: the overload was detected and then corrected as AGC took effect, with no ServiceNotResponding.

Not covered: a soak longer than a few hours (both test nodes are left running this build), and the staged rollout.

Seen during testing but not caused by this change: two applies about a minute apart (Auto-Calibrate's auto-save, then a config change) restarted sdrplay_apiService while a freshly started blah2 held the device. The service segfaulted while releasing it, and the new service never registered the RSPduo, so blah2 hung at API open, before any code this PR touches. The cron watchdog recovered it within 7 minutes. That restart path is tracked separately.

Docs

blah2-arm has no prose docs for RspDuo. The reasoning lives in the code comments, which this PR updates (the event callback, the mutex, the onset counters and the start-up block). The README does not describe the ack. retina-gui's docs/features/auto-calibrate.md reads ServiceNotResponding as a hard overload, which is largely this bug. That needs updating in retina-gui once a release with this fix is pinned.

Not in this PR

sdrplay-restart.sh uses pkill -x sdrplay_apiService, which can never match because comm is truncated to 15 characters. Fixing the match is not a one-liner: sdrplay.service restarts after 1 s, so a working kill would restart the service on every blah2 start with no settle time, which retina-gui found can wedge the device. It is tracked separately.

Test plan

• CI build and unit tests pass
• Start-up with an overload, 15 runs, on a test node
• Live frequency-jump retunes onto a strong signal, 10 jumps, on a test node
• Three reboots on an overloading config
• A full Auto-Calibrate run through retina-gui (3 towers, 278 retunes, 3 frequency changes)
• Overload onset counts not inflated by repeated events
• Staged rollout: nodes known to overload at start-up come back after the update without ServiceNotResponding

🤖 Generated with Claude Code