Hourly Spotify session storms cause repeated audio-key timeouts and zone unavailability with many zones

[tobsch]

Summary

When many zones each have their own librespot device (i.e. inputs.spotify.offload=false on all/most zones), Spotify-side disruptions arrive as synchronized 11-event bursts roughly every 60–110 minutes that knock all zones offline simultaneously and trip the rapid-failure cooldown introduced in #244.

After the recent beta-latest digest c7cb9f… (incl. #240 + #244), the engine no longer wedges — recovery via cooldown+retry is working as designed. But during each burst, all zones are unplayable for ~5+ minutes, and the bursts repeat all day.

Environment

• Image: ghcr.io/lox-audioserver/lox-audioserver:beta-latest digest sha256:c7cb9fd747e40c057e3ef8d83fefe31b940cd8833212c62bd4bb246dc39dad44 (post-fix: retry spotify connect restart after cooldown failure #244)
• 11 zones, all with inputs.spotify.enabled=true, all with inputs.spotify.offload=false
• Single Spotify Premium account → 11 simultaneous Spotify Connect devices registered to one account
• Network connectivity to Spotify endpoints from the container is healthy (apresolve.spotify.com returns 200)

Observed cadence (real production data, last 36h)

11 distinct burst windows. Each burst = exactly 11 simultaneous errors (= one per zone):

Start UTC	Errors	Gap from previous
2026-04-30 17:49:37	11	–
2026-04-30 18:16:27	11	+26 min
2026-04-30 19:24:42	11	+68 min
2026-04-30 21:16:36	11	+111 min
2026-04-30 22:16:41	11	+60 min
2026-04-30 23:27:51	11	+71 min
2026-05-01 00:55:06	11	+87 min
2026-05-01 02:43:46	11	+108 min
2026-05-01 05:51:58	12	+188 min
2026-05-01 05:55:00	12	+3 min
2026-05-01 05:59:44	24	+4 min

The fact that every burst is exactly 11 events, one per zone, within the same second is the smoking gun: this is account-wide, not per-zone — Spotify is disrupting every Connect device on the account at once.

Error pattern in each burst

Per zone within a burst:

[ERROR] librespot_core::session  Connection to server closed.
[ERROR] librespot_core::dealer::manager  failed sending dealer request channel closed
[ERROR] librespot_core::audio_key  Audio key response timeout
[ERROR] librespot_playback::player  Unable to read audio file: Symphonia Decoder Error: end of stream
[WARN]  Zones|Manager  reason='spotify Audio key response timeout' playback error
[WARN]  Input|Spotify  attempts=11 windowMs=30000  spotify connect restart suppressed after repeated failures

The audio-key timeout fires near-simultaneously across zones, the player bails on the encrypted segment, and the spirc reconnects from the dealer channel close. After 11 such events in 30s, #244's suppression kicks in (correctly).

What's working

• fix: detect and recover from rapid Spotify track change loops #240's rapid-track-change loop detection
• fix: retry spotify connect restart after cooldown failure #244's cooldown + post-cooldown retry — confirmed by [node-librespot] first pcm chunk: bytes=… sample_rate=44100 channels=2 arriving ~6 min after each burst, without manual intervention. Pre-fix: retry spotify connect restart after cooldown failure #244 (image b01dce…, 2026-03-07), the engine would stay wedged for hours after a burst.

What's still broken (the user-visible bug)

During each ~5-min cooldown window, all zones report "playback error" and audio cannot be initiated. With 11 bursts/36h, that's ~55 minutes/day of unplayable state, distributed across the day. From the user's perspective: "I press Play and nothing happens" several times a day.

Hypothesis on root cause

Eleven independent librespot Connect devices all live-attached to one Spotify account is well above what most listeners produce. We suspect Spotify's backend periodically performs an account-wide reset/rebalance (token refresh, dealer-region rotation, etc.) and disrupts all sessions simultaneously. This is not a librespot bug per se — librespot is correctly reconnecting — but the count of simultaneous devices is what makes the disruption visible and synchronized.

This matches what we'd expect from rate-limit / fair-use behavior: the more devices you stack on one account, the more attractive a target your account is for periodic resets.

Proposed fix

Make inputs.spotify.offload=true the default for new zones, and surface this in docs/admin UI:

1. With offload, only the primary zone runs librespot; the other zones receive audio via the internal output bus.
2. Reduces librespot device count from N (= number of zones) to 1 per Spotify-enabled bridge.
3. Eliminates the synchronized-burst surface area entirely; a single-device account is far less likely to be disrupted.
4. Matches the multiroom pattern used by Squeezebox/Snapcast/Music Assistant.

Specific suggestions:

• Schema default: change the offload default from false to true in extractZonesFromPayload() / defaultZoneInputs() so new zones built from the Miniserver descriptor get offload enabled
• Migration: on container start, if there are >3 Spotify-enabled zones with offload=false and they share the same accounts[] entry, log a warning recommending offload
• Admin UI: add a tooltip on the offload toggle explaining the trade-off (per-zone Spotify Connect device vs. centralized routing) and that offload is recommended for installs with >3 zones on one account
• Docs: add a "How many Spotify Connect devices is too many?" section to the wiki/README

Why this isn't fully solvable inside librespot

Even if librespot's reconnect loop were perfect, Spotify's account-side behavior would still produce visible gaps during reconnect. The only durable mitigation is reducing the device count, which has to happen at the lox-audioserver layer (since lox-audioserver decides how many librespot instances to spawn).

Workaround for affected users today

In data/config.json, edit each zone's inputs.spotify.offload: false → true for all but one zone. Restart container.

Repro

Set up ≥10 zones, all with inputs.spotify.enabled=true and inputs.spotify.offload=false. Run for >12 hours. Observe burst pattern in container logs.

---

Happy to share full anonymized logs or test patches.

[tobsch]

Update: clarifying the hypothesis & in-progress experiment

To make this more actionable for others hitting the same pattern:

Hypothesis being tested

The bursts are not "an audio-key bug" or "a librespot bug" per se — they correlate with the count of librespot Connect devices registered to one Spotify account. Spotify periodically resets/rebalances all sessions tied to an account; the more simultaneous Connect devices an account has, the larger and more synchronized the resulting disruption. The "exactly N errors per burst, where N == zone count" pattern in the original report supports this.

Why the proper upstream fix shouldn't require Music Assistant

The current inputs.spotify.offload=true mode does reduce librespot count to zero on offloaded zones, but expects audio to be rendered by an external Connect device — in practice the Music Assistant bridge with registerAll: true. That's a hard dependency on a separate addon, which most users running lox-audioserver natively (not in HAOS) won't have configured.

A cleaner architecture would be one shared librespot Connect host per audioserver process, registered with Spotify as e.g. "Lox AudioServer", with output internally routed to whichever zone Loxone selected. Spotify sees 1 device per account regardless of zone count; no external bridge required. Effectively: today's per-zone offload mode, but with the renderer built into lox-audioserver itself instead of delegated to MA.

Experiment in progress (started 2026-05-01 06:20 UTC)

To validate the device-count hypothesis without depending on offload semantics or the MA path, on the same production system from the OP:

• Before: 11 zones × offload:false → 11 librespot devices on one Spotify account
• After: disabled inputs.spotify.enabled on 5 lower-traffic zones (kids' rooms, guest bath, backup kitchen, secondary apartment) → 6 librespot devices on the account
• No code changes, no offload, no MA — just fewer librespot processes
• Watching for: change in burst frequency over the next 3-12 hours. Baseline was ~1 burst per 60-110 min. If the cadence drops materially, device count is implicated. If it stays the same, we need a different theory.

Will post results here. If the hypothesis holds, the upstream proposal narrows to "add an internal-shared-librespot rendering path so users can reduce device count without sacrificing per-zone Spotify availability and without an MA dependency."

[tobsch]

Result: device-count hypothesis confirmed (~5× drop in burst rate)

61 hours of data after disabling Spotify on 7 of 11 zones (4 librespot devices remaining):

Phase	librespot devices	Bursts	Errors per burst	Burst rate
Before	11	11 in 36 h	11	0.31 /hour
Phase 2 (4 enabled zones)	4	4 in 61.7 h	4	0.065 /hour

Smoking-gun details

1. Burst rate dropped ~5×, slightly better than the 11→4 device reduction predicts linearly.
2. "Errors per burst now equals 4" — exactly tracking the new device count, just like in the original report it was exactly 11. This is what you'd expect if each burst is "Spotify ran an account-level housekeeping pass and disconnected one connection per registered device", and very hard to explain otherwise.
3. 42-hour quiet stretch since the last burst (longest ever observed on this system).

Ruling out "user wasn't using Spotify"

A reasonable concern: maybe bursts dropped because the user happened to be away. The data doesn't support this:

• Before: bursts hit at all hours, including 00:55 / 02:43 / 05:51 / 05:55 / 05:59 UTC — clearly idle periods, nobody was actively using Spotify.
• After: bursts also hit at idle times — 00:13 / 04:21 UTC. Same scattered timing.
• The timing distribution of bursts didn't change; only the frequency and size did, and they scaled with device count.

If usage drove bursts, we'd expect them to disappear during quiet hours. Instead they're scattered across the day in both phases.

Implication for the fix

The OP suggested "default offload=true for new zones," but the existing offload mode requires Music Assistant (or some external Connect renderer) to actually produce audio. That's a hard add-on dependency most native installs won't satisfy.

A cleaner direction: add an internal-shared-librespot rendering path so a single librespot Connect host inside lox-audioserver can serve any zone Loxone selects. Spotify would see one device per audioserver instance, not one per zone. No external bridge required. Zones can keep their Loxone-side controls; the renderer is just shared internally.

This would give the same dramatic device-count reduction we measured here, while preserving per-zone Spotify availability and avoiding the MA dependency that current offload=true mode requires.

Happy to dig deeper into specific zones (e.g. extract per-zone error timing) if useful.

[rudyberends]

These Spotify issues are hard for me to troubleshoot, because I cannot reproduce it myself (new account with key error issue)

Right now every zone has (or can have) its own connect instance. Sending audio to that instance will play it on that zone. If we only keep one central connect instance, how would you route it to a specific zone?

What are the exact log message when this burst occurs? Is it not possible to make a fix to just ignore this?

[tobsch]

Withdrawing the shared-librespot proposal — wrong frame

@rudyberends, re-reading your reply: you were right on both points and I over-engineered this.

The shared-librespot architecture would solve the burst pattern but kill the per-zone Spotify Connect device picker, which is a core feature of a per-zone install (mine included — I just didn't think through what I'd lose). The "burst rate scales with device count" data was real, but reducing device count is the lever that's most legible from outside the audioserver, not the right one. The right lever is recovery quality.

Your "make a fix to just ignore this" was the right instinct. The bursts are Spotify-side events tied to account state, librespot already reconnects cleanly, the only thing in scope here is shortening the user-visible gap. Specifically: the per-zone restartStreak counter in scheduleRestart can't distinguish "one zone in a reconnect loop" (the case #244 was built for) from "every zone hit simultaneously by Spotify." For the latter, every zone independently trips the streak threshold off the same single account-side event and gets the 5-minute cooldown — by construction.

Patch sketch

Opened a draft PR: #257.

One file, 38 added lines. Adds a static cross-zone recentBurstErrors registry on SpotifyConnectInstance. When >= 3 distinct zones report Spotify errors within 3 s, it's classified as a burst (not a per-zone loop): restartStreak is reset, a short jittered + zoneId-staggered restart is scheduled (8–25 s) instead of the cooldown. Single-zone loop detection is unchanged.

Three open questions in the PR body that I'd like a sanity check on before taking it out of draft:

1. Which errorCode strings does node-librespot actually emit during a burst? The detector triggers on type === 'error'; if dealer-channel-close events surface as a different type, the burst threshold won't be reached on them and we won't catch the right moment.
2. Whether notifyOutputError produces visible UI flicker we'd want to suppress on the burst path (a brief "buffering" instead of "playback error → playing").
3. Whether to reuse computeStartupDelay-style connectOrder * N stagger for the burst-recovery path instead of the simpler (zoneId % 11) * 750 I used.

Patched JS is running on my container as of today; will edit the PR with a real-burst log slice once the next one hits (typical cadence is 60–110 min on this account).

If the framing in the PR makes sense, happy to address those three points in-PR and move it to ready-for-review.

[rudyberends]

Fixed at the node-librespot layer in v0.4.2, bumped into lox-audioserver in f762932.

Root cause: the connect-host log-observer surfaced any audio key / decryption key log line as a type: "error", errorCode: "audio_key_error" event immediately. During an account-side dealer-disconnect burst, that log line fires while spirc is already reconnecting internally — meaning lox-audioserver got a hard error in a window where the session was about to recover on its own.

The fix adds a 5s grace window before the error is forwarded to JS:

A shared healthy_seq counter is bumped by the player-event observer on playing and position_correction events (paused excluded — a paused stream doesn't prove the dealer is healthy).
When an audio-key log line matches, the emit is deferred 5s via a spawned task. If healthy_seq advanced in that window → spirc recovered, the error is suppressed and error_sent is re-armed so a later real fault in the same session can still report. If not → emit goes through as before.
Effects on your scenario:

Burst case: dealer drops → audio-key timeout log → grace window starts → spirc reconnects within seconds → playing event fires → emit suppressed → lox-audioserver sees nothing → no restartStreak increment, no cooldown, no per-zone teardown. User experiences a ~5–15s gap instead of ~5 min.
Single-zone permanent fault: no recovery event in the grace window → error fires after 5s → existing restart/cooldown path runs as today. Net cost is a 5s delay on real fault reports.
Detection works per-zone; no cross-zone coordination needed.
This obsoletes the cross-zone burst-detector approach in #257 — happy to close that draft. Your measurement methodology is still useful for verifying: after deploying the bump, the bursts themselves will still occur on Spotify's side, but they shouldn't propagate to lox-audioserver as errors anymore. If you do see audio_key_error events in the log post-upgrade during a burst, that means spirc didn't produce a playing / position_correction within 5s and the threshold likely needs tuning — would be good data to share.

Thanks for the thorough investigation and the burst-rate data; the "exactly N errors per burst tracking device count" pattern was the key signal that pointed at the right layer to fix this.