Skip to content

fix: add timeout to mqtt connect to avoid indefinite hang - #69

Merged
dahlb merged 1 commit into
dahlb:mainfrom
garymm:mqtt-connect-timeout
Jul 25, 2026
Merged

fix: add timeout to mqtt connect to avoid indefinite hang#69
dahlb merged 1 commit into
dahlb:mainfrom
garymm:mqtt-connect-timeout

Conversation

@garymm

@garymm garymm commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

get_rest_devices waits on the MQTT connect future with no timeout:

connect_future = await loop.run_in_executor(None, mqtt_connection.connect)
await loop.run_in_executor(None, connect_future.result)

If the Hatch cloud is unreachable in a way that doesn't fail fast (hung TCP, half-open connection during an outage), connect_future.result blocks its executor thread and the awaiting coroutine forever. In the ha_hatch integration this manifests as config entry setup hanging indefinitely during Hatch cloud outages.

This applies the same fix as #60 did for the shadow client operations: a bounded wait (30 s) that raises concurrent.futures.TimeoutError, which propagates through the existing except block in get_rest_devices so callers can retry with backoff.

Context

During a Hatch cloud outage on 2026-07-24 this hang (together with a companion bug in ha_hatch, PR incoming there) repeatedly locked up a Home Assistant instance from ~4:41 AM until the integration was disabled.

Testing

  • pytest tests/ — 23 passed

🤖 Generated with Claude Code

https://claude.ai/code/session_01KUv54T1tjc6TJYCsUXyPE2

connect_future.result was awaited with no timeout, so an unreachable or
half-open connection to the Hatch cloud blocked the executor thread and
the calling coroutine forever. Follows the same approach as the
MQTT_TIMEOUT added to ShadowClientSubscriberMixin in dahlb#60.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KUv54T1tjc6TJYCsUXyPE2
@yoniamir

Copy link
Copy Markdown

I independently observed the connect-side failure today on a separate production Home Assistant instance using hatch_rest_api 1.32.0. During repeated Core startups, get_rest_devices logged AWS_ERROR_MQTT_UNEXPECTED_HANGUP; Core then stopped answering its HTTP and WebSocket APIs. Safe mode restored service immediately, and disabling only the Hatch config entry allowed a normal stable restart.

A bounded Future.result(timeout=...) is the correct direction: failure of the Hatch cloud must make the integration unavailable, never leave config-entry setup pending indefinitely.

Please add regression tests covering:

  1. a mocked connect future that never resolves and raises TimeoutError within the configured bound;
  2. proof that the asyncio event loop remains responsive while the executor wait is pending; and
  3. propagation of a connect future that completes with an AWS exception, preserving the existing retry/error behavior.

It would also be useful to confirm that a timed-out native connect attempt is cleaned up or cannot accumulate across retries. Companion Home Assistant event-loop fix: dahlb/ha_hatch#320.

@dahlb
dahlb merged commit 21d7c6b into dahlb:main Jul 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants