fix(transport): implement persistent reconnection backoff in AsyncSmartTransport

[henfri]

Problem Description

When using AsyncSmartTransport in setups with a local proxy (e.g. modbus-proxy) or when a device's Modbus MCU is temporarily stalled or unresponsive, the TCP connection (_open()) succeeds immediately (<1 ms), which Tenacity considers a successful reconnect. However, subsequent PDU requests (send_and_receive()) time out.

Previously:

1. TimeoutError at the transaction boundary did not mark _must_reconnect = True.
2. Because each invocation of send_and_receive() executed under a separate call context, Tenacity's auto_reconnect attempt counter reset back to 1 on every invocation.
3. In integrations polling at regular intervals (such as huawei_solar), this resulted in continuous socket open/close cycles at ~1 Hz (>5,000 reconnects in 90 minutes), effectively hammering the proxy and hardware.

---

Key Invariants Preserved

• Lock & Pacing: _communication_lock and inter-frame pacing (wait_between_requests) remain strictly intact.
• Session Callbacks: on_reconnected remains at the canonical invocation point right after connection re-establishment and before any PDU transmission.
• Underlying Transports: async_tcp.py and other base transports remain untouched.

---

Proposed Solution

1. Persistent Recovery State:
   Introduced _recovery_failure_count and _next_reconnect_earliest across calls on AsyncSmartTransport.
2. Fast Recovery Semantics:
   • The initial communication failure during normal operation enters recovery mode immediately ($T_{\text{wait}} = 0,\text{s}$).
   • Subsequent consecutive recovery failures back off exponentially: $1,\text{s}, 2,\text{s}, 4,\text{s}, \dots$ capped at _RECONNECT_BACKOFF_MAX = 60.0\,\text{s}.
   • Backoff delay gating is checked before reconnection under _communication_lock.
3. Precise Exception Taxonomy:
   Expanded TRANSPORT_FAILURES to (TimeoutError, ModbusConnectionError, ConnectionError).
4. Application Success Coupling:
   Recovery state is reset upon receiving either:
   • A valid decoded response PDU, or
   • A decoded Modbus exception response (ModbusResponseError), confirming that the device MCU is alive and responding at the protocol level.
5. Clean Public Lifecycle:
   Public close() resets the recovery state, while internal transport drops during reconnection do not.

---

Tests & Verification

• Added 8 comprehensive unit tests in tests/transport/test_async_smart.py:
  • test_retry_with_new_connection_if_needed_includes_timeout_and_connection_error
  • test_timeout_at_transaction_boundary_marks_must_reconnect_and_closes_transport
  • test_persistent_backoff_progression_across_calls
  • test_application_success_resets_recovery_state
  • test_modbus_response_error_resets_recovery_state
  • test_tcp_connect_failure_preserves_existing_auto_reconnect_behavior
  • test_on_reconnected_called_at_canonical_point
  • test_public_close_resets_recovery_state
• All 45 transport unit tests pass (and full suite of 1,488 tests).
• Verified in a 60+ hour continuous production field test with Home Assistant Core 2026.9.0, modbus-proxy, and huawei_solar.

[wlcrs]

Hi, thank you for this PR. I'm currently on holiday with only a smartphone on me, so I'll review this in full when I return home in two weeks time.

One problem I have with your current approach is that it introduces an extra backoff mechanism on top of the tenacity one, which is not configurable by the user. I designed the API in such a way that all retry-logic can be expressed in tenacity-primitives, and I want to keep it that way. (For example: if an user of this library wants a constant backoff instead of an exponential one)

[henfri]

Hi @wlcrs,

I went ahead and refactored the PR to align with your Tenacity-based design:

1. Dropped custom backoff: Removed the hardcoded exponential formula and constants completely.
2. Tenacity-native RetryCallState: We now retain a persistent RetryCallState across calls and delegate backoff calculation directly to self.auto_reconnect.wait(state).
3. Exact Tenacity lifecycle: On consecutive failures, the state receives the exception on outcome, evaluates wait(state), triggers before_sleep(state) (handled async-safely with upcoming_sleep), and advances via prepare_for_next_attempt().
4. Full configurability: Custom user strategies like auto_reconnect=AsyncRetrying(wait=wait_fixed(5)) or custom exponential windows are now honored.

Enjoy the rest of your holiday, and take your time reviewing whenever you're back!
In the meantime I wil lbe testing this with my Huawei Sun2000 and your Homeassistant Component

[wlcrs]

Ok, now that I'm back from holiday I reviewed your PR. Maybe it's the jetlag, but it is very hard for me to get a mental image of the whole state machine that is proposed in this PR. And in terms of complexity that is where I draw the line: I must be able to explain it to myself so that I can also explain it to others.

I went back to the drawing board to try and solve your issue in a simpler manner. Can you please review and test https://github.com/wlcrs/tmodbus/compare/improve-smart-reconnect#diff-2dc315e3125f6327e87c2af74d57676af3b436c24b615b6e21126303b0e306eb instead?

In this alternate approach, we improve the existing retry-loop instead of creating a secondary loop on top of it. As we already have the _communication_lock that serializes the send_and_receive calls, these retry loops are nicely executed one after another. Simple and easy to explain.

[henfri]

Hi @wlcrs,

Thank you for looking into this and proposing the alternate approach in improve-smart-reconnect.

I completely understand your point regarding complexity and maintainability. Your solution is wonderfully simple and lean (~15 LOC), and relying on the existing response_retry_strategy instead of introducing an inter-call state machine is much easier to reason about.

I just tested your improve-smart-reconnect branch live in my production Home Assistant environment (Core with huawei_solar through modbus-proxy) for over 24 hours:

• All 41 transport tests pass with 100% coverage.
• The underlying socket is now properly closed on TimeoutError, and _retry_with_new_connection_if_needed triggers a clean reconnect within response_retry_strategy.
• In normal operation and transient recovery, everything runs smoothly with zero socket flapping (only a few isolated device timeouts over 24 hours, all of which recovered immediately on the next poll).

From my side, I am more than happy to close my PR in favor of your improve-smart-reconnect branch! Thank you for the great library and for taking the time to address this.

Greetings,
Hendrik

[wlcrs]

Thank you for confirming. I've merged the changes in #152 and they will be included in the next release.