WsClient: stale _wsa reference causes permanent RuntimeError on reconnect

[tobsh]

Hi @Voyz, first off thank you for maintaining ibind 🙏. It's become the
backbone of our IBKR integration and we really appreciate the work you put
into it.

I think I've tracked down a reconnect bug in WsClient and wanted to
share what we found in case it's useful.

Describe the bug

After a transient WebSocket disconnect, WsClient._new_websocket_app()
raises RuntimeError: WebSocketApp should be closed before attempting to create a new one on every reconnect attempt, because the stale _wsa
reference is never cleared. The client never recovers. It enters a retry
loop emitting repeated Thread already running: ws_client_thread-None
warnings until the process is restarted.

Two code paths seem to combine to cause this.

1. _new_websocket_app raises instead of recovering
(ibind/base/ws_client.py line 188 on master, v0.1.23):

if self._wsa is not None:
    raise RuntimeError(f'{self}: WebSocketApp should be closed before attempting to create a new one')

2. _handle_on_close returns early without clearing _wsa
(ibind/base/ws_client.py line 308–310 on master):

if not self._connected:
    _LOGGER.info(f'{self}: on_close event while disconnected')
    return   # self._wsa still points at the dead WSA

A close-while-disconnected leaves _wsa stale, then the next
_new_websocket_app() sees _wsa is not None, raises, and the reconnect
never proceeds.

Steps to Reproduce

1. Connect IbkrWsClient to a gateway.
2. Cause a disconnect (kill the gateway, let IBeam auth expire, or pull the network).
3. Wait for restart_on_close to trigger a reconnect.
4. Observe RuntimeError: WebSocketApp should be closed before attempting to create a new one.
5. Observe repeated WARNING: Thread already running: ws_client_thread-None.
6. Client never recovers. Only a process restart fixes it.

Expected Behaviour

Reconnection after a transient disconnect should succeed. A stale _wsa
reference from the previous connection cycle shouldn't block future
reconnects.

Additional context

We've been running a local patch with the change below for several weeks
in a live trading daemon. Before patching, we saw repeated bursts of
Thread already running warnings after every disconnect; after, the
reconnect path is clean and the client recovers on its own. Nothing else
appears to change.

We also have unit tests covering the patched behaviour (stale-WSA
force-close, close-failure handling, happy path, and the
_handle_on_close cleanup branch). Happy to port them if that's useful.

I also came across some earlier reports that mention overlapping symptoms
(#25, #81, #129). They were closed due to inactivity, so I wasn't sure
whether to resurrect one of those or open something new. Happy to do
whichever is easier on your end.

Possible Solution

A couple of small changes seem to address it.

In _new_websocket_app, force-close the stale WSA and join the dead
thread instead of raising:

def _new_websocket_app(self) -> bool:
    if self._wsa is not None:
        _LOGGER.warning(f'{self}: Force-closing stale WebSocketApp before reconnect')
        try:
            self._wsa.close()
        except Exception:
            _LOGGER.debug(f'{self}: Stale WSA close failed, abandoning', exc_info=True)
        self._wsa = None
        if self._thread is not None:
            self._thread.join(timeout=5)
            self._thread = None
    # ... proceed with existing WSA-creation logic

In _handle_on_close, clear _wsa on the early-return path when the
client isn't running:

if not self._connected:
    _LOGGER.info(f'{self}: on_close event while disconnected')
    if not self._running:
        self._wsa = None
    return

If this looks right to you, I'd be glad to put up a PR. Just let me know
whether you'd prefer that or to take it from here. Either works!

Your Environment

• IBind version: 0.1.22 (also reproduces against master, v0.1.23)
• Python version: 3.13
• Authentication method: IBeam
• Operating System and version: macOS (launchd-managed daemon)

Thanks again for all your work on this library 🙏

[Voyz]

Hey @tobsh once again thanks for the kind words!

Also thanks for the detailed description of the issue and the discussion around it. Honestly, exceptional work on reporting these bugs 🙌

In all fairness - I understand what you're observing but I don't have a clear indication on what may be the best way to fix it. I've recently been noticing similarly unexpected behaviours from the IbkrWsClient, and I'm considering rewriting it from scratch to simplify the lifecycle management.

I can propose two options:

1. Bear with me as I refactor the IbkrWsClient - although I don't have a roadmap for it yet - and use your internal patch in the meantime
2. Work on introducing this fix first - even if temporarily -, then look into the full refactor

Which one would sound more reasonable on your end?

[tobsh]

Hey @Voyz , highly appreciate the transparency and for laying out the options so clearly!
I'm happy to go with whatever works best for you. If it helps unblock others in the meantime, I'd be glad to open a PR with my monkey patch, no pressure, just if it's useful to have it documented while you figure out the bigger picture. Either way, looking forward to seeing where the refactor goes! 🙌

[Voyz]

Hey @tobsh Thanks for being open to waiting 🙌 I've just published a PR draft with the WebSocket refactor.

It comes with a complete refactor of the threading and lifecycle model - the restart_on_close is removed as a parameter. I believe it should address the issue you've been facing. I've tested it with a variety of disconnect scenarios and so far managed to recover gracefully.

#151

Let me know if you get a chance to try it out against your scenario - would be awesome to see if it worked 👍

[tobsh]

Hey @Voyz First: thank you for the refactor, and apologies it took us this long to come back to you, your comment slipped past me. This is awesome work. We read through the V2 design and the lifecycle/state-machine approach is a big step up from what anyone could bolt onto V1 from outside. We’ll run the RC against our reconnect scenario (we test by killing the gateway container mid-session) and report back with results.

One thing we’d genuinely love to discuss, because it’s where our production pain actually lives: session recovery responsibility. In our experience the reconnect itself is only half the story. When the gateway loses authentication (competing session, gateway restart), we found that detecting it isn’t enough for our use case. We run an active recovery on our side: re-tickle, reauthenticate, re-init the brokerage session, and then, this is the critical part, a REST backfill that replays any orders/executions that fired during the unauthenticated window. Without that replay, a fill that lands while the socket is down is silently gone, and for anything tracking positions against a local store, that’s the difference between “reconnected” and “actually consistent again”.

So the open question from our side: do you see active re-auth and gap-recovery as something ibind V2 should eventually offer (even just as hooks, e.g. a clean event when authenticated flips back, so apps can trigger their own replay), or is that deliberately app-side responsibility? Both answers are fine for us. We’re happy to keep that layer, but if you’re open to it living closer to the client, we’d gladly share what our recovery flow looks like in practice.

Either way, thanks again for picking this up, and we’ll get you concrete RC test feedback soon.

[Voyz]

Hello again @tobsh - no worries at all, in all fairness the update is a lot more mature now. Let me know how it holds up in your tests.

Regarding the synchronization after reconnect - you're absolutely right in that it is a critical part. I likewise include it in the systems I build for my clients. I appreciate you bringing it up and offering to talk more about your solution.

It's not immediately evident to me whether it is something that could be handled by IBind out of the box, or whether it should be left out to application code. It is tempting to offer if it can be done cleanly, however sticking a 'we'll handle it' sticker on a functionality that comes with a great deal of quirks and uncertainties could give users a false confidence. I wonder whether it can be released without having to eventually write 'but just in case we miss something, make sure to implement your own synchronisation anyway'. Maybe it can be done, I'm just being cautious.

Wanna open a new issue and let's discuss it there?

In particular I'm curious, where you write "a REST backfill that replays any orders/execution" - for example for "orders" topic, do you mean fetching from REST and then firing new events.Orders event?

As for:

even just as hooks, e.g. a clean event when authenticated flips back, so apps can trigger their own replay

That is already made available in V2: https://github.com/Voyz/ibind/blob/feat/ws_v2/docs/websocket/core-concepts/events.md

The WsAuthenticated is such an event, which can be supplemented with SubscriptionUpdated (DEGRADED -> ACTIVE). There isn't a bespoke ReconnectedSoMakeSureYouSynchronize event at the moment, as I think you can derive it from these other two.

eg.:

from ibind import CallbackSink, IbkrWsClientV2, events

def on_authenticated(event: events.WsAuthenticated):
    # trigger synchronisation here
    ...

sink = CallbackSink()
sink.on(events.WsAuthenticated, on_authenticated)
    
ws_client = IbkrWsClientV2(account_id='[YOUR_ACCOUNT_ID]', sinks=[sink])
ws_client.start()

If there's anything missing here, let me know.