Skip to content

fix(ws): pass skip_utf8_validation=True to run_forever to survive non-UTF-8 text frames#149

Merged
Voyz merged 1 commit into
Voyz:masterfrom
tobsh:fix/ws-skip-utf8-validation
May 3, 2026
Merged

fix(ws): pass skip_utf8_validation=True to run_forever to survive non-UTF-8 text frames#149
Voyz merged 1 commit into
Voyz:masterfrom
tobsh:fix/ws-skip-utf8-validation

Conversation

@tobsh

@tobsh tobsh commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Hi maintainers,

Closes #148.

This PR fixes a silent WebSocket failure where WsClient._run_websocket calls wsa.run_forever(...) without skip_utf8_validation=True. When IBKR emits a text-opcode frame containing bytes outside the UTF-8 range (we've observed 0x99 in the wild on order/trade channels), websocket-client's frame decoder raises UnicodeDecodeError inside its receive loop. The run_forever thread then dies before the close handshake completes, on_close never fires, the reconnect path is never triggered, and event delivery stops silently — no error, no reconnect, just a quiet stall.

Passing skip_utf8_validation=True lets raw bytes through to the application layer (where WsClient already wraps message handling in its own try/except), which is consistent with how the upstream websocket-client library expects this case to be handled.

Key changes:

  • ibind/base/ws_client.py: pass skip_utf8_validation=True to wsa.run_forever() and a short # why comment naming the failure mode.
  • test/integration/base/test_websocket_client_i.py: _verify_started now also asserts skip_utf8_validation=True is passed. Since this helper is shared by every happy-path test in the file (test_start_success, test_start_success_on_second_attempt, test_open_and_close, test_send, test_check_ping, …), the kwarg is regression-pinned. Also extended the local run_forever_exception mock signature in test_open_exception so the new kwarg doesn't TypeError-loop the reconnect path.
  • test/integration/base/websocketapp_mock.py: extended the shared run_forever mock signature with skip_utf8_validation: bool = False for the same reason.

Verification:

  • pytest test/unit test/integration → 78 passed locally.
  • make lint is unchanged on the modified files.

Notes:

  • Diff is intentionally minimal: 1 prod kwarg + 3 test/mock signature updates. No new behaviour, no API surface changes.
  • We've been running this patch in production for several weeks with no regressions on well-formed frames.

Happy to adjust scope, comment, or test layout based on review.

Thanks!

IBKR occasionally emits non-UTF-8 bytes (e.g. 0x99) in text-opcode
frames, which causes websocket-client's frame decoder to raise
UnicodeDecodeError inside its receive loop. The run_forever thread
then dies before the close handshake completes, so on_close never
fires and the reconnect path is never triggered — event delivery
stops silently.

Pass skip_utf8_validation=True so websocket-client lets raw bytes
through; decoding is already handled above this layer.

Closes Voyz#148

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Voyz Voyz marked this pull request as ready for review May 3, 2026 14:22
@Voyz

Voyz commented May 3, 2026

Copy link
Copy Markdown
Owner

LGTM, thanks for the contribution! 🙌

@Voyz Voyz merged commit badaf02 into Voyz:master May 3, 2026
8 checks passed
@tobsh tobsh deleted the fix/ws-skip-utf8-validation branch May 17, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WsClient: run_forever called without skip_utf8_validation, crashes on non-UTF-8 bytes in text frames

2 participants