Repository navigation
fix+perf(ws): seq rollback on pre-validation failure; drop per-frame shield; cache materialized orderbook - #278
Conversation
A malformed orderbook_delta/snapshot or order_group_updates frame (ValidationError, KeyError, programming bug) previously left the seq tracker over-advanced past a frame that was never applied and never delivered. The next legitimate frame's seq matched the over-advanced expected value, gap detection silently skipped, and the local orderbook desynced from the server with no resync trigger. Money risk on any strategy reading from the local book. Fix: - Widen `_process_frame`'s try/except to cover pre-validation + apply + dispatch as one unit. Any exception rolls back the seq watermark to `prev_seq` and re-raises, so the recv-loop's existing malformed-frame handler logs + continues the loop. - Propagate ValidationError out of `MessageDispatcher.dispatch` instead of swallowing it. order_group_updates (sequenced but validated inside dispatch) now also reaches the rollback path. Tests: - tests/ws/test_seq_rollback.py — end-to-end: snapshot seq=1, malformed delta seq=2, good delta seq=3 -> gap handler fires with expected=2 received=3 (both orderbook_delta and order_group_updates). - tests/ws/test_dispatch.py — update existing test that asserted the old swallow contract; it now asserts the re-raise behavior. Closes #241
…245) Hot path before: every WS frame allocated a Task, two Futures, and a contextvars copy via `asyncio.create_task(self._process_frame(raw))` + `asyncio.shield(inner)`, purely to make dispatch uninterruptible between recv and the next iteration. At >1k orderbook deltas/sec this dominated the recv-loop allocation budget and defeated the #199 in-place orderbook fast path the recv loop carefully preserves. Fix: - Replace cancel-based pause with cooperative pause. The recv loop polls `connection.recv()` with a 50ms timeout and checks a `_pause_pending` flag at the top of each iteration; on observing the request it sets `_pause_granted` and parks on `_resume_signal`. F-P-04 invariant (no in-flight frame dropped on pause) is preserved structurally: pause is only granted at the safe checkpoint between frames, so `_pause_recv_loop` blocks until any in-flight `_process_frame` returns. - Dispatch runs inline. No Task, no shield Future, no contextvars copy per frame. Only the wait_for TimerHandle remains, which is cancelled before firing on every non-quiet recv. - `_ensure_recv_loop` is now dual-purpose: resume a parked loop if one exists, otherwise spawn a fresh task. `_stop` sets `_resume_signal` so a parked loop wakes up and observes `_running=False` cleanly. Tests: - tests/ws/test_recv_loop_hardening.py — F-P-04 test renamed and updated to assert the new cooperative contract: pause_recv_loop blocks until in-flight dispatch finishes (positive assertion via pause_task.done() check before the dispatch barrier is released). Quiet-channel latency: bounded by _RECV_POLL_S (50ms). Busy-channel latency: zero — frame arrives before the timeout fires. Closes #245
Code Review — PR #278
Summary: Three distinct fixes bundled together — #241 (seq rollback), #245 (cooperative pause), #244 (orderbook cache). The correlation makes sense: all three touch the recv-loop / #241 — Seq rollback on pre-validation failureThe fix is correct and important. Moving the orderbook pre-validation inside the existing One question on the except Exception:
if tracked and sid is not None and self._seq_tracker:
self._seq_tracker.rollback(sid, prev_seq)
raise
#245 — Cooperative pause (drop per-frame shield)The cooperative design is cleaner and eliminates the per-frame Potential deadlock: recv loop dies while If the connection dies (e.g., The # Option A: timeout in _pause_recv_loop
try:
await asyncio.wait_for(self._pause_granted.wait(), timeout=2.0)
except asyncio.TimeoutError:
self._pause_pending = False # loop is gone, abandon pause
# Option B: _stop also sets _pause_granted
async def _stop(self) -> None:
self._running = False
self._pause_granted.set() # unblock any in-flight _pause_recv_loop
self._resume_signal.set()
...Option B is one line and directly reflects the invariant: if we're stopping, any pending pause should be considered "granted" (the loop won't come back anyway).
with contextlib.suppress(asyncio.CancelledError, Exception):
await asyncio.wait_for(self._recv_task, timeout=2.0)
with contextlib.suppress(asyncio.CancelledError, asyncio.TimeoutError):
await asyncio.wait_for(self._recv_task, timeout=2.0)Unexpected exceptions here are worth seeing, even at shutdown time.
In Python < 3.12, #244 — Materialized orderbook cacheClean implementation. The
_cached: Orderbook | None = field(default=None, repr=False, compare=False)Without
TestsOverall test quality is high:
Minor nit on Summary
The deadlock risk in |
Every `orderbook_delta` consumed via the high-level `subscribe_book` iterator previously paid an O(n log n) sort on both sides plus 2N `OrderbookLevel` validations + 1 `Orderbook` validation per delta. For a 200-level book at 100 deltas/sec that was ~40k Decimal validations/sec per ticker, completely defeating the #199 in-place hot path the recv loop carefully preserves. Approach (smallest surface area): memoize the materialized Orderbook on `_BookState._cached`. `_apply_delta_inplace` / `_apply_snapshot_inplace` invalidate via `state.invalidate()` when a side actually mutates. Subsequent `to_orderbook()` / `get()` calls return the same cached instance until the next mutation. Snapshot re-application allocates a fresh `_BookState` so the cache resets naturally without any explicit teardown. Trade-off: the previous `is not` identity guarantee between concurrent reads is no longer a contract (was an implementation detail of the always-rebuild path). The mutation-safety invariant holds at the *value* level — handed-out Orderbooks are never mutated in place by the manager. Tests: - tests/ws/test_orderbook_perf.py — identity-stability assertions: consecutive `get(ticker)` returns the same object; delta / snapshot invalidate the cache; `apply_delta`'s return matches the subsequent `get()` (same cached instance). - tests/ws/test_orderbook.py — `test_returned_book_is_snapshot_not_live_view` updated to assert the value-level mutation invariant (previously-handed-out book unchanged after later deltas) rather than the identity-distinctness mechanism. Closes #244
15c2e2f to
f544f82
Compare
Code Review — PR #278: seq rollback + cooperative pause + orderbook cacheSummary: Three focused fixes addressing real correctness and performance problems in the WebSocket layer. The code is well-reasoned and the intent is clear throughout. A few issues worth addressing before merge. #241 — Seq Rollback on Pre-validation FailureCorrect and well-structured. The two-part fix (re-raise in One subtle observation: the Partial-mutation / rollback asymmetry: if #245 — Drop Per-Frame Shield; Cooperative PauseThe cooperative pause design is cleaner than the previous shield approach. Replacing a per-frame Task allocation + contextvars copy with a simple boolean flag and two Events is a meaningful hot-path improvement, and the F-P-04 invariant is correctly preserved at the top-of-loop checkpoint. Issue — If
Result: deadlock. This is prevented in practice by Issue — missing test for The Minor — The comment on the #244 — Cached Materialized OrderbookClean and correct. The invalidation coverage is complete: snapshots create a new Minor — _cached: Orderbook | None = field(default=None, repr=False, compare=False)Without _cached: Orderbook | None = field(default=None, init=False, repr=False, compare=False)Test CoverageStrengths:
Gaps:
Summary
The core logic is sound. The |
Closes #241
Closes #245
Closes #244
Part of #273 (post-v2.4 audit), Wave 1 money-risk batch.