Skip to content

serve: check every server→engine frame write and drop the pending entry when one fails - #1721

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
monotophic:serve/checked-engine-writes
Sep 23, 2026
Merged

JustVugg merged 1 commit into
JustVugg:devfrom
monotophic:serve/checked-engine-writes

Conversation

@monotophic

Copy link
Copy Markdown
Contributor

Authored by Claude Fable 5.1 in Claude Code, analysis in partnership with @monotophic

The server writes SUBMIT, IMAGE, CANCEL and STOP frames to the engine's stdin without
looking at the result. A short write leaves the tail of a frame unsent and desynchronises the
engine's framing; a BrokenPipeError is a ConnectionError subclass, so it falls into the
client-hangup handler and the client sees a silent close instead of an error. This checks every
write and names the failure.

What changes. A small _write_all loops on short writes over a memoryview and fails closed on a
None or zero return. CANCEL and STOP go through Engine._write_frame, which drops the request's
pending-map entry on any failure and then raises: an OSError becomes a named RuntimeError, and
_write_all's own fail-closed error passes through; the SUBMIT and
IMAGE writes use the same checked loop inside the existing block, under the same single lock
acquisition, in the same order. A failed write while the response is uncommitted is an HTTP 500
engine_error; on a committed stream it ends the stream with one status line and nothing spliced
in.

What does not change. Every frame the server writes is byte-identical to dev, in the same
order, for plain, image, cancel and stop requests; three tests pin those sequences and pass on
dev. Engine.generate()'s signature is unchanged. A raise inside a decode callback still leaves
the pending entry behind, as on dev; this change covers failed writes only.

Tests (c/tests/test_openai_server.py, run from c/): fifteen added. WriteAllTest (four: a short
write reassembled whole, a short first write answered with the correct remainder, and the None and
zero returns that fail closed), PendingMapCleanupTest (four: a failed CANCEL, STOP and IMAGE,
and a CANCEL that fails with no OSError at all, a zero-byte return), WriteFailureHTTPTest
(four: a dead engine at SUBMIT on the non-streaming and
the streaming chat paths, a STOP failing before commit, a STOP failing on a committed stream), and
PlainRequestFrameOrderTest (three frame pins: the stop flow, the cancel flow, and IMAGE before
SUBMIT under one lock acquisition). Eleven fail on dev;
the three pins and the committed-stream test pass on dev by design, the last because dev already
ends the stream, only without the guarantee. test_openai_server.py and test_anthropic_messages.py:
240 green; test_openai_server.py wall time 24.2 → 24.3 s, no new test over 0.30 s.

Limits. A frame that was partly written before the failure stays on the pipe; nothing here
restarts or quarantines the engine, and a later request on that engine gets whatever the engine
makes of the partial frame. On dev the same failure left the request waiting for a reply the engine
would never send; here it is a named error.

Context. This is split out of #1353, which is being reduced to its logprobs/echo work. The
two are independent in behaviour. They meet in two places: both add module-level code just above
class Engine, and both append to the end of the test module, so whichever lands second takes a
small rebase there.

_write_all loops a memoryview slice over a short write and fails closed
(a named RuntimeError) on a None or zero return instead of spinning or
silently dropping the tail of a frame. The initial SUBMIT and IMAGE writes
use it inside their existing write_lock block and except-Exception/
pending.pop cleanup, unchanged in shape; each names itself correctly in
that RuntimeError regardless of which one failed or how.

Engine._write_frame gives CANCEL and STOP the same checked write, and
drops this request's own pending-map entry on any failure of that write
-- an OSError from the pipe, or _write_all's own fail-closed RuntimeError
-- since the dispatcher only does that on this id's own DONE/ERROR frame,
and neither arrives when the write that would have solicited one never
reached the engine. An OSError is additionally re-raised as a named
RuntimeError instead of left as itself, so it cannot fall into do_POST's
client-hangup handler and read as a silent close; do_POST's existing
exception handling already turns the named error into a clean 500
engine_error, or, once the response has committed, a clean stream end.

Dropping the pending entry is scoped to the write path: a raise from
inside a decode callback, or the duplicate-ACCEPT guard, still leaves the
entry behind, identically to base -- pre-existing and unrelated to this
change.
@JustVugg
JustVugg merged commit 97622ad into JustVugg:dev Sep 23, 2026
29 checks passed
monotophic added a commit to monotophic/colibri that referenced this pull request Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants