serve: check every server→engine frame write and drop the pending entry when one fails - #1721
Merged
Merged
Conversation
_write_all loops a memoryview slice over a short write and fails closed (a named RuntimeError) on a None or zero return instead of spinning or silently dropping the tail of a frame. The initial SUBMIT and IMAGE writes use it inside their existing write_lock block and except-Exception/ pending.pop cleanup, unchanged in shape; each names itself correctly in that RuntimeError regardless of which one failed or how. Engine._write_frame gives CANCEL and STOP the same checked write, and drops this request's own pending-map entry on any failure of that write -- an OSError from the pipe, or _write_all's own fail-closed RuntimeError -- since the dispatcher only does that on this id's own DONE/ERROR frame, and neither arrives when the write that would have solicited one never reached the engine. An OSError is additionally re-raised as a named RuntimeError instead of left as itself, so it cannot fall into do_POST's client-hangup handler and read as a silent close; do_POST's existing exception handling already turns the named error into a clean 500 engine_error, or, once the response has committed, a clean stream end. Dropping the pending entry is scoped to the write path: a raise from inside a decode callback, or the duplicate-ACCEPT guard, still leaves the entry behind, identically to base -- pre-existing and unrelated to this change.
monotophic
added a commit
to monotophic/colibri
that referenced
this pull request
Sep 23, 2026
…ngine writes (JustVugg#1721) under the logprobs work
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Authored by Claude Fable 5.1 in Claude Code, analysis in partnership with @monotophic
The server writes
SUBMIT,IMAGE,CANCELandSTOPframes to the engine's stdin withoutlooking at the result. A short write leaves the tail of a frame unsent and desynchronises the
engine's framing; a
BrokenPipeErroris aConnectionErrorsubclass, so it falls into theclient-hangup handler and the client sees a silent close instead of an error. This checks every
write and names the failure.
What changes. A small
_write_allloops on short writes over amemoryviewand fails closed on aNoneor zero return.CANCELandSTOPgo throughEngine._write_frame, which drops the request'spending-map entry on any failure and then raises: an
OSErrorbecomes a namedRuntimeError, and_write_all's own fail-closed error passes through; theSUBMITandIMAGEwrites use the same checked loop inside the existing block, under the same single lockacquisition, in the same order. A failed write while the response is uncommitted is an HTTP 500
engine_error; on a committed stream it ends the stream with one status line and nothing splicedin.
What does not change. Every frame the server writes is byte-identical to
dev, in the sameorder, for plain, image, cancel and stop requests; three tests pin those sequences and pass on
dev.Engine.generate()'s signature is unchanged. A raise inside a decode callback still leavesthe pending entry behind, as on
dev; this change covers failed writes only.Tests (
c/tests/test_openai_server.py, run fromc/): fifteen added.WriteAllTest(four: a shortwrite reassembled whole, a short first write answered with the correct remainder, and the
Noneandzero returns that fail closed),
PendingMapCleanupTest(four: a failedCANCEL,STOPandIMAGE,and a
CANCELthat fails with noOSErrorat all, a zero-byte return),WriteFailureHTTPTest(four: a dead engine at
SUBMITon the non-streaming andthe streaming chat paths, a
STOPfailing before commit, aSTOPfailing on a committed stream), andPlainRequestFrameOrderTest(three frame pins: the stop flow, the cancel flow, andIMAGEbeforeSUBMITunder one lock acquisition). Eleven fail ondev;the three pins and the committed-stream test pass on
devby design, the last becausedevalreadyends the stream, only without the guarantee.
test_openai_server.pyandtest_anthropic_messages.py:240 green;
test_openai_server.pywall time 24.2 → 24.3 s, no new test over 0.30 s.Limits. A frame that was partly written before the failure stays on the pipe; nothing here
restarts or quarantines the engine, and a later request on that engine gets whatever the engine
makes of the partial frame. On
devthe same failure left the request waiting for a reply the enginewould never send; here it is a named error.
Context. This is split out of #1353, which is being reduced to its
logprobs/echowork. Thetwo are independent in behaviour. They meet in two places: both add module-level code just above
class Engine, and both append to the end of the test module, so whichever lands second takes asmall rebase there.