Dingo router fix tool streaming - #73
Merged
Merged
Conversation
…ng until finish Client-side TTFT of pure tool-call streaming requests was effectively equal to the whole generation time: the post-processor only attached accumulated tool_calls to the finish chunk, and content-less deltas were suppressed, so no bytes reached the client during generation (Nexus TTFT/total median ~99.6%/97.4% vs ~17.2% for native SGLang). The post-processor now emits OpenAI tool_call deltas as the parser produces them: the id+name delta goes out with the first argument fragment and argument fragments stream as they arrive (the un-emitted tail of the accumulated arguments is the source of truth, so event granularity cannot produce duplicates). Correctness guards from the buffered design are retained: - names without any argument fragment are never emitted, so a misidentified prompt word cannot leak as a dangling call; - unknown tool names (not in the request's tool list) are suppressed mid-stream and purged at finish, exactly as before; - the finish-time full-text re-parse remains authoritative, now in merge mode: argument suffixes for partially streamed calls are patched (with streamed indices kept stable), calls the streaming parser missed are appended under fresh indices, and when nothing was streamed the re-parse rebuilds state exactly as before; - malformed (non-JSON) accumulated arguments of already-streamed calls can no longer be retracted, so they now trigger the finish-time re-parse for authoritative recovery instead of being silently dropped; finish_reason is still rewritten to tool_calls when a call was emitted. Tests: existing assertions now reassemble the delta stream per the client contract (semantic regression), plus new cases locking the TTFT contract (deltas arrive before the finish chunk, first entry carries id+name, distinct indices/ids for parallel calls) and the new malformed/unknown-name streaming contract. Not executed locally (no sglang env); run on gpu_1 CI.
githubgxll
had a problem deploying
to
external_collaborator
September 23, 2026 10:17 — with 
GitHub Actions
Failure
GitHub Actions
Failure
guojn1
approved these changes
Sep 25, 2026
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.