Skip to content

Output-budget restore is discarded by the watermark fastpath rebind, so the restored max_tokens never reaches upstream while the log reports success #6752

Closed

Description

@SAXEM1997

Description

When the client sends a shrunken output budget (max_tokens: 1), bili logs that it restored the budget, but the restored value never reaches the upstream: the forwarded body still carries max_tokens: 1. The upstream then streams a single token, and the client reports "output token limit reached, answer truncated".

Repro:

  1. dsh (DeepSeek Harness) sends a turn on a large session — 2487 messages in, max_tokens: 1 (shrunk from its own raw-history estimate).
  2. bili logs output budget restored 1 -> 64000 (#546 …).
  3. The forwarded body (captured with diagnostics.dumpBody) still has max_tokens: 1 and no max_completion_tokens.
  4. The upstream records completion_tokens = 1 for that request; every subsequent turn repeats the same way.

Expected: whatever restoreOutputBudget() decides is what gets forwarded.

bili version

0.1.214 (plugin native, embedded in dsh)

Usage mode

Plugin native (bili runs inside the dsh profile)

Client and version

dsh 0.2.2-rc.3 (DeepSeek Harness)

Operating system

Linux — Ubuntu-based container on a self-hosted NAS

Upstream and model

Self-hosted new-api v1.0.0-rc.48 (one-api fork) → SGLang serving DeepSeek-Flash, declared in bili as { "context": 512000, "output": 32768 }

Config

{
  "providers": {
    "https://<host>:23000": {
      "models": { "Deepseek-Flash": { "context": 512000, "output": 32768 } }
    }
  }
}

Log excerpt

18:54:28 [info] [session-…] output budget restored 1 -> 64000 (#546: client shrank it from its raw-history estimate; high-water 64000 below known output ceiling — floored (#1665/#1840))
18:57:12 [info] [session-…] output budget restored 1 -> 64000 (#546: …)
18:57:13 [info] [sess=session-…] forward POST → https://<host>:23000/v1/chat/completions

The body that was actually forwarded at 18:57:13:

keys: model,messages,stream,stream_options,store,max_tokens,tools,thinking,reasoning_effort
max_tokens: 1
max_completion_tokens: (absent)
messages: 270   (folded from 2487)
tools: 73
reasoning_effort: "max"

Where it looks like it goes wrong

Byte offsets in the published 0.1.214 dist/index.js (single-line bundle):

symbol offset
wireObj=parsed 5,660,237
const serialized=JSON.stringify(wireObj); wireBody=serialized; 5,660,463
commitGraph() — all four call sites 5,661,400 … 5,664,339
restoreOutputBudget(parsed, …) — the only call site 5,777,288

So the request body is serialized into the wireBody string, and every commitGraph() runs, before restoreOutputBudget() is called. restoreOutputBudget() mutates the parsed object, but nothing re-serializes it into wireBody afterwards — and wireBody is what is forwarded. The log2("info", … restored …) line is emitted regardless, so the failure is invisible.

This is consistent with [compat] dropFields working, since that path explicitly handles both representations: const graph = separatorManifest !== void 0 ? wireGraph() : null; → dropCompatFieldsJson(graph) + commitGraph(), otherwise applyCompatDropFields(wireBody).

Possibly related: the restore lands after the last commit, so even a commitGraph() there would need the mutation to target the same object the graph holds.

Workaround

"compat": { "dropFields": ["max_tokens", "max_completion_tokens"] }

plus a restart. The upstream then applies its own default output limit and completion_tokens immediately returned to normal (649 / 1183 / 355 / 117 across the next 8 requests, end_reason: done). The cost is that bili's own output-budget management is bypassed for that route.

(Side note, not filed separately: after saving compat.dropFields on a running instance there was no [compat] dropped line and no change in the forwarded body until the process restarted. We did not try POST /__bili/config/reload — it may well be the intended path; if so, a pointer in the docs would have saved a restart cycle.)

Activity

  1. SAXEM1997 commented on Nov 17, 2026

    @SAXEM1997
    Author

    Confirming on 0.1.215 (installed today): the same code shape is still there, so this is not fixed in the release.

    Byte offsets in the published 0.1.215 dist/index.js (single-line bundle, 6,263,710 bytes):

    symbol offset
    first wireBody= assignment 5,671,522
    last wireBody= assignment 5,722,944
    first commitGraph() 5,675,760
    last commitGraph() 5,678,699
    restoreOutputBudget(parsed, …) — only call site 5,791,648
    wireBody reads after the restore 5,885,180 / 5,885,632 / 5,885,654

    Same conclusion as 0.1.214: every write to wireBody and every commitGraph() happens ~113 KB before restoreOutputBudget() runs, and afterwards wireBody is only ever read. The restored value still cannot reach the forwarded body.

    I can't observe the failure end-to-end on 0.1.215 because the workaround is still active on this instance, but the path is unchanged.

    Two PRs reference this issue — #6766 and #6827. Happy to re-test and drop the workaround as soon as either lands.

    Minor nit: when the budget field is listed in compat.dropFields, the output budget restored 1 -> 64000 line is still logged even though the field is dropped moments later. That line sent me looking at the wrong layer for a while — suppressing it (or noting the drop) would make the log honest.

  2. ranxianglei commented on Nov 17, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 Thank you for the 0.1.215 confirmation and the precise byte-offset table — that pins the persistence of the defect across the release.

    Root cause (clarified by PR #6766): the issue title says "mutates the body after it is serialized," but the actual mechanism is different. On a watermark fastpath hit, prepareOpenai re-parses every scalar field (including max_tokens) from the raw wire text, silently discarding the restored budget that restoreOutputBudget had written into the parsed object. The log fires at restore time, before the rebind, so it reports success while the wire still carries the client's shrunk value. Your byte-offset analysis correctly identified that the restored value never reaches the forwarded body — the rebind is the producer, not a serialization-order bug.

    Status of the two PRs:

    Either landing will unblock your re-test. #6766 is the targeted fix; #6827 is the batch vehicle.

    On the log-honesty nit (dropFields): valid observation. The output budget restored 1 -> 64000 line at side-request.ts:871 fires unconditionally at restore time; the compat.dropFields drop happens later in relay.ts:683–709 and logs its own line ([compat] dropped 1 field(s) per compat.dropFields (max_tokens)). Both lines appear, but the first one is misleading when the field is going to be dropped. I'll add a note to the drop log in relay.ts so it explicitly calls out that it overrides the earlier restore — a one-line change I'll fold into #6766's branch.

  3. changed the title [-]Output-budget restore mutates the body after it is serialized, so the restored max_tokens never reaches upstream while the log reports success[/-] [+]Output-budget restore is discarded by the watermark fastpath rebind, so the restored max_tokens never reaches upstream while the log reports success[/+] on Nov 17, 2026
  4. SAXEM1997 commented on Nov 17, 2026

    @SAXEM1997
    Author

    Thanks for the correction — the watermark fastpath rebind explains what my offsets only showed as a symptom. I've renamed the issue to match the real mechanism, so anyone searching later lands on the right path:

    Output-budget restore is discarded by the watermark fastpath rebind, so the restored max_tokens never reaches upstream while the log reports success

    Keeping compat.dropFields in place until #6766 (or the #6827 batch) lands; I'll re-test and drop it as soon as a release with the fix is out, and report back either way.

  5. ranxianglei commented on Nov 17, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 Good — the title now names the mechanism, which will save the next reader the same offset-hunt you went through. The log-honesty fix is already on #6766's branch (commit 1fcc31d114), so the drop log will be honest in the same release that fixes the rebind. Standing by for your re-test result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions