Repository navigation
Description
Environment
| bili | 0.1.202 (commit 6deb4547), plugin native lane, 127.0.0.1:18790 |
| client | DSH desktop 0.2.1-rc.2 (Electron, win32 x64, Windows 11) |
| upstream | copilot.tencent.com / workbuddy relay → deepseek-v4.1-flash, declared 1,000,000, effective 872,000 (128k output reserve) |
| host RAM | 15.2 GB total, 1.4 GB free at the time of the incident |
| config | compress.externalSummary enabled (3 targets, all on one local forward gate) |
Session: session-7ffc1d1c-8c3d-4966-8e0c-8cf4cee3c8f4 — an image-heavy 3D-scene development conversation, 3,550 messages, 271 image parts / 264 unique attachments / 138,965,404 B (132.5 MiB) raw = ~176.7 MiB after base64.
Summary
This session worked perfectly for days at a 93% compression ratio (inbound=43.6MiB → outbound=2.9MiB). It then became permanently uncompressible after a sequence of completely ordinary user actions:
quota exhausted → user switches model & retries → attribution lost → DIRECT (uncompressed)
→ 413 request body too large → user retries with rewind
→ 3,250 covered ids drift → 3,455 dead refs
→ every `compress` call REFUSED (view=0) forever
→ preflight death spiral, session permanently slow
None of these steps is a misuse. The user did what any user does when they hit a quota wall: switch model, retry, undo the bad turn. The result is a conversation that can never be compressed again.
Phase 1 — healthy (compression at 93%)
2026-10-18T17:43:28Z request: 3540 msgs, inbound=43.6MiB, outbound=2.9MiB
Seconds later the upstream returned 429 and DSH surfaced QUOTA_EXCEEDED (turn 9). Compression was not the problem at this point — it was working exactly as designed.
Phase 2 — model switch → attribution loss → DIRECT uncompressed → 413
After the quota error the user switched provider/model and retried. bili then lost the session attribution and every request went out raw:
2026-10-19T04:14:36.170Z [warn] [dsh-client] bili-native-dsh: model request sent DIRECT (uncompressed)
— takeover gate refused http://127.0.0.1:7868/inference/v1/chat/completions:
no active initiator attribution (agentless/background lane, third-party in-process
caller, or stale attribution after host resume?)
2026-10-19T04:15:23.251Z … http://127.0.0.1:8897/… (same)
2026-10-19T04:17:22.110Z … http://127.0.0.1:8897/… (same)
2026-10-19T04:22:48.882Z … http://127.0.0.1:12322/… (same)
19 such DIRECT lines exist on 10-19, but most are an hourly sidecar (00:21, 01:21, 02:22, 03:22 … 09:58) and are unrelated noise. The ones that matter are the cluster at 04:14–04:22 UTC, which lines up to the second with the two 413s (Beijing time = UTC+8):
| DIRECT (uncompressed) | 413 in DSH | Δ |
|---|---|---|
04:17:22.110Z → :8897 |
12:19:11 = 04:19:11Z (turn 11) |
~2 min |
04:22:48.882Z → :12322 |
12:22:43 = 04:22:43Z (turn 12) |
~6 s |
2026-10-19 12:19:11 {"message":"request body too large","code":"CLIENT_ERROR","status":413} (turn 11)
2026-10-19 12:22:43 {"message":"request body too large","code":"CLIENT_ERROR","status":413} (turn 12)
And critically: zero bili log lines mention this session on 10-19. It was invisible to the proxy all day — which is exactly why nobody noticed compression had stopped.
Note the feedback loop: the bigger the session, the more the attribution loss hurts, and the more likely the user is to switch models — which is what causes the attribution loss. A 43.6 MiB session never 413s while compressed; it 413s immediately once compression silently stops.
Phase 3 — rewind → ref drift → 3,455 dead refs
The user retried and used rewind/withdraw to undo failed turns (7 times). When bili re-took over the session it found the fold anchors no longer matched:
[fold-reconcile] resent history drifted: 3250 covered id(s) missing — reanchored 3250
unannounced client history rewrite detected (1203/4998 incoming message(s) carry pre-turn refs of 4720 known)
— marking compaction boundary (#1001)
Result: 63 blocks (46 active) that claim to cover m00001–m04397, but ~3,455 refs are dead. The token view looks healthy (271,287 / 872,000 = 31%) while the wire body stays at 44.7 MiB — i.e. the panel says "compressed" and the request is still raw. That gap is the single most misleading thing in this whole incident.
Phase 4 — compress permanently REFUSED (this is the part that never heals)
The session weighs 444 MiB resident, and bili's resident pool budget is 512 MiB:
[session] evicted session-7ffc1d1c… from resident pool (weight 466297620 bytes, 1 session(s) remain, budget 536870912 bytes)
[session] evicted session-7ffc1d1c… from resident pool (weight 449962280 bytes, 1 session(s) remain, budget 536870912 bytes)
[session] evicted session-7ffc1d1c… from resident pool (weight 449375528 bytes, 1 session(s) remain, budget 536870912 bytes)
With a second 87.5 MiB session open, 444 + 87.5 = 532 MiB > 512 MiB → the big session is evicted every single time. And an evicted session has no live message view, so compress refuses every attempt:
[acp-proxy: compress REFUSED m00004–m00016, m00120–m00123, … — no visible context
(view=0 message(s), 5048 ref(s) mapped); refs left untouched and nothing recorded as dead
— the view self-heals on the next model request (#3126)]
The refusal message tells the user "after the next turn you can compress the same ranges again". That advice is wrong for this session: the next turn re-evicts it, so every retry fails identically. The user retried 3 times over 40 minutes and never succeeded.
This is the core structural defect I want to report: above a certain session size, the compress tool is permanently unusable, and nothing in the UI or the error message says so. The user is left repeatedly clicking a button that cannot work, on a session that cannot shrink without that button.
Phase 5 — preflight death spiral, plus two reporting bugs
After I cleared the stale ledger (backed up and moved both origin snapshots) so the session could relearn refs, preflight rebuilt from scratch and did work — 64 real folds, 2,012,855 → 914,731 tokens, every one via externalSummary:
[external-summary] completed (preflight) 4/4 range(s); calls=4; batch=finished result=success
[Compressed m02732–m02825 → 1 block(s), ~87282 tokens saved.]
… 64 lines total …
[preflight] round 9/32: payload ~914731 tok vs window 872000 (deficit 42731), 149 viable range(s)
Then it failed anyway, 42,731 tokens short, because the 32-call summarization budget ran out:
preflight fail-fast 502 (retryable=false): context ~652251 tokens (down from ~1153146 before preflight)
exceeds the model window 872000 (model=deepseek-v4.1-flash), with 149 compressible range(s) still visible
and preflight compression could not bring it under: the preflight summarization budget
(32 calls per protection regime) was exhausted before the payload fit the window.
The over-window payload was NOT forwarded.
Two separate reporting defects are visible right there, both already filed — but they made this incident much harder to diagnose, so I'm listing them as they hit me:
- [preflight] Failure headline mixes calibrated size and raw stats, and labels input budget as model window #3663 — the headline mixes calibrated and raw quantities and calls the input budget the model window. Here it printed
context ~652251 tokens … exceeds the model window 872000, which is self-contradictory on its face. The actual decision value in the same preflight was914,731. - [Bug][preflight] Local 413 is forgotten when the baseline becomes an estimate, reopening oversized probes #3846 — after that local 502 refusal, the next attempt forwarded on estimate only:
request: 3637 msgs, inbound=44.7MiB, outbound=32.2MiB→[acp-usage] input=620921. So the refusal was not durable; a later pass reopened the oversized forward.
Why I think this is a design gap, not user error
Every single trigger is normal usage:
| what the user did | is it reasonable? | consequence |
|---|---|---|
| hit a quota wall | yes, unavoidable | — |
| switch model and retry | yes, the obvious thing to do | attribution lost → uncompressed forward → 413 |
| rewind a failed turn | yes, the obvious thing to do | 3,250 covered ids drift → 3,455 dead refs |
| click "compress" to fix it | yes, that's what the button is for | permanently refused (session can't stay resident) |
The failure mode is also silently irreversible: once refs drift, the token view stays plausible (31%), so nothing warns the user that the session has become permanently unable to compress. The only visible symptom is "slow and expensive", which looks like a model problem, not a compression problem.
A session that has already been compressed 63 times should not be able to fall off a cliff like this. Graceful degradation would mean: detect that a large session cannot stay resident, and either (a) say so plainly instead of suggesting a retry that cannot work, (b) allow compress to operate on the on-disk snapshot without a resident view, or (c) evict smaller sessions first so the one that needs compression can stay resident.
Relationship to existing issues
| issue | state | covers | gap |
|---|---|---|---|
| #3909 | open | image session permanently re-sending 48.6 MiB, fold blocked by cadence, archived images still on wire (pi lane, 503) | different lane, different trigger; does not cover attribution loss → 413, nor the resident-eviction → compress REFUSED loop |
| #2438 | open | preflight compress/timeout loop on sustained mass fold-coverage loss has no graceful degrade | describes the loop, not the entry into it; no resident-pool angle |
| #3859 + PR #3863 | open | no-visible-context guard bypassed for ranges overlapping active blocks, marking live refs DEAD | adjacent, but here the guard did fire correctly — the problem is that it fires forever |
| #3137 | open | empty view marks request refs DEAD | defense-in-depth, not the permanent-refusal UX |
| #3724 | closed | host-side rewind/undo tail truncation unhandled (summaries survive rewinds) | closed, but rewind is still one of the drift sources here |
| #2975 | open | workbuddy reverse proxy can't compress — no active initiator attribution |
exact same mechanism as my Phase 2; owner asked for A/B evidence on 10-13, last activity 10-14, still open |
| #3651 / PR #3653 | open / open | takeover-broken banner false alarm from agentless sidecar lanes | addresses banner noise; PR still unmerged, so real attribution loss still has no user-visible signal |
| #3872 | open | mid-session wire/dialect switch sends one request far beyond the declared window; no send-time size guard | same class of "no byte guard before send" as my Phase 2 |
| #3846 | open | local 413 forgotten when baseline becomes estimate, reopening oversized probes | hit me in Phase 5 |
| #3663 | open | preflight failure headline mixes calibrated/raw size, labels input budget as model window | hit me in Phase 5 |
| #3588 / #3641 | closed | image-offload rewrites history → fold refs break → "substrate destroyed"; root fix = "snapshot as session", identity claim by order not hash | #3641 closed 2026-10-18; if the implementation is not yet in 0.1.202, this incident is a live instance of the exact defect #3641 was meant to root out |
I could not find an existing issue that ties these together as one causal chain reachable by ordinary usage. That is what I am reporting.
Expected behaviour
- Size-aware
compress: if a session cannot stay in the resident pool, say so explicitly and stop suggesting a retry that is guaranteed to fail. Better: allow compression against the persisted snapshot so the tool is not size-gated. - Resident eviction policy: evict by need, not by size — a session that the user is actively asking to compress should win over an idle smaller one.
- Make the drift visible: when refs die, the status panel must not keep reporting a healthy compression ratio. Today
271,287 / 872,000 = 31%coexists with a 44.7 MiB raw wire body. - Attribution is load-bearing but silent: losing it should degrade loudly (or better, fall back to the in-band session header chain that the server side already understands — see The D4.1F model accessed via the WorkBuddy reverse proxy cannot be compressed. | 使用workbuddy反代的D4.1F模型无法压缩 #2975's analysis). Right now the only trace is a
warnline the user never sees, and the consequence is an uncompressed 43 MiB request. - Durable local refusals: a local "too large" refusal must survive an estimate demotion ([Bug][preflight] Local 413 is forgotten when the baseline becomes an estimate, reopening oversized probes #3846). Otherwise the refusal is advisory.
- Preflight budget vs reality: when preflight has 64 successful folds and is 42,731 tokens short, running out of a hardcoded 32-call budget and returning a non-retryable 502 is a poor trade. Either extend the budget when progress is being made, or report "almost — N tokens short, retry will continue" instead of a terminal error.
What I did locally (in case it helps reproduce or triage)
- Appended one official DSH
image/offloadevent (seq=9771, 269 targets) to the session log, taking 269 of 271 images off the wire. DSH accepted it, seq stayed monotonic, projection replayed correctly. This works and is worth considering as a supported path for image-heavy sessions. - Backed up and moved both origin legacy snapshots so the session cold-loaded empty and relearned current refs. Preflight then produced 64 real folds — so the ledger rebuild approach works, it just does not survive the 32-call budget.
- After all of that the session is usable again (produces assistant messages and tool calls normally) but still sends ~44.7 MiB per round, and
compresson it still cannot succeed.
Happy to run any A/B or provide further dumps — I have the full bili.log, the session's session.v4.jsonl.zstd, and the ledger backups preserved.
Activity
[-][dsh] A healthy 93%-compressed image session becomes permanently uncompressed by ordinary usage: quota → model switch → attribution loss → 413 → rewind → 3,455 dead refs → compress REFUSED forever (0.1.202) | 一个健康压缩率 93% 的图片会话被普通操作永久废掉:额度→换模型→丢归因→413→撤回→3455 死 ref→compress 永远被拒[/-][+][dsh] A healthy 93%-compressed image session becomes permanently uncompressed by ordinary usage: quota → model switch → attribution loss → 413 → rewind → 3,455 dead refs → compress REFUSED forever (0.1.202) | 一个健康压缩率 93% 的图片会话被普通操作永久废掉:额度→换模型→丢归因→413→撤回→…[/+]ranxianglei commentedon Oct 20, 2026
(ai: vllm-qwen/qwen3.8-27b)
查重: 不重复(open+closed 均按失败签名检索过)。你列的关联表我逐条核了:Phase 2(归因丢失→DIRECT 不压缩)与 #2975 是同一机制、同一症状(takeover gate 无备选归因通道),并入 #2975 的 A/B/C 决策,本票不重开;Phase 3 的 id 漂移根因在 snapshot-render 谱系(#3756/#3641/#3724);持久本地拒发=#3846、标题口径混用=#3663,各有独立线程。本票自己的新机制:大会话被驻留池反复驱逐 → compress 永久 REFUSED + 误导性恢复建议 + preflight 有进展却报终局错误。检索无重复项(查重: 无)。
标签: bug, dsh, scope/kernel, p1(核心承诺失灵——该会话压缩能力永久不可用,按 2026-10-19 修订归 p1,不降档)。
版本核对(先定版本):你的 bili 0.1.202 = commit
6deb4547。我 diff 了 tag→master 的相关文件:src/stream.ts(refusal 逻辑)零改动,session.ts/preflight.ts/dsh-native.ts 仅有无关小改——下面的代码级结论对你的版本成立,不存在"master 已修"的情况。已在代码里钉死的三处缺陷(不是猜):
src/session.tsreleaseInFlight :1051-1057):每个模型请求结束后立即按字节预算回写驱逐;你的会话实测驻留 ~444 MiB + 另一会话 ~87.5 MiB > 512 MiB 预算,两个活跃会话互相驱逐,任何一次 compress 都落在"视图为空"窗口里。src/stream.tsnoVisibleContextRefusal :640):拒发回执固定说"等下一轮模型请求后重试即可"——对预算驱逐场景这句话永远不成立,下一轮照样被逐。这是把你卡死的直接文案。src/server.ts:3603-3608 +src/preflight.ts:2778):preflight 预算耗尽的 failure 不带 retryable 字段 → 一律 502 retryable=false,即使本轮已经真实折叠了 N 个 range(折叠是持久的,下一轮从更小基线继续)。你那 64 次有效折叠每次都报"终局",就是这条。处置计划(连贯修复,一个 PR 收口本机制族):
sessions.residentMaxBytes,CONFIGURATION.md 已有文档)+ 日志加 PERMANENT 标记便于 grep;② preflight 有持久化进展的非确定性耗尽 → retryable=true + 消息带上"本轮已折叠 N 个 range、还差约 X token、下轮继续"。PR 马上来,八问体例收尾。
ranxianglei commentedon Oct 20, 2026
(ai: vllm-qwen/qwen3.8-27b)
无决策依赖的部分已修完,两个根因方向等你拍板。 PR:#3943 (CI 运行中)
本次修了什么(两处方言,都是「判决说谎」)
修复 1 — 「compress 永远被拒」现在会明说「重试不会好」。
旧拒收回执对所有情况都说「等下一轮模型请求后重试即可」。但对装不进常驻池的大会话,这个承诺永远不成立:每次模型请求结束后它都被逐出(超预算即驱逐是既有设计),视图在请求之间从不存活。现在 bili 在每次驱逐时记下该会话的实测内存占用;若「本会话占用 + 其他常驻会话 > 池预算」,拒收消息改为:此拒收不是暂时的——现在或下一轮重试都会以同样方式失败;恢复途径:关掉另一个打开的会话,或把
sessions.residentMaxBytes提到约 X MiB 以上(按实测数字给出)并重启。操作日志同步加PERMANENT标记 + 三个数字。真正暂时的场景(小会话、刚重启)保持原文一字不改。修复 2 — preflight 预算耗尽但已有进展时,不再报「死局」。
原来预算耗尽一律
retryable=false。但已折叠的 range 持久在会话里,下一次尝试从更小的基线继续折——事故中 64 次有效折叠(2,012,855→914,731 tok)每一轮都被判死局。现在:非确定性耗尽 且 有成功折叠 →retryable=true+ 说明「N 个 range 已折叠并保留,距窗口还差约 X tokens」。零进展耗尽与确定性耗尽(含 #3461 的 413 路径)行为不变。结构化八问
progressContinues条件翻转 retryable 并追加进展说明。ctx has 0 message(s)时所有 ref 被判悬空(cannot-anchor),报错归因误导 #3126 六测全绿);零进展/确定性 preflight 路径不变(Three issues found during real-model stress testing: WS passthrough can't connect — HTTP upstream is guaranteed to die; | 真模型压测发现的三个问题:WS 直通连不上 HTTP 上游必死、超限时死锁+紧急提示风暴、compress 激活本身太重 #3461、preflight-multi-range、preflight-fail-fast、模型压缩输出是输入的数倍 #1841 futility、[Evidence] pi 车道切供应商(同 modelId)后 fold 不随会话迁移:裸上下文 2.23MB(≥298K est tokens)被 preflight 10 秒拒 3 次;账面 ≤157K 与裸水位背离 10.5h #2847 doom、[Bug] Codex 额度中断后恢复长会话,preflight SSE 摘要为空并持续 502 #726 cooldown 全绿,共 60 个定向测试过);驱逐行为本身未动(LRU 选择逻辑零改动,只在删除后加一行记录);无新配置面。sessions.residentMaxBytes/BILI_SESSION_RESIDENT_MAX_BYTES(CONFIGURATION.md en/zh 已收录)。合并判定:单模块范围的判决映射修正 + 双向回归测试齐备,但 retryable false→true 属错误语义变更(§7.4「fallback/默认值变更是产品决策」),请你人工合并:#3943
等你拍板的两项(本事故的根因方向)
决策 1 — 驱逐策略(你的预期行为 2)
决策 2 — 快照式压缩(你的预期行为 1b,真正的根治)
相关线程(未重复劳动)
标签维持首报:bug + dsh + scope/kernel + p1(核心承诺失灵:健康压缩会话被普通操作永久废掉)。
ranxianglei commentedon Oct 20, 2026
(ai: vllm-qwen/qwen3.8-27b)
CI 小修:首推的 head 上
unused-exports门禁红了——新加的ResidentPressure接口只有模块内使用(跨模块只消费函数返回的推断类型),已去掉export一词(af0956b5e,独立标注 commit),本地门禁 + typecheck 复跑全绿,已推上 PR 分支。核心测试矩阵在新 head 上运行中。fix: preflight progress note measures the shortfall against the targe…
fix: name the permanent resident-eviction case in compress refusals (#…
fix: preflight budget exhaustion with committed folds is not terminal (…
fix: preflight progress note measures the shortfall against the targe…
refactor(#3977): Step 1 — pure extraction of the size-evidence stores…
step 2: fitEvidence — the single adjudicator over persisted size evid…
fix: name the permanent resident-eviction case in compress refusals (r…
fix: preflight budget exhaustion with committed folds is not terminal (…
merge: PR ranxianglei#3943 — resident-eviction honest refusal + retry…
merge: PR ranxianglei#3867 — persist local preflight 413 evidence so …
Merge pull request #3943 from ranxianglei/2026-10-20_resident-evictio…
ranxianglei commentedon Oct 21, 2026
四组件修复全部进 master,关闭:
验收状态:相关测试族全绿(llm-stream 4/4、recall 2/2、native-intercept 64/64、compress-from-store 4/4、eviction honesty 8/8);CI 全绿。老会话的已损形状(如 rewind 后的死 ref)在新版首次加载时走快照/词袋认领自愈,内容不丢(CCR 保底)。
dsh: negotiate session identity back onto unattributed retries (ranxi…
dsh: carry session identity on the request object via llm/stream (ran…
fix: stop resident-pool evict/cold-load thrash that OOMs the proxy (r…
fix: mark structurally-exhausted tier-recovery deterministic (#5766)