You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Learned context window (self-heal) treats any truncated upstream stream as a window overflow: false positives permanently shrink the session window, persisting across restarts #570
billion-context v0.1.86 (npm), proxy mode; upstream = local llama.cpp server (llama-server) with a real 150,528-token window; client = dsh CLI; sessions also run concurrently against the same llama server (two slots, shared KV pool).
Symptom
The self-heal mechanism learned a wrong conservative window for the local model: 140000 鈫?121815, and it sticks (re-applied on every request, persisted in the session store; visible across restarts):
03:48:38 [<session-a>] weak overflow signal 1/3 (usage 96%, round 1: upstream stream truncated without a completion event)
03:53:45 [<session-a>] weak overflow signal 2/3 (usage 96%, round 1: upstream stream truncated without a completion event)
03:58:52 [<session-a>] weak overflow confirmed (3脳 high-usage failures 鈥? 鈥?learned conservative window 121815 for qwen3.8-27b-abliterated-q4_k_m-9fbdc3cd (was unset); arming emergency shrink
03:58:54 [<session-a>] self-healed context window: 140000 鈫?121815 (learned from an upstream overflow)
04:04:05 [<session-a>] self-healed context window: 140000 鈫?121815 (鈥?
04:16:12 / 04:16:24 / 04:26:23 / 04:39:10 [<session-a>] self-healed context window: 140000 鈫?121815 (鈥? 鈫?every request
Root cause of the truncations here was NOT the model window: with two ~100K-token sessions pre-filling concurrently, llama.cpp's shared KV pool ran out (failed to find free space in the KV cache, W srv try_clear_id: purging slot 0 with 112764 tokens, failed to find N available cells). The stream was truncated by the server's KV pressure, but the proxy counted it as "high-usage failure 鈫?overflow" and shrank the learned window 14% below the true window (121,815 vs real 150,528). Once learned, the window is min()'d into every future request config for that model, throttling the session below its true capacity; and since it is persisted with the session, restart does not clear it (a manual edit of the session store was needed to unstick it).
Not a one-off: earlier sessions had learned windows 115,951 / 91,575 / 99,032 for the same model under the same KV-contention conditions, all below the real 150,528.
Observations (facts only, no proposed fix)
The overflow signal is computed from stream truncation (upstream stream truncated without a completion event) + usage %, with no attempt to distinguish window overflow from any other reason a stream can die mid-flight (server-side KV exhaustion, OOM, adapter bug, network reset, upstream restart). All of those look identical here.
The heuristic then persists a per-model learned window into the session store (metadata.learnedContextLimits), i.e. a single misjudged session poisons the model's window for all later sessions on this proxy, across restarts, until manually deleted.
Self-heal re-application is per-request and unconditional (self-healed context window: 140000 鈫?121815 on every request), so even after the stack recovered (session-a later ran healthy at 41K input, window still reported 121815 at 04:39:10), the throttled window persists.
The real window (150,528) was visible in the upstream's own overflow error body (exceed_context_size_error (198,277 / 198,661 > 150,528)) 鈥?i.e. ground truth existed in-band, but the learned path never consulted it in these episodes.
Suggested repro
Run two llama.cpp sessions concurrently (2 slots, shared KV) with 100K+ prefill each so the KV pool is exhausted; upstream streams truncate with failed to find free space in the KV cache. Watch the proxy log accumulate weak overflow signal 1/3 鈥?2/3 鈥?confirmed, then observe the learned window drop below the true window and stay there.
(Log excerpts redacted to remove session identifiers.)
[bot] 🏷 Verified against the v0.1.86 source — this is real, and your root-cause layering is correct. Triage summary before I ship a fix:
What I verified
The weak-overflow signal (src/weak-overflow.ts, added for 高 usage 下的非 400 失败(流截断/超时)无任何 overflow 兜底 —— 会话直接死 #498/0.1.63 应用在DeepSeek Harness超出 固定阈值没进行压缩 #351/重启/会话分裂后上游 prompt cache 命中掉到 0% —— 每请求全量付费(成本回归) #499) counts any non-400 stream death — truncated SSE without a completion event, or any upstream error status — as long as usage ≥ 90%. There is no attempt to distinguish window overflow from KV exhaustion, OOM, network reset, or upstream restart; all of them land in the same noteWeakOverflow() call sites (src/plugin.ts passthrough pipes, src/loop/core.ts rounds). Your llama.cpp shared-KV scenario feeds it directly.
On the 3rd high-usage failure in 15 min it persists input → metadata.learnedContextLimits[model]shrink-only, which means it also overwrites larger previously-learned values — including ground truth learned from a real overflow error body. That's your observation feat: OpenAI Responses API support (/v1/responses) #4 made mechanical: a speculative value can clobber a confirmed one.
The only correction path (the upward self-heal in src/server.ts) is gated on nativeFromFallback — i.e. it only second-guesses low-confidence fallback windows. An authoritative-looking base (your 140k) plus a poisoned learned value never heals on its own; nothing retracts a learned limit when later turns succeed above it. Persistence via the session store explains the restart survival.
Fix design (implementing now)
Provenance split: strong evidence (an actual non-2xx overflow rejection, incl. a parseable window number) goes to new confirmedContextLimits; the weak truncation heuristic keeps writing learnedContextLimits. Resolution prefers confirmed over weak, so speculation can never clobber ground truth again.
Self-correction (retraction): a learned/confirmed window is a hypothesis. If a later turn succeeds with upstream-reported input above it (beyond a small estimation margin), the upstream demonstrably accepted more than the "window" — the value is stale (KV-pressure false positive, server resized, etc.) and gets deleted, restoring the configured window. False positives can no longer be permanent; true positives can't be contradicted (a session can't succeed above its real window).
Weak path respects known bounds: when a confirmed window already governs, a weak confirmation only arms the transient emergency shrink — it no longer writes a smaller speculative window.
llama.cpp ground truth in-band: exceed_context_size_error (198,277 / 198,661 > 150,528) currently matches neither the overflow marker list nor the window parser in src/util.ts, so even a clean 400 from llama.cpp taught us nothing. Adding both, so a genuine rejection from your server confirms 150,528 instead of guessing.
One honest limitation: while the session sits below a false value (your 41K-under-121,815 case), there is no in-band evidence the value is wrong — all those death modes look identical mid-stream. Retraction fires once the session outgrows the false value and succeeds; the no-op-release/validation rules in AGENTS.md don't apply here since src/update.ts is untouched, but I'll run the full suite + e2e preflight before opening the PR.
[bot] 🏷 Verified and fixed — PR: #572 (branch 2026-09-06_weak-overflow-self-correction, merge left to you).
What I verified (master code paths, matches your logs exactly)
The misclassification is real.noteWeakOverflow (src/weak-overflow.ts) counts any stream death without a completion event while input ≥ 90% of the resolved window; 3 in 15 min → metadata.learnedContextLimits[model] = lastFailingInput. The reason string carries no signal distinguishing window overflow from KV exhaustion — llama.cpp's failed to find free space in the KV cache is byte-indistinguishable from a real overflow here. Your 1/3 → 2/3 → confirmed progression is exactly this path.
Persistence + no correction path. That value is min()'d into every subsequent request for the model (self-heal block in src/server.ts handle()) and markDirty()'d, so it survives restarts. The only upward self-heal requires the session to have previously run under a fallback window (nativeFromFallback) — your configured 140000 is authoritative, so the poisoned 121815 can never be raised back. Hence self-healed context window: 140000 → 121815 on every request, including the healthy 41K turn at 04:39.
Structural defect beyond the trigger: the weak heuristic and the strong path (a real non-2xx overflow rejection, which writes the ground-truth window) share the same persistent fields — no provenance distinction, and nothing anywhere can falsify a learned value against the configured window. Separately, llama.cpp's in-band ground truth (exceed_context_size_error (198,277 / 198,661 > 150,528)) was not recognized by inspectContextOverflow, so even a genuine overflow from llama.cpp would fall through to the guessing path (observation feat: OpenAI Responses API support (/v1/responses) #4 confirmed).
Root-cause layering
Symptom: throttled window persisting across restarts.
Direct cause: non-overflow stream deaths counted as overflow.
Structural causes: (a) speculative and confirmed evidence share one field; (b) no retraction/falsification path exists for learned values; (c) available in-band ground truth unparsed. Fixing only the trigger (e.g. filtering by error text) would leave (b) in place — any future unparseable death mode poisons the window forever. So the PR addresses all three layers.
The fix (three parts)
Provenance split. Actual overflow rejections (incl. the payload-size upper bound) now land in metadata.confirmedContextLimits / confirmedContextLimit; the weak heuristic keeps writing only to learnedContextLimits / learnedContextLimit. Resolution order is confirmed > speculative, and a weak confirmation is skipped entirely when a confirmed window for that model already governs.
Retraction. Before each request's limit is resolved, any learned window (either kind) that a successful turn has exceeded beyond a margin (max(256 tokens, 3%)) is deleted. A true overflow cannot be contradicted — a turn cannot succeed above the real window — and the armed emergency value (== the learned value exactly) sits below the margin, so it never retracts itself.
llama.cpp ground truth.exceed_context_size_error is now recognized and the limit is parsed from the (N / M > W) shape — a real overflow learns 150528, not a guess.
Both compression modes were considered: the changed paths are upstream-facing (error inspection, per-request limit resolution, session metadata) and identical for plugin/proxy mode; no wire-format change.
Honest limitation
While a session stays below a false value (your 41K turn under 121815), nothing in-band contradicts it — retraction fires once the session outgrows the false value and succeeds, after which the configured 140000 applies again. Recovery is therefore automatic but usage-bounded; for an immediate reset, deleting the learnedContextLimits entry from the session store still works as before.
Pre-flight
npm run typecheck clean; npm run build success
npm test: 1056/1057 pass — the single failure (launcher.test.ts, codex resolving to /usr/bin/codex in this sandbox) is pre-existing on clean master and unrelated
New unit coverage: confirmed-governs precedence, weak-can't-clobber, all retraction cases (stale removed / margin respected / armed no-op / confirmed retracts too / other models untouched), llama.cpp body → window 150528
Real-client E2E (tests/e2e/, needs ACP_TEST_E2E=1 + live upstream) not runnable in this sandbox; the local e2e-style self-heal suites (e2e-overflow-selfheal, issue-393, issue496) all pass.
[bot] 🏷 Update from the PR review (#572): independent verification of the preflight passed, and the design held up — with one exception. Review found that retraction could be fooled by failed turns' high-water marks (a KV-pressure death above the real window would later delete the just-parsed confirmed window, contradicting the fix's own "true overflows cannot be contradicted" invariant). Fixed on the PR branch (arm at exactly the parsed window; cap weak-overflow arms at a governing confirmed window) with unit + e2e regression coverage. Details in the #572 thread. Merge remains yours: #572
问题本质:self-heal(自愈窗口)机制把「上游流中断」一律当成「上下文窗口溢出」,而流中断的原因有很多种——llama.cpp 共享 KV 池耗尽(failed to find free space in the KV cache)、OOM、网络重置——这些都不是窗口溢出,但在这个启发式眼里长得一模一样。
撤回生效(核心验证点):retracted stale context window(s) [learned 121815] for <model> — a later turn succeeded at <N> input tokens above them; using the configured window again —— 会话输入长过假值并成功一次后就会出现,之后窗口回到配置值。
自愈日志现在带来源标注:self-healed context window: A → B (confirmed by an upstream overflow error) vs (weak-overflow heuristic),一眼能看出当前生效的是 ground truth 还是推测值。
[bot] 🏷 更正上一条:我给的 npm i -g github:ranxianglei/billion-context#分支名 不可用——仓库的 bin 指向 dist/index.js 且没有 prepare 脚本,直接从 git 装不会触发构建,CLI 会缺失。想不等正式版本先验证,请从源码装:
git clone -b 2026-09-06_weak-overflow-self-correction https://github.com/ranxianglei/billion-context.git
cd billion-context && npm ci && npm run build
npm i -g .# 或 npm link
Activity
ranxianglei commentedon Sep 6, 2026
[bot] 🏷 Verified against the v0.1.86 source — this is real, and your root-cause layering is correct. Triage summary before I ship a fix:
What I verified
src/weak-overflow.ts, added for 高 usage 下的非 400 失败(流截断/超时)无任何 overflow 兜底 —— 会话直接死 #498/0.1.63 应用在DeepSeek Harness超出 固定阈值没进行压缩 #351/重启/会话分裂后上游 prompt cache 命中掉到 0% —— 每请求全量付费(成本回归) #499) counts any non-400 stream death — truncated SSE without a completion event, or any upstream error status — as long as usage ≥ 90%. There is no attempt to distinguish window overflow from KV exhaustion, OOM, network reset, or upstream restart; all of them land in the samenoteWeakOverflow()call sites (src/plugin.tspassthrough pipes,src/loop/core.tsrounds). Your llama.cpp shared-KV scenario feeds it directly.input → metadata.learnedContextLimits[model]shrink-only, which means it also overwrites larger previously-learned values — including ground truth learned from a real overflow error body. That's your observation feat: OpenAI Responses API support (/v1/responses) #4 made mechanical: a speculative value can clobber a confirmed one.src/server.ts) is gated onnativeFromFallback— i.e. it only second-guesses low-confidence fallback windows. An authoritative-looking base (your 140k) plus a poisoned learned value never heals on its own; nothing retracts a learned limit when later turns succeed above it. Persistence via the session store explains the restart survival.Fix design (implementing now)
confirmedContextLimits; the weak truncation heuristic keeps writinglearnedContextLimits. Resolution prefers confirmed over weak, so speculation can never clobber ground truth again.exceed_context_size_error (198,277 / 198,661 > 150,528)currently matches neither the overflow marker list nor the window parser insrc/util.ts, so even a clean 400 from llama.cpp taught us nothing. Adding both, so a genuine rejection from your server confirms 150,528 instead of guessing.One honest limitation: while the session sits below a false value (your 41K-under-121,815 case), there is no in-band evidence the value is wrong — all those death modes look identical mid-stream. Retraction fires once the session outgrows the false value and succeeds; the no-op-release/validation rules in AGENTS.md don't apply here since
src/update.tsis untouched, but I'll run the full suite + e2e preflight before opening the PR.ranxianglei commentedon Sep 6, 2026
[bot] 🏷 Verified and fixed — PR: #572 (branch
2026-09-06_weak-overflow-self-correction, merge left to you).What I verified (master code paths, matches your logs exactly)
noteWeakOverflow(src/weak-overflow.ts) counts any stream death without a completion event while input ≥ 90% of the resolved window; 3 in 15 min →metadata.learnedContextLimits[model] = lastFailingInput. The reason string carries no signal distinguishing window overflow from KV exhaustion — llama.cpp'sfailed to find free space in the KV cacheis byte-indistinguishable from a real overflow here. Your1/3 → 2/3 → confirmedprogression is exactly this path.src/server.tshandle()) andmarkDirty()'d, so it survives restarts. The only upward self-heal requires the session to have previously run under a fallback window (nativeFromFallback) — your configured 140000 is authoritative, so the poisoned 121815 can never be raised back. Henceself-healed context window: 140000 → 121815on every request, including the healthy 41K turn at 04:39.exceed_context_size_error (198,277 / 198,661 > 150,528)) was not recognized byinspectContextOverflow, so even a genuine overflow from llama.cpp would fall through to the guessing path (observation feat: OpenAI Responses API support (/v1/responses) #4 confirmed).Root-cause layering
The fix (three parts)
metadata.confirmedContextLimits/confirmedContextLimit; the weak heuristic keeps writing only tolearnedContextLimits/learnedContextLimit. Resolution order is confirmed > speculative, and a weak confirmation is skipped entirely when a confirmed window for that model already governs.max(256 tokens, 3%)) is deleted. A true overflow cannot be contradicted — a turn cannot succeed above the real window — and the armed emergency value (== the learned value exactly) sits below the margin, so it never retracts itself.exceed_context_size_erroris now recognized and the limit is parsed from the(N / M > W)shape — a real overflow learns 150528, not a guess.Both compression modes were considered: the changed paths are upstream-facing (error inspection, per-request limit resolution, session metadata) and identical for plugin/proxy mode; no wire-format change.
Honest limitation
While a session stays below a false value (your 41K turn under 121815), nothing in-band contradicts it — retraction fires once the session outgrows the false value and succeeds, after which the configured 140000 applies again. Recovery is therefore automatic but usage-bounded; for an immediate reset, deleting the
learnedContextLimitsentry from the session store still works as before.Pre-flight
npm run typecheckclean;npm run buildsuccessnpm test: 1056/1057 pass — the single failure (launcher.test.ts, codex resolving to/usr/bin/codexin this sandbox) is pre-existing on clean master and unrelatedtests/e2e/, needsACP_TEST_E2E=1+ live upstream) not runnable in this sandbox; the local e2e-style self-heal suites (e2e-overflow-selfheal,issue-393,issue496) all pass.fix: keep failed-turn arms from retracting learned overflow windows (#…
ranxianglei commentedon Sep 6, 2026
[bot] 🏷 Update from the PR review (#572): independent verification of the preflight passed, and the design held up — with one exception. Review found that retraction could be fooled by failed turns' high-water marks (a KV-pressure death above the real window would later delete the just-parsed confirmed window, contradicting the fix's own "true overflows cannot be contradicted" invariant). Fixed on the PR branch (arm at exactly the parsed window; cap weak-overflow arms at a governing confirmed window) with unit + e2e regression coverage. Details in the #572 thread. Merge remains yours: #572
ranxianglei commentedon Sep 6, 2026
简述下这个是什么问题
ranxianglei commentedon Sep 6, 2026
[bot] 🏷 简述:
问题本质:self-heal(自愈窗口)机制把「上游流中断」一律当成「上下文窗口溢出」,而流中断的原因有很多种——llama.cpp 共享 KV 池耗尽(
failed to find free space in the KV cache)、OOM、网络重置——这些都不是窗口溢出,但在这个启发式眼里长得一模一样。机制链条:
exceed_context_size_error (N / M > W)),却没被解析。修复(PR #572,已开待合并):三管齐下——① 证据分级:真实溢出拒绝写入独立的 confirmed 字段并优先于启发式的 learned 字段,互相不再覆盖;② 撤回机制:某轮成功且上报输入超过学习窗口 max(256 token, 3%) 以上,即证明该窗口过期,在下个请求解析配置前删除(真溢出不可能被成功轮反驳,所以不会误伤);③ 解析 llama.cpp 报错体里的真实窗口,真溢出直接学到准确值。
fix: separate confirmed vs speculative overflow window evidence (#570)
fix: separate confirmed vs speculative overflow window evidence (#570)
ranxianglei commentedon Sep 6, 2026
[bot] 🏷 (你这条消息中文部分转发时乱码成了
?,我按「逐条确认 PR #572 是否覆盖这三点」来理解,如有偏差请指正。)逐条对照 PR #572 的实际改动:
1. 误判(KV 压力截断被当成溢出)——触发条件没变,后果被限定了。
代理端看不到上游日志,KV 耗尽和真溢出在「流中断且无完成事件」上仍然无法区分——这个启发式对 sglang 类后端(真溢出也是静默断流、无报错体)仍然有用,所以保留。变的是命中之后:
metadata.learnedContextLimits(原字段名保留),不再和真实溢出错误的 ground truth 共用一个槽位;若该模型已有 confirmed 值,弱路径直接跳过写入。max(256 token, 3%)以上 ⇒ 判定过期,在下一个请求解析配置前删除。KV 压力场景下,等会话输入长过 121815×1.03≈125469 并成功后,配置值 140000 自动恢复。learnedContextLimits。2. 持久化污染(跨重启、每请求 min)——持久化保留(这是自愈的本意),但不再是单向不可逆。
markDirty()落盘,删除本身也持久化——重启后不会复活。3. ground truth 未解析——已修。
inspectContextOverflow(src/util.ts)新增exceed_context_size_error模式识别,并从(N / M > W)形状解析>之后的数作为真实窗口(你的例子即 150528)。llama.cpp 真溢出返回非 2xx 时,直接学到准确值进 confirmed 字段并优先于一切推测值;单测覆盖了「取 W 而不是 N/M」。PR #572 目前 open、等人工合并(系统规则禁止我合并);pre-flight:typecheck/build 干净,测试 1056/1057(唯一失败是沙箱环境的 launcher 既有问题,与本次无关)。你提到的 #571/#575 我这边没有上下文,如果它们和这个改动有交互(比如也动 learned 字段或 preflight 证据门),合并前建议一起看一遍。
shengbuding commentedon Sep 6, 2026
感谢确认!补充我们环境里的实测观察,佐证 #572 的修复方向:
触发场景:本地 llama.cpp 多槽共享 KV 池(n_slots=2, unified KV)并发 prefill 时出现
failed to find free space in the KV cache类流中断,叠加输入 ≥90% 窗口 → weak overflow 信号三连后误写 learned=121815(真实窗口 150528)。KV 空间不足与真实窗口溢出在「流无 completion 事件」层面不可区分,与 上游长时间 prefill(首 token 前无输出)被 undici 默认 bodyTimeout=300s 掐断,UPSTREAM_TIMEOUT_MS=10min 配置形同虚设,大上下文请求必超时 #551/Preflight compression holds the request with zero response bytes; client undici headersTimeout (default 300s) kills it -> repeated[preflight] summarization aborted: client disconnected5-minute death loop, payload never forwarded #568 属于同一盲区。持久化污染路径:learned 值存于会话存档
payload.metadata.learnedContextLimits,并以min(learned, effective, 配置)参与每请求窗口判定,跨重启残留;我们曾通过清理存档恢复窗口。fix: separate confirmed overflow evidence from weak truncation guesses in window self-heal (#570) #572 的撤回机制(某轮成功且输入超 learned 上界即删除)正好覆盖这条路径。报错体解析:llama.cpp 错误体带真实窗口(
exceed_context_size_error (N / M > W)),解析它是比 weak 信号更可靠的 ground truth。待 #572(以及 #571/#575)合并发布后,我们会升级验证。
ranxianglei commentedon Sep 6, 2026
[bot] 🏷 谢谢实测佐证!你们三点观察和 #572 的修复路径完全对上了。升级验证时可以直接 grep 这几条日志确认行为:
retracted stale context window(s) [learned 121815] for <model> — a later turn succeeded at <N> input tokens above them; using the configured window again—— 会话输入长过假值并成功一次后就会出现,之后窗口回到配置值。self-healed context window: A → B (confirmed by an upstream overflow error)vs(weak-overflow heuristic),一眼能看出当前生效的是 ground truth 还是推测值。confirmed window N governs … arming emergency shrink only, learned window untouched,不写 learned 字段。两个实用提示:
learnedContextLimits假值会在第一次被成功轮反驳时自动撤回(删除本身也落盘,重启不复活);想立即恢复的话手动删字段的老办法依然有效。npm i -g github:ranxianglei/billion-context#2026-09-06_weak-overflow-self-correction(会现场构建)。#571/#575 合并前建议确认一下是否也触碰了
learnedContextLimits/preflight 证据门这两处,避免和 #572 的字段拆分打架。等你们的验证结果。ranxianglei commentedon Sep 6, 2026
[bot] 🏷 更正上一条:我给的
npm i -g github:ranxianglei/billion-context#分支名不可用——仓库的bin指向dist/index.js且没有prepare脚本,直接从 git 装不会触发构建,CLI 会缺失。想不等正式版本先验证,请从源码装:其余内容(日志 grep 点、存量假值自动撤回、发布节奏)不变。
Merge pull request #572 from ranxianglei/2026-09-06_weak-overflow-sel…
fix(loop): client abort miscounted as weak overflow signal
fix: only an upstream-stated window is ever learned (ranxianglei#969)
fix: remove runtime context-window learning entirely (ranxianglei#987)
fix(window): stop trusting numbers in overflow error text; retry fail…
fix(window): stop trusting numbers in overflow error text — recogniti…
fix: client abort miscounted as weak overflow; opencode-local system …
test: add proxy-mode model-switch regressions for in-band overflow ar…