Skip to content

[PR] feat: observability root cause — nudge recovery rate in web chart/report + first-request dsh coverage readout (#6729 #6770) - #6779

Closed
ranxianglei wants to merge 0 commit into
masterfrom
2026-11-15_observability-root-cause
Closed

ranxianglei wants to merge 0 commit into
masterfrom
2026-11-15_observability-root-cause

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Root-cause fix: observability (aggregation of #6729 + #6770)

This PR is the single root-cause fix for the observability root-cause batch — two "diagnostic data exists but has no usable exit" gaps:

  1. Feature: show nudge-recommended vs actually-folded tokens per fold (recovery rate) in the web panel chart and session report | 功能:web 折线图与 session 报告显示每次压缩「推荐 vs 实际折叠」回收率 #6729 — nudge-recommended vs actually-folded (recovery rate) is invisible.
    The Every compression only trims a little: the post-fold floor ratchets up and a 1M-window session ends at 212K without recovering | 每次压缩只降一点:压缩后的地板越压越高,1M 窗口会话涨到 212K 不回落 #6720 investigation showed the model often folds only part of a nudge-recommended range, so the "post-compression floor" ratchets up turn after turn — but the recommended amount and the actually-folded amount had no visualization anywhere, so it took 30+ lines of log reading plus source reconciliation to pinpoint. This adds, per fold:

    • the recommended amount (pendingT1) recorded at nudge-injection time (only real injections — veto/skipped never book),
    • attribution (same-turn / next-turn / ≤10-min burst chain → nudge; no injection and not in a burst chain → active compression; >30-min stale nudges are not attributed),
    • recovery bars on the web session chart (outline = recommended, fill = folded, green ≥80% / yellow 40–80% / red <40%, blue = active; hollow = recommended but never folded — the direct ratchet signal),
    • the session-level average recovery rate (Σ folded / Σ recommended) on the chart and in the downloadable session report (svg/md) with a summary table.
      Persistence goes through the existing sqlite unified storage (session.metadata ledger JSON blob, sparse — the field is absent until the first nudge injection after upgrade); no backfill for pre-upgrade sessions; no new config; kernel/ untouched.
  2. [Diagnosis] Compression coverage reading for #3962 is unavailable on device: profileContext.startedBundles not detected | [诊断] #3962 的压缩覆盖读数在设备上取不到数:profileContext.startedBundles 探测不到 #6770 — the dsh.bundle.patch.yml gates four preset ids that may not exist — the miss is only visible in the loader log #3962 compression-coverage readout had no reachable exit.
    dshCompactionGateTelemetry().covered only ever rode the durable profile-bundle note, which stays silent wherever profileContext.startedBundles cannot be read (observed on both a desktop dsh build and the dsh-go runtime) — the reading was unreachable on every host checked. This emits the reading once per process on the first routed request via the already-confirmed [dsh-client] channel. The shared formatter compactionCoverageText() keeps the profile note byte-identical.

Provenance

Cherry-picked with -x (authorship preserved) from the two CI-green member PRs, whose directions match the owner's 2026-11-14 rulings:

This PR supersedes #6745 and #6772 (both left open; closing is yours).

Verification

  • typecheck clean; targeted local runs: dsh-native 77/77 (incl. the new wiring test), issue6770 formatter 4/4, issue6720 recovery 11/11, text-safe 18/18, session-report + web-i18n + cache-ledger 59/59.
  • Full-suite verdict: remote CI on this head.

Behavior changes

None beyond the new observability surfaces: one new [dsh-client] log line per process on the first routed dsh request; a new recovery section in the web session chart and the downloadable session report. No config surface, no wire/protocol change, no kernel/ change, no persistence migration (the recovery fields ride the existing session.metadata ledger blob).

Fixes #6729
Fixes #6770

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-15_observability-root-cause (42edd38)

Latest build — install this exact version

npm install -g billion-context@0.0.10508-pr.6779.9b9cacc

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
42edd38 billion-context@0.0.10508-pr.6779.9b9cacc build
41a113e billion-context@0.0.10195-pr.6779.44737c9 build
8612cab billion-context@0.0.10047-pr.6779.d8a8a0b build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6779.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title feat: observability root cause — nudge recovery rate in web chart/report + first-request dsh coverage readout (#6729 #6770) [PR] feat: observability root cause — nudge recovery rate in web chart/report + first-request dsh coverage readout (#6729 #6770) Nov 15, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Status report — PR #6779 (root-cause aggregation: #6729 nudge-recovery visibility + #6770 first-request dsh coverage readout).

I pulled head 8612cab46b and reconciled it against the description.

Commits (5, on current master f21b601893, 0 behind):

Diff: 14 files, +879/−19. No kernel/, no version touch, no config surface, no lockfile/generated churn.

CI (live, head 8612cab46b): all green — 13 guard lanes (kernel / version / config-surface / lane / architecture-docs / lock-sync / …), static-gates (ubuntu 24 + windows 22), all 11 test shards (ubuntu 22/24 × 3, windows 22 × 5), and mysql-contract pass. red-signal + the e2e lanes are skipped (conditional gates). 0 failures.

Labels: enhancement / p2 / scope/ui / root-cause — present.

Conclusion: the branch matches the PR description, CI is green on the current head, and the base is current master. Nothing to fix — ready for you to merge. After merge, #6745 and #6772 can be closed (yours to close).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

PR #6779 — merge conflict resolved (rebased onto current master), pushed.

What I did this run:

  1. Live state check: PR head 8612cab46b, base master at 2e3cfc973b — 87 commits / 105 files ahead of the PR's merge base f21b601893.
  2. Rebased the 5 PR commits onto current master. Six files were touched on both sides; three auto-merged cleanly (src/cache-ledger.ts, src/agent/dsh-native.ts, src/server/prepare-responses.ts), three conflicted on the same final-return region:
    • src/server/prepare-anthropic.ts:695, src/server/prepare-google.ts:430, src/server/prepare-openai.ts:663 — master's [PR] fix: un-show a nudge whose request the upstream rejected #6531 rewrite of the final Prepared return (adds the nudgeStampsBefore field) vs this PR's recordNudgeInject(session, nudge) insertion immediately before that same return line.
  3. Resolved all three as the union of intent: keep the PR's if (nudgeOutcome === "injected") recordNudgeInject(session, nudge); line AND master's return line carrying nudgeStampsBefore: nudgeOutcome === "injected" ? nudgeStampsBefore : undefined. No other hunks touched — the rebased diff vs master is the same scope as the original PR (14 files, +879/−19).
  4. Verified locally: typecheck clean, build clean, targeted suites green — issue6720-nudge-recovery + issue6770-compaction-coverage-text + cache-ledger + dsh-native + text-safe + session-report + web-i18n = 169/169, plus master's [PR] fix: un-show a nudge whose request the upstream rejected #6531 lane test rejected-nudge-rearm 4/4. Full-suite verdict: remote CI on the new head (running now; the mirror cache still shows the previous head's all-green 30-check set).
  5. Force-pushed lease-pinned to the old head 8612cab46b (unchanged at push time — no concurrent work lost). New head: 41a113ea6c. Live check after push: PR is now mergeable: true, state blocked = awaiting review approval (human-only), not a conflict.

Reviewer follow-up (non-blocking, for the PR author): one semantic interaction between this PR and master's newer #6531 (rejected-nudge re-arm). recordNudgeInject books the "n" event at prepare time; if the nudge-carrying request is then 4xx-rejected, rejectNudge (src/server/relay.ts:2161) rolls the nudge stamps back but the recovery ledger keeps the "n" event — a model-initiated fold within 30 min could then be attributed to a nudge the model never saw. Rare edge; it affects only the new recovery-rate metric's attribution, nothing functional. Minimal fix if wanted in this PR: drop the matching "n" event at the rejectNudge site; otherwise track as follow-up. I left behavior exactly as the PR designed it (union of both sides, nothing removed).

七问:

  1. 是什么问题: PR 分支落后 master 87 个提交,与 base 分支产生合并冲突(merge state: dirty),PR 无法合并。
  2. 什么原因导致的: PR 的 5 个提交基于旧 master(f21b601893)开发;期间 master 合入了 [PR] fix: un-show a nudge whose request the upstream rejected #6531(4xx 拒绝时回滚 nudge 时间戳)等 87 个提交,其中 3 个 prepare-* 文件改写了本 PR 插入 recordNudgeInject 调用的同一行(函数最终 return 语句),产生内容冲突。
  3. 影响范围与严重程度: 仅该 PR 分支本身——冲突期间 PR 无法合并;不影响任何已发布行为或运行中实例。严重程度:流程阻塞,非功能缺陷。
  4. 怎么修复的: 将 PR 分支 rebase 到当前 master;3 处冲突均按「双方意图并集」解决(保留 PR 的 recordNudgeInject 调用 + master 的 nudgeStampsBefore 返回字段),其余重叠文件自动合并;lease 锁定旧 head 后 force-push,未丢失任何并发提交。
  5. 是否有回归问题: 无。冲突解决未删除任何一方行为;本地 typecheck/build 干净,定向测试 169/169 + [PR] fix: un-show a nudge whose request the upstream rejected #6531 车道测试 4/4 全绿;全量以 CI 新 head 结果为准。唯一需 reviewer 留意的是上述 [PR] fix: un-show a nudge whose request the upstream rejected #6531×Feature: show nudge-recommended vs actually-folded tokens per fold (recovery rate) in the web panel chart and session report | 功能:web 折线图与 session 报告显示每次压缩「推荐 vs 实际折叠」回收率 #6729 语义交互(仅影响新恢复率指标在罕见 4xx 场景下的归因,非回归)。
  6. 是否破坏缓存: 不涉及——未触碰缓存、重试、增量游标、去重键逻辑。
  7. 是否改了设计方向 / 是否引入了新的配置: 不涉及——rebase 未改变 PR 的设计方向,未引入任何新配置(原 PR 本身也无配置面变更)。

收工判定: 冲突已解决并推送,PR 恢复可合并状态(待人工 review/merge);CI 在新 head 上运行中,结果出来前不宣称绿。

@ranxianglei
ranxianglei force-pushed the 2026-11-15_observability-root-cause branch from 41a113e to 42edd38 Compare November 16, 2026 21:57
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict resolved — PR is mergeable again (head 41a113ea6c → 42edd384b8).

Master moved 68 commits since the last rebase, and the decisive change: member PR #6745 was merged into master (merge commit 0821a81544), plus follow-up iterations on top of it (self-contained chart rework 51411796dc, baseline-100% line chart d643f0f8da, true line chart c5f573f7ed, dead-field cleanup 9a30dee80b, growth-recovery line 47e59a9deb, …). That means 3 of this PR's 5 commits (the #6729 recovery-visibility portion) are already on master in evolved form.

Resolution (union of intent):

What the PR now carries (diff vs master, 4 files, +154/−9):

  • src/agent/dsh-compaction-disable.ts (+11) — compactionCoverageText() formatter
  • src/agent/dsh-native.ts (±29) — first-routed-request coverage emission, once per process
  • tests/issue6770-compaction-coverage-text.test.ts (new, +75)
  • tests/dsh-native.test.ts (+48) — wiring test

Verified locally on the pushed head: npm run typecheck clean, npm run build clean, 103/103 targeted tests pass (issue6770-compaction-coverage-text, dsh-native, text-safe, rejected-nudge-rearm).

Live state (mirror cache was 2 days stale, so verified with a single GitHub API GET, as permitted): mergeable: true, mergeable_state: blocked (awaiting human review — unchanged), state open. The push is this branch's one full CI pass (no [ework:ci-held] marker).

Note for you (no action taken by me): the PR body still says "supersedes #6745 and #6772 (both left open; closing is yours)". #6745 is now closed (merged). #6772 is still open carrying the same #6770 content — after this PR merges it becomes a duplicate; closing it is yours to do. Also, the #6531×#6729 ledger-interaction finding I flagged in the previous run (a 4xx-rejected nudge keeps its prepare-time "n" event in the recovery ledger) now lives in master's merged #6745 code, not in this PR's diff.


八问

  1. 是什么问题:PR 分支与 master 再次冲突——master 前进了 68 个提交,其中 [PR] feat: show nudge-recommended vs actually-folded tokens (recovery rate) in web chart + session report | Closes #6729 #6745(本 PR 的 Feature: show nudge-recommended vs actually-folded tokens per fold (recovery rate) in the web panel chart and session report | 功能:web 折线图与 session 报告显示每次压缩「推荐 vs 实际折叠」回收率 #6729 成员部分)已被合入并继续迭代。
  2. 什么原因导致的:聚合 PR 的两个成员 PR 命运不同:[PR] feat: show nudge-recommended vs actually-folded tokens (recovery rate) in web chart + session report | Closes #6729 #6745 被直接合入 master(含后续重构),[PR] feat(dsh): emit compression coverage readout on the first routing request (#6770) | feat(dsh): 在首个路由请求上输出压缩覆盖读数 (#6770) #6772 仍开放;master 侧演进使本 PR 的 3 个 [PR] feat: show nudge-recommended vs actually-folded tokens (recovery rate) in web chart + session report | Closes #6729 #6745 衍生提交与 master 已演进版本冲突。
  3. 影响范围与严重程度:仅本 PR 分支的合并状态(此前 dirty 不可合);无用户面影响。
  4. 怎么修复的:rebase 到当前 master;3 个 [PR] feat: show nudge-recommended vs actually-folded tokens (recovery rate) in web chart + session report | Closes #6729 #6745 衍生提交按「master 已演进版本为准」跳过(保留 PR 旧快照会回滚已合入的重构,属回归);其余 2 个提交([Diagnosis] Compression coverage reading for #3962 is unavailable on device: profileContext.startedBundles not detected | [诊断] #3962 的压缩覆盖读数在设备上取不到数:profileContext.startedBundles 探测不到 #6770 覆盖读数 + 接线测试)干净落位,无需手工解冲突。
  5. 是否有回归问题:无。跳过方向是「取新弃旧」,不会回退 master 任何已合入行为;[Diagnosis] Compression coverage reading for #3962 is unavailable on device: profileContext.startedBundles not detected | [诊断] #3962 的压缩覆盖读数在设备上取不到数:profileContext.startedBundles 探测不到 #6770 部分与 master 的 [Design][dsh] Stop guessing dsh's auxiliary requests — the in-process plugin can declare each request's purpose explicitly #6880 改动同文件共存、互不触碰;定向测试 103/103 通过。
  6. 是否破坏缓存:不涉及(未触碰缓存/重试/增量游标/去重键路径)。
  7. 是否改了设计方向:不涉及。PR 的有效范围收窄为 [Diagnosis] Compression coverage reading for #3962 is unavailable on device: profileContext.startedBundles not detected | [诊断] #3962 的压缩覆盖读数在设备上取不到数:profileContext.startedBundles 探测不到 #6770 覆盖读数(Feature: show nudge-recommended vs actually-folded tokens per fold (recovery rate) in the web panel chart and session report | 功能:web 折线图与 session 报告显示每次压缩「推荐 vs 实际折叠」回收率 #6729 部分已由 [PR] feat: show nudge-recommended vs actually-folded tokens (recovery rate) in web chart + session report | Closes #6729 #6745 合入 master 覆盖),方向未变。
  8. 是否引入了新的配置:不涉及(无新 env/config/默认值)。

收工判定:冲突已解、本地验证通过、PR 恢复可合(待人工 review + CI 全量),本 lane 工作完成。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

No action needed this run — the "dirty" blocker is stale; the PR is live-verified clean.

I verified against live sources in this run (the mirror cache for this PR is 2 days stale, fetched_at 2026-11-15T13:40Z, so I confirmed with a single permitted GitHub API GET):

  • Branch head: 42edd384b8 — the final head pushed by the previous run (no [ework:ci-held] marker, so no mid-session death to finish).
  • Base: 301e1a5459 — master has not moved since the previous rebase; the branch is 0 commits behind, 2 ahead.
  • mergeable: true, mergeable_state: blocked (human review pending — not a conflict), state open.

No conflict exists between the branch and its base, so there is nothing to rebase or fix. The wake appears to have been generated from a pre-rebase snapshot. CI is running its one full pass on 42edd384b8; I'll act again only if a real blocker (new conflict or red CI) appears.


八问

  1. 是什么问题:系统再次报告 PR 与 base 冲突(dirty),需核实。
  2. 什么原因导致的:唤醒基于过期快照——镜像缓存停在 2026-11-15(上一轮 rebase 之前),而上一轮已把分支 rebase 到当前 master 并推送最终 head。
  3. 影响范围与严重程度:无实际影响;PR 当前可合(mergeable: true),仅「dirty」告警为误报。
  4. 怎么修复的:本轮无需修复——只做了 live 核实(fetch 比对 master/分支位置 + 单次 GitHub API GET 确认 mergeable 状态),未推送任何内容。
  5. 是否有回归问题:不涉及(零改动)。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

收工判定:核实完毕,PR 无冲突、无待办;等待 CI 全量结果与人工 review,本 lane 本轮无事可做。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment