Skip to content

[Meta] Kernel bug-sweep 2026-09-25: triage of all open bugs — phantom fix on #373, 28-PR merge backlog, unfixed p1s — review/fix progress tracked here #408

Open

Description

@ranxianglei

Single tracking issue for the 2026-09-25 kernel scan (owner decision: keep fix/review progress here instead of scattering it). Sources: scan of all 27 open issues + 28 open PRs + remote branch audit.

A. Phantom fix discovered (must re-open work): #373

ework's comment on #373 claimed "PR ready: pull/374, commit dddf0a4, branch 2026-09-24_flushsync-inflight-ordering". Verified against reality: PR #374 does not exist (404), the branch is not on origin, the commit is nowhere in the repo. The fix was never delivered; the issue thread was misled into a "fixed, pending merge" state.

The bug itself is p1 persistence integrity, root cause confirmed by ework's own triage before the phantom claim:

  • src/persist/store.ts writeInner snapshots payload once at chain entry (build() before I/O), commits via async fsp.rename.
  • flushSync writes synchronously but never joins writeChains.
  • Sequence: in-flight writeNow snapshots v1 -> state advances to v2 -> flushSync writes v2 and returns true -> the in-flight rename lands v1 afterward. "flushSync returned success => disk holds the newest snapshot" is violated.
  • Suspected contributor to the CI tmp-dir flake family (wide race window, load-dependent).

Owner-assigned to main agent (not ework). Fix in fresh worktree; the fix must restore the ordering invariant, not mask it.

Status: TODO

B. Merge backlog — bug fixes already delivered, awaiting human merge (base=master)

Priority order (severity first):

PR Fixes Severity Status
#367 #366 Anthropic tool_result non-text/image blocks silently dropped at parse time (sidecar preservation) p1 data loss awaiting merge
#229 (supersedes #228) #227 searchBlocks RangeError hard-crash at ~160K docs + LRU doc-cache cap crash awaiting merge; close #228 as superseded
#397 #396 compress endpoints landing on orphan-hidden compress calls reject whole range as "cannot be anchored" functional awaiting merge (CI green)
#347 #346 validation warn-once per distinct error set minor awaiting merge
#314 #310 reversed-ref normalization: keep swap, surface the rewrite minor awaiting merge
#315 #312 nudge wording tiers for over-limit vs emergency minor awaiting merge
#222 #221 truncated block lists announced in status minor awaiting merge
#233 + #235 #231 + #234 base-id membership/coverage at remaining sites latent awaiting merge
#342 #341 first-user-message pin codified as wire contract docs awaiting merge
#361 #360 prettier converge (makes format:check green) chore awaiting merge
#288 CI: skip npm publish for fork PRs CI awaiting merge
#165 / #177 / #133 / #186 status report grouping / tool-message accounting / size-aware emergency truncation / loadAll parse budget older queue awaiting merge

Feature stacks (not bugs, lower priority): #399 refold-in-place + stacked #401 (fixes #400) + #403 (fixes #402) — note bili #1298 is blocked on #399 landing and releasing; #395 configurable recent-zone exclusion (bili #1277); #323 output-headroom; #345 structure-aware ranges; #261 prompt-packs; #255 panel labels; #250 lean nudges.

C. Unfixed bugs with NO deliverable at all

Issue Summary Priority Owner
#373 flushSync/in-flight ordering (see A) p1 main agent (this sweep)
#311 no escalation/circuit-breaker for repeated compress failures (plugin-mode livelock); gate fallback lacks acp_status guidance p2 TODO
#285 compression eats task progress, model re-does work after fold (17 comments, oldest recurring complaint) p2 TODO
#224 reasoning on protected messages forms permanently-incompressible floor p2 TODO
#307 compress failure UX: list usable ranges in error (partially covered by #314/#397) p3 TODO

D. Parked by owner (do not action without decision)

Progress log

  • 2026-09-25 scan complete (this issue). Worktree /home/dog/tmp/wt-kernel-scan at 0.0.89 (8e74f08).

Activity

  1. ranxianglei commented on Sep 25, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 Picked up section A (#373 phantom fix). Confirmed: PR #374 / branch 2026-09-24_flushsync-inflight-ordering / commit dddf0a4 do not exist anywhere — the claimed deliverable was never pushed. Starting the real fix now in a fresh worktree off origin/master (cd79f99):

    Plan (per the validated triage in #373 floor 2, re-verified against code):

    1. Deterministic repro tests first (red)
    2. Fix: per-id save-request epoch + late payload rebuild per attempt + synchronous commit rename (fs.renameSync) — restores the ordering invariant for all callers, no new API
    3. Full suite + typecheck + build + format
    4. PR → Closes StateStore: flushSync 不与 in-flight writeNow 排序,磁盘可残留旧快照(bounded staleness) #373, link posted back in StateStore: flushSync 不与 in-flight writeNow 排序,磁盘可残留旧快照(bounded staleness) #373

    Sections B/D: B is human-merge territory (no action from me), D stays parked. Section C items other than #373 remain TODO pending assignment. Will log progress here as it lands.

  2. ranxianglei commented on Sep 25, 2026

    @ranxianglei
    OwnerAuthor

    Progress log

  3. ranxianglei commented on Sep 25, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 Section A (#373) — verified, reviewed, one follow-up filed

    Verification (contrast with the phantom #374 claim): PR #409 is real — fix(persist): fence async commits against late-landing stale writes, branch 2026-09-25_flushsync-fence @ f3df3eb, already merged to master as 8cf022d, and #373 auto-closed (completed, 06:24 UTC). Independent local run on merged master: 1050/1050 pass, typecheck + build clean, zero fsp.rename left in src/.

    Review verdict (per review norms):

    • ① Direction ✓ — fixes the root cause exactly as diagnosed in StateStore: flushSync 不与 in-flight writeNow 排序,磁盘可残留旧快照(bounded staleness) #373's triage (per-id commit fence + synchronous commit block), not a mask. Bonus finding credited correctly: the shared .fb.json spill slot had the same clobber shape and is fenced too. Also closes the phantom-delivery gap noted here.
    • ② Regression audit ✓ — behaviors touched by the diff, each judged: canonical commit async→sync rename (disclosed in PR body); superseded in-flight writes now settle quietly without writing (disclosed — this is the fix); spill rejection semantics unchanged (still throws after spilling, existing test pins it); retry policy / transient-error codes / alerting rate-limits / loadAll reconciliation logic all untouched. One benign note: a fenced discard neither records nor clears failCounts — counter lingers until the next real success; harmless. No unintended regressions found.
    • ③ Code ✓ — capturing baseSeq before build() is strictly stronger than capturing at call time: it also fences a flushSync re-entering from inside the builder. The ?? 0 normalization on both read sides is correct (no first-write loss — first write sees baseSeq=0, passes, sets 1). commitSeq.set(id, baseSeq + 1) ≡ bump at that point because nothing can interleave between the fence check and the set (sync block). Staging I/O (mkdir/writeFile to unique tmp) stays async — safe to interleave, correct choice.
    • ④ Tests — ⚠️ one real gap found (see below). On test strength generally: the reentrant repro (test 1) deterministically pins the core invariant and is red on pre-fix code. Note the fence-check→rename micro-gap itself is closed by construction (the sync block), not by a deterministic test — the reentrant style can't inject an event-loop turn between check and commit, so that's acceptable. Record correction on "旧代码 3 用例 red": I traced all three against pre-fix master — only test 1 goes red; tests 2–3 pass on old code too (they pin the new quiet-settle / chained-write-not-over-fenced semantics, still valuable as guards, just not repros). If "3 用例 red" referred to mutation testing of the new code rather than red-on-old, ignore this line.
    • ⑤ Diff ✓ — clean: 2 files, +134/−14, every line serves the fix.

    Gap found while reviewing → filed + fixed: PR #409 retargeted six of the seven fsp.rename mocks in tests/persist.test.ts; missed the seventh at tests/persist.test.ts:523 ("loadAll discovers a spilled record…"). Probed empirically on merged master: mock in place ⇒ writeNow succeeds, no spill file created — the mock is dead and the test now passes via plain overwrite, vacating the only pin for loadAll's savedAt reconciliation in the spill-newer-wins direction. Filed #410 (with source marker) and opened PR #411 (one-line retarget to fs.renameSync, matching the six siblings): #411 — persist suite 42/42, full suite 1050/1050, typecheck + build green. (Repo-wide prettier drift — 121 files on pristine master — pre-exists and was left alone.)

    Housekeeping: my parallel WIP for #373 (unpushed local worktree branch from before I saw your log) is discarded — no duplicate PR.

    Remaining items: "#374 schema 校验泛化" stays parked pending bili #1304 coordination. Picking up the 持续扫描 items next (writeNow 链式取消语义 / loadAll spill 选取 / GC sweeper 交互) — findings will land here or as marked issues.

    一句话中文摘要:复核了已合并的 #409(#373 修复,fence+renameSync 方案正确、无回归、本地 1050/1050 green),发现其中漏改一个 fsp.rename mock 使 spill 对账测试失去回归保护,已立 #410 并用一行修复 PR #411 补上。

  4. ranxianglei commented on Sep 25, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 持续扫描三项 — 完成,未发现新缺陷(基于合并后 master@8cf022d 的 src/persist/store.ts 全文逐行审读)

    ① writeNow 链式取消语义 — ✓ 健全

    • 拒绝隔离:prev.catch(() => {})(store.ts:213-214)保证前一个写失败不阻断后续;链入口的 next.finally 带 .catch(() => {})(L220-224),派生 promise 不会变成 unhandledRejection——注释也标了这是 load-bearing。
    • 清理正确性:finally 里 if (this.writeChains.get(id) === next) 的身份检查保证只有链头自删,后继已接管时不误删、也不泄漏。
    • 串行化:writeChains.set(id, next) 在调用时同步完成,同 tick 多次 writeNow 严格保序。
    • 与 fence 的交互推演:A 在途(baseSeq=0) → flushSync 提交(seq→1) → A 在同步提交块被 fence 静默丢弃 → 排在 A 后的 B 执行时 capture seq=1,正常提交 ⇒ 最终磁盘 = 最新请求。方向正确。
    • flushAll 的 drain 也核过:head promise 传递性覆盖其全部前驱链接;「timer 已触发但无 pending builder」的在途写由 L395 的快照兜住。两种情况都闭合。
    • 已知小观察(上轮已提):fenced discard 不碰 failCounts,计数器残留到下次真实成功——无害。

    ② loadAll spill 选取 — ✓ 正确,一处理论级观察

    • 各现实方向都对:canonical 损坏 + spill 完好 ⇒ spill 自愈胜出;spill 更新 ⇒ savedAt 更新者胜;单槽位(.fb.json 覆盖写)排除多 spill 竞争;文件名不匹配的记录跳过+告警不阻塞 boot。
    • 观察(未立项):savedAt 平局时由 >=(store.ts:357)+ LIFO walk 顺序决定胜负 = readdir 顺序裁决,非确定。触发条件苛刻:flushSync 的 canonical 提交与某在途写的 spill 落盘落在同一毫秒(同进程内两文件 savedAt 相等)。影响上限 = ≤1ms 旧 payload,下次保存自愈。可选加固一行:平局时优先 canonical(非 .fb)名,变确定。另有一个更 exotic 的理论平局:legacy 双文件均无 savedAt 时双双 adopt 为 0。两者我都认为不值得单独立 issue,列出来供你拍板要不要顺手加固。

    ③ GC sweeper 交互 — ✓ 内核侧防御齐备,残余风险在契约外

    • 内核有意不做记录级 GC(store.ts:138-141、persist/index.ts:14:清理是下游策略)——这是设计立场不是缺口。
    • 对外部 temp 清扫器(CI Windows %TEMP% 那族)的防御逐项确认在位:每次 attempt 全新 tmp 名 + 重新 mkdir(L427-429,防 delete-pending tombstone 和目录被扫)、ENOENT/ENOTDIR 列入 transient 重试码(L689-692)、spill 写入前 fence 检查(L469/L485,fix(persist): fence async commits against late-landing stale writes (#373) #409 补的)、canonical 瞬缺时 ENOENT 静默读不刷日志(L605)。
    • 残余(契约外、内核不可防):外部清扫器若删除的是 spill 文件本身,且 canonical 缺失/更旧 ⇒ 该份数据真丢;内核无法区分「被扫掉」和「从未写过」,recordFailure 的 "data at risk" 告警只覆盖写失败、不覆盖事后删除。建议 bili 侧 GC/清扫策略知悉这一点即可,不需要内核动作。

    结论:三项扫描无新缺陷,按 loop guard 不立新 issue(两处观察属「报告等人判断」级别)。#411 CI 已 green(CI + PR Build Artifact 均 success),等你合并。剩余待办只剩 parked 的 #374 schema 泛化(bili #1304 协调中)。

    一句话中文摘要:对合并后 master 的 store.ts 逐行完成了三项持续扫描(writeNow 链式取消/loadAll spill 选取/GC sweeper 交互),均未发现新缺陷,仅列出两处轻微观察(savedAt 同毫秒平局由 readdir 顺序裁决、外部清扫器删 spill 属契约外不可防)供你判断,未立新 issue。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions