Skip to content

[P0] 0.1.216 cold-load silently zeroed a 20-day compression ledger — a 3.35M-token session is now permanently unusable #7261

Open

Description

@RELLLEEE

This is not a "user misconfiguration" report. A session that worked for 20 days became permanently unusable after a routine restart, and the code that was supposed to save it refused to even try.

Summary

  • Session ran fine on 0.1.215 (the package had auto-updated on disk days earlier; the in-process code was still 0.1.215, so nothing broke).
  • One restart picked up 0.1.216.
  • First request after the restart: preflight fail-fast 413. Session dead. No recovery path exists.
  • The session is 3.35M tokens by the upstream's own count: This model's maximum context length is 1048576 tokens. However, you requested 3606982 tokens (3350982 in the messages, 256000 in the completion).

What the log says happened

[storage] unified FULL mode ENABLED ... index.db is the load path (unified-first, legacy fallback)
[session-order] hydrate failed — skeletons stay on the legacy file path
  ([storage] extension value exceeds 4194304 bytes — store large payloads in the content CAS instead)
[persist] ledger-aligned refs: kept=0 restamped=4624 rehomed=2514 blocked=206 superseded=7138
[persist] cold-loaded view is ~7021 min old (savedAt 2026-11-14T09:00:11Z) with 92 prior compressions

kept=0. Twenty days of compression state — zero of it survived the migration. The one file that carried the state, session-order.json, was 8,373,997 bytes — over the hard 4 MB single-value limit — so the unified store refused it and the ledger was rebuilt from nothing.

Then the recovery path made it worse

[preflight] payload ~3322881 tok vs window 996147.2 (~3.3x) — raising summarization budget 16 -> 32, rounds 16 -> 32
preflight compressed 32 range(s), ~563948 tokens saved (3339629 → 2706860) in 446292ms
preflight fail-fast 413: payload core text alone is ~9905030 bytes

You reported saving 563,948 tokens. The core text went from 9,905,030 bytes to 9,968,516 bytes. It went up.

The next attempt did not even run the fold — it hit the core-text gate in 1.6 seconds:

[preflight] round 1/32 ... round 32/32   (502 s spent, then 413)
[preflight] fail-fast 413                  (1.6 s, no folding attempted)

So the round budget is meaningless: the gate runs before the fold, and it reads a payload the fold never touches.

Questions I want answered

  1. Your fold reports success, and the payload does not change. Compressed m05899–m05965 → 1 block(s), ~28656 tokens saved. Then core text is byte-identical. Where does the folded view go, and why is the outbound payload built from something else? Is this a known bug, or is the receipt simply lying?
  2. A migration failure is silent, and it costs the user everything. session-order.json at 8.37 MB exceeds a 4 MB limit. Your own log says "store large payloads in the content CAS instead" — for the writer. For the reader, you dropped 20 days of compression state and logged one hydrate failed line. Why is there no migration-failure alert, no "this session cannot be safely loaded" refusal, and no export path for the ledger before upgrading?
  3. Is kept=0 an expected outcome for a real user? A user with a 20-day session who never touches the config gets restamped=4624 ... kept=0. That is not an edge case; that is exactly the session your product claims to support ("month-long single sessions").
  4. Why is there no degraded mode? If folding fails, the only sane fallback is "forward what fits, report what doesn't" or "refuse this request, keep the session." Instead the session is bricked: no upstream can accept it, so it was NOT forwarded even for evidence. The user cannot even recover by sending a short message.
  5. Why is the fix for this filed as a decision request and not a bug? [Bug] compress can silently fold another message after restart when a pre-restart ref is reused (0.1.209) #5196 reproduces it on 0.1.209. decision request: restamp reuses already-issued refs — replayed pre-restart citations silently compress the wrong message (#5199) #5204 describes the exact mechanism (restamp reuses issued refs → cold-load ledger mismatch). It is still a "decision request", and users are losing sessions in the meantime.

Environment

  • billion-context 0.1.216 (0.1.215 before restart), Windows x64, Node 24.21.0
  • model: deepseek-flash, window 1048576 (modelContextLimit=1048576)
  • storage: default (pure sqlite) at the time of the incident; storage.unified=legacy afterwards recovered
    92 blocks, but a view dated 2026-11-14 — i.e. 4.9 days stale, useless for a 3.35M-token session
  • session file: 41.3 MB compressed, 45,146 records

I am not asking for a workaround. I am asking why a product whose entire purpose is "keep the session alive forever" turns a routine restart into permanent data loss, and why the recovery path reports saving tokens it never saved.

Activity

  1. ranxianglei commented on Nov 19, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    Duplicate screening: Not a duplicate. The referenced #5196 and #5204 are both closed — #5196 was fixed by PR #5360 (slot-retention, merged 2026-10-31), which is included in 0.1.216 (verified: the slot-retirement logic is present in kernel/src/refs.ts at the v0.1.216 tag). Your report is a new incident (a 3.35M-token session bricked after a restart on 0.1.216), not a duplicate of the closed issues.

    Labels applied: bug, p1, scope/storage.

    What I verified (evidence, all at v0.1.216 — no commits since v0.1.216 touch these paths, so master code == the code you ran):

    1. The 4MB vs 8MB mismatch is a real bug. The order-snapshot write cap is 8MB (MAX_SNAPSHOT_BYTES, src/order-persist.ts:32), but the unified store's kv value limit is 4MB (EXT_VALUE_MAX_BYTES, src/storage/store.ts:361). Your 8.37MB session-order.json exceeds the 4MB limit, so the unified migration write fails. But it does not cause data loss — when the unified hydrate fails, kvReady stays false and all writes fall back to the legacy file path (src/order-persist.ts:186), so the skeletons survive on the file.

    2. Your attribution of session-order.json as "the one file that carried the state" is a misdiagnosis. The session-order.json file carries the resolver SKELETONS (id→kinds/frags/tagHash/lastSeen, Compression fails in long sessions while new sessions work fine | 压缩失败.新会话反而没事 #3397), NOT the compression ledger. The compression ledger (the 92 blocks) is stored in the unified store (SQLite) or the legacy file. Your log line cold-loaded view … with 92 prior compressions confirms the 92 blocks WERE loaded into the session state.

    3. The fail-fast 413 gate is working as designed. The core text is 9.9MB, which exceeds the doom threshold (max(2MB, 1M×7.5=7.47MB) = 7.47MB, src/server.ts:4096). So the payload is mathematically doomed — no upstream can accept a 9.9MB core text against a 1M window. The gate correctly refuses to forward it.

    4. The kept=0 is the After model switch / client-side history rewrite, all message identities (content hashes) are replaced: ACP numbers re-m | 切模型/客户端重写历史后,消息身份(内容哈希)全部更换:acp 编号重铸、折叠覆盖大规模重挂、死 ref、churn 区临时膨胀重折 #4316 ledger alignment. 4624 refs were renumbered to align with the storage ledger's seq+1 (src/persist.ts:2095, kernel/src/refs.ts:232). This is a massive realignment, but it doesn't directly cause data loss — the compression blocks reference messages by raw id (not ref number), so the realignment shouldn't affect the blocks themselves.

    The root cause (most likely, still verifying):

    Your session is 3.35M tokens raw, with 92 compression blocks. If the 92 blocks were applied, the folded view should be much smaller (under the 1M window). But after the restart, the outbound payload is 3.35M tokens (the full raw history). This suggests the 92 blocks are loaded into the session state but not being applied to the outbound payload after the cold load.

    I'm investigating how the folded view is built and why it's not applied after the cold load. I'll follow up with the specific mechanism.

    On your question #5: #5196 and #5204 are closed. #5196 was fixed by PR #5360 (slot-retention), which is in 0.1.216. #5204 was the decision request that led to #5360. So the "restamp reuses issued refs" mechanism you're referring to is already fixed in the version you're running. Your incident is a different (new) problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions