[KV Offload] Demote superseded recurrent states in the CPU tier

[Leonccaa]

Purpose

This is a follow-up to #598 (grouped hybrid RAM caching) for hybrid models with align-mode Mamba/GDN groups, such as Qwen3.8-Flash-Next. It fixes a long conversation that goes idle and can no longer be restored from the CPU tier.

Problem. Every retained replay-boundary state is handed to the CPU tier, so each conversation turn stores about two new states per recurrent group.

• The recurrent state pools are small: one state page covers all recurrent layers of a group, and [KV Offload] Grouped hybrid RAM caching, filesystem restoration and retention-sized Mamba pools (stacked on #617) #598 sizes the pools from the retention mask for a max_model_len-long reference request. With a 64 GiB tier on TP4 V100 this gives 16 state slots per group at max_model_len=524288, and 30 slots at a 262144-token reference.
• Plain LRU keeps the superseded states of active conversations. After a few dozen turns in other conversations, an idle conversation's newest state is evicted.
• Its attention blocks are still in RAM, but the prefix can no longer be served, because a hybrid hit must end at a recurrent checkpoint. The next request recomputes the whole context.
• Raising the slot count through mamba_state_slots_reference_tokens only postpones this: the demand grows with the number of turns, and every state slot costs several token slots.

Fix. When a request finishes, demote its recurrent states older than the last three block positions to the front of the eviction order. Those three positions hold the resend/extension replay boundaries. The demoted states are only superseded checkpoints of the same conversation, and blocks that are being loaded are still skipped by eviction. With demotion, the state demand becomes roughly two slots per cached conversation, independent of the number of turns.

The behavior is opt-in and nothing changes unless it is enabled:

"kv_connector_extra_config": {"demote_superseded_states": true, ...}

Changes:

• OffloadingManager.demote() (default no-op). It is implemented by the LRU policy, CPUOffloadingManager, GroupedCPUOffloadingManager and the tiering manager (primary tier only; secondary tiers keep their own policy). ARC keeps the no-op default.
• OffloadingConnectorScheduler.request_finished() demotes the pre-tail keys of every sliding-window-of-one group that requires exact boundary sources (align-mode Mamba).
• The grouped CPU pool logs its geometry at start-up: token and state slots, and page sizes.

Test Plan

• New tests/v1/kv_offload/cpu/test_demote_superseded_states.py:
  • LRU victim order after demotion;
  • referenced blocks are not evicted;
  • demotion is partitioned by group;
  • the scheduler hook demotes only pre-tail recurrent keys;
  • short requests demote nothing.
• tests/v1/kv_offload and tests/v1/kv_connector/unit/offloading_connector, run on CPU with and without this change on the same base.
• End to end on 4x V100 (Qwen3.8-Flash-Next AWQ, TP4, DCP2 ([Model][QSA] Target-only DCP2 for Qwen4Exp QSA on TP4 V100 #696), MTP4, E4M3 KV, YaRN 524288, prefix caching, 64 GiB grouped CPU tier with mamba_state_slots_reference_tokens=262144), fresh server per arm:
  • A ~509K-token conversation is built and then left idle.
  • Three other conversations (100K–320K tokens) run 117 turns, three in flight, each turn adding ~3.2K input tokens and 600 output tokens.
  • The idle conversation then sends one more turn. We record its cached tokens and time to first token (TTFT), plus the other conversations' hit rate.

Test Result

• Unit tests: the 5 new tests pass. On CPU, the offload and connector suites have identical failure sets with and without this change; the remaining failures need a GPU or are pre-existing on the base.
• End to end: measured on our deployment base ([KV Offload] Grouped hybrid RAM caching, filesystem restoration and retention-sized Mamba pools (stacked on #617) #598 + [Model][QSA] Target-only DCP2 for Qwen4Exp QSA on TP4 V100 #696 on main 1e90d17f2c); the change applies cleanly to current main.

	Baseline	This change
Idle conversation returns: GPU hit	0	0
Idle conversation returns: restored from the CPU tier	0	504,288 of 508,801 tokens (8.4 GiB)
Idle conversation returns: TTFT	264.8 s	5.7–5.8 s
Other conversations: cached	96.7%	96.7%
Other conversations: TTFT median	11.1 s	10.3–10.7 s
Preemptions / engine errors	0 / 0	0 / 0

• The GPU pool is exhausted in this scenario on both arms, so the whole difference comes from the CPU tier.
• A full acceptance run with the change enabled showed no difference from the baseline: quality × temperature, MTP4 block boundaries, CPU-tier restore parity, prefix TTFT, multimodal and memory stress, 5-needle retrieval at 211K/497K/2 × 400K tokens, and a 15-minute 4-way long-decode run.
• The logged pool geometry (7 recurrent groups on this model) was 2255 token slots and 30 state slots per group, with token pages of 7.21 MB and state pages of 30.9 MB per worker.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[yangzhuxinyzx]

I rebased the signed change onto current main in #798 and added coverage for default-on policy, the explicit disable override, and the request-completion hook. Demotion remains limited to eligible recurrent groups; pinned blocks and attention prefixes retain protection. The combined CPU gate passes (203 checks, five GPU skips), as does scoped pre-commit. Two unrelated sliding-window fixture failures reproduced on unmodified main and are documented.