[Bug] Materialization planner: fixed 5ms pressure-search budget causes root re-prefill fallback for large requests in multi-session states

[Gene0Liu]

Problem

With several large sessions resident, a request that arrives immediately after a completed turn gets cached_tokens = 0 and a full re-prefill (TTFT ≈ 142 s) even when its prompt is a byte-identical extension of the previous turn's prompt + response. The engine's own request-log diagnostics show what happened:

{"best_reuse_prompt_tokens": 130411, "budget_exhausted": true, "planning_elapsed_ns": 8768100,
 "predicted_future_loss_ns": 982660842110, "search_elapsed_ns": 6683300,
 "selected_degradation_units": 18, "selected_maximal_fallback": true,
 "stop_reason": "time_budget", "targets_evaluated": 10}

The candidate set contained a target reusing 130,411 of the 130,568 prompt tokens, but the bounded search verified only 10 targets within its budget and selected the root incumbent. Resending the identical request a few minutes later (engine settled) yields cached_tokens ≈ prompt and TTFT ≈ 0.7 s, so the failure is state-dependent at the moment of admission — which is why it appears intermittent.

Provenance: upstream code, not fork-specific

src/runtime/engine/materialization_planner.h:

const std::uint64_t search_budget_ns =
    std::min<std::uint64_t>(5'000'000ULL, incumbent.cost.total_ns / 20U);

• Introduced in upstream 3e903b70 ("feat(runtime): plan materialization under pressure", 2026-08-27); present verbatim at upstream tip ad0f3d38 (lines 257–258).
• The reproducing build is sergiuszm/ninfer-4090, branch rtx4090-port, HEAD 6f327f49 (upstream ad0f3d38 merged). That fork retunes sm_89 attention kernels and prefill schedules only; its maintainer ledger classifies every planner commit on this code path (3e903b70, 3d9fda22, 5e4bf313, da49c0d6, 138d76ae) as upstream.
• Planning is CPU-side (page-table projection and cost folding); no accelerator-specific code is involved, and no CLI flag adjusts this budget.

Reproduction

Environment: RTX 4090 (sm_89), 48 GB; Qwen3.8-27B .ninfer artifact, bf16 KV; server flags --max-context 262144 --kv-capacity 262144 --host-kv-mib 24576 --max-concurrency 3 --default-max-tokens 131072 --spec mtp --draft-tokens 3 --lm-head-draft --preserve-thinking --request-log-jsonl <path>. Two other large sessions resident with retained checkpoints.

1. Request A: ~128.5 K-token prompt, max_completion_tokens = 4096. Fresh process → root prefill, TTFT ≈ 139 s (expected; nothing to reuse yet).
2. Request B, immediately after A completes: A's messages + A's echoed assistant response + two user messages; max_completion_tokens = 131072. B's rendered prompt is a strict byte-prefix extension of A's (verified client-side; the response echo is replayed verbatim so rendered-token identity holds).
3. Result: B gets cached_tokens = 0, TTFT 142.6 s, and the diagnostics in the first section.
4. Control: the identical B a few minutes later → cached_tokens 130137/130142, TTFT 0.7 s.

Checks already performed:

• Reproduces identically with --host-kv-mib 8192 — host arena capacity is not the cause; verification time is.
• /slots shows all checkpoints retained at the exact prompt/completion boundaries — retention is working; admission planning is where the fallback occurs.
• The NoPressure fast path is unaffected, so small requests and settled engine states work normally.
• The constant has been present since 3e903b70, i.e. this is not a recent regression.

How it was found, and the local workaround

The failure was observed in long-running agent workflows: a multi-turn session whose context grows across turns (preserved reasoning, tool results, echoed responses), where each next turn is submitted as soon as the previous one completes. The growing prompt plus the output reservation eventually fills the pool, so every continuation lands on the pressure path, and the bug reproduces deterministically with the A→B sequence above.

The --request-log-jsonl materialization block identified the mechanism: stop_reason = time_budget + selected_maximal_fallback = true with a non-zero best_reuse_prompt_tokens means the planner found the reuse target, but its budget expired before that target was verified.

Why 5 ms is not enough here:

• For B, the active entitlement (prompt 130,568 + output reservation 131,072 ≈ 261,640) essentially equals the 262,144-token pool, so the reuse candidate is not identity-feasible and admission enters the bounded pressure search — the only path subject to the 5 ms cap.
• In this state, verifying one target costs ~0.3–0.7 ms, so 5 ms covers only ~10 targets.
• The incumbent/20 term does not bind: the root incumbent's cost is in seconds, so /20 ≫ 5 ms and the flat cap always dominates.

Locally I applied a workaround — search_budget_ns = min(250ms, max(5ms, incumbent.cost.total_ns / 20)). It resolves the bug in the workload: B then gets cached_tokens 130535/130692, TTFT 1.2 s, selected_maximal_fallback = false. Since the workaround was applied, the failure has not recurred: long agent sessions continue to reuse their prefixes, and no selected_maximal_fallback entries have appeared in the request log.

This report was drafted with LLM assistance.