Skip to content

[cache] LCP slot reuse discards another conversation without RAM preservation + lookup predicate gap (tracks novkien/llama-proxy#416) #1

Closed
Closed
[cache] LCP slot reuse discards another conversation without RAM preservation + lookup predicate gap (tracks novkien/llama-proxy#416)#1

Description

@novkien
novkien
opened on Sep 29, 2026

Current status - accepted by the owner and CLOSED (2026-09-30)

The owner explicitly accepted the deployed RAM-retention behavior and directed closure in the Codex task. The owner does not require exact generated-token equality between warm and cold Nex runs; the observed variation is retained as evidence rather than treated as a failing closure criterion. PR #2 is merged and installed in fork 95e6565970621aa508e80fd805dcf73af413093c. Maintained checks passed (33 native checks, 14 focused HTTP tests). Actual Qwen hybrid/DFlash2 retention and media append passed; actual Nex repeatedly restored 20015 tokens while processing 65 new tokens, including non-consuming cross-slot reuse. No claim is made that all warm/cold outputs are identical.

Optional SSD spill remains unimplemented and is not included in this accepted RAM delivery. The independent Nex cold-image draft metadata defect remains tracked in #6. The earlier OPEN/output-equivalence status below is historical and superseded by this explicit owner acceptance.

Current status - OPEN after deployed RAM acceptance (2026-09-30)

PR #2 is merged and deployed in 95e6565970621aa508e80fd805dcf73af413093c; 33 native checks and 14 focused HTTP tests passed. Actual Qwen 20000-token hybrid/DFlash2 preservation, non-consuming cross-slot restore, cache opt-out, exact warm/cold output and focused image append/restore passed.

Actual Nex restores 20015 tokens and processes 65, but fails exact output equivalence. Its matched no-RAM live control also differs from cold recomputation, and RAM warm differs from no-RAM warm. The cause is not isolated; mechanical cache hits do not justify closure. Read the latest self-contained acceptance/handoff comment before resuming. Nex cold-media failure is separately tracked by #6. Optional SSD spill remains unimplemented. GPU tests stopped at the owner's request; idle holds are off and the original Nex route is restored.

Tracks novkien/llama-proxy#416: [bug] Nex native LCP slot reuse discards another conversation cache before continuation
— https://github.com/novkien/llama-proxy/issues/416

This is the native-side tracking issue on this fork (novkien/llama.cpp-fork, forked from ggml-org/llama.cpp). The proxy issue stays the production-incident record; this issue tracks an isolated native reproduction + candidate repair reusing the existing RAM/checkpoint machinery. Please keep the two linked (comment on either when updating the other).

Production pin

Observed problem (from ggml-org#416, read-only production correlation 2026-09-28)

  • tools/server/server-context.cpp:get_available_slot selects a non-busy slot by LCP.
  • Selected-slot RAM preservation only runs when f_keep < 0.5 (or LRU path). Observed destructive selections had f_keep = 0.708, 0.668, 0.705 → save skipped, then the processing path restores a suitable recurrent checkpoint, deletes later live checkpoints and truncates the live token list.
  • Concrete case 11:17 +07: req-000008 leaves 71,352 tokens resident on slot 0 (checkpoints 69,476 / 70,496). req-000004 (different conversation, LCP 50,313) reuses slot 0, restores checkpoint 50,312, erases 69,476/70,496, no RAM save of the 71k branch. Continuation req-000018 (same logical conversation as req-000008) then finds only 50,606/74,135/52,792/55,873-token RAM candidates, restores 49,994 for an 84,328-token input → 34,334 prompt tokens.
  • Earlier same pattern: req-000026 (19,263 prefill) and req-000043 (20,018 prefill) with affinity_match=true but native_cached_tokens=0.
  • Positive controls: cross-slot RAM save/restore works (e.g. task 3595 restored 50,124/prefilled 192; task 3669 restored 73,169/prefilled 226).

Why threshold-only change is insufficient (from ggml-org#416 trace)

  • Preserve path exists: prompt_save (~server-context.cpp:299) → server_prompt_cache::alloc (~server-task.cpp:1711, clones tokens + checkpoints) → load (~server-task.cpp:1793) restores via llama_state_seq_set_data_ext into any id_slot.
  • But load lookup skips entries with f_keep<0.25 and requires BOTH f_keep AND f_sim strictly greater than resident candidate.
  • Arithmetic replay A=71,352 / B=50,468 / LCP(B,A-next)=50,313:
    • exact A extension wins, but
    • A-next with small suffix change LCP(A,A-next)=70,496: saved A keep=0.988 < B keep=0.997 → rejected despite ~20k more matching tokens;
    • B prefix-of-A-next: both keep=1.0 → strict > rejects longer saved A.
  • So a coherent candidate must address BOTH preservation-before-destructive-rollback AND lookup/selection by usable cached prefix/checkpoint benefit. Do not reintroduce naive forced id_slot pinning (see proxy Non-english languages? ggml-org/llama.cpp#339/Improve the Chat Mode with some tricks and considerations ggml-org/llama.cpp#353 context in Delete SHA256SUMS for now ggml-org/llama.cpp#416).

Origin

Proposed scope for this fork (needs isolated validation, no production activation here)

  • Isolated interleaved A/B/A fixture showing preserved A prefix on continuation after B borrows a similar native slot; counters vs baseline.
  • Define/test behavior when divergence precedes earliest hybrid checkpoint (shared prefix 7,458 vs checkpoints ≥8.2k case in Delete SHA256SUMS for now ggml-org/llama.cpp#416).
  • Preserve logical history priority / concurrency; bound RAM overhead (-cram -1 removes byte cap, not token cap — ~350,208-token budget observed); cover eviction/limits, latency/RAM measurements.
  • Reuse existing serialization + cross-slot restore; avoid unnecessary saves on simple continuations.

Status

Open. Proxy-side ggml-org#416 remains the authoritative incident/diagnosis log. Update both issues when a fixture or candidate patch lands here.

Activity

  1. novkien commented on Sep 29, 2026

    @novkien
    novkien
    on Sep 29, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Architecture: preserve replaced slot branches in RAM, with optional SSD spill

    Owner request and delivery scope: the owner explicitly requested this architecture comment and a draft implementation workspace in novkien/llama.cpp-fork. The requested behavior is to preserve an old slot context for later reuse, primarily in RAM and optionally on SSD below the configured slot-save directory. The workspace contains design/instructions, not an implemented native fix. No production changes, model restart, proxy policy activation, merge or upstream submission are included.

    Inspected source baseline: fork master at 526c43b8f7dfea9032e9f35e7a1be9183ca7cc20. Historical runtime evidence from novkien/llama-proxy#416 is not proof that a running process uses this fork revision.

    The complete versioned workspace is committed at 18a6e1e53c181d02dffa81d595e1c6795c505b12: start here, DESIGN, junior implementation guide, tests, research.

    1. Required behavior

    For A(100k) -> B(100k) -> A-next, A and B may use the same physical slot. Before B replaces A's live state, native preserves A's useful complete state in RAM when the configured resources allow retention. A-next can restore A into either the same slot or another available slot selected by native or by explicit id_slot.

    A slot is an execution destination, not a cache key. Multiple snapshots can originate from one slot. A RAM snapshot is not consumed when a different conversation borrows its common prefix. The 64 context checkpoints are rewind states within a prompt, not 64 independent full-conversation backups.

    The two paths share the same cache pipeline. Fixing only the empty-slot division, changing only f_keep < 0.5, increasing checkpoint count, or adding unconditional save/load is insufficient.

    2. Source defects to address

    The inspected get_available_slot() gates both saving and loading with update_cache. Its selected-slot heuristic can skip saving A when B shares more than half of A, even though B destroys A's suffix/checkpoints. Explicitly selected empty slots can skip RAM lookup entirely; disabling slot similarity must not preserve that bypass.

    The caller checks whether a slot is busy after selection, although selection can already mutate cache state. server_prompt_cache::load() clears serialized source buffers and consumes the entry after restoration. It ranks by two strict normalized ratios, not by usable token/checkpoint benefit. Allocation also treats token-prefix containment as sufficient for deduplication, without proving recurrent checkpoint coverage.

    Relevant baseline source: server-context.cpp, server-task.cpp, server-common.cpp.

    3. Unified pipeline

    request with optional id_slot
  -> select execution destination without mutation
  -> validate availability, configuration and parent/child capacity
  -> compute one read-only reuse plan across live/RAM/SSD metadata
  -> preserve outgoing branch before its first destructive mutation
  -> restore selected immutable snapshot, if it improves valid reuse
  -> apply a valid checkpoint / execution rewind
  -> prefill remaining tokens
  -> generate

    For explicit IDs, a busy slot is deferred, never mutated or silently replaced with another slot. Automatic selection considers available slots. Both paths use the same preparation/restore policy, including --slot-prompt-similarity 0.

    A destination reserved for disk I/O is unavailable to other requests but is not reported as generating. Other slots remain executable. Validate incoming LoRA/aLoRA and other execution-affecting configuration before choosing state; capture the outgoing state under its old identity before applying changes.

    Honor cache_prompt=false and existing non-completion behavior. Explicit slot erase does not trigger a new save and is not advertised as a global disk-archive purge.

    4. Snapshot contents and ownership

    Extend the existing server_prompt_cache; do not build a second cache manager. Each snapshot contains:

    • Epoch-qualified snapshot identity and source-slot provenance.
    • Exact server_tokens, including media serialization and position information.
    • Full target state, optional full draft state and current speculative implementation state when required.
    • Complete checkpoints: token count, position range, target/draft partial-state bytes and data_spec.
    • Source sequence coverage/capability metadata, compatibility descriptor, storage tier, size/token accounting and recency.

    Payloads are immutable after capture. Share large immutable checkpoint buffers while copying their lightweight metadata independently. A new checkpoint capture allocates fresh backing storage; it must not overwrite bytes shared with A. Keep speculative checkpoint bytes value-owned initially. Clone server_tokens rather than removing media support or using its deleted copy constructor.

    An internal lease/pin keeps source bytes alive during restore or spill. Removing an index entry does not imply its memory has been released. Count queued/in-progress buffers and retained leases, not only entries visible in the RAM list.

    5. Preserve before destructive reuse

    Replace the percentage-based preservation decision with a semantic one: will this operation remove or replace a useful branch that has not already been preserved?

    Compatible append-only continuation is a fast path. Replaying a final token for logits is not itself a reason to snapshot the entire context. Shorter prompts, genuine suffix divergence, state replacement and configuration changes require preservation when the outgoing branch remains useful.

    The preservation hook must run before seq_rm, chunk shifting, state setters, checkpoint erasure, keep_first or prompt_clear, including ordinary idle evacuation/purge. A hook after checkpoint deletion cannot recover the old branch.

    Capture is a transaction:

    1. Compute sizes and reserve RAM/staging capacity; protect the intended restore candidate.
    2. Allocate the whole unpublished entry, including token/media clone and checkpoint metadata.
    3. Serialize at a safe owner-thread boundary; verify target/draft returned byte counts and required speculative state.
    4. Publish only a complete snapshot.
    5. Only then deduplicate older equivalent entries and mutate the live slot.

    Use typed outcomes: SAVED, ALREADY_PRESERVED, DISABLED, BUDGET_REJECTED, ALLOCATION_FAILED, SERIALIZATION_FAILED, IO_PENDING, IO_FAILED. A duplicate, disabled cache and failed allocation are not one indistinguishable false.

    6. Valid reuse planning and non-consuming restore

    Extract the existing rewind decision into a pure helper. It returns raw LCP, actually reusable tokens, a chosen checkpoint/direct-reuse/reset action, next position, state-restore requirements and identity/version information.

    Preserve current position semantics: SWA coverage, target/draft sequence bounds, has_new_tokens, checkpoint pos_min/pos_max, media boundaries and the logits-token requirement. A simplistic min(LCP, checkpoint.n_tokens) is not sufficient. Unknown compatibility or rollback coverage means conservative zero reuse, not an optimistic hit.

    At this fork baseline get_common_prefix() returns the exact raw prefix; it does not itself subtract one logits token. Keep this separate from execution rewind even if an upstream PR description phrases it differently.

    Rank by actual reusable tokens. At equal benefit prefer live state, then RAM, then disk, with deterministic recency ties. Do not require both keep and similarity ratios to increase. Plan once; do not copy KV for every candidate or perform filesystem scans on each request.

    Restore reads an immutable snapshot without clearing its buffers, moving away its prompt or erasing its entry. Prepare destination metadata first, restore target/draft/speculative state, check all results, then publish the live prompt and execute the same verified plan. Live checkpoint pruning affects only the live metadata.

    Restoration is not hardware-atomic. If target restore succeeds and draft restore fails, retain the source; clear only the affected destination sequence and its sampler/speculative metadata, then safely cold-process or return the existing native error. Never generate with mixed target/draft state or restart other requests. Record actual cache_n only after successful execution validation.

    7. Deduplication, RAM limits and spill pressure

    A longer token prefix does not necessarily cover a shorter snapshot's recurrent checkpoints. Initially deduplicate exact compatible snapshots only with proven state/checkpoint coverage. Do not delete A solely because B contains A's token prefix.

    Retain existing RAM-only byte/token cap semantics. -cram -1 still has the separate token-cap behavior in the baseline. The optional per-entry token policy proposed in upstream ggml-org#28046 is not silently included.

    Choose deterministic least-recently-used eligible victims. Before evicting an unpersisted RAM victim, spill it to SSD if enabled. Keep RAM backing until durable commit acknowledgement. A committed disk copy can remain indexed after its RAM backing is released. Restore/write-pinned entries are not eviction victims.

    If ordinary capture can fit after a healthy scheduled spill, keep its outgoing slot reserved and unchanged until the spill releases capacity; other slots continue. Queue-full handling waits on capacity-change notifications without spinning or growing the queue. Cancellation releases the reservation. Do not drop the source simply because the spill has not finished yet.

    Retention is bounded, not unlimited. Terminal allocation/serialization/I/O failure, an individually oversized snapshot, or exhausted configured tiers can produce a clearly reported best-effort retention loss and later cold prefill. Keep inference correct and memory bounded; do not report a successful save, loop forever or force a restart. Optional idle evacuation may be skipped if preservation fails. Unavoidable capacity purges must name the loss reason.

    8. SSD location and configuration

    The existing option is --slot-save-path PATH, not save-slots-path. The proposed fork option is --cache-spill-mib N, default 0 (disabled), positive finite values enable spill, negatives are invalid. Enablement requires an explicit slot-save path and positive finite RAM budget for capture/read staging. Merely setting a slot-save path must not enable persistence.

    <slot-save-path>/
  existing manual slot files              # never modified by auto cache
  prompt-cache-v1/
    <compatibility-digest>/
      <engine-epoch>-<counter>.lpc         # committed snapshot
      .<unique-id>.tmp                    # unpublished write

    Keep one nonblocking filesystem ownership lock per compatibility namespace. A second process targeting the same namespace falls back to RAM-only with a warning; it does not steal ownership. Different model/config directories coexist. This is file-store coordination, not a global source-worktree or inference lock.

    Use private file/directory permissions, generated names, no-follow opened-file checks and strict parser limits. Never delete foreign namespaces, manual slot files or symlink targets.

    9. Persistent compatibility, format and commit protocol

    Build a canonical descriptor from strong content identities of model shards, draft/mmproj/adapters/control vectors and effective scales, native build/state ABI, effective context capacities, KV types/layout, SWA/recurrent parameters, RoPE/YaRN and speculative implementation settings. Use the repository's existing vendor::hash SHA-256 implementation. Hash loaded immutable model files during initialization, not on the request hot path; account for startup I/O. Path/mtime alone is not a durable equality proof. Initially require the same tested build/backend/layout.

    Use a versioned server envelope around existing native state bytes, not a new core KV format. Include magic/version/byte order/total length and full descriptor; packed media-aware tokens; source coverage metadata; full target/draft/current-speculative state; all checkpoint descriptors and target/draft/speculative blobs; per-section lengths/digests and exact EOF validation. Reset imported checkpoint task IDs to -1.

    Reject overflow, impossible positions, oversized lengths/counts, invalid tokens/media, truncation, incompatible formats and checksum mismatches before installation. A malformed checkpoint section invalidates an automatic snapshot; do not silently accept just base KV and claim full reuse. SHA-256 detects corruption but does not authenticate files against someone who can rewrite the directory.

    Keep legacy manual /slots APIs compatible. Their files are not automatically treated as complete snapshots; the automatic spill format must actually contain all hybrid/draft state.

    Write to a private same-directory temporary file, handle short writes, flush/sync/close, atomically rename and sync the directory with platform-appropriate helpers. Only committed entries become catalog-visible, and the last RAM-only backing remains alive until commit is acknowledged.

    Do not capture GPU state directly into a shared file mapping. Capture into ordinary host buffers on the owner thread; a worker performs ordinary file I/O. SSD storage does not eliminate the snapshot-sized host staging requirement.

    Startup scans only the owned namespace and builds a bounded metadata catalog. Initial limits are 64 MiB of indexed metadata and 4096 entries; excess nonindexed entries produce an honest index-capacity miss. Load KV payloads lazily. Ignore incomplete temporary files; only identifiable stale files owned by this namespace/format may be cleaned.

    This is spill persistence, not automatic write-through of every turn. Only committed SSD entries survive process restart. RAM-only/active states are not promised crash durable. Restart tests must first prove a real spill commit.

    10. Disk worker, cancellation and lifetime

    Use one worker with at most two pending jobs initially. Queued/in-flight bytes count against RAM/staging. Worker jobs hold immutable buffers, generated paths and numeric identity fields, never live context/slot pointers.

    SSD reads reserve only their destination task/slot. Results post a typed internal SERVER_TASK_TYPE_CACHE_IO_DONE through the existing server_queue::post. Apply state changes only on non-yielding owner iterations; reject stale engine epoch, slot generation or task ID before installation. Cancellation must prevent late results from generating a response or replacing a newly assigned slot.

    Do not hold the task mutex across I/O or worker join. Shutdown stops submissions/installs, handles pending work, joins the worker, drains results and destroys owners in a safe order. Portable C++ cannot impose a hard deadline on arbitrary kernel syscalls stuck on a failed filesystem; do not claim a job cap cures that. Use local SSD storage, not a network filesystem assumption.

    11. Upstream research informing this design

    As inspected on September 29, these five PRs were open/unmerged. Their implementation/reporting scopes differ; do not cherry-pick them as one bundle.

    Reference Reuse / lesson
    #28992 Independent lookup and pinned-empty regression; not a complete branch-retention/ranking fix.
    #24143 Non-consuming restore and A/X/C/A regression. Older struct layout needs a manual port.
    #27451 Share immutable checkpoint backing, catch all capture-entry allocations; not native recovery.
    #26004 Serialize checkpoint/draft/speculative state; RAM can mask broken disk restore; test restored divergence against a live reference.
    #28092 Persistent cache and compatibility concepts; avoid copying its disk-versus-RAM policy, unsafe shared-directory cleanup or mmap capture path.

    Important comments, not just PR titles:

    • ngxson, #28092: a local block-hash alternative and desired refactoring explain why deeper review of that PR was paused. This does not prove the alternative is published or available.
    • sammcj, #28092: shared-directory cleanup could wipe another model's cache; libc++ file-clock representation caused a build failure.
    • author, #28092: flags a reported ROCm GPU-DMA/shared-mmap/writeback hang risk. It was not reproduced by this task.
    • wagi-sho, #26004: RAM/checkpoint state can create a false-positive file restore test.
    • terisuke, #26004: byte-count reporting must include appended checkpoint bytes; a small cross-backend result is not a general portability claim.

    The research file records inspected heads, selected diffs, relevant discussion and nonselected adjacent work. ggml-org#28092's current diff names --cache-dir/--cache-dir-max despite its --cache-disk title. Our new spill flag is explicitly a proposed fork option.

    12. Junior implementation map and acceptance

    The draft has eight coherent slices, each with exact files/symbols, ordered instructions and completion checks:

    1. Deterministic baseline regressions in the maintained server tests.
    2. Immutable checkpoint/snapshot ownership, complete capture allocation and typed outcomes.
    3. Pure reuse planner and common pinned/automatic slot preparation.
    4. Preserve-before-mutate, non-consuming restore and conservative dedup/eviction.
    5. Full snapshot codec, configuration, namespace and atomic SSD store.
    6. Bounded worker, task completion, cancellation/restart/shutdown handling.
    7. RAM/disk/hybrid/draft/media tests and matched performance measurements.
    8. Implementation documentation and review; native deployment/proxy activation remain separate.

    Primary changes belong in tools/server/server-context.cpp, server-task.h/.cpp, narrowly affected checkpoint code in common/common.h/.cpp, new subordinate server-cache-storage.h/.cpp, common/arg.cpp, server CMake and existing test helpers/files. Preserve current media serialization. No CUDA or recurrent-algorithm rewrite is planned.

    TESTING defines 24 cases covering destructive LCP replacement, RAM-entry borrowing, pinned-empty and pinned-busy behavior, similarity on/off, logits replay, missing checkpoints, allocation and target/draft failure, byte/token quotas, SSD restart with no inherited RAM source, malformed files, interrupted writes, namespaces, symlinks, stale I/O completion, slow disk and shutdown.

    Measure actual cached/processed tokens, output correctness, capture/restore/read/write bytes and latency, peak RSS, pending budget and other-slot progress. Ordinary append continuations must not introduce full-state copies after warm-up. A dense toy result does not establish Nex acceptance; a RAM hit does not establish SSD correctness.

    Current validation: documentation structure, local links, shell-block syntax and requirement/test coverage checked. Not run: native compilation, inference, disk behavior or performance. The issue remains open. Draft PR publication follows this comment; this design is not a repair claim.

    AI assistance: ChatGPT researched and authored this fork-specific architecture/workspace at the owner's explicit request. No comment or PR was submitted to upstream ggml-org/llama.cpp.

  2. novkien commented on Sep 29, 2026

    @novkien
    novkien
    on Sep 29, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    Draft workspace published

    Draft PR #2 is open on design/issue-1-kv-cache-retention, head 18a6e1e53c181d02dffa81d595e1c6795c505b12, targeting this fork's master.

    The published diff was verified: one documentation-only commit, exactly five added files, no runtime source/configuration changes:

    • docs/kv-cache-retention/README.md
    • docs/kv-cache-retention/DESIGN.md
    • docs/kv-cache-retention/IMPLEMENTATION.md
    • docs/kv-cache-retention/TESTING.md
    • docs/kv-cache-retention/RESEARCH.md

    The full architecture is in the preceding comment. Begin with the workspace README, then IMPLEMENTATION S1. S1-S8 provide the ordered source changes and completion checks; TESTING specifies T01-T24, including SSD restart with RAM unable to mask a broken restore.

    Validation completed: local Markdown links, balanced fences, whitespace/ASCII checks, shell-block syntax, requirement/test coverage, and matching local/published Git blob hashes. The named maintained unit/test_slot_save.py was also verified at the pinned source baseline. These are document checks, not native behavioral tests.

    Native compilation, implementation, inference, SSD behavior and performance remain NOT RUN. No merge, deployment, model restart, proxy pinning activation or upstream publication occurred. Issue #1 remains open and PR #2 remains draft for the owner's implementation work.

  3. novkien
    added 4 commits that reference this issue on Sep 29, 2026

    ci: export the exact cache review source for isolated validation

    c2b7e95

    ci: collect the public tiny model for offline cache regressions

    6b598d2

    ci: complete offline Python 3.13 review dependencies

    fa04884

    ci: validate and publish the hash-checked cache review repair

    09a8b3b
  4. github-actions
    added a commit that references this issue on Sep 29, 2026

    server: harden RAM cache retention across slot reuse

    e8ef91d
  5. novkien
    added a commit that references this issue on Sep 29, 2026

    ci: remove the temporary cache review transport

    705ee67

  6. novkien commented on Sep 29, 2026

    @novkien
    novkien
    on Sep 29, 2026 – with ChatGPT Codex Connector
    OwnerAuthor

    PR #2 RAM review repair published; full issue remains open

    The owner pushed the initial native change and explicitly requested direct review repairs, stopping before their own merge/build/live test. PR #2 is now ready for review at 705ee67a9aaf75409cad53c5a5c334ec61b64be9, with mergeable=true at recheck. The tested native repair is e8ef91dc691f1197f9a8f69b586563de377d6278; the final commit removes only the temporary validation transport workflow.

    Detailed findings, repairs, evidence and limits. Non-consuming snapshots, checkpoint-aware reuse, cache opt-out, safe capture/restore, failed-idle-save retention and immutable checkpoint sharing are implemented with focused regressions. The new HTTP fixtures demonstrate 7 failures on the owner's initial bytes and 12/12 passes on the repair; expanded local tests report 53 passes and the C++ native fixture 33 checks. A fresh GitHub run rebuilt and tested the exact repair before non-force publication.

    Remaining: optional SSD spill/codec/worker are not implemented, and actual Nex/GPU/hybrid/draft/media plus matched long-context runtime acceptance remain unverified. Documentation explicitly retains these incomplete criteria. Existing RAM budgets and manual slot-file semantics remain. Do not close this issue at PR merge or treat --cache-spill-mib as an available flag.

    No merge, deployment, model restart, proxy pinning activation or upstream submission was performed. The owner retains the requested merge/build/live-trial endpoint.

  7. novkien
    added a commit that references this issue on Sep 30, 2026

    test: restore cache retention assertion

    12ac804

  8. novkien commented on Sep 30, 2026

    @novkien
    novkien
    on Sep 30, 2026
    OwnerAuthor

    Work report - long-context Qwen RAM retention verified

    Owner and scope: Codex root task, merged PR #2 delivered in fork revision 95e6565970621aa508e80fd805dcf73af413093c. The owner's 2026-09-30 request supersedes the earlier stop-before-merge/build boundary: build/deploy the fork, update Qwen-27B, test PR #2 and issue #3, and close accepted work. This acceptance covers PR #2's RAM repair; optional SSD spill remains unimplemented and is not claimed.

    Done and evidence: actual Qwen-27B/DFlash2 on all four V100-SXM2 16 GB GPUs, four slots, q8_0 KV, 64 checkpoints/min step 2048 and -cram -1. Following later owner steering, the production trial has -c 200000 -ts 29,29,27,15; native context is aligned to 200192 with unified KV, and pipeline parallelism remains enabled through long prefill. The child hash and installed source manifests match the published fork; UUID order is 5ca3,b2e,e149,d4d.

    The fixture sends exact token arrays: A=20000, B=20000 with approximately 12000 shared tokens, then continuations containing the actual A/B generated tokens plus 64 new tokens. B borrows A's slot and legitimately processes cold because usable hybrid checkpoint coverage is unavailable. A then restores on another slot with cache_n=20015, prompt_n=65. Erasing the borrower destination and restoring A again gives the same counters, demonstrating that the RAM source is not consumed. B's live continuation also reuses 20015 tokens. Warm continuations take 1.036-1.063 seconds; matched cache-disabled cold references take approximately 22.47-22.49 seconds. All four deterministic warm/cold generated token comparisons matched exactly. DFlash reports 13/13 accepted/generated draft tokens for these continuation requests. Cache opt-out followed by another borrower/new destination reports cache_n=0 as required.

    Native evidence contains 19 preservation events, eight checkpoint restores and 22 draft-acceptance events, with no capture/restore failure, CUDA error or assertion. A preservation record contains 20015 tokens and 1235481076 serialized bytes. An observed native process RSS was 13612360 KiB; this is an observation, not a claimed campaign peak. The tested long prefill left 1749-2813 MiB free per GPU.

    Media fixture correction: the first 1034-token inline-image request preserved a complete 1049-token branch with zero recurrent checkpoints. Asking for the shorter original prompt requires rewinding the 15 generated tokens, so a cold media replay is valid and does not prove a backend defect. That initial fixture's blanket near-full reuse expectation was incorrect for this hybrid case. The full harness therefore reported PARTIAL for media while all text/draft/log checks passed. A focused media append/restore fixture will establish the applicable case rather than weakening the RAM checks.

    Remaining and next: finish that media fixture and real Nex hybrid/RPC acceptance, plus relevant finite-budget/opt-out evidence already covered by maintained native/HTTP tests. Preserve current Qwen profile for the owner, publish the measured acceptance record and close only against the applicable completed checks. Proxy incident ggml-org#416 remains linked; its own route deployment endpoint is separate.

  9. novkien commented on Sep 30, 2026

    @novkien
    novkien
    on Sep 30, 2026
    OwnerAuthor

    Deployed RAM acceptance: Qwen PASS; Nex unresolved, issue remains OPEN

    Owner-authorized PR #2 merge/build is complete. The installed native/RPC source is 95e6565970621aa508e80fd805dcf73af413093c, which includes merged PR #2. The 33 native cache checks and 14 focused HTTP tests passed. Optional SSD spill is not implemented and is not claimed by this delivery.

    Qwen acceptance completed

    The actual four-GPU Qwen/DFlash2 hybrid trial passed the 20000-token A/B preservation, cross-slot restore, second non-consuming restore, usable checkpoint, opt-out and deterministic warm/cold output checks documented in the preceding report. A preserved 20015-token state needed only 65 processed tokens on continuation; four warm/cold output-token comparisons matched exactly. The later focused inline-image append fixture also passed two cross-slot restores: cache_n=1062, cold cache_n=0, exact warm/cold output tokens. The initial shorter-original-image replay was an invalid full-reuse expectation because the generated suffix required a rewind without a covering recurrent checkpoint; no runtime patch was made for that fixture.

    Nex failed the output acceptance criterion

    Nex ran with its real Qwen35-MoE hybrid target, external DFlash draft on RPC0, three native CUDA devices plus RPC0, original -c 400000, eight slots, q8_0 KV, 64 checkpoints/min step 2048 and the published fork. Mechanical preservation/restore worked: 20015 cached tokens, 65 processed continuation tokens, repeated non-consuming reuse and no native capture/restore error. However the deterministic warm output differed from the full cold reference.

    We repeated the exact original 20000-token fixture with RAM disabled (cache_ram=0, actual idle RAM evacuation disabled) and compared its live-slot continuation to cold recomputation. The A/A0 inputs, append, digest and sampling seed were identical across modes; A0 generated tokens matched. These are input controls, not a correctness PASS:

    Comparison First generated-token divergence, zero-based index
    No-RAM live warm vs full cold 4
    RAM-restored warm vs full cold 4
    RAM-restored warm vs no-RAM live warm 4

    At index 0, both warm modes chose token 33578 but its reported log probability differed by approximately 1.13e-5. This small difference does not explain or excuse later different tokens. The no-RAM control shows that warm/cold divergence is not unique to RAM restoration. It does not prove the RAM implementation is correct: RAM and live warm also diverged. Numerical/layout differences, speculative state and an actual state-restoration defect have not been separated. Do not report a confirmed RAM root cause or close this issue based only on cache_n.

    Raw evidence is on the GPU host: /tmp/issue1-nex-live.json, /tmp/issue1-nex-live-native.log, /tmp/issue1-nex-noram-original.jsonl, /tmp/issue1-nex-ram-original.jsonl and /tmp/issue1-nex-original-comparison.json. The comparison's PASS label refers to matched input control only; it does not certify restored output correctness.

    Independent media failure and next work

    Nex's first cold-image request fails before any RAM restore because the old DFlash GGUF lacks M-RoPE sections; after skipped image embeddings, its one-position validator rejects the text-position gap. This is separately recorded with source/model evidence in issue #6. No metadata derivative, causal override, model replacement or runtime patch was applied.

    For a later owner-authorized GPU session, discriminate the Nex text failure with the exact saved fixture: compare restored and untouched-live continuation logits/state at the same input, first with speculation disabled, and include repeated same-layout controls before attributing it to RAM. Recheck current source and native/RPC manifests; preserve the model/profile and original evidence. Any new runtime repair requires the owner's decision after a demonstrated cause. SSD acceptance remains a separate unimplemented scope.

    The owner explicitly requested release of idle shutdown. GPU trials stopped; Nex was safely unloaded and its original backend/specification restored. Qwen's approved persistent fork/DFlash2 profile remains configured (-c 200000, -ts 29,29,27,15, four slots, selected NVLink pairs) for the next ordinary load. The idle lease and renewal process are off, the block gate is false, and the proxy reported idle_shutdown_active=true. No new GPU work is pending in this task. This issue remains OPEN.

  10. novkien commented on Sep 30, 2026

    @novkien
    novkien
    on Sep 30, 2026
    OwnerAuthor

    Owner acceptance and closure

    On 2026-09-30 the owner explicitly directed Codex to close issue #1 and accepted the measured RAM-retention result, stating that generated-output variation is normal for this acceptance. This supersedes the prior exact warm/cold token-equality requirement and pending Nex text investigation; it does not fabricate equal outputs or an isolated numerical root cause.

    PR #2 is merged and installed in fork 95e6565970621aa508e80fd805dcf73af413093c. The 33 native checks and 14 focused HTTP tests passed. Actual Qwen 20000-token hybrid/DFlash2 retention, repeated cross-slot restore, opt-out and focused media append passed. Actual Nex preserved/restored 20015 cached tokens and processed only 65 new tokens, including repeated non-consuming reuse. The previously recorded warm/cold variation remains available in the preceding evidence.

    The owner accepts this RAM scope as complete. Optional SSD spill is not implemented or claimed. Independent Nex cold-image draft metadata issue #6 remains open. Closing #1 as completed under the owner's explicit accepted endpoint; no further Nex output-equality trials are pending.

  11. novkien
    closed this as completedon Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions