From 18a6e1e53c181d02dffa81d595e1c6795c505b12 Mon Sep 17 00:00:00 2001 From: LE TRUNG KIEN Date: Tue, 29 Sep 2026 09:34:10 +0700 Subject: [PATCH 1/9] docs : prepare KV cache retention design and junior implementation guide Addresses novkien/llama.cpp-fork#1 Documentation-only workspace for shared automatic/pinned cache handling, branch-preserving RAM snapshots and optional SSD spill. No runtime changes. Assisted-by: ChatGPT --- docs/kv-cache-retention/DESIGN.md | 203 ++++++++++++++++++++++ docs/kv-cache-retention/IMPLEMENTATION.md | 172 ++++++++++++++++++ docs/kv-cache-retention/README.md | 45 +++++ docs/kv-cache-retention/RESEARCH.md | 80 +++++++++ docs/kv-cache-retention/TESTING.md | 109 ++++++++++++ 5 files changed, 609 insertions(+) create mode 100644 docs/kv-cache-retention/DESIGN.md create mode 100644 docs/kv-cache-retention/IMPLEMENTATION.md create mode 100644 docs/kv-cache-retention/README.md create mode 100644 docs/kv-cache-retention/RESEARCH.md create mode 100644 docs/kv-cache-retention/TESTING.md diff --git a/docs/kv-cache-retention/DESIGN.md b/docs/kv-cache-retention/DESIGN.md new file mode 100644 index 000000000000..40d2be705a9c --- /dev/null +++ b/docs/kv-cache-retention/DESIGN.md @@ -0,0 +1,203 @@ +# Branch-preserving RAM cache with optional SSD spill + +Status: implementation design for [issue 1](https://github.com/novkien/llama.cpp-fork/issues/1), not implemented. Source baseline: `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20`. Evidence and upstream comparisons are in [RESEARCH.md](RESEARCH.md); execution instructions are in [IMPLEMENTATION.md](IMPLEMENTATION.md). + +## 1. Requirements and scope + +The owner's requested sequence is A(100k) -> B(100k) -> A-next, with A and B allowed to use the same physical slot. A must remain reusable after B replaces the live slot state when the configured cache can retain it. This applies both to automatic selection and explicit `id_slot` requests. + +R1. Preserve a reusable outgoing branch before rollback, replacement, idle evacuation or an ordinary idle-slot purge destroys its only copy. + +R2. Search the cache independently of the outgoing-state save heuristic. An empty pinned slot and `--slot-prompt-similarity 0` must not bypass lookup. A busy slot must not be mutated. + +R3. A borrowed snapshot remains available. Score candidates by actually reusable state, not two independently normalized similarity ratios. + +R4. Optional SSD spill lives below the configured `--slot-save-path`, preserves the complete server snapshot and permits reuse of committed entries after process restart. + +R5. Bound RAM, pending I/O, metadata and disk usage; preserve normal concurrency, cancellation and correctness. Do not synchronously write SSD files on the inference loop. + +R6. Keep normal append-only continuations cheap. Do not capture full KV on every request, force every request through a file or turn this into a proxy-side `/slots` RPC loop. + +The current draft contains documentation only. Future implementation includes the RAM path and SSD path described here, but not a CUDA/kernel rewrite, distributed cache, cross-machine portability guarantee, compression, native model recovery, or proxy scheduling changes. `llama-proxy#419` recovery/order work and `#386` admission optimizations are not silently incorporated. Future proxy pinning integration remains separate. + +## 2. Existing mechanism and required changes + +The inspected `get_available_slot()` gates both save and load with `update_cache`; selected-slot saving normally depends on `f_keep < 0.5` or the LRU path. Pinned empty slots can miss this gate. The caller checks busy state after selection. `server_prompt_cache::load()` restores and then consumes the saved entry. Allocation deduplicates by token-prefix containment, which does not prove checkpoint coverage. These are source findings, not newly observed runtime incidents. + +Reuse the existing `server_prompt_cache`, `server_tokens`, `common_prompt_checkpoint` and `llama_state_seq_*_ext` APIs. Do not create a second cache manager. Add a small storage/codec helper for disk operations; it must not own or access live llama contexts. + +```text + explicit id_slot or automatic choice + | + available target slot + | + shared, read-only reuse planner + / | \ + live state RAM snapshot SSD catalog + \ | / + preserve outgoing branch if needed + | + restore chosen immutable snapshot + | + valid rewind + suffix prefill + TG +``` + +## 3. Ownership and data model + +A `server_slot` owns one live sequence. A snapshot describes one reusable prompt branch, not one agent and not one slot. `source_slot_id` is diagnostic provenance only; it is never the lookup key or a reason to restore incompatible data. + +Extend the existing cache entry with: + +- An internal monotonic snapshot ID, engine epoch, compatibility descriptor and source-slot provenance. +- Exact `server_tokens` including media data, plus a token-position map where needed. +- Full target state, full draft state when present, and current speculative implementation state when required by the active drafter. +- The full checkpoint list: token counts, position ranges, target/draft partial-state bytes and `data_spec`. Persisted task IDs are reset to `-1` on import. +- Source sequence position bounds and memory capability metadata required to plan valid rollback. +- Storage location, byte/token accounting, last-use sequence, and transient pins held by restore/write operations. + +Payload bytes become immutable after capture. Copy checkpoint metadata by value but share immutable large target/draft checkpoint buffers, following the ownership idea in upstream #27451. Updating a checkpoint allocates a fresh backing buffer; it must never rewrite a buffer shared by a saved branch. Keep `data_spec` value-owned initially. `server_tokens` needs `clone()`, not a deleted copy constructor or a text-only conversion. + +A cache lease keeps an entry alive during planning/restore/spill. Erasing its index entry cannot invalidate a pointer being used by a pending operation. Count pending owned buffers as well as indexed RAM entries; removing an entry from a list is not proof that its memory has been freed. + +## 4. One slot-preparation pipeline + +### 4.1 Select without mutation + +Refactor selection to return a target and a reason only. Preserve existing explicit-ID normalization/validation behavior unless separately changed and tested. For an explicit request, do not silently choose another slot. For auto mode, consider only available slots and retain existing scheduling tie behavior where reuse benefit is equal. + +Check busy state before save, clear, restore or checkpoint mutation. Reserve an idle target while preparing an asynchronous disk restore. A reserved target is unavailable to other tasks but is not falsely reported as generating. Parent/child task capacity must be checked before changing any participating slot. + +Validate the incoming execution configuration, including LoRA/aLoRA handling, before selecting compatible state. Snapshot the outgoing state under its old compatibility descriptor before applying a configuration change. Do not relabel old KV with the new adapter identity. + +Honor `cache_prompt=false`: do not restore a cached candidate or archive that request's resulting state. Preserve a previously cache-eligible outgoing branch when appropriate. Non-completion tasks keep their existing non-cache behavior. Explicit slot erase must not trigger a new snapshot; it is not advertised as a global disk-archive purge. + +### 4.2 Plan reuse, once + +Extract the existing rollback decision into a pure helper used for live, RAM and disk candidates. Its proposed result is: + +```text +cache_reuse_plan + candidate_id / live-slot identity / version + raw_lcp_tokens + reusable_tokens + checkpoint index, or direct reuse / cold reset + next token position + needs_target_restore / needs_draft_restore + configuration compatibility +``` + +Use the existing position semantics, not `min(LCP, checkpoint.n_tokens)` as a substitute. The source algorithm uses sequence position minima, SWA coverage, `has_new_tokens`, checkpoint `pos_min`/`pos_max`, media positions and the requirement to evaluate a token for logits. Apply equivalent constraints to target and draft state. When metadata cannot establish a safe rewind, report zero reuse; never claim a speculative hit. + +At this baseline `server_tokens::get_common_prefix()` returns the exact common prefix. It does **not** subtract a logits token. Keep raw prefix matching separate from the execution-only replay adjustment. + +Rank compatible candidates primarily by `reusable_tokens`. For equal benefit choose live state, then RAM, then disk; use last-use order as a deterministic secondary tie. Do not require both `f_keep` and `f_sim` to increase. Do not restore each candidate merely to evaluate it. A candidate with no useful state must not trigger a costly copy. + +Disk candidates have bounded in-memory metadata containing exact token/media identity and checkpoint descriptors. Metadata lookup performs no per-request directory scan or KV-file read. A cold catalog entry not admitted to the bounded index may miss; report that limitation rather than pretending all disk bytes are searchable without memory cost. Block-hash indexing can be added later without changing correctness. + +### 4.3 Preserve before the first destructive operation + +An append-only compatible request does not need a full snapshot merely because execution replays a final token. A semantic prefix truncation, divergence, configuration replacement or switch to another saved state does require preservation of useful outgoing state unless an equivalent snapshot is already retained. + +Use raw prefix and compatibility for this distinction. Preserve before `seq_rm`, chunk shifting, state replacement, checkpoint erasure, `keep_first`, or `prompt_clear`; a hook after any of them is too late. A conservative save is correct for an uncertain semantic divergence; do not guess that different tokenization is harmless. + +Capture transaction: + +1. Compute full serialized target/draft sizes and accounted tokens/checkpoints. +2. Reserve cache/staging capacity and protect the selected restore candidate from eviction. +3. Allocate every part in an unpublished temporary entry. Catch allocation failure around vectors, token clone, checkpoint metadata and container insertion. +4. Serialize the live state while no decode is using it; require every returned byte count to match. Capture the required speculative implementation state through its existing API. +5. Publish only the completed snapshot. Only now mark the outgoing branch preserved and perform deduplication that removes older equivalents. + +Use typed results: `SAVED`, `ALREADY_PRESERVED`, `DISABLED`, `BUDGET_REJECTED`, `ALLOCATION_FAILED`, `SERIALIZATION_FAILED`, `IO_PENDING`, `IO_FAILED`. A duplicate and a failed save are not the same result. + +### 4.4 Restore without consuming + +Pin the chosen source. Prepare token/checkpoint metadata before mutating live state. Read SSD bytes into bounded ordinary host memory, verify them, then restore target/draft on the inference owner thread. Verify all byte counts and restore required speculative state. Publish slot metadata only when the whole operation succeeds. + +Do not clear source byte vectors, move the source prompt away, or erase the source merely because restoration succeeded. Live checkpoint metadata may be pruned without altering the saved copy. + +A restore is not hardware-atomic. If target restoration succeeds and draft restoration fails, preserve the source snapshot, clear only the affected destination sequence and its speculative/sampler metadata, and use the existing safe cold-processing path if possible. If safe cold processing cannot proceed, return the existing native error. Never continue with mismatched target/draft state or restart other requests. A failure is not a cache hit. + +### 4.5 Execute the verified plan + +Recheck task/slot/engine epoch after asynchronous work. Apply the planned rewind and process the remaining prompt. Record actual `cache_n` only from the state accepted by execution. Log planned and actual counts separately so a metadata estimate cannot masquerade as reuse. + +## 5. Deduplication, eviction and resource failure + +Do not delete A merely because B's tokens contain A as a prefix. For recurrent/SWA models B may lack the checkpoint needed to rewind to A. Initially deduplicate exact compatible snapshots only when the retained entry's position/checkpoint coverage covers the replaced entry. Keep distinct branches until ordinary eviction. + +Preserve existing `--cache-ram` byte and token-limit semantics in RAM-only mode, including the fact that `-1` does not disable the separate aggregate token cap in this baseline. Optional per-entry token policy from upstream #28046 is not bundled automatically. + +Use deterministic least-recently-used eligible victims, updating recency after real use rather than scanning. Before evicting an unpersisted RAM victim, try SSD spill when enabled. Keep its RAM backing alive until disk commit is acknowledged. An already persisted victim can release RAM immediately, subject to leases. Disk victims are chosen only within this server's owned namespace and never while read-pinned. + +When a normal capture cannot reserve RAM but a spill can free it, keep the outgoing slot reserved and unchanged, submit an eligible victim spill, and resume capture after commit releases capacity. Other slots continue. If the queue is full, attach the target to the next capacity-change notification rather than spin or grow the queue. Cancellation releases this wait. A source must not be dropped merely because a healthy, already scheduled spill has not finished. + +The retention promise is conditional on resources. If both tiers are full, a snapshot exceeds the finite capture budget, or a save fails, preserve inference correctness and bounded memory: retain existing copies where possible, then use an explicit best-effort cache-loss outcome rather than an infinite wait or a forced server restart. A request may cold-prefill after this outcome. Emit the exact reason and affected snapshot; do not label it a successful save. Optional idle evacuation can be skipped when preservation fails; an unavoidable capacity purge must be logged as loss. This design does not promise zero loss under OOM, disk failure or exhausted quotas. + +## 6. SSD contract + +### 6.1 Configuration and directory + +The existing option is `--slot-save-path PATH`. The new proposed option is **`--cache-spill-mib N`**, with `N=0` by default and `N>0` enabling a finite disk budget. Negative values are rejected. Enablement requires an explicit slot-save path and a finite positive `--cache-ram` budget for capture/read staging; invalid combinations fail startup with an explanation. Setting a slot-save path alone must not enable automatic persistence. + +```text +/ + existing-user-slot-files # never touched by this cache + prompt-cache-v1/ + / + -.lpc # committed complete snapshot + ..tmp # unpublished, owner-created only +``` + +Use a process-lifetime nonblocking filesystem lock for a single compatibility directory. Another server using that exact namespace disables its SSD tier with a warning and continues RAM-only; it does not steal the lock or delete files. Different model/config namespaces can coexist. This lock protects cache files, not source worktrees or unrelated inference. + +### 6.2 Compatibility and trust + +Build a canonical descriptor from model shard content digests, draft/mmproj/adapter/control-vector identities and effective scales, native build/state-format version, model architecture, effective context capacities, KV types/layout, SWA/recurrent settings, RoPE/YaRN parameters, and speculative implementation/configuration. Use the existing vendored SHA-256 implementation under `vendor/hash/sha256`; do not use `std::hash` or path/mtime alone as durable identity. + +Hash immutable model files once during model initialization for persistent-cache compatibility, not on the request path. This costs startup I/O and must be measured. Detect files changing during fingerprinting and decline persistent reuse. Loaded weights must not be modified in place. Initially require the same tested build/backend/layout; a user's cross-backend success is not a portability contract. + +Store the full descriptor and compare it, not only its directory hash. Raw token/media equality still determines prompt matching. Sampling temperature is not a substitute for execution compatibility. No conversation ID is required for the single trusted proxy deployment; a multi-tenant cache/authentication redesign is out of scope. + +Cache files contain sensitive derived state and prompt material. Use private directory/file permissions, generated filenames, strict relative paths, no symlink following, and checks on opened file handles. Treat files as untrusted input to the parser even in a private directory. SHA-256 checksums detect corruption; they are not authentication against an attacker who can rewrite the directory. + +### 6.3 Complete file format + +Use a server-owned versioned envelope around existing serialized sequence bytes; do not change the core llama state format. Explicit little-endian integers and checked lengths describe: + +1. Magic `LPCACHE1`, envelope version, byte order, total length, compatibility descriptor and snapshot identity. +2. `server_tokens::serialize()` payload, exact token count, source position bounds and planner capability metadata. +3. Full target state and optional full draft state, each with length and digest. +4. Current speculative implementation bytes and checkpoint count. +5. For every checkpoint: `n_tokens`, `pos_min`, `pos_max`, target/draft partial-state blobs and `data_spec`, each length-delimited. On import, `id_task=-1`. +6. A checksum covering metadata and all payload sections, and an exact end-of-file check. + +Reject overflow, oversized counts/blobs, impossible positions, incompatible versions, truncation, checksum mismatch and invalid token/media payload before calling native setters. A malformed checkpoint appendix invalidates the automatic snapshot; never silently restore only its base KV and claim checkpointful reuse. + +Keep manual `/slots` save/restore files and their existing API compatible. They are not automatically imported as complete snapshots. Upstream #26004 is a reference for a future manual-file adapter and for checkpoint fields, not permission to call a legacy slot file equivalent to this envelope. + +### 6.4 Publication and restart + +Write a private temporary file in the same namespace; loop for short writes, flush/sync, close, then atomically rename and sync the directory using platform-appropriate helpers. Publish the catalog entry and release the last RAM-only copy only after durable commit acknowledgement. If commit fails, keep RAM when possible and report failure. Do not replace a valid file with a partially written one. + +Do not DMA GPU state directly into a shared file mapping. Capture into ordinary host buffers on the inference owner thread; the storage worker writes those immutable buffers using ordinary file I/O. The ROCm mmap/writeback report is a reason to avoid that path, not proof that all mmap reads are defective. + +On startup, scan only the owned namespace, validate bounded metadata and build the index. Never delete a foreign namespace or arbitrary files under slot-save-path. Ignore unfinished temporary files; remove only identifiable stale files belonging to this format and namespace after lock acquisition. Bound indexed metadata to 64 MiB and 4096 entries initially; report entries not indexed. KV payloads are loaded lazily. + +This is spill persistence, not write-through persistence of every turn. Only committed SSD entries survive a crash/restart. RAM-only and active states are not promised durable. Tests must force a real spill and confirm commit before testing restart recovery. + +## 7. Threading, I/O lifecycle and cancellation + +The server context/queue already controls safe decode boundaries. Keep all live KV, checkpoint and speculative-state operations on that owner path. The storage worker receives immutable byte buffers, generated filenames and numeric IDs, never a `llama_context *`, slot pointer or mutable prompt. + +Use one disk worker and at most two pending jobs initially. Both queued and in-progress owned bytes count toward the finite RAM/staging budget. A full queue cannot grow unbounded; skip optional idle spill, or let required capture wait on an existing capacity-changing job without growing the queue. Terminal I/O failure or impossible budgets use the documented pressure fallback. Do not block the inference loop waiting for a write. + +A disk read reserves only the target task/slot, allowing other slots to decode. Completion posts a typed internal `SERVER_TASK_TYPE_CACHE_IO_DONE` through `server_queue::post`. Process state-changing completion only when the queue is not yielding a decode; otherwise decline it unchanged for later handling. Validate engine epoch, slot generation and task ID before installation. Cancellation releases the reservation/lease; a late I/O result is discarded without generation or success reporting. + +Do not hold the task-queue mutex while doing filesystem I/O, joining the worker or invoking sleep callbacks that need queue progress. Shutdown stops new submissions, cancels pending installs, joins the worker, drains results and only then destroys storage/context owners. Normal local-file errors must be handled; portable C++ cannot guarantee a hard deadline for a kernel syscall stuck on a failing filesystem. Do not use network filesystems or claim that a job-count cap can cure kernel hangs. + +## 8. Observability and acceptance + +Add compact events/counters for capture reason/result, snapshot ID, source/target slot, raw LCP, planned/actual reusable tokens, RAM/disk hit, missing checkpoint, compatibility rejection, copy/read/write milliseconds, bytes, eviction reason and queue/staging use. Do not log prompt text or token arrays by default. IDs must be scoped by engine epoch. + +Acceptance requires deterministic branch-loss and pinned-empty regressions; correct output as well as counters; RAM-only, SSD-only-after-restart and mixed-tier evidence; cancellation/failure isolation; and matched cold/warm/alternating performance measurements. Ordinary append continuations must not add full-state save/restore operations. Neither native builds nor these tests have been run for this documentation draft. See [TESTING.md](TESTING.md). diff --git a/docs/kv-cache-retention/IMPLEMENTATION.md b/docs/kv-cache-retention/IMPLEMENTATION.md new file mode 100644 index 000000000000..6014e79ffb30 --- /dev/null +++ b/docs/kv-cache-retention/IMPLEMENTATION.md @@ -0,0 +1,172 @@ +# Junior implementation guide + +This is an ordered implementation guide, not an implemented patch. Complete the RAM and SSD slices on the draft branch. Keep [DESIGN.md](DESIGN.md) authoritative for behavior and [TESTING.md](TESTING.md) authoritative for evidence. Names introduced below are proposed names. + +## 0. Establish a safe local starting point + +Run these read-only commands in the checkout you intend to use: + +```bash +pwd +git rev-parse --show-toplevel +git remote -v +git branch --show-current +git status --short +git rev-parse HEAD +``` + +The intended repository is `novkien/llama.cpp-fork`, not `ggml-org/llama.cpp` or `novkien/llama-proxy`. This design's source baseline is `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20`. Newer commits are not automatically wrong: inspect intervening changes to the listed symbols and reconcile them. Never reset to that SHA or switch another person's shared branch to make the commands match. + +A fresh clone is an optional way to get this draft without disturbing an existing checkout. Use the head branch shown by the actual PR. Do not push an upstream PR as a side effect of working on this fork. + +Read root/scoped AGENTS, CONTRIBUTING, `tools/server/README-dev.md` and `tools/server/tests/README.md`. The current owner request authorizes this fork's design workspace and draft publication; implementation and production activation must be reported separately. Keep complete source files, not placeholder stubs or intentionally failing TODO binaries, in implementation commits. + +## 1. Read the source in execution order + +Use symbol searches rather than stale line numbers: + +```bash +git grep -n -E 'get_available_slot|process_single_task|launch_slot_with_task|try_clear_idle_slots' -- tools/server +git grep -n -E 'prompt_save|prompt_load|server_prompt_cache::|struct server_prompt' -- tools/server +git grep -n -E 'get_common_prefix|keep_first|serialize\(|deserialize\(' -- tools/server/server-common.cpp +git grep -n -E 'common_prompt_checkpoint|data_spec' -- common +git grep -n -E 'slot.save.path|cache.ram|ctx.checkpoints' -- common/arg.cpp common/common.h +git grep -n -E 'cleanup_pending_task|yield_to_queue|on_sleeping_state' -- tools/server/server-queue.* +``` + +| File/symbol | What to understand before editing | +|---|---| +| `server-context.cpp`: `server_slot::prompt_save/load/clear` | Full target/draft serialization and live-state destruction. | +| `get_available_slot`, `process_single_task` | Selection, update_cache gate, busy checks, parent tasks and task ownership. | +| Prompt-start section of `update_slots()` | Raw matching, chunk shifts, recurrent/SWA checkpoint selection, logits replay and final `cache_n`. | +| `server-task.h/.cpp`: `server_prompt_cache` | Entry allocation, prefix deduplication, consumption on load, byte/token limits. | +| `server-common.cpp`: `server_tokens` | Exact LCP, media-aware token cloning, packed serialization and validation. | +| `common/common.h/.cpp`: checkpoints | Target/draft state and speculative bytes; updates must not mutate shared snapshots. | +| `server-queue.h/.cpp` | `post`, deferred tasks, decode yielding, cancellation and shutdown. | +| `vendor/hash/CMakeLists.txt` | Existing `vendor::hash` target and vendored SHA-256; no new crypto dependency needed. | + +Write down one A -> B -> A trace and one pinned-empty trace before modifying their conditions. A blank slot with a matching RAM entry is not a cache miss. A nonzero LCP without a valid recurrent checkpoint is not a reusable-state hit. + +## 2. Build a disposable baseline + +The following creates only an external build/test environment. It does not launch production inference. It may download Python packages and the maintained test fixtures when tests run. + +```bash +ROOT="$(git rev-parse --show-toplevel)" +WORK="$(mktemp -d -t llama-cache-issue1.XXXXXX)" +printf 'Evidence directory: %s\n' "$WORK" +python3 -m venv "$WORK/venv" +. "$WORK/venv/bin/activate" +python -m pip install -r "$ROOT/tools/server/tests/requirements.txt" +cmake -S "$ROOT" -B "$WORK/build" \ + -DGGML_CUDA=OFF -DGGML_METAL=OFF -DGGML_VULKAN=OFF \ + -DCMAKE_BUILD_TYPE=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS=ON +cmake --build "$WORK/build" --target llama-server -j 2 +unset DEBUG_EXTERNAL +export LLAMA_SERVER_BIN_PATH="$WORK/build/bin/llama-server" +export LLAMA_CACHE="$WORK/model-cache" +export PYTEST_WORKERS=1 +mkdir -p "$LLAMA_CACHE" +bash "$ROOT/tools/server/tests/tests.sh" \ + unit/test_completion.py -k prompt_cache -v -x +``` + +The maintained runner changes into its own tests directory. `DEBUG_EXTERNAL` must not point tests at a real service. Use test-helper-owned ports and temporary slot directories. A CPU fixture can prove scheduler/cache behavior; it cannot validate Nex hybrid checkpoints or production GPU performance. + +Record baseline commit, flags, model digest, test node, actual counters, output and logs. Rebuild into a different directory for comparisons, or record exactly which rebuilt candidate replaced the baseline binary. Do not call an unrun command PASS. + +## S1. Add deterministic failing reproductions + +Edit existing `tools/server/tests/unit/test_completion.py` and `test_slot_save.py` where appropriate; extend `tests/utils.py` only for options the new fixtures need. Do not copy an old upstream test helper over the current helper. + +Implement T01-T05 from TESTING first. Use explicit token arrays or a tokenizer-derived stable fixture to control the branch split. Avoid random prompt selection and sleeps as the only evidence of busy state. Use `n_predict=0` for pure cache-selection fixtures where supported; multi-turn fixtures must include the actual generated tokens in the next prompt. For the consumption case use A -> unrelated X -> prefix-sharing C -> A-next so that A exists only in RAM when C borrows it. For pinned-empty, prove the requested slot was evacuated before the continuation. + +Assert that the tests fail for the intended reason on the baseline and retain their raw counters. A test failing because the model is missing, the port is occupied or the request JSON is wrong is not a reproduced cache defect. + +Completion: baseline failures are discriminating; existing prompt-cache tests still have recorded results. + +## S2. Make snapshot ownership and capture safe + +Change `common/common.h/.cpp` for immutable shared target/draft checkpoint backing buffers, adapting the small ownership pattern in upstream #27451. `reset(size)` must allocate fresh storage before exposing a writable capture buffer. Existing copies must still see the old bytes after reset or clear. Keep checkpoint metadata independent; keep speculative bytes value-owned initially. Search every use of `data_tgt` and `data_dft` and adapt it without dropping bounds checks. + +In `server-task.h`, add snapshot metadata, a lease/pin representation and typed capture/restore outcomes. Prefer straightforward structs and RAII ownership. A lease means "this operation holds the bytes alive"; it is not a mutex held across GPU or disk work. + +Replace the current alloc-then-publish pattern with reserve -> construct temporary -> fill -> publish. Include token/media clone, checkpoint allocation, container insertion and byte-count verification in error handling. Deduplication removal occurs only after the replacement snapshot is complete. Do not leave an indexed entry with allocated but unwritten bytes after an exception. + +Add current target/draft position/capability metadata and an engine epoch. Source-slot ID is provenance. Use an epoch-qualified sequence for snapshot IDs so process restarts cannot overwrite older filenames. + +Completion: T06-T08 pass for ownership/allocation; serialization returns precise success/failure; original slot remains untouched during a failed capture. + +## S3. Extract the reuse planner and unify slot preparation + +In `server-task.h/.cpp`, define the pure `cache_reuse_plan` and candidate descriptor from DESIGN. Move the decision part of the prompt-start rewind logic into a helper. Do not move native state setters into this helper. + +Translate the existing position algorithm literally before optimizing it. Inputs include raw LCP, token-to-position conversion, whether new prompt tokens exist, target/draft minimum/maximum positions, SWA coverage, checkpoint descriptors and configuration identity. Output includes the specific usable checkpoint, next position and reusable token count. Preserve media chunk boundaries and the final-logits adjustment. Compare helper results against the old live-state path on fixtures before using it for RAM/disk. + +Refactor `get_available_slot()` to selection-only. In `process_single_task`, validate availability/configuration/parent capacity before any cache mutation. Explicit ID requests must never modify busy slots or fall through to another slot. A disk-pending reservation must be distinguishable from idle and generating and included in availability/cancellation logic. + +Perform one candidate selection and retain its plan/lease. Remove `update_cache` as the gate for lookup. Do not merely add a zero-denominator guard inside the similarity-enabled block; that still misses similarity=0. Replace the `f_keep && f_sim` comparison with the DESIGN ordering. + +Wire execution to the same verified plan. On epoch/version change discard and replan before mutation. Keep the existing errors for invalid request/capacity; this refactor is not authority to reset the engine. + +Completion: T01-T05 and T09-T11 pass; no state API call is made by the pure planner or on a busy slot. + +## S4. Preserve branches and stop consuming snapshots + +Add an `ensure_preserved` call before the first destructive change in ordinary replacement, idle evacuation and idle purge. Audit ALL matching call sites: `prompt_clear`, `seq_rm`, `seq_add`, `keep_first`, checkpoint erase and `llama_state_seq_set_data_ext`. Explicit erase/cancellation/model teardown have different intent; do not accidentally turn an explicit cache erase into another save. + +Compute semantic divergence from raw tokens and execution compatibility, not the post-logits `n_past`. A pure extension must not create a full-KV snapshot every turn. A shorter request, a changed suffix or another compatible branch must preserve the useful outgoing branch if not already represented. When preservation cannot succeed, follow the explicit pressure/failure contract and count the loss, not a false successful save. + +Change `load` to read immutable source buffers. Clone tokens and copy independent checkpoint metadata into a prepared destination object. Remove source `clear()`, `shrink_to_fit()` and `states.erase()` from successful restore. Do not copy full KV byte vectors a second time just to keep the RAM source. Maintain leases until all setters and metadata installation finish. + +Restore errors must not publish a half-restored prompt. Clear only the target sequence and its draft/speculative/sampler state when needed, then safely cold-process or return the existing error. Do not delete a healthy source after an unrelated destination capacity failure. + +Use exact-equivalence/coverage checks for deduplication. Maintain deterministic eligible-victim recency and existing RAM-only caps. Test shared-buffer lifetime after eviction; a removed index entry is not necessarily freed memory. + +Completion: A survives both destructive slot reuse and another branch's RAM restore. T12-T14 pass, and append-only capture/restore counts remain zero after initial warm-up where no evacuation occurs. + +## S5. Implement the codec and opt-in SSD store + +Add `tools/server/server-cache-storage.h/.cpp` as a subordinate codec/file-I/O helper, not a second scheduler or cache. Register it in `tools/server/CMakeLists.txt` and use the existing `vendor::hash` target. Keep all `llama_context` calls out of this module. + +Implement DESIGN's version-1 envelope using checked little-endian reads/writes, existing `server_tokens::serialize/deserialize/validate`, and complete target/draft/spec/checkpoint sections. Use length-delimited data, never `write(sizeof(struct))`. Validate lengths against actual file size and reserved RAM before allocation. Initial parser limits: at most 1024 checkpoints and at most the configured capture/read budget for a snapshot; unsupported larger snapshots return a reason, not an allocation attempt. Validate position ranges and token counts before narrowing to native integer types. + +Use SHA-256 for payload integrity and compatibility namespaces. Link the already vendored implementation; do not invent cryptography or add an HTTP dependency. Include all model shards and execution-affecting inputs in the canonical descriptor. Cache model fingerprints only for the loaded immutable model lifetime. Do not accept path/mtime alone as proof of equal weights. + +In `common/common.h` and `common/arg.cpp`, add proposed `cache_spill_mib`, default 0, and `--cache-spill-mib N`. Reject negatives; require a configured `--slot-save-path` and positive finite RAM budget when enabled. Keep RAM-only legacy flags untouched. Update the generated/maintained server option documentation according to its current generation procedure. + +Implement the private namespace, exclusive nonblocking namespace lock, generated paths, no-follow file opens, atomic temporary-file publication and exact total-byte accounting. Never delete manual slot files, foreign model directories or symlink targets. Use ordinary buffered writes from host memory, NOT shared mmap as the destination for GPU state capture. + +Build the bounded startup metadata index. Do not eagerly read all KV payloads or scan the directory during each request. If metadata exceeds 64 MiB or 4096 entries, keep a deterministic recent subset and expose an index-capacity miss. Do not claim the nonindexed files were corrupt. + +Completion: codec tests reject malformed input without calling state setters; T15-T19 establish isolated restart reuse and file safety. The SSD format does not silently upgrade old manual slot files. + +## S6. Integrate bounded disk I/O without blocking other slots + +Keep one worker and two jobs maximum. Define native storage jobs/results holding immutable buffer leases, numeric IDs, engine epoch and generated file paths. The worker may perform read/write/hash/fsync/rename; it cannot access a live slot or call any llama state function. + +Add a typed internal `SERVER_TASK_TYPE_CACHE_IO_DONE` payload in the existing task mechanism. The worker uses the verified `server_queue::post(server_task&&)` entry point. Process the completion on a non-yielding owner iteration; decline it unchanged while decoding. Do not bypass the queue's safety boundary just because disk I/O has finished. + +For a read, reserve only the destination and remember task ID, slot generation and engine epoch. Add that pending ownership to cancellation and cleanup paths, including `cleanup_pending_task`. If cancelled, release the reservation; a late result releases bytes and does not resume inference. Prevent parent/child requests from being partially installed. + +For a spill, pin the RAM source until commit succeeds. Pending bytes remain accounted; no "evict then write through dangling pointer". After acknowledgement mark the disk copy committed and release RAM only if no live lease needs it. On failure keep RAM where possible, record the specific cause, and avoid retry loops on the same broken path. + +Handle queue-full and snapshot-too-large without unbounded allocations or waits. These resource-loss paths must remain distinguishable from branch-loss bugs. Under normal pressure the selected recent outgoing snapshot should replace eligible older entries, spilling them first when possible. A required capture that can fit after a scheduled spill waits with only its source slot reserved and unchanged; wake it on committed capacity change, not a polling loop. Do not discard the outgoing branch merely because the worker has not finished. Do not let a queued optional spill block all incoming requests. + +Integrate orderly stop and model epoch changes. Do not join a disk worker while holding the server task mutex or from a callback that prevents its completion processing. Stop submissions and installs before destroying contexts or queues. Only SSD-committed snapshots are crash durable; this version does not flush every active/RAM-only state on shutdown. + +Completion: T20-T24 prove cancellation, stale completion rejection, other-slot progress and bounded memory. Document the filesystem/kernel hang limit rather than promising hard cancellation of arbitrary blocking syscalls. + +## S7. Finish regression and performance evidence + +Run the existing cache, slot save, cancellation and relevant media/speculation tests along with TESTING's deterministic cases. Enable the real hybrid model tests only on isolated hardware/capacity the owner intends for testing. Preserve the exact native model/build/KV/draft configuration in the report. + +Measure cold baseline, ordinary warm continuation, unrelated A/B alternation and prefix-sharing A/C alternation. Separate capture, disk write, disk read, restore, prefill and generation. Use counters and deterministic content/logit checks, not latency alone. Inspect peak RSS, accounted cache/staging bytes and other-slot progress under slow disk injection. + +A dense toy test is not acceptance for Nex. A successful RAM restore is not acceptance for SSD. A successful build is not acceptance for either. Report each matrix row PASS/PARTIAL/FAILED with evidence and untested boundaries. + +## S8. Finish the implementation PR, not production activation + +Update this checklist and the runtime README with actual implemented option names, resource behavior and remaining limitations. Remove stale claims if the implementation differs, or revise the design explicitly before changing its contract. Link issue 1 and the precise upstream references used; preserve attribution/licenses for adapted code. + +Check the exact diff and run `git diff --check`. Commit only intended paths; no broad staging, force push or unrelated cleanup. Keep the PR draft until its promised implementation evidence exists. Do not mark issue 1 fixed because this design or a compiling patch was merged. Proxy policy integration and native deployment require their own explicit endpoint and runtime verification. diff --git a/docs/kv-cache-retention/README.md b/docs/kv-cache-retention/README.md new file mode 100644 index 000000000000..0aa5c236ea12 --- /dev/null +++ b/docs/kv-cache-retention/README.md @@ -0,0 +1,45 @@ +# Issue 1: branch-preserving KV cache workspace + +Status: **design and implementation instructions only; no runtime fix implemented**. + +Addresses [novkien/llama.cpp-fork#1](https://github.com/novkien/llama.cpp-fork/issues/1). Historical production evidence: [novkien/llama-proxy#416](https://github.com/novkien/llama-proxy/issues/416). + +Prepared on 2026-09-29 against fork `master` at `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20`. This SHA is the source baseline, not a claim about a deployed binary. + +## Outcome + +When a new prompt replaces a slot's old branch, preserve the old branch's complete reusable state in RAM before destructive changes. Restore it later into either a native-selected slot or an explicitly requested `id_slot`. Optionally spill RAM victims to an SSD directory below the existing **`--slot-save-path`**, including recurrent/SWA checkpoints and draft state. + +A physical slot is an execution destination, not a cache key. Multiple saved branches can originate from the same slot. A saved branch is not consumed when another request borrows its prefix. + +Retention is bounded by configured resources. Cache eviction, incompatible state and genuinely changed prompts remain possible; this is not unlimited storage or a promise that every request avoids prefill. + +## Start here + +| File | Purpose | +|---|---| +| [DESIGN.md](DESIGN.md) | Settled behavior, algorithms, state ownership, RAM/SSD contracts, failures and boundaries. | +| [IMPLEMENTATION.md](IMPLEMENTATION.md) | Junior-oriented implementation order, exact files/symbols, build commands and completion checks. | +| [TESTING.md](TESTING.md) | Deterministic regressions, disk-isolated tests, failure injection and performance evidence. | +| [RESEARCH.md](RESEARCH.md) | Dated upstream PRs, inspected comments, source anchors and reuse decisions. | + +Read DESIGN first, then implement the numbered slices in IMPLEMENTATION. Update the checklist below as real code and tests land on this draft branch. Do not replace failed behavioral tests with a successful build or a larger cache counter. + +## Implementation checklist + +- [ ] S1: reproduce baseline defects with deterministic fixtures. +- [ ] S2: safe snapshot ownership, complete allocation handling and typed results. +- [ ] S3: pure reuse planner, pinned/automatic pipeline and preserve-before-mutate. +- [ ] S4: non-consuming restore, conservative deduplication and bounded retention. +- [ ] S5: complete snapshot codec and opt-in namespaced SSD store. +- [ ] S6: bounded disk worker, cancellation, restart and shutdown integration. +- [ ] S7: RAM/disk/hybrid/draft/media regressions and matched performance measurements. +- [ ] S8: implementation documentation, review and owner-selected native/proxy integration. + +These are implementation slices, not extra approval stages. The present delivery stops at a draft documentation PR. It does not authorize production restarts, model changes, proxy pinning activation, upstream publication, merge or deployment. + +## Workspace safety + +Use the existing checkout if appropriate. Inspect its actual remote, branch and modifications before editing; preserve other people's work. Do not reset, stash, clean, force-push or switch a shared checkout merely to follow these instructions. A separate clone is an optional convenience, not a requirement for all contributors. + +All new CLI names, structs, task types and event fields in this package are **proposed implementation contracts**, not options available in the baseline binary. Existing interfaces are identified explicitly. This package was authored with ChatGPT assistance at the fork owner's request; it is not an upstream-maintainer endorsement or a native test report. diff --git a/docs/kv-cache-retention/RESEARCH.md b/docs/kv-cache-retention/RESEARCH.md new file mode 100644 index 000000000000..221e7af6b42f --- /dev/null +++ b/docs/kv-cache-retention/RESEARCH.md @@ -0,0 +1,80 @@ +# Source and upstream research record + +Inspected on **2026-09-29**. PR status is a dated observation, not a promise of merge or future availability. The implementation package is an original integration design informed by these sources; none of the following PRs was cherry-picked, built or benchmarked during this documentation task. + +## 1. Fork baseline and evidence boundary + +Repository: [novkien/llama.cpp-fork](https://github.com/novkien/llama.cpp-fork). Default branch observed: `master`. Source baseline: [`526c43b8f7dfea9032e9f35e7a1be9183ca7cc20`](https://github.com/novkien/llama.cpp-fork/commit/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20). + +[Issue 1](https://github.com/novkien/llama.cpp-fork/issues/1) tracks the native branch-loss and lookup problems from [llama-proxy#416](https://github.com/novkien/llama-proxy/issues/416). Historical native build/revision evidence in that incident is not proof that today's running process uses this fork baseline. No production process was inspected or changed in this task. + +The newest owner request adds an implementation-ready design for RAM-first retention and optional SSD spill below the configured slot-save path, and authorizes a full architecture comment plus a draft PR in this fork. It does not request upstream publication or a production rollout. + +## 2. Relevant opened PRs inspected + +| PR and dated state | Inspected head | What the design takes from it | What it does not establish | +|---|---|---|---| +| [#28992: skipped prompt-cache lookups](https://github.com/ggml-org/llama.cpp/pull/28992), open/unmerged | `94bc965b57b7a1f84eb01e591345226b71e05925` | Independent metadata lookup; pinned-empty denominator guard; two useful regression shapes. | Does not change the strict two-ratio ranking or preserve all outgoing branches. No SSD tier. | +| [#24143: non-consuming load](https://github.com/ggml-org/llama.cpp/pull/24143), open/unmerged; conflicts reported | `6fca89bf228d8f7c7a744629f8f84c1da8a8bcde` | Keep serialized source bytes after restore; A/X/C/A fixture exposes borrowed-entry consumption. | Uses an older prompt/entry layout. Its dense-model report does not validate current Nex/draft/SSD behavior. | +| [#27451: shared checkpoints and OOM handling](https://github.com/ggml-org/llama.cpp/pull/27451), open/unmerged | `5c7018ed7b419197ffda429999a2334cab5df939` | Immutable shared checkpoint backing buffers; catch allocation failure through complete entry construction. | Does not fix ranking, branch preservation or persistence; its validation is not a complete runtime acceptance matrix. | +| [#26004: checkpoint-preserving slot files](https://github.com/ggml-org/llama.cpp/pull/26004), open/unmerged | `06d9d0ff54b586514a59268e2c780abc08473daa` | Checkpoints must travel with saved state; bounded appendix parsing; compare restored divergence with live divergence. | Manual save/restore is not automatic RAM/SSD tiering. Reported cross-machine success is not a portability guarantee. | +| [#28092: persistent prompt cache](https://github.com/ggml-org/llama.cpp/pull/28092), open/unmerged | `f2431a23a2cc059c1d1ed599396aa3b5d6a4342a` | Complete persistent state, compatibility metadata, startup catalog, corruption handling and disk budgets. | The inspected implementation chooses disk versus RAM rather than our RAM-first spill contract. Its mmap write path and namespace deletion require different handling. | + +The #28992, #24143, #27451 and #28092 implementation diffs were inspected for the relevant behavior. #26004 was examined through its detailed current implementation/test description and relevant user/author discussion; it must still receive a line-by-line port review before code reuse. Do not merge these PRs as a bundle or assume they apply cleanly to this baseline. + +Searches included `cache disk` across open/closed PRs and `"prompt cache" in:title` among opened PRs, plus direct reads of the linked issues/PRs and discussions. This is a bounded relevant search, not a claim that every cache PR has been audited. + +## 3. Comments that materially change the architecture + +### Disk design is already being discussed upstream + +In [#28092, comment 5515802845](https://github.com/ggml-org/llama.cpp/pull/28092#issuecomment-5515802845), maintainer ngxson said on September 2 that a local alternative used chained hashes over fixed token blocks, needed refactoring, and was intended eventually to replace that PR. The maintainer explicitly stopped deeper review of that proposal. This is evidence of a design/refactoring reason for that particular PR's review status, not a hardware prohibition and not proof the unpublished alternative is available today. + +Decision: retain a replaceable metadata lookup boundary. Begin with an exact, bounded metadata index; do not invent or depend on an unavailable upstream branch. Avoid scanning or reading all KV files on every request. + +### A shared disk directory must not destroy another model's cache + +[sammcj's comment 5517314832](https://github.com/ggml-org/llama.cpp/pull/28092#issuecomment-5517314832) reported that mismatched-key cleanup could delete another model's files when router children inherited one cache path. The same report identified a libc++ `file_clock` streaming build failure. [Author response 5518591168](https://github.com/ggml-org/llama.cpp/pull/28092#issuecomment-5518591168) described the numeric cast fix and acknowledged the namespace issue. + +Decision: compatibility-specific directories, no foreign-directory deletion, exact opened-file/path checks, and portable validated metadata encodings. Do not carry over a broad startup `remove_files` rule. + +### GPU-to-file-mapping capture has a reported kernel-hang risk + +[Author comment 5518710877](https://github.com/ggml-org/llama.cpp/pull/28092#issuecomment-5518710877) points to a ROCm report on #26408: GPU state capture directly into shared file-mapped pages could interact badly with writeback. The author said #28092 used the same pattern and had not reproduced it on their CUDA/WSL setup. This is an attributed report, not a universal proven failure across backends. + +Decision: ordinary host capture buffers followed by worker-owned file writes. No direct GPU DMA into shared file mappings in this implementation. Snapshot-sized host staging remains a real resource requirement; SSD capacity does not eliminate it. + +### RAM can hide a broken file restore + +[wagi-sho's comment 5055464245](https://github.com/ggml-org/llama.cpp/pull/26004#issuecomment-5055464245) explains that an apparently warm file restore was actually masked by existing RAM/checkpoint state. The reporter compared a restored divergent prompt with a live reference and checked output content. [terisuke's comment 5077635955](https://github.com/ggml-org/llama.cpp/pull/26004#issuecomment-5077635955) supplied a small cross-backend experiment and noted the mismatch between reported and actual file bytes when checkpoint data was appended. + +Decision: new-process SSD tests with a confirmed committed spill, no inherited RAM source, full checkpoint/speculative serialization and exact byte reporting. Do not use another user's timings as this fork's benchmark. + +### Non-consuming restore and shared-buffer ownership are separate fixes + +#24143's [discussion](https://github.com/ggml-org/llama.cpp/pull/24143#issuecomment-4625637662) contains a bot policy warning and an author AI-disclosure response; no substantive technical rejection was present in the inspected discussion. Its actual diff stops consuming source entries but reflects older structs. Port the behavior and test shape, not the old struct assignments. + +#27451's [September 26 comment](https://github.com/ggml-org/llama.cpp/pull/27451#issuecomment-5843549146) discusses recurrent/draft exhaustion over RPC. That is a separate capacity/failure problem, not proof that immutable buffer sharing solves native recovery. Decision: keep target/draft failure injection, but do not import unrelated purge/retry policy or #419 runtime recovery into this feature. + +## 4. Historical foundations and nonselected adjacent work + +[#16391](https://github.com/ggml-org/llama.cpp/pull/16391), merged October 9, 2025, introduced host-memory prompt caching. [The author's intended workload](https://github.com/ggml-org/llama.cpp/pull/16391#issuecomment-3377934532) included large agent context interleaved with auxiliary calls. This establishes that saving old context for later reuse is an upstream goal, not a novel hardware capability. + +[#16560](https://github.com/ggml-org/llama.cpp/pull/16560), merged October 14, 2025, adjusts token limits to a finite memory budget. The baseline still has separate byte/token accounting. Do not infer that `-cram -1` means unlimited token retention. + +[#28046](https://github.com/ggml-org/llama.cpp/pull/28046) proposes a per-entry token-limit policy; [#23666](https://github.com/ggml-org/llama.cpp/pull/23666) proposes second-chance eviction. These were located as adjacent options, not selected or fully audited here. The design keeps RAM-only cap semantics and uses a simple eligible-victim policy. Sleep-only retention PRs and recurrent-kernel changes are also outside this branch-replacement design. + +## 5. Exact source anchors and terminology corrections + +All source links below are pinned to the fork baseline: + +- [Slot selection and cache mutation](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/tools/server/server-context.cpp): `get_available_slot`, `process_single_task`, `prompt_save/load/clear`, prompt-start rollback and idle evacuation. +- [RAM alloc/load/update](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/tools/server/server-task.cpp): consume-on-load, ratio comparison, prefix dedup and quota behavior. +- [Token/media handling](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/tools/server/server-common.cpp): exact `get_common_prefix`, packed `serialize`, validation and `clone`. +- [Checkpoint state](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/common/common.h): target, draft and `data_spec` bytes; 64 checkpoints are not 64 complete conversations. +- [Queue and threading](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/tools/server/server-queue.h): `post`, decode yielding, cancellation and mutex-held sleep callbacks. +- [Server development contract](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/tools/server/README-dev.md): filesystem features opt-in; heavy owner-thread work affects other sequences. +- [Test runner](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/tools/server/tests/tests.sh) and [test instructions](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/tools/server/tests/README.md). +- [Existing hash target](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/vendor/hash/CMakeLists.txt): `vendor::hash` already includes SHA-256. + +Two discrepancies matter when implementing. First, #28992's prose describes a one-token difference, but the inspected fork's `get_common_prefix()` returns the raw exact prefix; the logits adjustment belongs to execution. Second, #28092's title says `--cache-disk`, while its inspected diff defines `--cache-dir` and `--cache-dir-max`. Our proposed `--cache-spill-mib` is deliberately named and documented as a new fork option using existing `--slot-save-path`, not presented as an existing upstream option. diff --git a/docs/kv-cache-retention/TESTING.md b/docs/kv-cache-retention/TESTING.md new file mode 100644 index 000000000000..cb0224190363 --- /dev/null +++ b/docs/kv-cache-retention/TESTING.md @@ -0,0 +1,109 @@ +# Verification plan and acceptance evidence + +Status: all implementation results are **NOT RUN** in the documentation draft. The table specifies required tests, not claimed passes. Use the maintained server pytest harness and disposable files/ports. See [IMPLEMENTATION.md](IMPLEMENTATION.md) for setup. + +## 1. Evidence rules + +Record exact source/head, binary identity, model digest, backend, context capacity, parallel count, KV types, draft settings, cache budgets, checkpoint settings and fixture identity. Compare before/after using the same fixture and hardware. Preserve stdout/stderr and request counters without publishing private production prompts. + +For every completion check HTTP/result status, `prompt_n`, `cache_n`, their sum against actual tokenized input, returned slot/task identity and output correctness. Raw LCP is not actual cache reuse. Planned reuse must match execution or be explicitly downgraded on failure. Do not infer "no prefill" from low wall time or count response text as proof of native cache residency. + +For a deterministic model/backend fixture, compare output tokens or logits with an independently cold-evaluated reference where feasible. A valid checkpoint-granularity rollback may process more than one token. The reference for a disk-restored divergent hybrid prompt is the same divergent prompt on a live un-evacuated state, not a universal one-token expectation. + +Tests must prove the intended path was exercised through compact cache event tags/counters. A cache miss caused by disabled cache cannot pass a test meant to cover failed restore. Failure injection must target a specific operation, not randomly corrupt unrelated shared memory. + +## 2. Deterministic RAM and slot cases + +Use stable token arrays where possible. Give A and C a controlled shared prefix P and distinct suffixes. X has negligible overlap. Choose sizes that actually fit the selected fixture and budgets. Test 0/1/2-token boundaries separately from long-context fixtures. + +| ID | Fixture / setup | Required observation | +|---|---|---| +| T01 | A -> C -> A-next; A/C share over 50% of A but diverge before A's suffix. | Baseline demonstrates skipped save; candidate captures A before destructive reuse and reuses A's valid prefix on return. | +| T02 | A -> X -> C -> A-next in one slot; A is RAM-only when C borrows P. | A snapshot still exists after C restore; A-next reuses beyond P when its state supports it. | +| T03 | A fixed to slot 0, another slot starts and evacuates 0, A-next fixed to 0. | Prove slot 0 is empty and A is in RAM; restore into 0 rather than cold prefill. | +| T04 | Repeat T03 with slot-prompt-similarity 0 and nonzero. | Both use the same restore policy; no division-by-zero or hidden LRU dependency. | +| T05 | Submit a second request to a confirmed busy explicit slot. | No save/clear/restore of its running state; request waits or cancels according to existing semantics. | +| T06 | Copy checkpoint handles, replace/clear the live checkpoint and evict an entry. | Old snapshot bytes remain unchanged until the last lease is released; no use-after-free. | +| T07 | Inject allocation failure in target allocation, draft allocation, token/media clone, checkpoint metadata and list insertion separately. | No partial published snapshot, crash or successful-save counter; source live state remains valid. | +| T08 | Inject target/draft capture byte-count mismatch. | Typed serialization failure, no indexed unwritten bytes; later unrelated requests remain correct. | +| T09 | RAM A has longer useful prefix but lower normalized keep fraction than resident B; also test equal keep fractions. | Candidate chosen by actual reusable tokens, not the old strict ratio conjunction. | +| T10 | Hybrid/SWA divergence before the first usable checkpoint. | Zero/limited true reuse reported as appropriate; no false hit; borrowing cannot delete the original A snapshot. | +| T11 | Identical prompt, pure extension, shorter prompt, single-token replay and changed suffix. | Raw-prefix and logits-replay semantics differ correctly; pure extension does not snapshot every turn. | +| T12 | Target restore succeeds, draft restore fails. | Source retained; destination metadata not published as valid; safe cold fallback or explicit native error, never mixed-state generation. | +| T13 | Longer token prefix has worse/missing checkpoint coverage than a shorter saved entry. | Deduplication does not delete the only state capable of the shorter rewind. | +| T14 | Byte limit, aggregate token limit, disabled cache, oversized snapshot and all eligible victims pinned. | Bounded memory/termination, correct explicit eviction/failure reasons, no silent unlimited `-cram -1` assumption. | + +Do not use `sleep(2)` as the only proof that generation is active. Wait for a server-owned processing/task event or bounded status condition, then release/cancel only that fixture's work. + +## 3. SSD cases with RAM unable to mask failures + +Each case uses a fresh temporary `--slot-save-path`. The future SSD switch is `--cache-spill-mib`; it is not present in the baseline binary. Set finite RAM small enough to force spill but large enough for one complete snapshot/read staging. Confirm an actual durable commit event and file before stopping the source server. + +| ID | Fixture / setup | Required observation | +|---|---|---| +| T15 | Capture A, force A from RAM to SSD with X, stop process, start a new process using the same compatible model/config and directory, send A-next. | New process has no inherited RAM source; disk-read/restore events occur; counters and output match the live-reference behavior. | +| T16 | Repeat T15 with hybrid/SWA, divergent suffix and a compatible draft/MTP configuration. | Target, draft, checkpoint and speculative bytes survive; compare against the same live divergent prompt. | +| T17 | Truncate each section, corrupt checksum/magic/version, forge length/count/position, introduce integer overflow and invalid packed media. | Parser rejects before native state mutation; bad entry cannot erase healthy entries or trigger huge allocations. | +| T18 | Interrupt after temporary creation, mid-write, before rename and after rename; inject short writes, ENOSPC and permission failure. | Only fully committed snapshots enter the catalog. No success acknowledged for incomplete durability; RAM source retained where possible. | +| T19 | Two model/config namespaces share one parent directory; same namespace has two concurrent server processes; include manual slot files and symlinks. | Different models coexist; second same-namespace writer cannot acquire ownership; manual/foreign/symlink-target files are never deleted. | +| T20 | Cancel during SSD read; reuse the slot for another task; complete the old read afterwards. | Epoch/task/slot-generation mismatch discards old result without installing state or generating an extra response. | +| T21 | Inject slow disk while another slot generates. | No inference-thread file write/read wait or global task mutex stall; other slot progresses. GPU-to-host capture latency is measured separately. | +| T22 | Fill the two-job queue, RAM staging budget, disk quota, metadata budget and entry cap. | Counts/bytes remain bounded, no retry spin, catalog-cap misses distinguished from corrupted state, eligible eviction deterministic. | +| T23 | Model reload or shutdown with queued/running reads and writes. | No callback into destroyed contexts/queues, no join under task mutex, no orphan install, committed entries still readable later. | +| T24 | Change weights, draft/mmproj, adapters/scales, KV layout, RoPE settings or state ABI while reusing directory. | Incompatible entries ignored in separate namespace, not restored or deleted as foreign garbage. | + +For manual `/slots` regression tests, set RAM cache to zero and restart between phases where the existing API permits it. Otherwise a RAM hit can hide a missing checkpoint appendix. Manual-file compatibility remains separate from automatic spill acceptance. + +## 4. Regression surfaces + +Run existing `unit/test_completion.py` and `unit/test_slot_save.py`, plus maintained cancellation, multimodal and speculative tests applicable to changed code. Do not replace existing media-aware token serialization with text-only buffers. Test different media with identical placeholder token positions; it must not falsely share a cache entry. Test parent/child completion ownership and cache-disabled behavior. + +After adding new tests to existing files, a targeted invocation is: + +```bash +# Run from the repository root after the disposable build/venv setup. +export PYTEST_WORKERS=1 +unset DEBUG_EXTERNAL +bash tools/server/tests/tests.sh unit/test_completion.py \ + -k 'prompt_cache or cache_retention' -v -x +bash tools/server/tests/tests.sh unit/test_slot_save.py -v -x +``` + +The new test names should contain `cache_retention`; the selector is a naming instruction, not proof those tests exist yet. Keep expensive hybrid/draft tests marked and explicitly invoked; a skipped hybrid test yields PARTIAL coverage, not PASS for Nex. + +Use ASan/UBSan builds for ownership/parser failures where supported. Run affected C++ translation units under GCC/libstdc++ and Clang/libc++ when available. The disk research identified a `file_clock` representation difference; use fixed-width validated representations rather than assuming its native count streams portably. + +## 5. Matched performance experiment + +Measure four shapes using baseline and candidate: cold A; warmed append-only A; unrelated A/B/A; shared-prefix A/C/A. Use both explicit ID and automatic selection. Add SSD-spilled A after restart as a separate scenario. Warm-up runs and model/token fixtures must be identical; report contention and sample count rather than quoting unmatched medians as a causal speedup. + +Required measurements: + +- Native processed/cached token counts, valid checkpoint position, output correctness and request success. +- Capture count/bytes/time, restore count/bytes/time, SSD read/write time and total first-token latency. +- Aggregate throughput and latency of other concurrently active slots. +- Accounted RAM entries, staging/pinned bytes, catalog size, disk bytes, queue depth, and peak process RSS. +- Eviction, oversize, incompatible, missing-checkpoint, allocation and I/O failure counts. + +A fixture with no destructive change or evacuation must add **zero full-state capture/restore operations after warm-up**. For other cases, show how reduced prefill compares with added copies. Do not impose an invented universal percentage threshold; record the measured tradeoff and whether the intended workload improves without correctness or unrelated-slot regression. Re-run affected measurements after relevant source changes. + +## 6. Result record + +Use one row per test/scenario: + +```text +Case: +Verdict: PASS | PARTIAL | FAILED +Source/head and binary: +Model/draft digests and complete flags: +Fixture/test node: +Baseline result and evidence: +Candidate result and evidence: +Actual cached/processed tokens and selected slot: +Capture/restore/disk timings and bytes: +Output correctness check: +Peak RSS and accounted cache/staging: +What remains untested or changed: +``` + +PASS means the specific case passed against the delivered candidate. PARTIAL includes unavailable model/backend, unrun disk restart, skipped failure injection or unresolved evidence. FAILED includes incorrect output, lost required branch under ample budget, busy-slot mutation, a false cache hit, unbounded resource growth or a reproducible regression. A documentation-only draft has no native PASS verdict. From 8cfa7bb52fe69e904abc53c86b775c01a6ec3b0b Mon Sep 17 00:00:00 2001 From: novkien Date: Tue, 29 Sep 2026 10:16:52 +0700 Subject: [PATCH 2/9] server: preserve slot branch before LCP reuse and rank RAM cache by lcp Save live branch when incoming tokens would discard it, instead of f_keep < 0.5 gate. Always try RAM lookup for completion tasks including pinned slots. Rank cache entries by longest reusable prefix, keep 0.25 keep guard. --- tools/server/server-context.cpp | 22 +++++++--------------- tools/server/server-task.cpp | 18 ++++++++---------- 2 files changed, 15 insertions(+), 25 deletions(-) diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp index 2036319c1381..7fec3edf8417 100644 --- a/tools/server/server-context.cpp +++ b/tools/server/server-context.cpp @@ -1528,8 +1528,6 @@ struct server_context_impl { server_slot * get_available_slot(const server_task & task) { server_slot * ret = nullptr; - bool update_cache = false; - // if a specific slot is requested, use it (still goes through cache update logic below) if (task.id_slot != -1) { ret = get_slot_by_id(task.id_slot); @@ -1582,11 +1580,6 @@ struct server_context_impl { SLT_INF(*ret, "selected slot by LCP similarity, f_sim_best = %.3f (> %.3f thold), f_keep = %.3f\n", f_sim_best, slot_prompt_similarity, f_keep); } - - // if we are about to lose a large portion of the existing context - save it in the prompt cache - if (f_keep < 0.5f) { - update_cache = true; - } } } @@ -1609,23 +1602,22 @@ struct server_context_impl { if (ret != nullptr) { SLT_INF(*ret, "selected slot by LRU, t_last = %" PRId64 "\n", t_last); - - update_cache = true; } } if (ret) { - update_cache = update_cache && prompt_cache; - - // cache prompts only for completion tasks - update_cache = update_cache && task.type == SERVER_TASK_TYPE_COMPLETION; + const bool can_cache = prompt_cache && task.type == SERVER_TASK_TYPE_COMPLETION && !ret->is_processing(); - if (update_cache) { + if (can_cache) { SRV_TRC("%s", "updating prompt cache\n"); const int64_t t_start = ggml_time_us(); - ret->prompt_save(*prompt_cache); + // save when reuse would discard live tokens + const size_t lcp_slot = ret->prompt.tokens.get_common_prefix(task.tokens); + if (!ret->prompt.tokens.empty() && lcp_slot < ret->prompt.tokens.size()) { + ret->prompt_save(*prompt_cache); + } if (!ret->prompt_load(*prompt_cache, task.tokens)) { ret->prompt_clear(); diff --git a/tools/server/server-task.cpp b/tools/server/server-task.cpp index 0d3beb313cea..f61755da4d4c 100644 --- a/tools/server/server-task.cpp +++ b/tools/server/server-task.cpp @@ -1793,19 +1793,18 @@ server_prompt_cache_state * server_prompt_cache::alloc(const server_prompt & pro bool server_prompt_cache::load(server_prompt & prompt, const server_tokens & tokens_new, llama_context * ctx_tgt, llama_context * ctx_dft, int32_t id_slot) { const int lcp_best = prompt.tokens.get_common_prefix(tokens_new); - float f_keep_best = prompt.tokens.size() > 0 ? float(lcp_best) / prompt.tokens.size() : -1.0f; // empty slot: any cache entry wins - float f_sim_best = float(lcp_best) / tokens_new.size(); + int lcp_win = lcp_best; - SRV_TRC(" - looking for better prompt, base f_keep = %.3f, f_sim = %.3f\n", f_keep_best, f_sim_best); + SRV_TRC(" - looking for better prompt, base lcp = %d\n", lcp_best); auto it_best = states.end(); - // find the most similar cached prompt, that would also preserve the most context + // find the cached prompt with the longest reusable prefix for (auto it = states.begin(); it != states.end(); ++it) { const int lcp_cur = it->prompt.tokens.get_common_prefix(tokens_new); - const float f_keep_cur = float(lcp_cur) / it->prompt.tokens.size(); - const float f_sim_cur = float(lcp_cur) / tokens_new.size(); + const float f_keep_cur = it->prompt.tokens.empty() ? 0.0f : float(lcp_cur) / it->prompt.tokens.size(); + const float f_sim_cur = tokens_new.empty() ? 0.0f : float(lcp_cur) / tokens_new.size(); SRV_TRC(" - prompt with length %7zu, lcp = %7d, f_keep = %.3f, f_sim = %.3f\n", it->prompt.tokens.size(), lcp_cur, f_keep_cur, f_sim_cur); @@ -1814,16 +1813,15 @@ bool server_prompt_cache::load(server_prompt & prompt, const server_tokens & tok continue; } - if (f_keep_best < f_keep_cur && f_sim_best < f_sim_cur) { - f_keep_best = f_keep_cur; - f_sim_best = f_sim_cur; + if (lcp_cur > lcp_win) { + lcp_win = lcp_cur; it_best = it; } } if (it_best != states.end()) { - SRV_TRC(" - found better prompt with f_keep = %.3f, f_sim = %.3f\n", f_keep_best, f_sim_best); + SRV_TRC(" - found better prompt with lcp = %d (base %d)\n", lcp_win, lcp_best); { auto & data = it_best->data.main; From c2b7e953a366642aac537266734f0d3b49861baa Mon Sep 17 00:00:00 2001 From: LE TRUNG KIEN Date: Tue, 29 Sep 2026 10:32:51 +0700 Subject: [PATCH 3/9] ci: export the exact cache review source for isolated validation Addresses novkien/llama.cpp-fork#1 Assisted-by: ChatGPT --- .github/workflows/kv-cache-review-export.yml | 31 ++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 .github/workflows/kv-cache-review-export.yml diff --git a/.github/workflows/kv-cache-review-export.yml b/.github/workflows/kv-cache-review-export.yml new file mode 100644 index 000000000000..57228faa9e29 --- /dev/null +++ b/.github/workflows/kv-cache-review-export.yml @@ -0,0 +1,31 @@ +name: Cache review source export + +on: + push: + branches: [design/issue-1-kv-cache-retention] + paths: [.github/workflows/kv-cache-review-export.yml] + +permissions: + contents: read + +jobs: + source: + if: github.repository == 'novkien/llama.cpp-fork' + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v6 + with: + persist-credentials: false + - name: Export tracked review source + run: | + mkdir -p "${RUNNER_TEMP}/cache-review" + git rev-parse HEAD > "${RUNNER_TEMP}/cache-review/HEAD.txt" + git archive --format=tar.gz HEAD > "${RUNNER_TEMP}/cache-review/source.tar.gz" + sha256sum "${RUNNER_TEMP}/cache-review/source.tar.gz" > "${RUNNER_TEMP}/cache-review/SHA256SUMS" + - uses: actions/upload-artifact@v4 + with: + name: cache-review-source + path: ${{ runner.temp }}/cache-review/ + retention-days: 1 + if-no-files-found: error From 6b598d272f45d1843192243f69d47ffeb62ae24f Mon Sep 17 00:00:00 2001 From: LE TRUNG KIEN Date: Tue, 29 Sep 2026 10:40:27 +0700 Subject: [PATCH 4/9] ci: collect the public tiny model for offline cache regressions Addresses novkien/llama.cpp-fork#1 Assisted-by: ChatGPT --- .github/workflows/kv-cache-review-export.yml | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/.github/workflows/kv-cache-review-export.yml b/.github/workflows/kv-cache-review-export.yml index 57228faa9e29..de27c934a2c9 100644 --- a/.github/workflows/kv-cache-review-export.yml +++ b/.github/workflows/kv-cache-review-export.yml @@ -17,15 +17,18 @@ jobs: - uses: actions/checkout@v6 with: persist-credentials: false - - name: Export tracked review source + - name: Collect public test assets run: | - mkdir -p "${RUNNER_TEMP}/cache-review" + mkdir -p "${RUNNER_TEMP}/cache-review/wheels" git rev-parse HEAD > "${RUNNER_TEMP}/cache-review/HEAD.txt" - git archive --format=tar.gz HEAD > "${RUNNER_TEMP}/cache-review/source.tar.gz" - sha256sum "${RUNNER_TEMP}/cache-review/source.tar.gz" > "${RUNNER_TEMP}/cache-review/SHA256SUMS" + curl --fail --location --retry 2 --max-time 90 \ + https://huggingface.co/ggml-org/models/resolve/main/tinyllamas/stories260K.gguf \ + -o "${RUNNER_TEMP}/cache-review/stories260K.gguf" + python -m pip download --dest "${RUNNER_TEMP}/cache-review/wheels" 'wget==3.2' 'openai~=2.14.0' 'pytest-xdist~=3.6' + sha256sum "${RUNNER_TEMP}/cache-review/stories260K.gguf" > "${RUNNER_TEMP}/cache-review/SHA256SUMS" - uses: actions/upload-artifact@v4 with: - name: cache-review-source + name: cache-review-test-assets path: ${{ runner.temp }}/cache-review/ retention-days: 1 if-no-files-found: error From fa048840b2426a7e4fd21cb26646f1eb10aa6849 Mon Sep 17 00:00:00 2001 From: LE TRUNG KIEN Date: Tue, 29 Sep 2026 10:52:32 +0700 Subject: [PATCH 5/9] ci: complete offline Python 3.13 review dependencies Addresses novkien/llama.cpp-fork#1 Assisted-by: ChatGPT --- .github/workflows/kv-cache-review-export.yml | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/.github/workflows/kv-cache-review-export.yml b/.github/workflows/kv-cache-review-export.yml index de27c934a2c9..6db339de503a 100644 --- a/.github/workflows/kv-cache-review-export.yml +++ b/.github/workflows/kv-cache-review-export.yml @@ -14,21 +14,14 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 5 steps: - - uses: actions/checkout@v6 - with: - persist-credentials: false - - name: Collect public test assets + - name: Collect Python 3.13 wheels run: | mkdir -p "${RUNNER_TEMP}/cache-review/wheels" - git rev-parse HEAD > "${RUNNER_TEMP}/cache-review/HEAD.txt" - curl --fail --location --retry 2 --max-time 90 \ - https://huggingface.co/ggml-org/models/resolve/main/tinyllamas/stories260K.gguf \ - -o "${RUNNER_TEMP}/cache-review/stories260K.gguf" - python -m pip download --dest "${RUNNER_TEMP}/cache-review/wheels" 'wget==3.2' 'openai~=2.14.0' 'pytest-xdist~=3.6' - sha256sum "${RUNNER_TEMP}/cache-review/stories260K.gguf" > "${RUNNER_TEMP}/cache-review/SHA256SUMS" + python -m pip download --dest "${RUNNER_TEMP}/cache-review/wheels" \ + --only-binary=:all: --python-version 313 --platform manylinux2014_x86_64 --implementation cp --abi cp313 'jiter>=0.10,<1' - uses: actions/upload-artifact@v4 with: - name: cache-review-test-assets + name: cache-review-python313 path: ${{ runner.temp }}/cache-review/ retention-days: 1 if-no-files-found: error From 09a8b3b1374e00aaad5b87ed3edd5d3021f60892 Mon Sep 17 00:00:00 2001 From: LE TRUNG KIEN Date: Tue, 29 Sep 2026 11:16:13 +0700 Subject: [PATCH 6/9] ci: validate and publish the hash-checked cache review repair Addresses novkien/llama.cpp-fork#1 Assisted-by: ChatGPT --- .github/workflows/kv-cache-review-export.yml | 109 +++++++++++++++++-- 1 file changed, 99 insertions(+), 10 deletions(-) diff --git a/.github/workflows/kv-cache-review-export.yml b/.github/workflows/kv-cache-review-export.yml index 6db339de503a..08ca1366e674 100644 --- a/.github/workflows/kv-cache-review-export.yml +++ b/.github/workflows/kv-cache-review-export.yml @@ -1,4 +1,4 @@ -name: Cache review source export +name: Cache review validation and publication on: push: @@ -6,22 +6,111 @@ on: paths: [.github/workflows/kv-cache-review-export.yml] permissions: - contents: read + contents: write jobs: - source: + validate: if: github.repository == 'novkien/llama.cpp-fork' runs-on: ubuntu-latest - timeout-minutes: 5 + timeout-minutes: 20 + env: + GH_TOKEN: ${{ github.token }} + REVIEW_BRANCH: design/issue-1-kv-cache-retention steps: - - name: Collect Python 3.13 wheels + - uses: actions/checkout@v6 + with: + ref: ${{ github.sha }} + - name: Install the exact reviewed payload + run: | + set -euo pipefail + mkdir -p "${RUNNER_TEMP}/cache-review" + test "$(git ls-remote origin "refs/heads/${REVIEW_BRANCH}" | cut -f1)" = "$GITHUB_SHA" + python3 - <<'PY' + import base64, hashlib, json, lzma, os, pathlib, subprocess + chunks = [ + 'e61ee5e62d9cb087e83c9785b4fdb021f3481cb1', + '0768ea48e0ea3ca49df96f221aaf77ab7f3a40b5', + '6ae71d11bbb7c533b1bfdfc4e4fcc52e81f72f38', + '0a42bc60c4859517681ab43d19a7e4fe633e6b3b', + ] + def blob_sha(data): + return hashlib.sha1(b'blob ' + str(len(data)).encode() + b'\0' + data).hexdigest() + encoded = [] + for sha in chunks: + result = json.loads(subprocess.check_output(['gh', 'api', f'repos/{os.environ["GITHUB_REPOSITORY"]}/git/blobs/{sha}'])) + data = base64.b64decode(result['content']) + assert blob_sha(data) == sha, 'transport checksum mismatch' + encoded.append(data) + payload = json.loads(lzma.decompress(b''.join(encoded))) + manifest = payload['manifest'] + expected_paths = { + 'common/common.cpp', 'common/common.h', + 'docs/kv-cache-retention/DESIGN.md', 'docs/kv-cache-retention/IMPLEMENTATION.md', + 'docs/kv-cache-retention/README.md', 'docs/kv-cache-retention/TESTING.md', + 'tests/CMakeLists.txt', 'tests/test-server-prompt-cache.cpp', + 'tools/server/server-context.cpp', 'tools/server/server-task.cpp', + 'tools/server/server-task.h', 'tools/server/tests/unit/test_completion.py', + 'tools/server/tests/utils.py', + } + assert {r['path'] for r in manifest} == expected_paths + for record in manifest: + path = pathlib.Path(record['path']) + actual = blob_sha(path.read_bytes()) if path.exists() else None + assert actual == record['before'], f'concurrent source change: {path}' + patch = payload['patch'].encode() + subprocess.run(['git', 'apply', '--check', '-'], input=patch, check=True) + subprocess.run(['git', 'apply', '-'], input=patch, check=True) + for record in manifest: + assert blob_sha(pathlib.Path(record['path']).read_bytes()) == record['after'], record['path'] + subprocess.run(['git', 'diff', '--check'], check=True) + subprocess.run(['git', 'add', '--', *sorted(expected_paths)], check=True) + staged = set(subprocess.check_output(['git', 'diff', '--cached', '--name-only'], text=True).splitlines()) + assert staged == expected_paths + out = pathlib.Path(os.environ['RUNNER_TEMP']) / 'cache-review' + (out / 'manifest.json').write_text(json.dumps(manifest, indent=2)) + PY + git config user.name 'ChatGPT review assistant' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git commit -m 'server: harden RAM cache retention across slot reuse' -m 'Addresses novkien/llama.cpp-fork#1' -m 'Assisted-by: ChatGPT' + git rev-parse HEAD | tee "${RUNNER_TEMP}/cache-review/candidate-sha.txt" + - name: Build the reviewed CPU candidate + run: | + set -euo pipefail + cmake -S . -B build-review -G Ninja \ + -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF \ + -DLLAMA_USE_PREBUILT_UI=OFF -DLLAMA_BUILD_EXAMPLES=OFF \ + -DLLAMA_BUILD_APP=OFF -DLLAMA_FATAL_WARNINGS=ON \ + 2>&1 | tee "${RUNNER_TEMP}/cache-review/configure.log" + cmake --build build-review --target llama-server test-server-prompt-cache -j2 \ + 2>&1 | tee "${RUNNER_TEMP}/cache-review/build.log" + - name: Run native and HTTP cache regressions + run: | + set -euo pipefail + python3 -m pip install pytest requests wget 'openai~=2.14.0' + curl --fail --location --retry 2 --max-time 90 \ + https://huggingface.co/ggml-org/models/resolve/main/tinyllamas/stories260K.gguf \ + -o "${RUNNER_TEMP}/stories260K.gguf" + echo "270cba1bd5109f42d03350f60406024560464db173c0e387d91f0426d3bd256d ${RUNNER_TEMP}/stories260K.gguf" | sha256sum --check + ctest --test-dir build-review -R '^test-server-prompt-cache$' --output-on-failure \ + 2>&1 | tee "${RUNNER_TEMP}/cache-review/ctest.log" + build-review/bin/test-server-prompt-cache "${RUNNER_TEMP}/stories260K.gguf" \ + 2>&1 | tee "${RUNNER_TEMP}/cache-review/native.log" + cd tools/server/tests + LLAMA_CACHE_TEST_MODEL="${RUNNER_TEMP}/stories260K.gguf" \ + LLAMA_SERVER_BIN_PATH="${GITHUB_WORKSPACE}/build-review/bin/llama-server" \ + PYTHONPATH=. python3 -m pytest --confcutdir=unit unit/test_completion.py \ + -k cache_retention -q --tb=short \ + 2>&1 | tee "${RUNNER_TEMP}/cache-review/http.log" + - name: Publish without overwriting concurrent work run: | - mkdir -p "${RUNNER_TEMP}/cache-review/wheels" - python -m pip download --dest "${RUNNER_TEMP}/cache-review/wheels" \ - --only-binary=:all: --python-version 313 --platform manylinux2014_x86_64 --implementation cp --abi cp313 'jiter>=0.10,<1' + set -euo pipefail + test "$(git ls-remote origin "refs/heads/${REVIEW_BRANCH}" | cut -f1)" = "$GITHUB_SHA" + git push origin "HEAD:refs/heads/${REVIEW_BRANCH}" + git rev-parse HEAD | tee "${RUNNER_TEMP}/cache-review/published-sha.txt" - uses: actions/upload-artifact@v4 + if: always() with: - name: cache-review-python313 + name: cache-review-validation path: ${{ runner.temp }}/cache-review/ - retention-days: 1 + retention-days: 3 if-no-files-found: error From e8ef91dc691f1197f9a8f69b586563de377d6278 Mon Sep 17 00:00:00 2001 From: ChatGPT review assistant <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 29 Sep 2026 04:16:30 +0000 Subject: [PATCH 7/9] server: harden RAM cache retention across slot reuse Addresses novkien/llama.cpp-fork#1 Assisted-by: ChatGPT --- common/common.cpp | 8 +- common/common.h | 24 +- docs/kv-cache-retention/DESIGN.md | 6 +- docs/kv-cache-retention/IMPLEMENTATION.md | 4 +- docs/kv-cache-retention/README.md | 26 +- docs/kv-cache-retention/TESTING.md | 29 ++- tests/CMakeLists.txt | 4 + tests/test-server-prompt-cache.cpp | 135 ++++++++++ tools/server/server-context.cpp | 199 ++++++--------- tools/server/server-task.cpp | 284 +++++++++++---------- tools/server/server-task.h | 42 ++- tools/server/tests/unit/test_completion.py | 138 ++++++++++ tools/server/tests/utils.py | 3 + 13 files changed, 624 insertions(+), 278 deletions(-) create mode 100644 tests/test-server-prompt-cache.cpp diff --git a/common/common.cpp b/common/common.cpp index 6f443f1bf022..23113891d814 100644 --- a/common/common.cpp +++ b/common/common.cpp @@ -2362,9 +2362,9 @@ void common_prompt_checkpoint::update_tgt( const size_t ckpt_size = llama_state_seq_get_size_ext(ctx, seq_id, flags); - data_tgt.resize(ckpt_size); + uint8_t * data = data_tgt.reset(ckpt_size); - const size_t n = llama_state_seq_get_data_ext(ctx, data_tgt.data(), ckpt_size, seq_id, flags); + const size_t n = llama_state_seq_get_data_ext(ctx, data, ckpt_size, seq_id, flags); if (n != ckpt_size) { GGML_ABORT("checkpoint size mismatch: expected %zu, got %zu\n", ckpt_size, n); } @@ -2380,9 +2380,9 @@ void common_prompt_checkpoint::update_dft( const size_t ckpt_size = llama_state_seq_get_size_ext(ctx, seq_id, flags); - data_dft.resize(ckpt_size); + uint8_t * data = data_dft.reset(ckpt_size); - const size_t n = llama_state_seq_get_data_ext(ctx, data_dft.data(), ckpt_size, seq_id, flags); + const size_t n = llama_state_seq_get_data_ext(ctx, data, ckpt_size, seq_id, flags); if (n != ckpt_size) { GGML_ABORT("checkpoint size mismatch: expected %zu, got %zu\n", ckpt_size, n); } diff --git a/common/common.h b/common/common.h index 45b15def71f5..222d9ebe8b78 100644 --- a/common/common.h +++ b/common/common.h @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -1229,6 +1230,25 @@ enum ggml_opt_optimizer_type common_opt_get_optimizer(const char *); // prompt utils // +class common_shared_state_buffer { +public: + size_t size() const { return storage ? storage->size() : 0; } + bool empty() const { return size() == 0; } + const uint8_t * data() const { return storage ? storage->data() : nullptr; } + + uint8_t * reset(size_t size) { + auto next = std::make_shared>(size); + storage = std::move(next); + return storage->data(); + } + + void clear() { storage.reset(); } + bool shares_storage(const common_shared_state_buffer & other) const { return storage == other.storage; } + +private: + std::shared_ptr> storage; +}; + struct common_prompt_checkpoint { int64_t n_tokens; @@ -1238,8 +1258,8 @@ struct common_prompt_checkpoint { llama_pos pos_min; llama_pos pos_max; - std::vector data_tgt; - std::vector data_dft; + common_shared_state_buffer data_tgt; + common_shared_state_buffer data_dft; // (optional) speculative-decoding implementation state stashed with the checkpoint // (e.g. eagle3's deferred-boundary g_embd row) diff --git a/docs/kv-cache-retention/DESIGN.md b/docs/kv-cache-retention/DESIGN.md index 40d2be705a9c..cbe48f055ce5 100644 --- a/docs/kv-cache-retention/DESIGN.md +++ b/docs/kv-cache-retention/DESIGN.md @@ -1,6 +1,6 @@ # Branch-preserving RAM cache with optional SSD spill -Status: implementation design for [issue 1](https://github.com/novkien/llama.cpp-fork/issues/1), not implemented. Source baseline: `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20`. Evidence and upstream comparisons are in [RESEARCH.md](RESEARCH.md); execution instructions are in [IMPLEMENTATION.md](IMPLEMENTATION.md). +Status: target architecture for [issue 1](https://github.com/novkien/llama.cpp-fork/issues/1). The RAM subset is implemented; SSD and broader acceptance remain pending. [README.md](README.md) records the current delivery boundary. Source baseline: `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20`. Evidence and upstream comparisons are in [RESEARCH.md](RESEARCH.md); execution instructions are in [IMPLEMENTATION.md](IMPLEMENTATION.md). ## 1. Requirements and scope @@ -18,11 +18,11 @@ R5. Bound RAM, pending I/O, metadata and disk usage; preserve normal concurrency R6. Keep normal append-only continuations cheap. Do not capture full KV on every request, force every request through a file or turn this into a proxy-side `/slots` RPC loop. -The current draft contains documentation only. Future implementation includes the RAM path and SSD path described here, but not a CUDA/kernel rewrite, distributed cache, cross-machine portability guarantee, compression, native model recovery, or proxy scheduling changes. `llama-proxy#419` recovery/order work and `#386` admission optimizations are not silently incorporated. Future proxy pinning integration remains separate. +The branch now contains a RAM-path implementation. The complete target includes the RAM and SSD paths described here, but not a CUDA/kernel rewrite, distributed cache, cross-machine portability guarantee, compression, native model recovery, or proxy scheduling changes. `llama-proxy#419` recovery/order work and `#386` admission optimizations are not silently incorporated. Future proxy pinning integration remains separate. ## 2. Existing mechanism and required changes -The inspected `get_available_slot()` gates both save and load with `update_cache`; selected-slot saving normally depends on `f_keep < 0.5` or the LRU path. Pinned empty slots can miss this gate. The caller checks busy state after selection. `server_prompt_cache::load()` restores and then consumes the saved entry. Allocation deduplicates by token-prefix containment, which does not prove checkpoint coverage. These are source findings, not newly observed runtime incidents. +The original baseline `get_available_slot()` gates both save and load with `update_cache`; selected-slot saving normally depends on `f_keep < 0.5` or the LRU path. Pinned empty slots can miss this gate. The caller checks busy state after selection. `server_prompt_cache::load()` restores and then consumes the saved entry. Allocation deduplicates by token-prefix containment, which does not prove checkpoint coverage. These are source findings, not newly observed runtime incidents. Reuse the existing `server_prompt_cache`, `server_tokens`, `common_prompt_checkpoint` and `llama_state_seq_*_ext` APIs. Do not create a second cache manager. Add a small storage/codec helper for disk operations; it must not own or access live llama contexts. diff --git a/docs/kv-cache-retention/IMPLEMENTATION.md b/docs/kv-cache-retention/IMPLEMENTATION.md index 6014e79ffb30..605da79b7ab2 100644 --- a/docs/kv-cache-retention/IMPLEMENTATION.md +++ b/docs/kv-cache-retention/IMPLEMENTATION.md @@ -1,6 +1,6 @@ # Junior implementation guide -This is an ordered implementation guide, not an implemented patch. Complete the RAM and SSD slices on the draft branch. Keep [DESIGN.md](DESIGN.md) authoritative for behavior and [TESTING.md](TESTING.md) authoritative for evidence. Names introduced below are proposed names. +This is the ordered guide for the complete architecture. The RAM repair is now in source; read [README.md](README.md) for implemented coverage and [TESTING.md](TESTING.md) for actual results before continuing the remaining slices. SSD is not implemented. Keep [DESIGN.md](DESIGN.md) authoritative for behavior and [TESTING.md](TESTING.md) authoritative for evidence. Names introduced below are proposed names. ## 0. Establish a safe local starting point @@ -19,7 +19,7 @@ The intended repository is `novkien/llama.cpp-fork`, not `ggml-org/llama.cpp` or A fresh clone is an optional way to get this draft without disturbing an existing checkout. Use the head branch shown by the actual PR. Do not push an upstream PR as a side effect of working on this fork. -Read root/scoped AGENTS, CONTRIBUTING, `tools/server/README-dev.md` and `tools/server/tests/README.md`. The current owner request authorizes this fork's design workspace and draft publication; implementation and production activation must be reported separately. Keep complete source files, not placeholder stubs or intentionally failing TODO binaries, in implementation commits. +Read root/scoped AGENTS, CONTRIBUTING, `tools/server/README-dev.md` and `tools/server/tests/README.md`. The owner subsequently authorized direct review repairs on the fork PR and retained merge/build/live-trial responsibility. Source repair and production activation are separate actions. Keep complete source files, not placeholder stubs or intentionally failing TODO binaries, in implementation commits. ## 1. Read the source in execution order diff --git a/docs/kv-cache-retention/README.md b/docs/kv-cache-retention/README.md index 0aa5c236ea12..06ae0e3f433e 100644 --- a/docs/kv-cache-retention/README.md +++ b/docs/kv-cache-retention/README.md @@ -1,6 +1,6 @@ # Issue 1: branch-preserving KV cache workspace -Status: **design and implementation instructions only; no runtime fix implemented**. +Status: **RAM-path implementation and CPU regression evidence are present. Optional SSD spill is not implemented.** Addresses [novkien/llama.cpp-fork#1](https://github.com/novkien/llama.cpp-fork/issues/1). Historical production evidence: [novkien/llama-proxy#416](https://github.com/novkien/llama-proxy/issues/416). @@ -23,23 +23,33 @@ Retention is bounded by configured resources. Cache eviction, incompatible state | [TESTING.md](TESTING.md) | Deterministic regressions, disk-isolated tests, failure injection and performance evidence. | | [RESEARCH.md](RESEARCH.md) | Dated upstream PRs, inspected comments, source anchors and reuse decisions. | -Read DESIGN first, then implement the numbered slices in IMPLEMENTATION. Update the checklist below as real code and tests land on this draft branch. Do not replace failed behavioral tests with a successful build or a larger cache counter. +Read the current delivery section below before continuing the remaining numbered slices in IMPLEMENTATION. Update the checklist as complete slice acceptance, not just source changes, is established. Do not replace failed behavioral tests with a successful build or a larger cache counter. + +## Current RAM delivery + +The review of owner commit `8cfa7bb52fe69e904abc53c86b775c01a6ec3b0b` found that the initial two-file patch still consumed snapshots on load, ranked raw LCP rather than valid checkpoints, retained the 25% exclusion, and archived cache-opt-out requests. Optional idle evacuation also cleared state after a failed save. + +The repair keeps selection read-only, prepares the cache only after slot availability and family capacity checks, preserves outgoing branches before replacement, restores without consuming snapshots, and shares immutable checkpoint backing. `server_prompt::plan_reuse` is used by both candidate ranking and execution. Exact compatible snapshots, not contained token prefixes, are deduplicated. Capture checks byte counts and protects every allocation; restore prepares metadata before native setters. RAM limits remain in force and actual restoration moves an entry to the most-recently-used end. + +`cache_prompt=false` disables incoming reuse and automatic archiving of that request. Infill keeps its existing live-state reuse path without becoming an automatic RAM archive. LoRA identity is checked before candidate selection and old state is saved under its old identity. A failed optional idle save keeps live state; a necessary pressure purge reports any preservation loss. + +The RAM source has local CPU build and regression evidence in [TESTING.md](TESTING.md). This is not a claim about Nex/GPU, media, real MTP, disk persistence or production behavior. The requested owner merge/build/live trial remains separate. `--cache-spill-mib` is **not an available flag** in this implementation; no automatic file writing or disk restore was added. ## Implementation checklist -- [ ] S1: reproduce baseline defects with deterministic fixtures. -- [ ] S2: safe snapshot ownership, complete allocation handling and typed results. -- [ ] S3: pure reuse planner, pinned/automatic pipeline and preserve-before-mutate. -- [ ] S4: non-consuming restore, conservative deduplication and bounded retention. +- [ ] S1: RAM regressions reproduce seven failing cases on the owner's initial patch; full hybrid fixture coverage remains. +- [ ] S2: core RAM ownership/capture implemented; exhaustive per-allocation failure injection remains. +- [ ] S3: target rewind planner and pinned/automatic pipeline implemented; real hybrid/draft/media acceptance remains. +- [ ] S4: non-consuming RAM restore, conservative deduplication and budget protection implemented; SSD-pressure integration remains. - [ ] S5: complete snapshot codec and opt-in namespaced SSD store. - [ ] S6: bounded disk worker, cancellation, restart and shutdown integration. - [ ] S7: RAM/disk/hybrid/draft/media regressions and matched performance measurements. - [ ] S8: implementation documentation, review and owner-selected native/proxy integration. -These are implementation slices, not extra approval stages. The present delivery stops at a draft documentation PR. It does not authorize production restarts, model changes, proxy pinning activation, upstream publication, merge or deployment. +These are implementation slices, not extra approval stages. The current repair stops at reviewed PR source and isolated checks. It does not perform production restarts, model changes, proxy pinning activation, upstream publication, merge or deployment. Unchecked items preserve the full architecture acceptance rather than hiding unfinished work. ## Workspace safety Use the existing checkout if appropriate. Inspect its actual remote, branch and modifications before editing; preserve other people's work. Do not reset, stash, clean, force-push or switch a shared checkout merely to follow these instructions. A separate clone is an optional convenience, not a requirement for all contributors. -All new CLI names, structs, task types and event fields in this package are **proposed implementation contracts**, not options available in the baseline binary. Existing interfaces are identified explicitly. This package was authored with ChatGPT assistance at the fork owner's request; it is not an upstream-maintainer endorsement or a native test report. +Names in DESIGN/IMPLEMENTATION describe the complete target architecture; consult the current delivery above and actual source for the implemented RAM subset. SSD flags, I/O tasks and event fields remain **proposed contracts**, not available options. Existing interfaces are identified explicitly. This package was authored with ChatGPT assistance at the fork owner's request; it is not an upstream-maintainer endorsement or a native test report. diff --git a/docs/kv-cache-retention/TESTING.md b/docs/kv-cache-retention/TESTING.md index cb0224190363..94fceefb97ad 100644 --- a/docs/kv-cache-retention/TESTING.md +++ b/docs/kv-cache-retention/TESTING.md @@ -1,6 +1,33 @@ # Verification plan and acceptance evidence -Status: all implementation results are **NOT RUN** in the documentation draft. The table specifies required tests, not claimed passes. Use the maintained server pytest harness and disposable files/ports. See [IMPLEMENTATION.md](IMPLEMENTATION.md) for setup. +Status: the CPU RAM-path results below are verified. The complete T01-T24 table remains an acceptance plan, not a blanket PASS; SSD, real hybrid/GPU/MTP/media and exhaustive failure injection are still unverified. Use the maintained server pytest harness and disposable files/ports. See [IMPLEMENTATION.md](IMPLEMENTATION.md) for setup. + +## Current review evidence (2026-09-29) + +- Baseline native source: owner commit `8cfa7bb52fe69e904abc53c86b775c01a6ec3b0b`. Temporary export-only commits did not change native bytes. Baseline and repaired builds used GCC 14.2.0, CMake 3.31.6, Release, CPU, `GGML_NATIVE=OFF`, `LLAMA_FATAL_WARNINGS=ON`, and no embedded UI in an offline disposable environment. +- Model: public `tinyllamas/stories260K.gguf`, 1,185,376 bytes, SHA-256 `270cba1bd5109f42d03350f60406024560464db173c0e387d91f0426d3bd256d`. +- New HTTP cache regressions: **12 passed** on the repair, compared with **7 failed / 5 passed** on the initial owner patch. The failing baseline cases were borrowed-snapshot loss in automatic and explicit-slot modes at both similarity settings, the 25% exclusion, opt-out archiving, and failed idle save followed by clear. +- Expanded existing completion/manual text-slot regressions plus the new cases: **53 passed, 1 skipped, 1 slow test excluded**. The excluded test requires a different Phi model; it was not claimed tested using the tiny model. The harness timing override is not evidence of production throughput. +- `test-server-prompt-cache`: **18 model-independent checks** for actual planner behavior and immutable ownership; **33 checks total** with the same tiny model supplied, including native capture, duplicate capture, protected-candidate byte-budget rejection, cross-slot restore, non-consumption, draft-state mismatch rejection before installation and cache opt-out. Its CTest registration passed. +- Build and `git diff --check` passed. These are source-level/isolated results, not a deployed native or GPU acceptance report. + +The focused HTTP cases use a supplied local model and ephemeral loopback ports. This command avoids the parent conftest's all-model preload while running the maintained completion test module and each cache fixture's own teardown: + +```bash +ROOT="$(git rev-parse --show-toplevel)" +MODEL="/absolute/path/to/stories260K.gguf" +BUILD="/absolute/path/to/disposable-build" +cmake --build "$BUILD" --target llama-server test-server-prompt-cache -j2 +ctest --test-dir "$BUILD" -R '^test-server-prompt-cache$' --output-on-failure +"$BUILD/bin/test-server-prompt-cache" "$MODEL" +LLAMA_CACHE_TEST_MODEL="$MODEL" \ +LLAMA_SERVER_BIN_PATH="$BUILD/bin/llama-server" \ +PYTHONPATH="$ROOT/tools/server/tests" \ +python -m pytest --confcutdir="$ROOT/tools/server/tests/unit" \ + "$ROOT/tools/server/tests/unit/test_completion.py" -k cache_retention -q +``` + +When comparing separate builds that use shared libraries, point `LD_LIBRARY_PATH` at that build's own `bin` directory so a baseline executable cannot accidentally load repaired libraries. On other platforms use the corresponding library-resolution mechanism. Keep `cache_ram=0` and a process restart for any future disk-only test so RAM cannot mask a broken file path. ## 1. Evidence rules diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 96d910a752b7..4ecdf1b9e29d 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -163,6 +163,10 @@ if (NOT WIN32 OR NOT BUILD_SHARED_LIBS) llama_build_and_test(test-chat.cpp WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}) target_include_directories(test-chat PRIVATE ${PROJECT_SOURCE_DIR}/tools/server) target_link_libraries(test-chat PRIVATE server-context) + + llama_build_and_test(test-server-prompt-cache.cpp) + target_include_directories(test-server-prompt-cache PRIVATE ${PROJECT_SOURCE_DIR}/tools/server) + target_link_libraries(test-server-prompt-cache PRIVATE server-context) llama_build_and_test(test-json-schema-to-grammar.cpp) if (NOT GGML_BACKEND_DL) diff --git a/tests/test-server-prompt-cache.cpp b/tests/test-server-prompt-cache.cpp new file mode 100644 index 000000000000..ff59593aa876 --- /dev/null +++ b/tests/test-server-prompt-cache.cpp @@ -0,0 +1,135 @@ +#include "server-task.h" + +#include +#include + +static int n_checks = 0; + +static void check(bool condition, const char * message) { + ++n_checks; + if (!condition) { + std::fprintf(stderr, "FAIL: %s\n", message); + std::exit(1); + } +} + +static server_prompt prompt(size_t n, llama_token token = 11) { + server_prompt p; + p.tokens = server_tokens(llama_tokens(n, token), false); + return p; +} + +static void checkpoint(server_prompt & p, int64_t n, llama_pos lo, llama_pos hi) { + common_prompt_checkpoint c{}; + c.update_pos(n, lo, hi); + c.data_tgt.reset(8)[0] = 7; + p.checkpoints.push_back(std::move(c)); +} + +static void native_roundtrip(const char * model_path) { + common_params params; + params.model.path = model_path; + params.n_ctx = 2048; + params.n_batch = 512; + params.n_ubatch = 512; + params.n_parallel = 2; + params.n_gpu_layers = 0; + params.kv_unified = true; + params.warmup = false; + params.fit_params = false; + params.cpuparams.n_threads = 2; + params.cpuparams_batch.n_threads = 2; + auto init = common_init_from_params(params); + auto * ctx = init->context(); + check(ctx != nullptr, "initialize native test context"); + const auto decode = [&](const server_prompt & p, int id_slot) { + llama_batch batch = llama_batch_init(p.n_tokens(), 0, 1); + for (int i = 0; i < p.n_tokens(); ++i) { + common_batch_add(batch, p.tokens[i], i, { id_slot }, i + 1 == p.n_tokens()); + } + const int result = llama_decode(ctx, batch); + llama_batch_free(batch); + check(result == 0, "decode native test prompt"); + }; + server_prompt_cache cache(64, 0); + const std::vector lora; + auto a = prompt(192, 11); + decode(a, 0); + check(cache.save(a, ctx, nullptr, 0, lora, nullptr) == SERVER_PROMPT_SAVED, "capture complete native state"); + const auto * saved = &cache.states.front(); + const auto bytes = saved->data.main; + check(!bytes.empty() && saved->pos_max == 191, "captured sequence bounds and bytes"); + check(cache.save(a, ctx, nullptr, 0, lora, nullptr) == SERVER_PROMPT_ALREADY_SAVED, "exact duplicate capture"); + auto b = prompt(64, 41); + llama_memory_seq_rm(llama_get_memory(ctx), 0, -1, -1); + decode(b, 0); + cache.limit_size = saved->size(); + check(cache.save(b, ctx, nullptr, 0, lora, nullptr, saved) == SERVER_PROMPT_SAVE_FAILED, "protect chosen restore source at budget limit"); + check(cache.states.size() == 1 && saved->data.main == bytes, "failed capture retains source"); + cache.limit_size = 64 * 1024 * 1024; + check(cache.save(b, ctx, nullptr, 0, lora, nullptr, saved) == SERVER_PROMPT_SAVED, "save replacement branch"); + server_prompt restored; + check(cache.load(restored, *saved, ctx, nullptr, 1, nullptr), "restore into another native slot"); + check(cache.states.size() == 2 && saved->data.main == bytes, "restore does not consume snapshot"); + check(llama_memory_seq_pos_max(llama_get_memory(ctx), 1) == 191 && restored.n_tokens() == 192, "install consistent native state and tokens"); + check(!cache.load(restored, *saved, ctx, ctx, 1, nullptr), "reject missing draft state before target mutation"); + check(llama_memory_seq_pos_max(llama_get_memory(ctx), 1) == 191, "failed restore leaves native state unchanged before installation"); + a.cache_eligible = false; + check(cache.save(a, ctx, nullptr, 1, lora, nullptr) == SERVER_PROMPT_NOT_CACHED, "opt-out bypasses archive"); +} + +int main(int argc, char ** argv) { + auto p = prompt(1000); + auto plan = p.plan_reuse(700, 1100, 0, 0); + check(plan.n_past == 700 && !plan.checkpoint && plan.pos_next == 700, "dense direct reuse"); + check(p.plan_reuse(1000, 1000, 0, 0).usable_tokens(1000) == 999, "logits replay"); + check(p.plan_reuse(0, 1000, -1, 0).n_past == 0, "empty input prefix"); + check(p.plan_reuse(500, 1000, -1, 0).n_past == 0, "no native state"); + check(p.plan_reuse(700, 1100, 999, 0).n_past == 0, "recurrent cache without checkpoint"); + checkpoint(p, 600, 599, 599); + checkpoint(p, 900, 899, 899); + plan = p.plan_reuse(700, 1100, 999, 0); + check(plan.n_past == 600 && plan.pos_next == 600 && plan.checkpoint, "recurrent rollback"); + check(p.plan_reuse(500, 1100, 999, 0).n_past == 0, "divergence before earliest checkpoint"); + check(p.plan_reuse(700, 1100, 300, 128).n_past == 700, "SWA resident coverage"); + check(p.plan_reuse(700, 1100, 600, 128).n_past == 0, "SWA checkpoint does not cover window"); + + auto copy = p.clone(); + check(copy.checkpoints.front().data_tgt.data() == p.checkpoints.front().data_tgt.data(), "checkpoint backing shared"); + copy.checkpoints.front().data_tgt.reset(8)[0] = 9; + check(p.checkpoints.front().data_tgt.data()[0] == 7, "capture does not overwrite old snapshot"); + copy.checkpoints.clear(); + check(p.checkpoints.size() == 2, "live checkpoint pruning is independent"); + + server_prompt_cache cache(64, 0); + const std::vector lora; + auto resident = prompt(1000); + resident.tokens.set_token(800, 12); + // Resident raw LCP is longer but has no usable recurrent checkpoint. + cache.states.emplace_back(); + auto & useful = cache.states.back(); + useful.prompt = prompt(900); + useful.prompt.tokens.set_token(700, 12); + useful.pos_min = 899; + checkpoint(useful.prompt, 600, 599, 599); + const auto input = server_tokens(llama_tokens(1000, 11), false); + check(cache.find_best(resident, input, 999, 0, lora, 1000) == &useful, "rank usable checkpoints, not raw LCP"); + check(cache.find_best(resident, input, 0, 0, lora, 1000) == nullptr, "prefer superior live state"); + auto empty = prompt(0); + useful.prompt = prompt(1000); + useful.pos_min = 0; + const auto short_input = server_tokens(llama_tokens(100, 11), false); + check(cache.find_best(empty, short_input, -1, 0, lora, 100) == &useful, "no arbitrary 25 percent exclusion"); + useful.lora.emplace_back(); + check(cache.find_best(empty, input, -1, 0, lora, 1000) == nullptr, "adapter mismatch is excluded"); + useful.lora.clear(); + check(cache.find_best(useful.prompt, input, 0, 0, lora, 1000) == nullptr, "resident wins equal benefit"); + check(cache.find_best(empty, input, -1, 0, lora, 0) == nullptr, "zero benefit does not restore"); + if (argc == 2) { + llama_backend_init(); + native_roundtrip(argv[1]); + llama_backend_free(); + } + std::printf("%d prompt-cache checks passed\n", n_checks); + return 0; +} diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp index 7fec3edf8417..58ae2c8191c0 100644 --- a/tools/server/server-context.cpp +++ b/tools/server/server-context.cpp @@ -255,39 +255,9 @@ struct server_slot { server_prompt prompt; - bool prompt_save(server_prompt_cache & prompt_cache) const { - if (prompt.tokens.size() == 0) { - return false; - } - - const size_t cur_size_tgt = llama_state_seq_get_size_ext(ctx_tgt, id, LLAMA_STATE_SEQ_FLAGS_NONE); - const size_t cur_size_dft = ctx_dft ? llama_state_seq_get_size_ext(ctx_dft, id, LLAMA_STATE_SEQ_FLAGS_NONE) : 0; - - const size_t cur_size = cur_size_tgt + cur_size_dft; - - SRV_TRC(" - saving prompt with length %d, total state size = %.3f MiB (draft: %.3f MiB)\n", - (int) prompt.tokens.size(), cur_size / (1024.0 * 1024.0), cur_size_dft / (1024.0 * 1024.0)); - - auto * cur = prompt_cache.alloc(prompt, cur_size_tgt, cur_size_dft); - if (cur == nullptr) { - return false; - } - - llama_state_seq_get_data_ext(ctx_tgt, cur->data.main.data(), cur_size_tgt, id, LLAMA_STATE_SEQ_FLAGS_NONE); - if (ctx_dft) { - llama_state_seq_get_data_ext(ctx_dft, cur->data.drft.data(), cur_size_dft, id, LLAMA_STATE_SEQ_FLAGS_NONE); - } - - return true; - } - - bool prompt_load(server_prompt_cache & prompt_cache, const server_tokens & tokens) { - bool res = prompt_cache.load(prompt, tokens, ctx_tgt, ctx_dft, id); - if (!res) { - SLT_WRN(*this, "%s", "failed to load prompt from cache\n"); - } - - return res; + bool prompt_save(server_prompt_cache & cache, const server_prompt_cache_state * keep = nullptr) const { + const auto result = cache.save(prompt, ctx_tgt, ctx_dft, id, lora, spec, keep); + return result == SERVER_PROMPT_SAVED || result == SERVER_PROMPT_ALREADY_SAVED; } void prompt_clear() { @@ -1528,12 +1498,12 @@ struct server_context_impl { server_slot * get_available_slot(const server_task & task) { server_slot * ret = nullptr; - // if a specific slot is requested, use it (still goes through cache update logic below) if (task.id_slot != -1) { ret = get_slot_by_id(task.id_slot); if (ret) { SLT_INF(*ret, "selected slot by id (%d)\n", task.id_slot); } + return ret; } // find the slot that has at least n% prompt similarity @@ -1605,30 +1575,6 @@ struct server_context_impl { } } - if (ret) { - const bool can_cache = prompt_cache && task.type == SERVER_TASK_TYPE_COMPLETION && !ret->is_processing(); - - if (can_cache) { - SRV_TRC("%s", "updating prompt cache\n"); - - const int64_t t_start = ggml_time_us(); - - // save when reuse would discard live tokens - const size_t lcp_slot = ret->prompt.tokens.get_common_prefix(task.tokens); - if (!ret->prompt.tokens.empty() && lcp_slot < ret->prompt.tokens.size()) { - ret->prompt_save(*prompt_cache); - } - - if (!ret->prompt_load(*prompt_cache, task.tokens)) { - ret->prompt_clear(); - } - - prompt_cache->update(); - - SRV_TRC("prompt cache update took %.2f ms\n", (ggml_time_us() - t_start) / 1000.0); - } - } - return ret; } @@ -1652,6 +1598,12 @@ struct server_context_impl { if (slot.prompt.n_tokens() > 0) { SRV_WRN("purging slot %d with %zu tokens\n", slot.id, slot.prompt.tokens.size()); + if (prompt_cache && slot.prompt.cache_eligible && !slot.prompt_save(*prompt_cache)) { + SRV_WRN("slot %d cache lost during required KV pressure purge\n", slot.id); + } + if (prompt_cache) { + prompt_cache->update(); + } slot.prompt_clear(); res = true; @@ -1678,27 +1630,17 @@ struct server_context_impl { } bool launch_slot_with_task(server_slot & slot, server_task && task) { - // process per-request lora adapters - if (!task.params.lora.empty()) { - auto task_loras = construct_lora_list(task.params.lora); - if (!are_lora_equal(task_loras, slot.lora)) { - // if lora has changed, check to see if the cache should be cleared - if (lora_should_clear_cache(slot.lora, task_loras)) { - SLT_TRC(slot, "clearing cache for lora change. %zu loras -> %zu loras\n", slot.lora.size(), task.params.lora.size()); - slot.prompt.clear(); - } else { - SLT_TRC(slot, "keeping cache for alora. %zu target loras\n", task_loras.size()); - } - slot.lora = task_loras; - } - } else { - slot.lora = params_base.lora_adapters; + GGML_ASSERT(!slot.is_processing()); + if (!task.tokens.validate(ctx_tgt)) { + send_error(task, "Prompt contains invalid tokens", ERROR_TYPE_INVALID_REQUEST); + return false; } + auto task_loras = task.params.lora.empty() ? params_base.lora_adapters : construct_lora_list(task.params.lora); // if using alora, make sure it's only a single one requested and active size_t alora_invocation_start = task.tokens.size(); - if (lora_all_alora(slot.lora)) { - const auto & enabled_ids = lora_get_enabled_ids(slot.lora); + if (lora_all_alora(task_loras)) { + const auto & enabled_ids = lora_get_enabled_ids(task_loras); // TODO: This will error out if a user requests two aloras, but only // provides the activation string for one. We could, instead search // for all requested alora activation strings and then either keep @@ -1707,7 +1649,7 @@ struct server_context_impl { send_error(task, "Cannot run multiple aLoRAs in a single request", ERROR_TYPE_INVALID_REQUEST); return false; } - const auto & lora = slot.lora[enabled_ids[0]].ptr; + const auto & lora = task_loras[enabled_ids[0]].ptr; // get the pointer and count for the invocation tokens const uint64_t n_invocation_tokens = llama_adapter_get_alora_n_invocation_tokens(lora); @@ -1736,18 +1678,12 @@ struct server_context_impl { // if the activation string is not found, disable the alora if (alora_invocation_start == task.tokens.size()) { SLT_DBG(slot, "alora %zu requested, but not found. deactivating\n", enabled_ids[0]); - slot.lora[enabled_ids[0]].scale = 0.0f; + task_loras[enabled_ids[0]].scale = 0.0f; } else { SLT_DBG(slot, "alora %zu activated starting at %zu\n", enabled_ids[0], alora_invocation_start); - slot.alora_invocation_start = alora_invocation_start; } } - if (!task.tokens.validate(ctx_tgt)) { - send_error(task, "Prompt contains invalid tokens", ERROR_TYPE_INVALID_REQUEST); - return false; - } - SLT_DBG(slot, "launching slot : %s\n", safe_json_to_str(slot.to_json()).c_str()); // initialize samplers @@ -1787,6 +1723,44 @@ struct server_context_impl { slot.smpl.reset(); } + const bool same_lora = are_lora_equal(slot.lora, task_loras); + const bool clear_lora = !same_lora && lora_should_clear_cache(slot.lora, task_loras); + const bool can_reuse = task.params.cache_prompt && !task.is_child() && + (task.type == SERVER_TASK_TYPE_COMPLETION || task.type == SERVER_TASK_TYPE_INFILL); + const server_prompt_cache_state * candidate = nullptr; + const size_t reuse_limit = lora_all_alora(task_loras) && alora_invocation_start > 0 + ? alora_invocation_start - 1 : task.tokens.size(); + if (prompt_cache && can_reuse && task.type == SERVER_TASK_TYPE_COMPLETION) { + const server_prompt empty; + candidate = prompt_cache->find_best(!clear_lora && slot.prompt.cache_reusable ? slot.prompt : empty, + task.tokens, llama_memory_seq_pos_min(llama_get_memory(ctx_tgt), slot.id), + n_swa, task_loras, reuse_limit); + } + const size_t lcp = slot.prompt.tokens.get_common_prefix(task.tokens); + const bool replace = !same_lora || !can_reuse || candidate || lcp < slot.prompt.tokens.size(); + if (prompt_cache && replace && slot.prompt.cache_eligible && !slot.prompt.tokens.empty()) { + if (!slot.prompt_save(*prompt_cache, candidate)) { + SLT_WRN(slot, "%s", "outgoing branch could not be preserved in RAM\n"); + } + } + if (clear_lora || !slot.prompt.cache_reusable || !can_reuse) { + slot.prompt_clear(); + } + slot.lora = std::move(task_loras); + slot.alora_invocation_start = lora_all_alora(slot.lora) ? (int32_t) alora_invocation_start : -1; + if (candidate && !prompt_cache->load(slot.prompt, *candidate, ctx_tgt, ctx_dft, slot.id, spec.get())) { + slot.prompt_clear(); + llama_set_sampler(ctx_tgt, slot.id, nullptr); + slot.smpl.reset(); + send_error(task, "Failed to restore prompt cache state", ERROR_TYPE_SERVER); + return false; + } + slot.prompt.cache_eligible = task.type == SERVER_TASK_TYPE_COMPLETION && task.params.cache_prompt; + slot.prompt.cache_reusable = can_reuse || (task.is_child() && task.params.cache_prompt); + if (prompt_cache) { + prompt_cache->update(); + } + // the per-request limit takes priority over the global one slot.n_predict_max = task.params.n_predict != -1 ? task.params.n_predict : params_base.n_predict; @@ -2411,12 +2385,13 @@ struct server_context_impl { if (!slot.is_processing()) { SLT_TRC(slot, "%s", "saving idle slot to prompt cache\n"); - if (slot.prompt_save(*prompt_cache)) { + const bool saved = slot.prompt_save(*prompt_cache); + if (saved) { SLT_DBG(slot, "%s", "__TEST_TAG_CACHE_IDLE_SLOT__\n"); prompt_cache->update(); } - if (params_base.kv_unified) { + if (params_base.kv_unified && (saved || !slot.prompt.cache_eligible)) { // [TAG_IDLE_SLOT_CLEAR] slot.prompt_clear(); } @@ -2611,6 +2586,8 @@ struct server_context_impl { slot->prompt.clear(); slot->prompt.tokens = std::move(restored); + slot->prompt.cache_eligible = true; + slot->prompt.cache_reusable = true; } catch (const std::exception & err) { slot->prompt_clear(); send_error(task, std::string("Unable to restore slot: ") + err.what(), ERROR_TYPE_INVALID_REQUEST); @@ -3263,12 +3240,6 @@ struct server_context_impl { llama_pos pos_next = slot.prompt.tokens.pos_next(n_past); - // ref: https://github.com/ggml-org/llama.cpp/pull/24110 - const bool has_new_tokens = (n_past < slot.task->n_tokens()); - - // the largest pos_min required for a checkpoint to be useful - const auto pos_min_thold = std::max(0, pos_next - n_swa - (has_new_tokens ? 0 : 1)); - if (n_past > 0 && n_past <= slot.prompt.n_tokens()) { const auto pos_min = llama_memory_seq_pos_min(llama_get_memory(ctx_tgt), slot.id); if (pos_min == -1) { @@ -3319,43 +3290,17 @@ struct server_context_impl { SLT_WRN(slot, "%s\n", st1.str().c_str()); } - if (pos_min >= pos_min_thold) { - // search for a context checkpoint - const auto it = std::find_if( - slot.prompt.checkpoints.rbegin(), - slot.prompt.checkpoints.rend(), - [&](const auto & cur) { - // guarantee that a checkpoint will result in at least one token being processed [TAG_PROMPT_LOGITS] - SLT_TRC(slot, "checking checkpoint with [%d, %d] against %d...\n", cur.pos_min, cur.pos_max, pos_min_thold); - // workaround for [TAG_CHECKPOINTS_FIX_POS_MIN] - if (cur.pos_max > pos_next) { - return false; - } - return cur.pos_min < pos_min_thold || cur.pos_min == 0; - } - ); - - bool do_reset = it == slot.prompt.checkpoints.rend(); - - if (!do_reset) { - // restore the context checkpoint - it->load_tgt(ctx_tgt, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY); - it->load_dft(ctx_dft, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY); - // restore the draft's speculative state - common_speculative_set_state(spec.get(), slot.id, it->data_spec); - - pos_next = std::min(pos_next, std::max(it->pos_min + 1, it->pos_max)); - n_past = std::min(slot.prompt.tokens.size_up_to_pos(pos_next), (size_t) it->n_tokens); - SLT_TRC(slot, "restored context checkpoint (pos_min = %d, pos_max = %d, n_tokens = %" PRId64 ", n_past = %d, size = %.3f MiB)\n", it->pos_min, it->pos_max, it->n_tokens, n_past, (float) it->size() / 1024 / 1024); - } - - if (do_reset) { - SLT_TRC(slot, "forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see %s)\n", - "https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055"); - pos_next = 0; - n_past = 0; - } + const auto plan = slot.prompt.plan_reuse(n_past, slot.task->n_tokens(), pos_min, n_swa); + if (plan.checkpoint) { + plan.checkpoint->load_tgt(ctx_tgt, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY); + plan.checkpoint->load_dft(ctx_dft, slot.id, LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY); + common_speculative_set_state(spec.get(), slot.id, plan.checkpoint->data_spec); + SLT_TRC(slot, "restored context checkpoint, n_past = %zu\n", plan.n_past); + } else if (plan.n_past == 0) { + SLT_TRC(slot, "%s", "forcing full prompt re-processing due to lack of cache data\n"); } + pos_next = plan.pos_next; + n_past = plan.n_past; } { diff --git a/tools/server/server-task.cpp b/tools/server/server-task.cpp index f61755da4d4c..7d70cf16576a 100644 --- a/tools/server/server-task.cpp +++ b/tools/server/server-task.cpp @@ -11,6 +11,8 @@ #include "server-common.h" #include +#include +#include // // task_params @@ -1708,161 +1710,187 @@ size_t server_prompt_cache::n_tokens() const { return res; } -server_prompt_cache_state * server_prompt_cache::alloc(const server_prompt & prompt, size_t state_size_tgt, size_t state_size_dft) { - // first check if the current state is contained fully in the cache - for (auto it = states.begin(); it != states.end(); ++it) { - const int cur_lcp_len = it->prompt.tokens.get_common_prefix(prompt.tokens); - - if (cur_lcp_len == (int) prompt.tokens.size()) { - SRV_TRC("%s", " - prompt is already in the cache, skipping\n"); - return nullptr; - } +server_prompt_reuse server_prompt::plan_reuse(size_t prefix, size_t n_input, llama_pos pos_min, int32_t n_swa) const { + server_prompt_reuse plan; + plan.n_past = std::min(prefix, tokens.size()); + plan.pos_next = tokens.pos_next(plan.n_past); + if (plan.n_past == 0) { + return plan; } - - // calculate checkpoints size to see if it will fit with the prompt - size_t checkpoints_size = 0; - for (const auto & ckpt : prompt.checkpoints) { - checkpoints_size += ckpt.size(); + if (pos_min < 0) { + return {}; } - const size_t state_size_new = state_size_tgt + state_size_dft + checkpoints_size; - - // skip over-limit entries to avoid disturbing the cache - if (limit_size > 0 && state_size_new > limit_size) { - SRV_WRN(" - prompt state size %.3f MiB exceeds cache size limit %.3f MiB, skipping\n", - state_size_new / (1024.0 * 1024.0), limit_size / (1024.0 * 1024.0)); - return nullptr; + const bool has_new_tokens = plan.n_past < n_input; + const llama_pos threshold = std::max(0, plan.pos_next - n_swa - (has_new_tokens ? 0 : 1)); + if (pos_min < threshold) { + return plan; } - // remove any cached prompts that are fully contained in the current prompt - for (auto it = states.begin(); it != states.end();) { - const int len = it->prompt.tokens.get_common_prefix(prompt.tokens); - - if (len == (int) it->prompt.tokens.size()) { - SRV_TRC(" - removing obsolete cached prompt with length %d\n", len); - - it = states.erase(it); - } else { - ++it; + for (auto it = checkpoints.rbegin(); it != checkpoints.rend(); ++it) { + if (it->pos_max > plan.pos_next || (it->pos_min >= threshold && it->pos_min != 0)) { + continue; } + plan.checkpoint = &*it; + plan.pos_next = std::min(plan.pos_next, std::max(it->pos_min + 1, it->pos_max)); + plan.n_past = std::min(tokens.size_up_to_pos(plan.pos_next), (size_t) it->n_tokens); + return plan; } + return {}; +} - if (limit_size > 0) { - // make room before allocating the new vectors to avoid breaching the limit - while (!states.empty() && size() + state_size_new > limit_size) { - SRV_WRN(" - making room for prompt cache entry, removing oldest entry (size = %.3f MiB)\n", - states.front().size() / (1024.0 * 1024.0)); +const server_prompt_cache_state * server_prompt_cache::find_best(const server_prompt & prompt, + const server_tokens & tokens_new, llama_pos pos_min, int32_t n_swa, + const std::vector & lora, size_t reuse_limit) const { + const size_t prefix = std::min(prompt.tokens.get_common_prefix(tokens_new), reuse_limit); + size_t best = prompt.plan_reuse(prefix, tokens_new.size(), pos_min, n_swa).usable_tokens(tokens_new.size()); + const server_prompt_cache_state * result = nullptr; - states.pop_front(); + for (const auto & state : states) { + // A shorter entry cannot beat the resident state. Avoid a full prefix scan on warm turns. + if (state.prompt.tokens.size() <= best || !are_lora_equal(state.lora, lora)) { + continue; + } + const size_t lcp = std::min(state.prompt.tokens.get_common_prefix(tokens_new), reuse_limit); + const auto plan = state.prompt.plan_reuse(lcp, tokens_new.size(), state.pos_min, n_swa); + const size_t reusable = plan.usable_tokens(tokens_new.size()); + if (reusable > best || (result && reusable == best)) { + best = reusable; + result = &state; } } + return result; +} - std::vector state_data_tgt; - std::vector state_data_dft; - - // check if we can allocate enough memory for the new state - try { - state_data_tgt.resize(state_size_tgt); - state_data_dft.resize(state_size_dft); - } catch (const std::bad_alloc & e) { - SRV_ERR("failed to allocate memory for prompt cache state: %s\n", e.what()); - - limit_size = std::max(1, 0.4*size()); - - SRV_WRN(" - cache size limit reduced to %.3f MiB\n", limit_size / (1024.0 * 1024.0)); - - update(); - - return nullptr; +static bool server_prompt_checkpoints_equal(const server_prompt & a, const server_prompt & b) { + if (a.checkpoints.size() != b.checkpoints.size()) { + return false; + } + auto ai = a.checkpoints.begin(); + auto bi = b.checkpoints.begin(); + for (; ai != a.checkpoints.end(); ++ai, ++bi) { + if (ai->n_tokens != bi->n_tokens || ai->pos_min != bi->pos_min || ai->pos_max != bi->pos_max || + !ai->data_tgt.shares_storage(bi->data_tgt) || !ai->data_dft.shares_storage(bi->data_dft) || + ai->data_spec != bi->data_spec) { + return false; + } } - - states.push_back({ - /*.prompt =*/ { - /*.tokens =*/ prompt.tokens.clone(), - /*.checkpoints =*/ prompt.checkpoints, - }, - /*.data =*/ { - /*.main =*/ std::move(state_data_tgt), - /*.drft =*/ std::move(state_data_dft), - }, - }); - - return &states.back(); + return true; } -bool server_prompt_cache::load(server_prompt & prompt, const server_tokens & tokens_new, llama_context * ctx_tgt, llama_context * ctx_dft, int32_t id_slot) { - const int lcp_best = prompt.tokens.get_common_prefix(tokens_new); - - int lcp_win = lcp_best; - - SRV_TRC(" - looking for better prompt, base lcp = %d\n", lcp_best); - - auto it_best = states.end(); - - // find the cached prompt with the longest reusable prefix - for (auto it = states.begin(); it != states.end(); ++it) { - const int lcp_cur = it->prompt.tokens.get_common_prefix(tokens_new); - - const float f_keep_cur = it->prompt.tokens.empty() ? 0.0f : float(lcp_cur) / it->prompt.tokens.size(); - const float f_sim_cur = tokens_new.empty() ? 0.0f : float(lcp_cur) / tokens_new.size(); - - SRV_TRC(" - prompt with length %7zu, lcp = %7d, f_keep = %.3f, f_sim = %.3f\n", it->prompt.tokens.size(), lcp_cur, f_keep_cur, f_sim_cur); +server_prompt_save_result server_prompt_cache::save(const server_prompt & prompt, + llama_context * ctx_tgt, llama_context * ctx_dft, int32_t id_slot, + const std::vector & lora, common_speculative * spec, + const server_prompt_cache_state * keep) { + if (prompt.tokens.empty() || !prompt.cache_eligible) { + return SERVER_PROMPT_NOT_CACHED; + } - // don't trash large prompts - if (f_keep_cur < 0.25f) { - continue; + try { + const auto memory = llama_get_memory(ctx_tgt); + const llama_pos pos_min = llama_memory_seq_pos_min(memory, id_slot); + const llama_pos pos_max = llama_memory_seq_pos_max(memory, id_slot); + if (pos_min < 0 || pos_max < pos_min) { + return SERVER_PROMPT_SAVE_FAILED; } - - if (lcp_cur > lcp_win) { - lcp_win = lcp_cur; - - it_best = it; + std::vector spec_state; + common_speculative_get_state(spec, id_slot, spec_state); + for (auto & state : states) { + if (state.pos_min == pos_min && state.pos_max == pos_max && are_lora_equal(state.lora, lora) && + state.prompt.tokens.size() == prompt.tokens.size() && + state.prompt.tokens.get_common_prefix(prompt.tokens) == prompt.tokens.size() && + server_prompt_checkpoints_equal(state.prompt, prompt) && state.data.spec == spec_state) { + return SERVER_PROMPT_ALREADY_SAVED; + } } - } - - if (it_best != states.end()) { - SRV_TRC(" - found better prompt with lcp = %d (base %d)\n", lcp_win, lcp_best); - { - auto & data = it_best->data.main; - - const size_t size = data.size(); - const size_t n = llama_state_seq_set_data_ext(ctx_tgt, data.data(), size, id_slot, 0); - if (n != size) { - SRV_ERR("failed to restore state with size %zu\n", size); - - return false; + const size_t n_tgt = llama_state_seq_get_size_ext(ctx_tgt, id_slot, LLAMA_STATE_SEQ_FLAGS_NONE); + const size_t n_dft = ctx_dft ? llama_state_seq_get_size_ext(ctx_dft, id_slot, LLAMA_STATE_SEQ_FLAGS_NONE) : 0; + if (n_tgt == 0 || (ctx_dft && n_dft == 0)) { + return SERVER_PROMPT_SAVE_FAILED; + } + size_t required = spec_state.size(); + const auto add_size = [&](size_t n) { + if (n > std::numeric_limits::max() - required) { + throw std::length_error("prompt cache size overflow"); } - - data.clear(); - data.shrink_to_fit(); + required += n; + }; + add_size(n_tgt); + add_size(n_dft); + for (const auto & ckpt : prompt.checkpoints) { + add_size(ckpt.size()); + } + if (limit_size > 0 && (required > limit_size || (keep && keep->size() > limit_size - required))) { + SRV_WRN("%s", "prompt cache capture rejected by byte budget\n"); + return SERVER_PROMPT_SAVE_FAILED; } - { - auto & data = it_best->data.drft; - - if (!data.empty()) { - GGML_ASSERT(ctx_dft); - - const size_t size = data.size(); - const size_t n = llama_state_seq_set_data_ext(ctx_dft, data.data(), size, id_slot, 0); - if (n != size) { - SRV_WRN("failed to restore state with size %zu\n", size); - - return false; + // Allocate metadata before evicting anything; keep the entry private until capture succeeds. + std::list pending; + pending.emplace_back(); + auto & entry = pending.back(); + entry.prompt = prompt.clone(); + entry.lora = lora; + entry.pos_min = pos_min; + entry.pos_max = pos_max; + entry.data.spec = std::move(spec_state); + if (limit_size > 0) { + while (size() > limit_size - required) { + auto victim = states.begin(); + while (victim != states.end() && &*victim == keep) { + ++victim; } - - data.clear(); - data.shrink_to_fit(); + if (victim == states.end()) { + return SERVER_PROMPT_SAVE_FAILED; + } + SRV_WRN(" - prompt cache byte budget: evicting %zu bytes\n", victim->size()); + states.erase(victim); } } - - prompt = std::move(it_best->prompt); - - states.erase(it_best); + entry.data.main.resize(n_tgt); + entry.data.drft.resize(n_dft); + if (llama_state_seq_get_data_ext(ctx_tgt, entry.data.main.data(), n_tgt, id_slot, LLAMA_STATE_SEQ_FLAGS_NONE) != n_tgt || + (ctx_dft && llama_state_seq_get_data_ext(ctx_dft, entry.data.drft.data(), n_dft, id_slot, LLAMA_STATE_SEQ_FLAGS_NONE) != n_dft)) { + SRV_WRN("%s", "prompt cache capture failed: incomplete state\n"); + return SERVER_PROMPT_SAVE_FAILED; + } + states.splice(states.end(), pending); + SRV_TRC(" - preserved prompt with %zu tokens, %zu bytes\n", prompt.tokens.size(), required); + return SERVER_PROMPT_SAVED; + } catch (const std::exception & e) { + SRV_WRN("prompt cache capture failed: %s\n", e.what()); + return SERVER_PROMPT_SAVE_FAILED; } +} - return true; +bool server_prompt_cache::load(server_prompt & prompt, const server_prompt_cache_state & state, + llama_context * ctx_tgt, llama_context * ctx_dft, int32_t id_slot, common_speculative * spec) { + try { + server_prompt restored = state.prompt.clone(); + const auto & main = state.data.main; + const auto & draft = state.data.drft; + if (main.empty() || bool(ctx_dft) != !draft.empty()) { + return false; + } + if (llama_state_seq_set_data_ext(ctx_tgt, main.data(), main.size(), id_slot, LLAMA_STATE_SEQ_FLAGS_NONE) != main.size() || + (ctx_dft && llama_state_seq_set_data_ext(ctx_dft, draft.data(), draft.size(), id_slot, LLAMA_STATE_SEQ_FLAGS_NONE) != draft.size())) { + return false; + } + common_speculative_set_state(spec, id_slot, state.data.spec); + prompt = std::move(restored); + // A successful read changes recency, not ownership of the immutable source. + for (auto it = states.begin(); it != states.end(); ++it) { + if (&*it == &state) { + states.splice(states.end(), states, it); + break; + } + } + return true; + } catch (const std::exception & e) { + SRV_WRN("prompt cache restore failed: %s\n", e.what()); + return false; + } } void server_prompt_cache::update() { diff --git a/tools/server/server-task.h b/tools/server/server-task.h index 9c99143f8e19..a8bc9fcaa77f 100644 --- a/tools/server/server-task.h +++ b/tools/server/server-task.h @@ -563,10 +563,24 @@ struct server_task_result_apply_lora : server_task_result { virtual json to_json() override; }; +struct server_prompt_reuse { + size_t n_past = 0; + llama_pos pos_next = 0; + const common_prompt_checkpoint * checkpoint = nullptr; + + size_t usable_tokens(size_t n_input) const { + return n_past == n_input && n_past > 0 ? n_past - 1 : n_past; + } +}; + struct server_prompt { server_tokens tokens; std::list checkpoints; + bool cache_eligible = true; + bool cache_reusable = true; + + server_prompt_reuse plan_reuse(size_t prefix, size_t n_input, llama_pos pos_min, int32_t n_swa) const; void clear() { tokens.clear(); @@ -581,6 +595,8 @@ struct server_prompt { return server_prompt { tokens.clone(), checkpoints, + cache_eligible, + cache_reusable, }; } }; @@ -588,15 +604,19 @@ struct server_prompt { struct server_prompt_data { std::vector main; std::vector drft; + std::vector spec; size_t size() const { - return main.size() + drft.size(); + return main.size() + drft.size() + spec.size(); } }; struct server_prompt_cache_state { server_prompt prompt; server_prompt_data data; + std::vector lora; + llama_pos pos_min = -1; + llama_pos pos_max = -1; size_t size() const { size_t res = data.size(); @@ -609,6 +629,15 @@ struct server_prompt_cache_state { } }; +enum server_prompt_save_result { + SERVER_PROMPT_SAVED, + SERVER_PROMPT_ALREADY_SAVED, + SERVER_PROMPT_NOT_CACHED, + SERVER_PROMPT_SAVE_FAILED, +}; + +struct common_speculative; + struct server_prompt_cache { server_prompt_cache(int32_t limit_size_mib, size_t limit_tokens) { this->limit_size = 1024ull*1024ull*(limit_size_mib < 0 ? 0 : limit_size_mib); @@ -627,9 +656,16 @@ struct server_prompt_cache { size_t n_tokens() const; - server_prompt_cache_state * alloc(const server_prompt & prompt, size_t state_size_main, size_t state_size_drft); + const server_prompt_cache_state * find_best(const server_prompt & prompt, const server_tokens & tokens_new, + llama_pos pos_min, int32_t n_swa, const std::vector & lora, + size_t reuse_limit) const; + + server_prompt_save_result save(const server_prompt & prompt, llama_context * ctx_tgt, llama_context * ctx_dft, + int32_t id_slot, const std::vector & lora, common_speculative * spec, + const server_prompt_cache_state * keep = nullptr); - bool load(server_prompt & prompt, const server_tokens & tokens_new, llama_context * ctx_tgt, llama_context * ctx_dft, int32_t id_slot); + bool load(server_prompt & prompt, const server_prompt_cache_state & state, + llama_context * ctx_tgt, llama_context * ctx_dft, int32_t id_slot, common_speculative * spec); void update(); }; diff --git a/tools/server/tests/unit/test_completion.py b/tools/server/tests/unit/test_completion.py index 01732eb16313..d796a9bc38bf 100644 --- a/tools/server/tests/unit/test_completion.py +++ b/tools/server/tests/unit/test_completion.py @@ -2,6 +2,9 @@ import requests import time import random +import os +import socket +from pathlib import Path from openai import OpenAI from utils import * @@ -666,3 +669,138 @@ def test_completion_prompt_cache(): assert "prompt_n" in timings and timings["prompt_n"] + timings["cache_n"] == n_prompt assert "predicted_n" in timings and timings["predicted_n"] == n_predict assert "tokens" in res.body and isinstance(res.body["tokens"], list) + + +@pytest.fixture +def cache_retention_server(tmp_path): + global server + with socket.socket() as probe: + probe.bind(("127.0.0.1", 0)) + server.server_port = probe.getsockname()[1] + server.debug = True + server.n_ctx = 4096 + server.n_batch = 512 + server.n_slots = 2 + server.cache_ram = 64 + server.no_cache_idle_slots = True + server.temperature = 0.0 + server.log_path = str(tmp_path / "cache.log") + # A local model permits a focused offline run without downloading every preset. + model = os.environ.get("LLAMA_CACHE_TEST_MODEL") + if model: + server.model_file = model + server.model_hf_repo = None + server.model_hf_file = None + yield server + server.stop() + + +def cache_retention_complete(tokens, id_slot, cache_prompt=True, n_predict=0): + data = { + "prompt": tokens, "cache_prompt": cache_prompt, + "n_predict": n_predict, "temperature": 0, "ignore_eos": True, + "return_tokens": True, + } + if id_slot is not None: + data["id_slot"] = id_slot + result = server.make_request("POST", "/completion", data=data) + assert result.status_code == 200, result.body + timings = result.body["timings"] + assert timings["cache_n"] + timings["prompt_n"] == len(tokens) + return result.body + + +@pytest.mark.parametrize("similarity", [0.0, 0.1]) +def test_cache_retention_pinned_empty(cache_retention_server, similarity): + server.slot_prompt_similarity = similarity + server.start() + a = [11, 12, 13, 14] * 64 + x = [71, 72, 73, 74] * 64 + assert cache_retention_complete(a, 0)["timings"]["cache_n"] == 0 + cache_retention_complete(x, 0) + restored = cache_retention_complete(a + [31, 32, 33], 1) + assert restored["timings"]["cache_n"] >= len(a) - 1 + + +@pytest.mark.parametrize("similarity", [0.0, 0.1]) +@pytest.mark.parametrize("id_slot", [0, None]) +def test_cache_retention_borrowed_snapshot(cache_retention_server, similarity, id_slot): + server.slot_prompt_similarity = similarity + server.n_slots = 1 + server.start() + common = [11, 12, 13, 14] * 48 + a = common + [21, 22, 23, 24] * 32 + x = [71, 72, 73, 74] * 80 + c = common + [41, 42, 43, 44] * 32 + cache_retention_complete(a, id_slot) + cache_retention_complete(x, id_slot) + borrowed = cache_retention_complete(c, id_slot) + assert borrowed["timings"]["cache_n"] == len(common) + resumed = cache_retention_complete(a + [51, 52, 53], id_slot, n_predict=8) + assert resumed["timings"]["cache_n"] >= len(a) - 1 + cold = cache_retention_complete(a + [51, 52, 53], id_slot, cache_prompt=False, n_predict=8) + assert cold["timings"]["cache_n"] == 0 + assert resumed["tokens"] == cold["tokens"] + + +def test_cache_retention_low_fraction_prefix(cache_retention_server): + server.start() + common = [11, 12, 13, 14] * 12 + a = common + [21, 22, 23, 24] * 88 + x = [71, 72, 73, 74] * 80 + c = common + [41, 42, 43, 44] * 32 + cache_retention_complete(a, 0) + cache_retention_complete(x, 0) + assert cache_retention_complete(c, 1)["timings"]["cache_n"] == len(common) + + +def test_cache_retention_opt_out(cache_retention_server): + server.start() + a = [11, 12, 13, 14] * 64 + x = [71, 72, 73, 74] * 64 + cache_retention_complete(a, 0, cache_prompt=False) + cache_retention_complete(x, 0) + assert cache_retention_complete(a, 1)["timings"]["cache_n"] == 0 + + +def test_cache_retention_warm_append_does_not_save(cache_retention_server): + server.start() + a = [11, 12, 13, 14] * 64 + cache_retention_complete(a, 0) + # Raw token arrays avoid a text re-tokenization mismatch at the append boundary. + cache_retention_complete(a + [31, 32, 33], 0) + log = Path(server.log_path).read_text() + assert "preserved prompt with" not in log + + +@pytest.mark.parametrize("similarity", [0.0, 0.1]) +def test_cache_retention_idle_evacuation(cache_retention_server, similarity): + server.no_cache_idle_slots = False + server.kv_unified = True + server.server_slots = True + server.slot_prompt_similarity = similarity + server.start() + a = [11, 12, 13, 14] * 64 + cache_retention_complete(a, 0) + cache_retention_complete([71, 72, 73, 74] * 64, 1) + status = server.make_request("GET", "/slots") + assert status.status_code == 200 + slot = next(item for item in status.body if item["id"] == 0) + assert slot["n_prompt_tokens"] == 0 + assert cache_retention_complete(a + [31, 32, 33], 0)["timings"]["cache_n"] >= len(a) - 1 + + +def test_cache_retention_failed_idle_save_keeps_live_state(cache_retention_server): + server.no_cache_idle_slots = False + server.kv_unified = True + server.server_slots = True + server.cache_ram = 1 + server.start() + a = [11, 12, 13, 14] * 500 + cache_retention_complete(a, 0) + cache_retention_complete([71, 72, 73, 74] * 64, 1) + status = server.make_request("GET", "/slots") + assert status.status_code == 200 + slot = next(item for item in status.body if item["id"] == 0) + assert slot["n_prompt_tokens"] == len(a) + assert cache_retention_complete(a + [31, 32, 33], 0)["timings"]["cache_n"] >= len(a) - 1 diff --git a/tools/server/tests/utils.py b/tools/server/tests/utils.py index 3a50ae5c3f48..dd7feb744b86 100644 --- a/tools/server/tests/utils.py +++ b/tools/server/tests/utils.py @@ -114,6 +114,7 @@ class ServerProcess: sleep_idle_seconds: int | None = None cache_ram: int | None = None no_cache_idle_slots: bool = False + slot_prompt_similarity: float | None = None log_path: str | None = None ui_mcp_proxy: bool = False backend_sampling: bool = False @@ -279,6 +280,8 @@ def start(self, timeout_seconds: int = DEFAULT_HTTP_TIMEOUT) -> None: server_args.extend(["--cache-ram", self.cache_ram]) if self.no_cache_idle_slots: server_args.append("--no-cache-idle-slots") + if self.slot_prompt_similarity is not None: + server_args.extend(["--slot-prompt-similarity", self.slot_prompt_similarity]) if self.ui_mcp_proxy: server_args.append("--ui-mcp-proxy") if self.server_tools: From 705ee67a9aaf75409cad53c5a5c334ec61b64be9 Mon Sep 17 00:00:00 2001 From: LE TRUNG KIEN Date: Tue, 29 Sep 2026 11:22:50 +0700 Subject: [PATCH 8/9] ci: remove the temporary cache review transport The reviewed RAM source was built, tested and published as e8ef91dc691f1197f9a8f69b586563de377d6278. No permanent workflow change is needed. Addresses novkien/llama.cpp-fork#1 Assisted-by: ChatGPT --- .github/workflows/kv-cache-review-export.yml | 116 ------------------- 1 file changed, 116 deletions(-) delete mode 100644 .github/workflows/kv-cache-review-export.yml diff --git a/.github/workflows/kv-cache-review-export.yml b/.github/workflows/kv-cache-review-export.yml deleted file mode 100644 index 08ca1366e674..000000000000 --- a/.github/workflows/kv-cache-review-export.yml +++ /dev/null @@ -1,116 +0,0 @@ -name: Cache review validation and publication - -on: - push: - branches: [design/issue-1-kv-cache-retention] - paths: [.github/workflows/kv-cache-review-export.yml] - -permissions: - contents: write - -jobs: - validate: - if: github.repository == 'novkien/llama.cpp-fork' - runs-on: ubuntu-latest - timeout-minutes: 20 - env: - GH_TOKEN: ${{ github.token }} - REVIEW_BRANCH: design/issue-1-kv-cache-retention - steps: - - uses: actions/checkout@v6 - with: - ref: ${{ github.sha }} - - name: Install the exact reviewed payload - run: | - set -euo pipefail - mkdir -p "${RUNNER_TEMP}/cache-review" - test "$(git ls-remote origin "refs/heads/${REVIEW_BRANCH}" | cut -f1)" = "$GITHUB_SHA" - python3 - <<'PY' - import base64, hashlib, json, lzma, os, pathlib, subprocess - chunks = [ - 'e61ee5e62d9cb087e83c9785b4fdb021f3481cb1', - '0768ea48e0ea3ca49df96f221aaf77ab7f3a40b5', - '6ae71d11bbb7c533b1bfdfc4e4fcc52e81f72f38', - '0a42bc60c4859517681ab43d19a7e4fe633e6b3b', - ] - def blob_sha(data): - return hashlib.sha1(b'blob ' + str(len(data)).encode() + b'\0' + data).hexdigest() - encoded = [] - for sha in chunks: - result = json.loads(subprocess.check_output(['gh', 'api', f'repos/{os.environ["GITHUB_REPOSITORY"]}/git/blobs/{sha}'])) - data = base64.b64decode(result['content']) - assert blob_sha(data) == sha, 'transport checksum mismatch' - encoded.append(data) - payload = json.loads(lzma.decompress(b''.join(encoded))) - manifest = payload['manifest'] - expected_paths = { - 'common/common.cpp', 'common/common.h', - 'docs/kv-cache-retention/DESIGN.md', 'docs/kv-cache-retention/IMPLEMENTATION.md', - 'docs/kv-cache-retention/README.md', 'docs/kv-cache-retention/TESTING.md', - 'tests/CMakeLists.txt', 'tests/test-server-prompt-cache.cpp', - 'tools/server/server-context.cpp', 'tools/server/server-task.cpp', - 'tools/server/server-task.h', 'tools/server/tests/unit/test_completion.py', - 'tools/server/tests/utils.py', - } - assert {r['path'] for r in manifest} == expected_paths - for record in manifest: - path = pathlib.Path(record['path']) - actual = blob_sha(path.read_bytes()) if path.exists() else None - assert actual == record['before'], f'concurrent source change: {path}' - patch = payload['patch'].encode() - subprocess.run(['git', 'apply', '--check', '-'], input=patch, check=True) - subprocess.run(['git', 'apply', '-'], input=patch, check=True) - for record in manifest: - assert blob_sha(pathlib.Path(record['path']).read_bytes()) == record['after'], record['path'] - subprocess.run(['git', 'diff', '--check'], check=True) - subprocess.run(['git', 'add', '--', *sorted(expected_paths)], check=True) - staged = set(subprocess.check_output(['git', 'diff', '--cached', '--name-only'], text=True).splitlines()) - assert staged == expected_paths - out = pathlib.Path(os.environ['RUNNER_TEMP']) / 'cache-review' - (out / 'manifest.json').write_text(json.dumps(manifest, indent=2)) - PY - git config user.name 'ChatGPT review assistant' - git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - git commit -m 'server: harden RAM cache retention across slot reuse' -m 'Addresses novkien/llama.cpp-fork#1' -m 'Assisted-by: ChatGPT' - git rev-parse HEAD | tee "${RUNNER_TEMP}/cache-review/candidate-sha.txt" - - name: Build the reviewed CPU candidate - run: | - set -euo pipefail - cmake -S . -B build-review -G Ninja \ - -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF \ - -DLLAMA_USE_PREBUILT_UI=OFF -DLLAMA_BUILD_EXAMPLES=OFF \ - -DLLAMA_BUILD_APP=OFF -DLLAMA_FATAL_WARNINGS=ON \ - 2>&1 | tee "${RUNNER_TEMP}/cache-review/configure.log" - cmake --build build-review --target llama-server test-server-prompt-cache -j2 \ - 2>&1 | tee "${RUNNER_TEMP}/cache-review/build.log" - - name: Run native and HTTP cache regressions - run: | - set -euo pipefail - python3 -m pip install pytest requests wget 'openai~=2.14.0' - curl --fail --location --retry 2 --max-time 90 \ - https://huggingface.co/ggml-org/models/resolve/main/tinyllamas/stories260K.gguf \ - -o "${RUNNER_TEMP}/stories260K.gguf" - echo "270cba1bd5109f42d03350f60406024560464db173c0e387d91f0426d3bd256d ${RUNNER_TEMP}/stories260K.gguf" | sha256sum --check - ctest --test-dir build-review -R '^test-server-prompt-cache$' --output-on-failure \ - 2>&1 | tee "${RUNNER_TEMP}/cache-review/ctest.log" - build-review/bin/test-server-prompt-cache "${RUNNER_TEMP}/stories260K.gguf" \ - 2>&1 | tee "${RUNNER_TEMP}/cache-review/native.log" - cd tools/server/tests - LLAMA_CACHE_TEST_MODEL="${RUNNER_TEMP}/stories260K.gguf" \ - LLAMA_SERVER_BIN_PATH="${GITHUB_WORKSPACE}/build-review/bin/llama-server" \ - PYTHONPATH=. python3 -m pytest --confcutdir=unit unit/test_completion.py \ - -k cache_retention -q --tb=short \ - 2>&1 | tee "${RUNNER_TEMP}/cache-review/http.log" - - name: Publish without overwriting concurrent work - run: | - set -euo pipefail - test "$(git ls-remote origin "refs/heads/${REVIEW_BRANCH}" | cut -f1)" = "$GITHUB_SHA" - git push origin "HEAD:refs/heads/${REVIEW_BRANCH}" - git rev-parse HEAD | tee "${RUNNER_TEMP}/cache-review/published-sha.txt" - - uses: actions/upload-artifact@v4 - if: always() - with: - name: cache-review-validation - path: ${{ runner.temp }}/cache-review/ - retention-days: 3 - if-no-files-found: error From 990a75740c7b4b4b2d708cc479cf53d213050771 Mon Sep 17 00:00:00 2001 From: LE TRUNG KIEN Date: Wed, 30 Sep 2026 13:56:20 +0700 Subject: [PATCH 9/9] docs: prepare selective CUDA P2P implementation Refs #3 Assisted-by: ChatGPT --- docs/cuda-p2p-allowlist/DESIGN.md | 107 ++++++++++++++++ docs/cuda-p2p-allowlist/IMPLEMENTATION.md | 114 +++++++++++++++++ docs/cuda-p2p-allowlist/PREFLIGHT-RESULTS.md | 123 +++++++++++++++++++ docs/cuda-p2p-allowlist/PREFLIGHT.md | 88 +++++++++++++ docs/cuda-p2p-allowlist/README.md | 42 +++++++ docs/cuda-p2p-allowlist/RESEARCH.md | 52 ++++++++ docs/cuda-p2p-allowlist/TESTING.md | 93 ++++++++++++++ 7 files changed, 619 insertions(+) create mode 100644 docs/cuda-p2p-allowlist/DESIGN.md create mode 100644 docs/cuda-p2p-allowlist/IMPLEMENTATION.md create mode 100644 docs/cuda-p2p-allowlist/PREFLIGHT-RESULTS.md create mode 100644 docs/cuda-p2p-allowlist/PREFLIGHT.md create mode 100644 docs/cuda-p2p-allowlist/README.md create mode 100644 docs/cuda-p2p-allowlist/RESEARCH.md create mode 100644 docs/cuda-p2p-allowlist/TESTING.md diff --git a/docs/cuda-p2p-allowlist/DESIGN.md b/docs/cuda-p2p-allowlist/DESIGN.md new file mode 100644 index 000000000000..41287c5fcc6c --- /dev/null +++ b/docs/cuda-p2p-allowlist/DESIGN.md @@ -0,0 +1,107 @@ +# Design: physical-pair allowlist with explicit host fallback + +This is the proposed implementation contract for [issue #3](https://github.com/novkien/llama.cpp-fork/issues/3), not a description of an existing option. Source and primary-document references are indexed in [RESEARCH.md](RESEARCH.md). + +## 1. Problem and non-goals + +Recorded tests distinguish working NVLink transfers from a corrupt PCIe peer transfer on the same four-V100 system. The aim is to avoid the latter while retaining all four GPUs. A two-GPU workload is not acceptance. This feature does not repair IOMMU/ACS/switch firmware, rearrange layers for an alleged speedup, implement tensor parallelism, or monitor physical links continuously. + +The pair selector operates in the CUDA backend. It does not belong in the HTTP request path, model token loop, or a proxy-side topology poll. No NVML or shell command is needed per copy. + +## 2. What "return to the normal path" means + +There are two different paths in the reviewed source: + +1. Legacy P2P environment absent: CUDA copy callbacks still call `cudaMemcpyPeerAsync` when compiled in. Explicit peer enablement is absent, but the runtime selects the transport. The recorded peer-off experiment was correct; that does not prove every allocation type follows an identical route. +2. Proposed excluded pair: both CUDA callbacks decline before enqueueing work. The existing generic backend uses host storage and source get/destination set. CUDA buffer get/set use D2H/H2D and synchronize their streams. + +The proposal deliberately chooses path 2 for excluded pairs. It preserves a correct fallback algorithm, not necessarily the exact latency, buffering, or overlap of path 1. Do not advertise identical performance or infer that every `cudaMemcpyPeerAsync` call itself proves physical P2P was active. + +Layer split determines where weights and compute reside. The selector controls transfers between the resulting backend buffers; it must not change layer placement. A transfer can also connect nonadjacent GPUs, especially with draft/target graphs or outputs. Apply policy to actual endpoints, not layer numbers or adjacency. + +## 3. Configuration and identity + +Proposed environment variable: + +```text +GGML_CUDA_P2P_PAIRS=0-1,2-3 +``` + +Endpoints are physical CUDA runtime ordinals after `CUDA_VISIBLE_DEVICES`, with range `[0, physical_device_count)`. They are not host `nvidia-smi` indices, indices into a shortened `--device` list, virtual ggml device IDs, or RPC IDs. Use the existing logical-to-physical mapping for backend copies. Startup diagnostics must show runtime ordinal, full UUID, and PCI identity so an operator can check the interpretation. + +The recorded UUID-prefix order is `5ca3,b2e,e149,d4d`; recover full UUIDs from current evidence rather than reconstructing them. If visibility changes, the same pair text can select different hardware. CUDA/NVML ordinals must not be equated. + +Every pair is symmetric configuration expanded into two directional permissions. It is not transitive: `0-1,1-2` does not permit `0-2`. Reversed/repeated pairs normalize to one pair. Same-physical transfers always use local D2D, including different virtual backends sharing one GPU. + +For this deployment: + +| Physical source/destination | 0 | 1 | 2 | 3 | +|---|---|---|---|---| +| 0 | LOCAL | PEER | HOST | HOST | +| 1 | PEER | LOCAL | HOST | HOST | +| 2 | HOST | HOST | LOCAL | PEER | +| 3 | HOST | HOST | PEER | LOCAL | + +There are four allowed and eight denied ordered distinct-device edges. + +### Grammar + +Parse a nonempty comma-separated list of decimal `a-b` pairs, allowing ASCII whitespace around tokens/separators. Accept decimal leading zeros and normalize them. Reject signs, overflow, self-pairs, empty entries, incomplete pairs, trailing garbage, and out-of-range endpoints. Validate everything before the first enable call. This resolves the earlier plan's optional wording about leading zeros into one testable proposed behavior. + +An explicitly requested unsupported direction is an initialization error, not permission to enable a larger set. Check capability in both directions. Capability is necessary but not proof of a correct link. + +### Compatibility decisions + +| Setting | Proposed result | +|---|---| +| New key absent | Keep legacy behavior unchanged. | +| New key present and valid | Activate immutable restrictive policy. | +| Old and new P2P keys both present | Error, including old value `0` or empty. | +| New key empty | Error, not enable-all or legacy mode. | +| `GGML_CUDA_NO_PEER_COPY` build with new key | Error for unsupported opt-in. | +| Managed-memory opt-in with new key | Error in v1; not covered by the transport contract. | +| `GGML_CUDA_NO_PINNED` present | Preserve it; use the ordinary fallback, without an application pinned pool. | +| NVIDIA CUDA, layer split | Intended supported feature scope. | +| CUDA collective initialization with new key | Reject before NCCL/internal/meta transport setup in v1. | +| HIP/MUSA | Diagnose unsupported attempted opt-in; no behavior change when absent. | +| RPC | No remote transport/enumeration changes. | + +The old key is presence-based: `GGML_CUDA_P2P=0` still activates its legacy branch. Do not silently reinterpret it in this feature. + +## 4. One authoritative policy + +Keep capability, requested permission, and successfully enabled state distinct. Publish the policy only after selected directions initialize successfully. Read it without mutation from allocation and copy paths. A fixed-size matrix and per-owner access lists are sufficient; avoid a generalized routing framework. + +During `ggml_cuda_init`, build from the local `info` under construction. The existing physical-device helper calls `ggml_cuda_info`; calling it recursively during that same function-local static initialization is unsafe. After initialization, use the normal helper. + +No per-request enable/disable, device reset, or hot environment reload. A new configuration requires a fresh child process. This is an application transport policy, not a security isolation boundary against arbitrary third-party CUDA code in the same process. + +## 5. Memory access invariant + +For a VMM mapping owned by physical GPU `owner`, grant READWRITE to the owner and only readers for which `policy(reader, owner)` is enabled. Always include the owner. Deduplicate physical IDs. This direction describes who may access the allocation, not simply the direction in which a tensor is being copied. + +The selective branch takes precedence over the current `GGML_USE_NCCL` compile-time widening. Retain the original branch when selective mode is absent. Apply permissions to each new span before it is returned. Verify growth/remap behavior and query flags with `cuMemGetAccess` in native tests. + +Omission from a new access list must not be confused with revoking previously granted access. Immutable pre-allocation policy avoids that ambiguity. GPU VMM allocations using `CU_MEM_ALLOCATION_TYPE_PINNED` with a DEVICE location are not host staging buffers. + +Ordinary ggml tensor buffers and temporary VMM pools are different allocation families. Both need native tests. Managed memory, CUDA async memory pools, external mappings, and IPC need separate access review if introduced later. + +## 6. Copy invariant and synchronization + +For a denied distinct-device edge, both callbacks return false before any copy or completion event is submitted. The generic fallback then completes producer work, protects destination reuse according to its caller, reads into host memory, and writes to the destination. A return value of false must never mean "some work was already enqueued; try again elsewhere." + +For allowed edges, retain the existing peer-copy source stream and event record/destination wait. For local edges, retain D2D. Do not replace a forbidden peer call with a cross-device `cudaMemcpyDefault`, D2D call, remote-pointer kernel, or external collective that bypasses the policy. + +Do not globally disable events or CUDA graphs to implement selection. Scheduler input copies and the ordinary async split-input copies are separate entry paths. Their synchronization contracts are not interchangeable. + +## 7. NCCL scope + +The source initializes communicators through a separate communication context; a NCCL build alone does not prove a communicator runs in layer mode. Keep the existing build usable. However, actual NCCL transport can independently enable/map peers. In v1, a restrictive-mode collective attempt must fail explicitly before transport initialization. Returning null and silently running an unreviewed meta fallback is not that rejection. + +`NCCL_P2P_LEVEL=NVL` only constrains NCCL topology selection; it does not implement this arbitrary pair matrix or control llama.cpp's copy callbacks. Internal AllReduce is not a four-GPU substitute in the reviewed implementation. + +## 8. Performance and completion + +Start with existing blocking host fallback. Only profile-driven work should add a bounded reusable staging pool, and it must respect `NO_PINNED`. Buffer reuse after D2H but before H2D completion is forbidden; cancellation does not cancel already queued CUDA operations. Keep bytes unchanged and avoid new precision conversions. + +Correctness, selected transport, and workload speed are separate outcomes. Existing NVLink bandwidth does not forecast Qwen tokens/s. Native four-GPU evidence is required to promote the feature; the local software-double experiment only establishes branch feasibility. See [PREFLIGHT.md](PREFLIGHT.md) and [TESTING.md](TESTING.md). diff --git a/docs/cuda-p2p-allowlist/IMPLEMENTATION.md b/docs/cuda-p2p-allowlist/IMPLEMENTATION.md new file mode 100644 index 000000000000..6f9b3918309c --- /dev/null +++ b/docs/cuda-p2p-allowlist/IMPLEMENTATION.md @@ -0,0 +1,114 @@ +# Implementation map and engineering notes + +No code below is an implemented patch. This is the work map for the future coder, based on `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20`. Read the current source before using these symbol names. The feature contract is in [DESIGN.md](DESIGN.md); sources are in [RESEARCH.md](RESEARCH.md). + +## 1. Trace the whole layer handoff + +```text +llama model tensor placement + -> per-layer backend/buffer selection + -> graph scheduler split and input-copy allocation + -> ggml_backend_sched_compute_splits + user input: synchronous tensor_copy directly + ordinary split input: backend cpy_tensor_async directly + success: compute consumer after backend's ordering + false: synchronize producer + destination reuse + -> ggml_backend_tensor_copy + -> destination buffer cpy_tensor + success: stop + false: host allocation -> get -> set -> free +``` + +In `src/llama-model.cpp`, `get_layer_buft_list` selects a device with cumulative split weights and `std::upper_bound`; the repeating-layer loop and output-layer selection use it. Inspect actual selected backends rather than assuming the split percentages equal exact layer counts. Input/output placement, tensor overrides, CPU fallback, and the MTP graph may add transfers outside the three adjacent boundaries. + +In `ggml/src/ggml-backend.cpp`, the scheduler calls the backend interface directly. It does not always pass through `ggml_backend_tensor_copy_async`. It also has an immediate-copy path for `GGML_TENSOR_FLAG_INPUT`. Existing destination-use events and producer completion must remain valid. + +## 2. Exact change targets + +All CUDA symbols in this table are in `ggml/src/ggml-cuda/ggml-cuda.cu` unless stated otherwise. + +| Target | Existing behavior | Required future work | Primary failure risk | +|---|---|---|---| +| `ggml_cuda_init` | Presence of old flag enables all capable ordered physical pairs. | Parse/validate once; enable exactly the selected directions; publish immutable state. | Partial policy, accidentally enabled PCIe edge, recursive static initialization. | +| `ggml_cuda_get_physical_device` / `ggml_cuda_info` | Map logical backend IDs to runtime physical IDs. | Reuse after initialization; use local `info` during initialization. | Mixing host, runtime, virtual, and route-list indices. | +| `ggml_cuda_pool_vmm::alloc` | Old flag or NCCL build broadens peer descriptors. | Selective precedence; owner plus allowed readers for every new mapping. | NCCL macro silently widens policy; owner omitted; direction reversed. | +| `ggml_backend_cuda_cpy_tensor_async` | CUDA-to-CUDA copy with source event and destination wait. | Classify after validity/ownership checks; decline denied edge before work. | Copy enqueued before false; invalid lifetime or event ordering. | +| `ggml_backend_cuda_buffer_cpy_tensor` | Separate synchronous callback also calls peer copy. | Apply the same physical-pair exclusion. | Async fallback re-enters peer copy through this callback. | +| `ggml_backend_cuda_comm_init` and dispatch | Select NCCL/internal/meta reductions. | Explicitly reject unsupported restrictive collective use before setup. | Null/fallback is mistaken for rejection; external mapping bypass. | +| `ggml/src/ggml-cuda/common.cuh` | Internal device/context declarations. | Add only necessary internal policy representation/declarations. | Public ABI growth or unnecessary routing subsystem. | +| `ggml/src/ggml-backend.cpp` | Generic/scheduler fallback algorithms. | Review and test; initially no algorithm change. | Broad refactor obscures the focused CUDA change. | +| `docs/build.md` | Maintained build/config instructions. | Document the delivered option when it exists. | Docs advertise an unsupported binary. | +| Existing backend tests and test registration | Backend operation validation infrastructure. | Add focused policy/copy integration coverage using existing conventions. | Mock-only tests presented as physical GPU validation. | + +Locate the relevant code without relying on stale line numbers: + +```sh +git grep -n -E 'GGML_CUDA_P2P|cudaDeviceEnablePeerAccess|cuMemSetAccess|cudaMemcpyPeer|cudaMemcpyDefault|ncclCommInitAll' -- ggml/src +git grep -n -E 'get_layer_buft_list|LLAMA_SPLIT_MODE_TENSOR' -- src/llama-model.cpp +git grep -n -E 'ggml_backend_tensor_copy|ggml_backend_sched_compute_splits|GGML_TENSOR_FLAG_INPUT' -- ggml/src/ggml-backend.cpp +``` + +These are read-only inventory commands, not a guarantee that the search terms exhaust future CUDA APIs. + +## 3. Initialization recipe + +First resolve the physical runtime count and logical mapping. Parse the entire option and conflicting environment/build settings into temporary state. Reject invalid syntax/counts before CUDA peer side effects. Deduplicate undirected pairs, expand directions, and validate both capabilities. + +Then select the correct CUDA current device and enable each allowed edge exactly once. Handle only documented `cudaErrorPeerAccessAlreadyEnabled` as an expected idempotent case. Unexpected errors must preserve the exact pair/API/error in the diagnostic and stop this opt-in initialization. Restore the previously current device as appropriate. Do not suppress every CUDA error or reset devices. + +Do not expose a "configured" matrix as an "enabled" matrix before all requested directions succeed. Do not continue serving with half the requested policy. Policy is fixed for the process; no allocations or graph capture should precede its establishment. + +A plain pair list is not proof of NVLink. Log identities and selected edges; topology preflight must verify the known-good physical pairs. The first implementation does not need auto-discovery or NVML as a new mandatory runtime dependency. + +## 4. Allocation recipe + +Prepare per-owner access lists from initialized policy. The local owner always has READWRITE. A remote reader gets READWRITE only for its allowed reader-to-owner direction. Keep the original allocation, reservation, mapping, and handle-release lifecycle. + +Implement selective-mode precedence ahead of the legacy NCCL/P2P widening. Preserve the old code path exactly when the option is absent. Test growing a pool and remapping; an allocation-level assertion is stronger evidence than merely seeing the new environment variable in logs. + +Ordinary tensor storage uses the device allocation helper while VMM temporary storage comes from the pool. A raw `cudaMalloc` probe is not coverage of the pool. Do not add a managed/async-pool allocation as an incidental optimization. + +## 5. Copy recipe + +Maintain existing backend/buffer ownership checks before interpreting CUDA contexts. Resolve valid views through backing storage. Do not widen accepted layouts or treat every cross-backend copy as cross-physical. + +In the async callback: same physical means existing local D2D; selected distinct physical means existing peer copy and event handoff; denied distinct physical means false before enqueue. In the synchronous buffer callback: apply the identical denial before its peer API. That allows the existing generic fallback to run. + +The public async wrapper synchronizes both backends when a copy is declined. The scheduler instead synchronizes the producer and the destination copy-slot reuse event when available. Preserve each caller's existing contract rather than adding an unconditional global device synchronization. + +Returning false after allocating/enqueueing half of a staged copy risks duplicate transfers and buffer lifetime errors. The initial implementation avoids that by adding no async staging resources. In a later staging optimization, a partially submitted failure is an error to handle/drain, not a request to retry through another callback. + +## 6. Sensitive implementation details + +| Detail | Safe reasoning rule | +|---|---| +| Virtual aliases | Classify by backing physical GPU; preserve local D2D between different backend instances. | +| Event lifetime | Preserve source event record and destination wait; software call order does not prove CUDA completion timing. | +| Destination reuse | Honor scheduler copy-slot events, including pipeline copies and graph reuse. | +| Source lifetime | Do not release/reuse source data while a relevant copy still reads it. | +| Cancellation | Queued CUDA work persists; retain/drain resources rather than freeing staging immediately. | +| Views/layout | Test valid views with populated backing buffers, offsets, and guards; do not claim arbitrary malformed views are supported. | +| VMM permissions | Reader/owner direction differs from a casual source/destination naming convention. | +| Old flag | Presence, not a parsed boolean; conflicting old/new keys must be rejected. | +| Graph capture | No blocking fallback inside an already invalidated capture; inspect real transfer/capture boundaries. | +| Diagnostics | Counters/traces should be opt-in and bounded; no per-token logging overhead in production. | +| External libraries | Never assume another library respects this private policy. Guard unreviewed communicator use. | +| `NO_PINNED` | No hidden application pinned pool; driver-internal staging is a separate CUDA behavior. | + +## 7. Work sequence + +1. Inventory current code and instruction scope; preserve concurrent edits. Read the issue and results, including negative controls. +2. Implement internal policy and parser with focused pure tests, including all incompatibilities. Do not touch layer placement or request handling. +3. Gate runtime enablement and VMM access with one policy. +4. Gate both copy callbacks, retaining legacy/local/allowed paths. Exercise the generic and direct-scheduler entry points. +5. Add the collective rejection boundary; verify no transport setup occurs first. +6. Run native tests and profiling in [TESTING.md](TESTING.md). Add optimized staging only if its measured benefit is needed and its lifetime is proven. +7. Update maintained configuration docs and the narrowly scoped launcher dependency. Keep missing runtime evidence explicit. + +## 8. Managed-route dependency + +The reviewed proxy and bridge allow `GGML_CUDA_P2P` but not the proposed pair key. Before managed acceptance, a companion change must add the new key and matching bounded grammar/conflict checks to `src/llama_proxy/launcher_spec.py` and `bridge/server.py`, with tests in `tests/test_launcher_spec.py` and `tests/test_bridge_ports.py`. Recheck their current branch and instructions before that separate repository work. + +Keep the child authoritative for its actual CUDA count/capability. Preserve full UUID visibility, split/MTP/RPC arguments, `NO_PINNED`, envelope hashing, and revision-checked configuration writes. Check the effective inherited child environment for an old/new conflict. Route readback alone does not prove the binary recognizes the option; require its policy diagnostic and the intended binary identity. + +This docs PR makes none of those changes. It records the dependency so a CLI-only demonstration cannot be mistaken for end-to-end delivery. diff --git a/docs/cuda-p2p-allowlist/PREFLIGHT-RESULTS.md b/docs/cuda-p2p-allowlist/PREFLIGHT-RESULTS.md new file mode 100644 index 000000000000..35419f466cf5 --- /dev/null +++ b/docs/cuda-p2p-allowlist/PREFLIGHT-RESULTS.md @@ -0,0 +1,123 @@ +# Pre-implementation result: host-only source-branch probe + +## Verdict + +**PASS for the tested control-flow scope; PARTIAL for native feature feasibility.** No CUDA/NVLink/VMM/NCCL or Qwen success is claimed. The feature is not implemented in this PR. + +The owner requested a docs-only draft plus any feasible pre-code test in the current environment. A disposable C++ probe was compiled and executed outside the repository. Its final run completed at **2026-09-30T06:48:32Z**. + +## Actual environment and access limits + +```text +2026-09-30T06:38:12Z +Linux dc64cbaacd3c 6.18.44 #1 SMP Sat Sep 26 20:02:31 UTC 2026 x86_64 GNU/Linux + +GPU tools: + +GPU device nodes: + +CUDA driver library: + +Build tools: +cmake version 3.31.6 +c++ (Debian 14.2.0-19) 14.2.0 +git version 2.47.3 + +Runtime GPU probe: +find_library(cuda)= None +CUDA_LOAD_ERROR: libcuda.so.1: cannot open shared object file: No such file or directory +``` + +The container has neither `nvidia-smi`/`nvcc`, NVIDIA device nodes, nor a loadable CUDA driver library. A remote-terminal plugin search returned an available but uninstalled integration; no connected GPU host was established or operated. A full archive download failed in this container, so no full-source CUDA or CPU llama.cpp build was claimed. Source specimens came from successful connected GitHub reads. + +## Exactly what executed + +The source baseline is `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20`, with the CUDA and generic-backend blob identities in RESEARCH. Five function bodies were copied from those source reads and compiled with minimal test types: + +- `ggml_backend_buffer_copy_tensor`; +- `ggml_backend_tensor_copy`; +- `ggml_backend_tensor_copy_async`; +- `ggml_backend_cuda_buffer_cpy_tensor`; +- `ggml_backend_cuda_cpy_tensor_async`. + +Two scheduler blocks were compiled in fixture wrappers: ordinary split-input async/fallback and immediate user-input copy. The outer scheduler, graph split/allocation machinery, model layer assignment, real CUDA backend registration, peer initialization, VMM allocator, and NCCL were **not** executed. + +The CUDA APIs, memory accessors, physical mapping helpers, streams/events, and proposed policy decorators were software test doubles. Peer API doubles copied bytes with CPU memory operations and recorded the requested endpoints. The decorators hard-coded the proposed two pairs; they did not test a production environment parser. Original callback bodies were not edited to implement a feature. + +This is an executable source-specimen experiment, not an independent end-to-end CUDA implementation or a full scheduler test. It answers whether the existing copied branches route to host fallback when both callback seams decline. + +## Commands and final result + +These commands reproduce execution with the separately archived disposable probe files. The probe is not installed as a repository test target. + +```sh +c++ -std=c++17 -O1 -g -Wall -Wextra -Werror -fsanitize=undefined probe.cpp -o probe +./probe +c++ -std=c++17 -O1 -g -Wall -Wextra -Werror -fsanitize=address,undefined probe.cpp -o probe-asan +ASAN_OPTIONS=detect_leaks=1 ./probe-asan +``` + +Both final builds and executions returned exit code 0. Output was identical; AddressSanitizer/UndefinedBehaviorSanitizer emitted no diagnostics for this harness run. An initial harness build rejected an unused variable and misleading indentation under `-Werror`; those scaffold-only issues were corrected before the reported runs. This is not a full codebase sanitizer result. + +```text +HOST-ONLY CONTROL-FLOW PROBE; CUDA API, GPU memory, events, and callback policy are test doubles. +Executed: 5 source function bodies + 2 scheduler copy blocks; not whole ggml scheduler/model. +PASS directional_entrypoint_size_cases=1728 byte_mismatches=0 forbidden_peer_dispatches=0 +PASS same_backend_cases=24 +PASS physical_alias_cases=48 +PASS populated_buffer_view_cases=72 +PASS negative_control=async_only forbidden_cases_detected=48/48 (bytes alone still match in doubles) +PASS negative_control=buffer_only forbidden_cases_detected=24/48 (bytes alone still match in doubles) +PASS negative_control=ungated forbidden_cases_detected=48/48 (bytes alone still match in doubles) +TRACE excluded_scheduler: async_decline | backend_sync:1 | event_sync | buffer_decline | D2H:1 | H2D:2 +TRACE allowed_scheduler: peer_API:0->1 | record_copy_event | wait_copy_event +TRACE negative_async_only: async_decline | backend_sync:1 | event_sync | peer_API:1->2 | stream_sync +PASS synthetic_layer_handoff_chain=0->1->2->3 dispatch=PEER,HOST,PEER consumer_matches_cpu_oracle=true +PASS synthetic_layer_handoff_chain=3->2->1->0 dispatch=PEER,HOST,PEER consumer_matches_cpu_oracle=true +SUMMARY positive_cases=1874 negative_controls=3 hardware_verdict=PARTIAL reason=no_CUDA_driver_or_GPU +``` + +## Test population and negative controls + +The 1,728 directional combinations are 12 directed physical pairs x 6 entry paths x 8 sizes x 3 patterns/iterations. Sizes are 0, 1, 15, 16, 17, 4096, 65537, and 1048576 bytes. The additional 24 same-backend, 48 physical-alias, 72 populated-buffer-view, and 2 synthetic chain cases give 1,874 positive combinations. These are scenario combinations, not 1,874 independent production regression tests. + +All source/destination buffers were CPU allocations. Guard ranges and copied bytes matched. The zero-length cases check branch/no-data behavior in the doubles, not native CUDA zero-copy semantics. View coverage uses valid populated buffer pointers; malformed views are not covered. + +For each incomplete-policy negative control, 48 combinations cover the eight denied physical directions through six entry paths: + +| Deliberately incomplete seam | Detected forbidden peer-call cases | Meaning | +|---|---:|---| +| Async only | 48/48 | The original buffer callback can re-enter peer copy, including direct synchronous entry paths. | +| Buffer only | 24/48 | Async-success paths bypass the guarded buffer callback. | +| Neither | 48/48 | An ungated peer-call path is not selective. | + +The negative controls still copy matching CPU bytes. The detector therefore checks forbidden path selection independently, rather than calling every byte-correct result a success. It does not recreate or measure the host's PCIe corruption. + +## Interpretation for the intended layer chain + +In the synthetic forward and reverse chains, the same source scheduler blocks selected PEER, HOST, PEER and the final CPU consumer matched an independently transformed expected value. This supports the proposed double-gate design at those branch seams. + +It does **not** establish that llama.cpp's full graph allocator will place these operations on four real GPUs, that CUDA events complete in the required order, that NVLink is physically used, or that the actual VMM pool has denied mappings. It also does not prove equality with the legacy implicit staging path's performance. + +The strongest valid conclusion is: **the existing generic fallback can be reached without replacing the layer scheduler, but both CUDA copy callbacks must be gated.** Native experiments B/C from PREFLIGHT and candidate tests from TESTING remain required. + +## Reproducibility manifest + +The original probe files and raw logs are delivered as a separate conversation artifact, not executable additions to this docs PR. The hashes below identify that local artifact content; they are not hashes of complete upstream source files. To recreate the experiment without that artifact, use the five pinned source bodies and two scheduler blocks listed above, explicit doubles, the independent matrix and negative controls described here, and preserve the evidence boundary. + +```text +6f1190e459dfa83e655e267f0829992dc1e9880e65466bdeadb8fc0bda94d6da environment.txt +e58e8d0228b4c3e7038bf77c7a7d2f866c0792d2da98b472de5897d785d0a45a probe.cpp +9fe0e8380dd9799612afeca7c0e90d867d8963621e23c0c2318c56f0fc131e49 source_excerpts.inc +fc6c755f81d77aaee830a737e08803853ac45aefa907596ef632a16a66c1b18e scheduler_excerpt.inc +372766983e43a0952fc65a4ec66aa66e5fe42b62371807db08a639aa20d56cf4 scheduler_input_excerpt.inc +e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 build.log +1d0565adac46281cc2cc3ed69327465ded24581d1fa85a45965c2b3516ed616d probe-output.txt +e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 build-asan.log +1d0565adac46281cc2cc3ed69327465ded24581d1fa85a45965c2b3516ed616d probe-asan-output.txt +5ca3fff463f816bef183bd3c2470b596e5b0c2e88d1925008051cbab42161140 run-finished-utc.txt +``` + +## Unrun work + +Native CUDA initialization and directional enablement; real NVLink/PCIe transfers; all actual allocator permissions; whole ggml graph execution; asynchronous race/capture/cancellation behavior; candidate parser and collective rejection; launcher integration; and four-GPU Qwen/MTP performance are unrun. No source patch, deployment, restart, route edit, or host-setting change was made by this experiment. diff --git a/docs/cuda-p2p-allowlist/PREFLIGHT.md b/docs/cuda-p2p-allowlist/PREFLIGHT.md new file mode 100644 index 000000000000..eeaf1394754f --- /dev/null +++ b/docs/cuda-p2p-allowlist/PREFLIGHT.md @@ -0,0 +1,88 @@ +# Pre-implementation experiment: falsify the layer-transfer design + +## Question and evidence levels + +Can the existing llama.cpp layer handoff retain the selected CUDA peer path and deliberately reach the normal generic host fallback for excluded physical pairs, without dropping GPUs or changing layer placement? + +Three experiments answer different parts. Do not combine their verdicts into a false hardware PASS. + +| Experiment | Can run before feature code? | What it establishes | +|---|---|---| +| A. Extracted source control flow with test doubles | Yes; executed locally. | Existing callback/fallback branches can implement the proposed dispatch. | +| B. Four-GPU CUDA/VMM transport microprobe | Yes, with a disposable native harness. | These physical pairs, explicit host fallback, and restricted VMM descriptors work on the actual host. | +| C. Whole ggml scheduler with actual CUDA backends | Yes using clearly identified test-only callback seams, or later using candidate code. | Real graph allocation, layer-like partitioning, producer/consumer ordering, and transfer paths work together. | + +The eventual four-GPU Qwen/MTP acceptance remains separate. No unchanged binary implements an environment key it does not recognize. Test wrappers or standalone allocations must never be described as the production feature. + +## A. Local branch experiment + +[Results and limitations](PREFLIGHT-RESULTS.md) record the executed host-only experiment. It compiles the existing generic copy functions, both CUDA copy callback bodies, and two exact scheduler blocks with minimal types and mocked CUDA/memory/event helpers. External callback decorators supply the proposed deny decision without changing those source bodies. + +Test six entry paths: generic async, generic sync, scheduler ordinary input with reuse event, scheduler ordinary input without reuse event, and scheduler user input with/without reuse event. Exercise all 12 ordered distinct-device pairs, valid populated-buffer views, local copies, and virtual aliases. + +Negative controls intentionally apply only one decorator, or neither. Detect forbidden peer-call dispatch even if the software-copy output matches. This prevents a bytes-only checker from approving a routing error. + +This does not execute full graph splitting, real CUDA event timing, VMM, initialization, NCCL, a model, or the proposed parser. Those are explicitly not established by A. + +## B. Native four-GPU microprobe, without a feature patch + +This is a test specification, not an existing executable or a command that was run in the ChatGPT container. Use a disposable process and small allocations; do not enable unsafe PCIe pairs even for a negative control. + +### Preconditions and provenance + +Record the exact source/build/toolkit/driver identities, full GPU UUIDs, PCI identities, and physical topology. Verify four GPUs and the two intended active NVLink pairs under the process's effective visibility. Preserve host settings. Do not change BIOS, IOMMU, ACS, driver, RPC, or a running production child. + +Use the existing compatible V100 toolchain; documentation from a newer CUDA release is not an upgrade instruction. A missing GPU/library produces PARTIAL with the actual probe error, not PASS or a fabricated bandwidth result. + +Capture selected runtime environment keys without secrets. The old global P2P variable must be absent. Do not rely on the unimplemented new environment key. The disposable harness explicitly enables the four selected directions through the CUDA runtime. + +### Exact operation sequence + +1. Call `cudaGetDeviceCount`, enumerate UUID/PCI identities, and validate the selected mapping. Check `cudaDeviceCanAccessPeer` in both directions of selected pairs only as capability evidence. +2. Enable only 0->1, 1->0, 2->3, 3->2 in their proper current-device contexts. Fail on unexpected errors. Never enable a denied direction, and never use a sample that loops over all capable pairs. +3. Allocate independent source/destination buffers on all four devices with `cudaMalloc`. Fill each source with a pattern encoding source, destination, offset, and iteration; prefill destination with another sentinel. +4. For a selected edge, invoke `cudaMemcpyPeerAsync` on the producer stream; record its completion event; make the destination stream wait; enqueue a destination-side consumer/checksum and verify output on the host. +5. For an excluded edge, explicitly wait for the producer and protect destination reuse, then D2H to ordinary host memory and H2D to destination. Synchronize completion before validation. Do not call any cross-device peer/default/D2D API on that edge. +6. Repeat the same matrix using separate native VMM buffers: DEVICE allocation properties, granularity-rounded `cuMemCreate`, reserve/map, owner plus permitted reader descriptors, then `cuMemSetAccess`. Query each physical device's flags with `cuMemGetAccess`. This tests the VMM concept, not the actual ggml pool. +7. Run the complete forward chain 0->1->2->3 and reverse chain 3->2->1->0 with a deterministic producer transformation and consumer validation at every boundary. Run nonadjacent edges too. +8. Synchronize owned work and release only the harness's resources. Exit the process. Do not reset devices to perform cleanup. + +Use bounded sizes: zero-handling separately, 1, 15, 16, 17, 4096, 65537, 1048576, and 16777216 bytes. For VMM allocate at least the required granularity even for a small logical payload. Compare every logical byte plus guard ranges. Add repeated and alternating transfers; cap peak allocation and iteration counts. + +### What counts as success + +All tested bytes match, no invalid access/timeouts, and instrumentation shows no denied enable, grant, or peer-copy API. Both directions of each selected pair must be correct. Report one-directional throughput separately from bidirectional aggregate throughput, after warmup, without converting it to predicted tokens/s. + +For cross-device end-to-end timing, use host monotonic timing around a completed operation or correctly scoped device timings. Do not subtract CUDA event timestamps from different devices. Preserve the route's `NO_PINNED` choice in the first comparison; a pinned-host variant is a separately named experiment. + +If B passes, the conclusion is "native transport/VMM mechanism is feasible for this host," not "llama.cpp already implements it." + +## C. Whole-scheduler native seam test + +### Build the real graph path + +Use actual ggml CUDA backends plus a CPU backend last, consistent with the scheduler contract. Construct a small four-stage graph using exactly representable FP32 values, for example each stage computes `y = 2*x + bias_i` with bounded integer inputs. Put each stage's operations and its bias on the intended GPU. Use `ggml_backend_sched_set_tensor_backend` before graph allocation. + +Run `ggml_backend_sched_alloc_graph` and `ggml_backend_sched_graph_compute`; inspect `ggml_backend_sched_get_n_splits`, each node's chosen backend, and tagged transfer endpoints. Require that the four compute stages actually run on four distinct CUDA backends. A CPU fallback, a graph collapsed to one device, or only two devices is not a pass. + +Trace actual inter-stage tensors. Expected adjacent activation handoffs are PEER/HOST/PEER. Do not infer this merely from their intended assignment or count unrelated CPU input/bias transfers as peer violations. Compare each stage and final output against an independently calculated CPU oracle. + +Repeat reverse order, a nonadjacent cross-group edge, both input-copy paths, valid views, and repeated graph execution. Test the scheduler's pipeline-copy setting both ways; it is not the same setting as server request `--parallel 4`. + +### Before feature implementation + +A native test harness can decorate the existing backend `cpy_tensor_async` and buffer `cpy_tensor` interfaces, delegating selected/local edges to their original callbacks and declining excluded edges. It must cover buffers allocated by the scheduler, not only manually created tensors. Keep original interface entries and ownership intact. No process-global callback replacement or hot mutation during work is acceptable. + +This uses internal APIs and requires a version-specific disposable adapter; none is claimed implemented here. It proves a seam, not a supported public customization API. The current VMM allocator's NCCL-driven widening remains a separate problem: a callback-only seam is NOT validation of the full VMM policy. Either instrument an explicitly identified test-only VMM seam in a disposable source copy, or mark actual-pool permission coverage PARTIAL. Do not quietly rebuild without NCCL/VMM and claim production-equivalent success. + +Do not use `LD_PRELOAD` as the primary method: intercepted API visibility and linkage cannot by themselves prove every backend path was covered. Do not run a patched production server merely to test this seam. + +### After feature implementation + +Remove test-only policy decorators and run C directly against the exact candidate. Observe native enable/grant/copy operations and VMM pool behavior, not only debug statements that report intended policy. The experiment must still detect an async-only or buffer-only regression through safely mocked negative controls, not by enabling the known-bad PCIe hardware path. + +## Decision rule before coding + +A passing A with B/C unavailable means PARTIAL overall: the source path is suitable, but native integration is unconfirmed. A passing B plus actual-scheduler C is strong evidence to proceed with the bounded feature; it is still not final Qwen acceptance. Any mismatch or forbidden-path use invalidates the relevant candidate. Missing evidence is not an implementation defect and must not be converted into a hardware failure claim. + +The current run stops at A because CUDA is absent. The next native experiment and its required evidence are completely specified above; there is no claim that a real GPU test ran in this environment. diff --git a/docs/cuda-p2p-allowlist/README.md b/docs/cuda-p2p-allowlist/README.md new file mode 100644 index 000000000000..1ff7599fc5ee --- /dev/null +++ b/docs/cuda-p2p-allowlist/README.md @@ -0,0 +1,42 @@ +# Selective CUDA P2P: coder preparation pack + +Status: documentation-only draft. No backend feature, executable test, build configuration, workflow, launcher, or production change is included in this PR. + +Tracks [llama.cpp-fork #3](https://github.com/novkien/llama.cpp-fork/issues/3). The starting design is the [owner-requested implementation plan](https://github.com/novkien/llama.cpp-fork/issues/3#issuecomment-5904416014). These documents organize that plan for implementation and add an explicitly limited pre-implementation experiment. They do not mark the proposal approved or the feature deployed. + +## Intended outcome + +Keep all four GPUs in layer-split inference. With the verified process mapping, allow peer transfers only between CUDA0/CUDA1 and CUDA2/CUDA3. Every other distinct local CUDA pair must take an explicit host-memory fallback. Preserve the model, layer distribution, MTP draft placement, context, parallelism, and RPC configuration from the issue. + +```text +Target layer groups: CUDA0 -> CUDA1 -> CUDA2 -> CUDA3 +Transfer choice: PEER HOST PEER +Reverse transfers: PEER HOST PEER +``` + +The proposed `GGML_CUDA_P2P_PAIRS=0-1,2-3` option does not exist in the reviewed backend. Merely setting it on that binary proves nothing. + +## Read in this order + +| Document | What the coder gets | +|---|---| +| [DESIGN.md](DESIGN.md) | Exact policy contract, identity mapping, normal-path clarification, supported scope, and invariants. | +| [IMPLEMENTATION.md](IMPLEMENTATION.md) | Symbol-level change map, source flow, sensitive code, implementation sequence, and integration dependency. | +| [RESEARCH.md](RESEARCH.md) | Primary-source CUDA/NCCL knowledge and what each source does and does not establish. | +| [PREFLIGHT.md](PREFLIGHT.md) | Experiments to falsify the design before feature implementation, including a native four-GPU seam test. | +| [PREFLIGHT-RESULTS.md](PREFLIGHT-RESULTS.md) | Actual local experiment, transcript, negative controls, and limits. | +| [TESTING.md](TESTING.md) | Candidate regression matrix, real layer-graph validation, Qwen acceptance, and completion checklist. | + +## Current evidence + +The accessible default branch was rechecked at `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20` on 2026-09-30. This is a source baseline, not the installed host binary. The issue's older `2138591d...` reference previously failed to resolve; do not treat it as a usable implementation base without checking it. + +A disposable host-only C++ probe executed five copied source function bodies and two scheduler copy blocks with software test doubles. It passed 1,874 positive scenario combinations and detected the intended incomplete-gate negative controls. It did not compile the whole scheduler, execute a CUDA API, or run a model. Native feasibility remains PARTIAL because this environment has no CUDA driver/device. See the results document before quoting any number. + +## Working boundary + +Continue development on this draft's branch when implementation is requested. Read the current checkout's applicable instructions and reconcile newer changes before editing. Do not overwrite another coder's work or rewrite a shared branch to match this dated baseline. + +Only Markdown files in this folder are introduced. The temporary local probe is evidence gathering, not a shipped implementation. No merge, production restart, BIOS/IOMMU/ACS change, or automatic issue closure is authorized by this docs PR. Link the issue with `Refs #3`, not a closing keyword. + +The first implementation can stay small: one policy, filtered peer initialization, filtered VMM access, both copy callbacks gated, and existing host fallback. A new asynchronous staging subsystem is not a prerequisite unless measured performance requires it. diff --git a/docs/cuda-p2p-allowlist/RESEARCH.md b/docs/cuda-p2p-allowlist/RESEARCH.md new file mode 100644 index 000000000000..bc8de0c7ea1e --- /dev/null +++ b/docs/cuda-p2p-allowlist/RESEARCH.md @@ -0,0 +1,52 @@ +# Research and source map + +Reviewed for this documentation pack on 2026-09-30. API documentation establishes contracts; fork source establishes call sites; recorded host experiments establish only their tested conditions. None proves an unimplemented feature works on production. + +## Fork source baseline + +Base commit: `526c43b8f7dfea9032e9f35e7a1be9183ca7cc20` on the accessible fork's `master`. The existing implementation PR #2 concerns RAM prompt-cache retention, not selective P2P. Do not build this work on that unrelated feature branch or include its changes accidentally. + +| Reference | Verified relevance | +|---|---| +| [S1: CUDA backend](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/ggml/src/ggml-cuda/ggml-cuda.cu) | Peer enable loop, VMM descriptors, ordinary allocation, both copy callbacks, communicator setup. Blob `b50d05d5f76fb7010d921b8e71bd4ff83af5b23c`. | +| [S2: Generic backend/scheduler](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/ggml/src/ggml-backend.cpp) | Async/sync fallback, direct scheduler callback, user-input immediate copy and reuse events. Blob `20bf965017e31338f579c1d8d1a64b0ab56c78ee`. | +| [S3: Model placement](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/src/llama-model.cpp) | `get_layer_buft_list`, cumulative split lookup, repeating and output layer assignment. Blob `ab5e744b5d1076c2ad9c4700051ae3bdb9edb024`. | +| [S4: Backend public API](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/ggml/include/ggml-backend.h) | Real scheduler assignment/allocation/compute/synchronization APIs; reset invalidates allocated tensors. | +| [S5: CUDA context](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/ggml/src/ggml-cuda/common.cuh) | Internal device/context declarations and VMM compile guards. | +| [S6: Existing tests](https://github.com/novkien/llama.cpp-fork/blob/526c43b8f7dfea9032e9f35e7a1be9183ca7cc20/tests/CMakeLists.txt) | Existing backend test target and registration patterns to reuse. | + +Read the applicable repository instructions. This pack adds no competing AGENTS policy and exports no ChatGPT project canonical into runtime files. It is limited to this feature's design and evidence. + +## Primary API knowledge + +| Source | Contract used | What not to infer | +|---|---|---| +| [N1: Runtime peer access](https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__PEER.html) | Enablement is directional; capability query and expected already-enabled error are documented. | Capability does not establish correctness, bandwidth, or NVLink topology. | +| [N2: CUDA multi-GPU guide](https://docs.nvidia.com/cuda/cuda-programming-guide/03-advanced/multi-gpu-systems.html) | Cross-device event waits are supported; event recording/timing has device association constraints; PCIe/IOMMU caveats matter. | A software trace of wait calls does not establish real GPU completion order. | +| [N3: Volta NVLink section](https://docs.nvidia.com/cuda/volta-tuning-guide/index.html#nvlink-interconnect) | Transfers between NVLink-connected endpoints use NVLink transparently within CUDA. | An allowlist is not a cable selector or continuous link-health monitor. | +| [N4: VMM driver API](https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__VA.html) and [VMM guide](https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/virtual-memory-management.html) | Allocation location, mapping, and per-device access are separate operations. | Filtering runtime enable calls alone does not restrict all VMM allocations. | +| [N5: Synchronization behavior](https://docs.nvidia.com/cuda/cuda-runtime-api/api-sync-behavior.html) | Pageable host copies may synchronize despite the Async suffix. | Explicit host fallback necessarily has the same performance as implicit runtime staging. | +| [N6: NCCL environment](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-p2p-level) | `NVL` is a NCCL P2P topology cutoff; direct-user-buffer access has a separate control. | NCCL settings apply to arbitrary llama.cpp CUDA copies or to every private endpoint allowlist. | +| [N7: Stream-ordered allocator](https://docs.nvidia.com/cuda/archive/13.1.1/cuda-programming-guide/04-special-topics/stream-ordered-memory-allocation.html) | Async memory pools have their own access controls. | A future allocator can be substituted without revisiting policy coverage. | + +The retrieved documentation versions need not match the installed V100 toolchain. Verify applicability during native testing; do not upgrade the host just to match a web document. + +## Upstream implementation context + +[Merged upstream PR #21891](https://github.com/ggml-org/llama.cpp/pull/21891) moved NCCL communicator management into a separately initialized context. This supports checking actual communicator use rather than treating a NCCL build as proof of all-device communicator creation. + +[NVIDIA NCCL P2P source](https://github.com/NVIDIA/nccl/blob/master/src/transport/p2p.cc), inspected during the issue research, contains same-process peer-enable and VMM/IPC mapping paths. It motivates a guard around unreviewed collective use. The moving link is a reference, not a claim about the host's installed NCCL revision. + +## Recorded host evidence, not repeated here + +[Issue #3](https://github.com/novkien/llama.cpp-fork/issues/3) and [the direct copy result in proxy #470](https://github.com/novkien/llama-proxy/issues/470#issuecomment-5904023781) report zero mismatches for CUDA0->1 and CUDA2->3 with peer access, but severe mismatches for CUDA1->2 with peer access. Access was enabled symmetrically; the published data-transfer measurements cover those three directions only. + +[Rollback evidence](https://github.com/novkien/llama-proxy/issues/470#issuecomment-5903851666) records recovery to about 45 tokens/s with the old flag removed. The exact IOMMU/ACS/switch/driver subcause is not isolated. No new host benchmark is claimed in this docs PR. + +## Integration reference + +[Proxy PR #472](https://github.com/novkien/llama-proxy/pull/472) demonstrates matching proxy/bridge validation for the old P2P flag. The previously read launcher and bridge lacked the new pair key. Recheck their live source before a companion change; adding docs here does not change their allowlists or deployed state. + +## Design conclusions versus facts + +The fail-closed parser, pair matrix, v1 collective guard, initial explicit host fallback, and decision to defer optimized staging are proposed engineering choices. Primary documentation makes them implementable; it does not mandate this exact UI or prove its speed. The local probe establishes reachability of the existing fallback under test-supplied guards. Native transport, actual VMM pool permissions, graph timing, and Qwen performance remain to be tested. diff --git a/docs/cuda-p2p-allowlist/TESTING.md b/docs/cuda-p2p-allowlist/TESTING.md new file mode 100644 index 000000000000..9ad9dd87f6d0 --- /dev/null +++ b/docs/cuda-p2p-allowlist/TESTING.md @@ -0,0 +1,93 @@ +# Candidate regression and acceptance plan + +These are requirements for future implementation. The docs-only PR satisfies documentation delivery, not the unchecked runtime requirements below. The pre-code test and actual local results are in [PREFLIGHT.md](PREFLIGHT.md) and [PREFLIGHT-RESULTS.md](PREFLIGHT-RESULTS.md). + +## 1. Evidence must identify the candidate + +Record source commit plus any local diff, compiled binary identity, architecture/build flags, CUDA driver/runtime/compiler, NCCL library version when relevant, full GPU UUID/PCI mapping, model identity, and exact launch configuration. Freshly resolve the intended remote/base branch; a previous SHA is not proof of current deployment. + +Separate same-binary option-off/option-on tests from old-production-binary/new-candidate comparisons. Otherwise a source-version difference is confounded with P2P selection. Build/test results follow actual candidate bytes; rerun affected checks after relevant edits. + +## 2. Focused test matrix + +| Area | Required cases | Failure detector | +|---|---|---| +| Parser | Empty, signs, overflow, garbage, missing endpoint, duplicate/reversed pairs, decimal leading zeros, whitespace, count changes. | Complete parse or explicit error; zero enables before validation completes. | +| Policy | All 12 distinct directed edges, diagonal, physical aliases, non-transitivity, all denied cross-group edges. | Independent expected matrix, not production classifier reused as its own oracle. | +| Initialization | Both directions supported, one direction unsupported, expected already-enabled, unexpected failure. | No partial-serving state; exact failing pair/API; no forbidden enable. | +| VMM | Owner, selected readers, excluded readers, growth/remap, VMM off. | Native `cuMemGetAccess` and allocation/operation traces, plus correct consumers. | +| Copy dispatch | Both CUDA callbacks, generic sync/async, scheduler direct async, scheduler input sync. | Peer/host call counts and bytes by actual physical pair; no bypass. | +| Data | Odd/zero sizes, alignment/granularity boundaries, valid views, guards, different patterns and iterations. | Byte-for-byte raw copies; exact or justified numeric oracle for graph ops. | +| Lifecycle | Producer work, consumer waits, reuse slots, reverse traffic, graph replay, cancel, teardown. | Native completion correctness, no use-after-free/hang or unbounded memory. | +| Scope | NCCL build on/off, actual collective attempted, managed mode conflict, peer-copy-disabled build, HIP/MUSA compile guards. | Unsupported selective combinations rejected before unsafe work; legacy unaffected. | +| Integration | Route and bridge validation, launch hash/environment, binary recognizes option. | Exact readback plus effective child state and native policy evidence. | + +For the current `0-1,2-3` matrix, inspect eight denied directions, not only 1->2. Independently count expected edges. Keep same-physical aliases local even across different backend instances. + +For raw-copy tests, initialize destination sentinels each iteration and validate guards. Bytes alone are not sufficient to prove selected transport: the local negative controls deliberately demonstrate this limitation. + +## 3. Real layer-like graph + +Run experiment C from PREFLIGHT against real backends with no policy decorators. Force and verify four compute stages across four physical GPUs. Include independent expected numeric results and tagged per-boundary transfers. Check forward/reverse and nonadjacent edges. + +Actual VMM scratch pools must be exercised separately with operations that allocate temporary storage; a four-stage add/scale graph may not use the CUDA VMM pool at all. Record observed pool allocations and permission queries rather than assuming coverage from a successful graph. Avoid adding a broad graph-optimizer rewrite solely for this test. + +Exercise scheduler events and copy-slot reuse without replacing them with one global device synchronization. Test input flags separately because they select a different entry path. A host-only excerpt probe cannot establish asynchronous timing, capture compatibility, or race freedom. + +## 4. Optional staging optimization + +Only add a reusable pinned staging path if profiling justifies it. Respect `GGML_CUDA_NO_PINNED`; keep any pinning experiment separately named. Verify pool cap, allocation failure before submission, capacity pressure, large-copy chunking, slot/event generation reuse, opposite-direction dependencies, cancellation, and teardown. + +The source data must outlive the D2H read; the staging slot must outlive H2D completion; the destination must not be consumed before completion. A source-side completion observer must retain the documented copy lifetime semantics. Avoid cycles when adding cross-stream waits. Return false only when nothing was submitted. + +No FP16/BF16 compression, global graph disable, or global synchronization may conceal a correctness defect. Test data transfer byte-for-byte. + +## 5. Four-GPU Qwen acceptance + +Preserve the exact target workload from issue #3, after resolving the full current launch configuration: + +```text +--device CUDA0,CUDA1,CUDA2,CUDA3 +-sm layer +-ts 28,28,29,15 +--spec-type draft-mtp +--spec-draft-device CUDA3 +--spec-draft-n-max 3 +--spec-draft-n-min 1 +-c 400000 +--parallel 4 +``` + +Keep the actual model, UUID order, RPC configuration, host settings, other environment values, and prompt-cache/warmup policy constant. Do not replace acceptance with two GPUs, a smaller context/model, MTP off, or a different split. Diagnostic target-only/small-model runs are useful but separately labeled. + +Use fresh children for baseline -> selective -> rollback. Do not add the already-broken global-P2P condition. First reproduce the bounded deterministic 256-token request; then run several repeats and the configured four-slot workload. Recover the original prompt from the recorded evidence; do not invent a prompt and compare its acceptance counts to the historical run as though they were identical. + +Record individual outputs/token IDs when available, finish reasons, draft generated/accepted counts, target verification work, native decode and prompt throughput, TTFT, client elapsed, copy counts/bytes/time, host-memory usage, clocks/temperature/competing load, errors, cancellation, and cleanup. Report all repeats and their median/range rather than one selected run. + +The historical healthy reference is approximately 44-45 decode tokens/s and 159/285 accepted draft tokens for the recorded request. These are context-specific observations, not universal required counts. Use a fresh healthy same-binary baseline. Establish baseline variability before interpreting a difference. Raw-copy mismatches always fail; large unexplained output/draft divergence is not dismissed as normal nondeterminism. + +A transfer bandwidth gain need not improve token generation if those copies occupy little runtime or the explicit host fallback adds more cost. Report correctness, selected transport, and performance separately. Native correct routing with unresolved performance gain remains PARTIAL for production promotion. + +## 6. Publication and rollback + +A documentation PR or implementation merge does not close the issue. No production action is included in the current draft. When native execution is authorized, use existing managed drain/unload/reload controls rather than killing a busy bridge child. Preserve other routes and rollback to the verified healthy binary/environment. Remove new/old P2P keys as required; setting the old key to `0` is not disabling it. + +The companion proxy/bridge change needs matching validation and launch-hash tests. Verify the effective child, not only route JSON. An old binary ignoring the new key cannot pass activation. + +## 7. Completion checklist for the eventual implementation + +- [ ] Exact delivered candidate and runtime identities recorded. +- [ ] Parser, mapping, conflict, and fail-closed initialization tests pass. +- [ ] Both copy callbacks and VMM allocator obey one immutable pair policy. +- [ ] No forbidden enable, mapping grant, or peer-copy dispatch on any of the eight denied directions. +- [ ] All 12 directed native copy cases pass, including reverse NVLink transfers and relevant buffer families. +- [ ] Whole-scheduler four-GPU layer-like graph passes with native backend assignment and independent output oracle. +- [ ] Actual VMM pool coverage, producer/consumer ordering, reuse, cancellation, and teardown are evidenced. +- [ ] NCCL build remains usable for layer mode; unsupported selective collectives fail before initialization. +- [ ] Legacy behavior without the new key and relevant build/platform guards are checked. +- [ ] Required managed-launcher dependency and documentation match the implementation. +- [ ] Unchanged four-GPU Qwen/MTP baseline/selective/rollback and four-slot results recorded. +- [ ] No correctness/performance gap is hidden by workload downgrade or mock-only PASS. +- [ ] Owner-reviewed acceptance outcome is explicit; issue remains open while required evidence is absent. + +Use PASS for an actually satisfied tested scope, FAILED for an observed violation, and PARTIAL for incomplete/undecidable coverage. Do not replace unrelated tool/process status fields with this verdict vocabulary.