Repository navigation
fix(llama): size llama.cpp's RAM prompt cache to the host (16 GB WSL OOM kills) - #6769
Open
Lightheartdevs wants to merge 3 commits into
Open
Lightheartdevs wants to merge 3 commits into
Lightheartdevs wants to merge 3 commits into
Conversation
…p's 8 GiB llama.cpp b9014 keeps earlier prompts (with their context checkpoints) in a host-RAM prompt cache capped at 8192 MiB by default. No ODS memory check counts it: the container limit, the VRAM fit and the catalog profiles size weights, KV cache and checkpoints only, and only CPU runtime profiles set LLAMA_ARG_CACHE_RAM. In a 16 GB WSL VM the kernel OOM-killed llama-server at about 10 GB anon RSS three times, dropping Pixel requests while the model reloaded. When neither the model's runtime profile nor .env sets it, the Linux installer (phase 06), the Windows installer and dashboard model activation now write a size for the Docker llama-server: a quarter of the memory left after 6 GiB for the rest of ODS (Docker's memory when smaller than the host's), at most a quarter of the llama-server container limit, at least 512 MiB. At 8192 MiB or more the key stays unset, so large hosts keep llama.cpp's default. An existing value is kept; a profile's value wins. Lemonade (AMD) and native macOS are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A live sample on the 16 GB WSL laptop (Qwen3.5-9B, 64K, b9014) showed that a Pixel conversation is cached with all its context checkpoints: a 48,465-token prompt with 32 checkpoints took 2,464 MiB, a 52,019-token one 2,523 MiB. A quarter of the headroom (2304 MiB) is below one such entry, so the next short prompt saved after it evicts it (b9014 evicts oldest-first once the total exceeds the limit). A third of the memory left after 6 GiB gives 3072 MiB in that VM, which is also a quarter of the 12G container limit of its runtime profile: one full agent conversation plus a few short prompts. Hosts from about 30 GiB keep llama.cpp's default unless their container limit is smaller. Also keeps the Windows test's new assertions away from the lines #6617 changes, and restores LF endings in the new Python test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sensitive Files DetectedTrigger: Security-sensitive files detected: ods/installers/ Files flagged: Extra human review is recommended for this PR. Claude Code Review | Sensitive File Detection | ~$1.50 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
llama.cpp b9014 keeps earlier prompts, with their context checkpoints, in a host-RAM prompt cache that may grow to 8192 MiB (
--cache-ram,LLAMA_ARG_CACHE_RAM). No ODS memory check counts it. The container limit, the VRAM fit and the catalog profiles size only weights, KV cache and checkpoints. Only CPU runtime profiles, plus one 4B NVIDIA profile, setLLAMA_ARG_CACHE_RAM. On the 16 GB WSL laptop, the kernel OOM-killed llama-server three times with the default cache.This PR makes ODS size the cache for the Docker llama-server when neither the model's runtime profile nor
.envalready sets a size:docker infoMemTotal. Inside WSL both readings come from the VM; under Docker Desktop the VM reading is the smaller one.docker statsshows 3.6 GiB for the other 22 containers.LLAMA_SERVER_MEMORY_LIMIT. It comes from the profile, the owner or the NVIDIA default; failing those, the backend's compose default applies (nvidia 64G, cpu 6G, intel/arc 24G). Capping the cache at a quarter of L leaves three quarters of the container for weights, KV cache and checkpoints..env, then the computed default. An owner's value is never overwritten.env-generator.ps1), and dashboard model activation (host agent). Activation computes a size only when the key is absent..envdoes not change.OOM evidence (laptop, read-only)
Host and configuration
docker inforeports MemTotal 16,469,123,072 B.nvidia-8gb-64k-q8-kv, imageserver-cuda-b9014.docker inspect: Memory 12G, MemorySwap 24G. The command line has no--cache-ram, andLLAMA_ARG_CACHE_RAMis passed through unset. The server log readsprompt cache is enabled, size limit: 8192 MiB.Kernel log (
journalctl -k, EDT; UTC is +4 h). Every kill hasconstraint=CONSTRAINT_NONE ... global_oom: the VM ran out of memory. The container's 12G cgroup limit was never reached.The 03:36:43 EDT kill (07:36:43Z) made two Pixel sweep prompts fail with "pixel request rejected": the router got a 503 while the model reloaded. From the kernel's task dump at that kill:
The largest of the others were python3 (1,920 MiB), litellm (625), hermes (385), openclaw (324) and text-embeddings (233). The dump reported
Free swap = 0kB.Why the prompt cache is what grows
b9014 source (
tools/server/server-task.{h,cpp},server-context.cpp):server_prompt::size()counts both.update()evicts the oldest entries while the total exceeds the limit, but always keeps at least one entry.Live sample on the laptop. The model restarted at 08:18Z and a UI phase and sweep were running. I sampled the llama log
cache stateline and the container cgroup'sanononce a minute, read-only:The cache is most of llama-server's anonymous memory, and it grows toward the 8192 MiB cap as distinct Pixel, Hermes and Open WebUI prompts arrive. The rest is the active slot's checkpoints plus about 0.5 GiB of buffers. At the growth rate in the table, the running laptop is on course for another kill once the cache nears 8 GiB.
Entry sizes from the same log:
Under this memory pressure, saving a 2.5 GiB entry took 9.7 s once and 41.6 s another time (
prompt cache update took 9652.90 ms/41586.09 ms).Arithmetic at the kill. The fixed host buffers in this configuration are about 0.2–0.6 GiB, and 32 checkpoints are 1.57 GiB. That leaves at least about 7 GiB of the 9.6–10.1 GB anon RSS for the prompt cache. The killed containers' own logs are gone because the container was recreated, so this part is inferred.
On the 16 GB Mac mini, the same default cache reached 8,017 MiB (#6757).
What happens today (before this PR)
LLAMA_ARG_CACHE_RAMwhenqwen3.5-4b nvidia-8gb-64k-q4-kv(2048). The 8 GB-GPU profiles for 9B and Ministral setLLAMA_SERVER_MEMORY_LIMIT=12Gor nothing, and no cache size.docker-compose.base.yml)env-generator.ps1).envvalue was dropped on rerun.--cache-ramonly when.envsets it.The NVIDIA container default is memory − 3 GiB (<16 GiB) or − 4 GiB (12G on this VM). That is above the VM's real headroom, and compose gives the container 2× that in swap, so the cgroup limit never bounds llama-server here.
What each fleet host class gets
LLAMA_ARG_CACHE_RAM.envchange)--cache-ram 0(with--parallel 2 --kv-unified --ctx-checkpoints 64) on ≤16 GiB Macs; larger Macs keep 8192Other classes:
The tradeoff: why not 0 on GPU hosts, and why not 8192 on small ones
With one slot (
LLAMA_PARALLEL=1), Pixel interleaves with Open WebUI task calls, Hermes and Talk, and each switch evicts the conversation from the slot. For hybrid models (Qwen3.5/3.6) the RAM cache plus checkpoints is what lets it come back without a full re-prefill.found better prompt with f_keep = 0.352, sim = 0.624restored the 48,465-token entry for the next Pixel turn.--cache-ram 0.--cache-ram 0: the foreign chat evicted the main prefix.Why 3072 rather than a smaller cap in the 16 GB VM. One 64K agent conversation with its 32 checkpoints is about 2.5 GiB. With 3072 the cache keeps that entry plus a few short prompts. A smaller cap such as 2304 still keeps the newest entry, but the next short prompt saved after it pushes the total over the limit, and eviction is oldest-first, so the big entry goes. Replaying the live sequence above with 2304, the
found better promptrestore would have missed.Peak memory. The cache is bounded by max(cap, largest entry) plus one transient new entry, so about 5.5 GiB at worst on the laptop instead of about 10.5 GiB. At the 03:36 kill that is roughly 5 GiB less demand against the 19.3 GiB of VM RAM plus swap.
Container limit: considered, not changed
LLAMA_SERVER_MEMORY_LIMITtoward the VM's headroom would not prevent the crash. The container gets swap equal to its limit (MemorySwap 24G), so llama-server would page, then be cgroup-OOM-killed instead of global-OOM-killed. The kernel already picks llama-server, so the other processes survive either way.system_ram_min_gb15). Changing it needs its own validation.Conflicts with open PRs
.env.schema.jsonLLAMA_ARG_CACHE_RAM.description, where both PRs rewrite the same line. Whichever merges second keeps both sentences: this PR's sizing sentence, and macOS: shared-slot llama-server layout for hybrid models on 16 GB Macs (candidate, needs live A/B) #6757's "except that the native macOS shared-slot layout (see LLAMA_PARALLEL) sets 0. Setting it keeps native macOS on one slot." The semantics do not overlap. This PR never writesLLAMA_ARG_CACHE_RAMon Apple:installers/macosis untouched and activation skipsapple. So macOS: shared-slot llama-server layout for hybrid models on 16 GB Macs (candidate, needs live A/B) #6757's "all three keys unset" condition still holds on Macs.git merge-treeagainst this branch is clean. fix(installer): persist the installer tier as TIER in .env #6617's Windows test hunk is kept apart by placing the new assertions elsewhere.main.Tests
Passing locally:
bash tests/test-llama-memory-budget.sh: memory-limit parser, the rule table, and phase 06's own block run against fixtures (precedence, Docker VM lower bound, every skipped backend). Mutation-checked: removing the.envfallback or letting AMD through fails it.tests/test-windows-llama-memory-budget.ps1: rule table, fresh NVIDIA install, rerun keeps an owner value, profile wins over.env, 15 GiB VM → 3072, 128 GiB → unset, AMD/cloud/CPU cases. Passes on Windows PowerShell 5.1 (pwsh is not installed here) and is mutation-checked.test-linux.yml,lint-powershell.yml); neither ran in CI before.tests/test_llama_cache_budget.py(24), plus 16 activation tests intest_model_activate.py. The activation tests cover the Docker VM lower bound, the container-limit bound, keeping a.envvalue, profile precedence, large-host unset, the host-RAM fallback, and the skipped runtimes (apple, amd, lemonade, cloud, Windows native).main. The branch's failures are a subset of main's: Windows-only, or fastapi/symlink/bash-path environment issues. After the second commit changed the fraction, I re-ran the changed test files:test_llama_cache_budget.py24 passed;test_model_activate.py,test_model_switch_speed.pyandtest_model_transaction.pyhad only the 2 failures that also fail onmainhere.tests/contracts/test-llama-spec-default.py,test-llama-runtime-tunables.py,tests/test-safe-env.sh,tests/test-validate-env.shandtests/test-cpu-only-path.shpass.Not run cleanly here (Windows-only):
test-preserve-active-model.py(bash path handling),test-phase06-env-rerun-quoting.sh(docker compose --env-file) and the installer-contracts symlink case. They fail identically onmainon this machine, so Linux CI is authoritative for them.Live validation needed (laptop)
.envshould containLLAMA_ARG_CACHE_RAM=3072.docker inspect ods-llama-servershould showLLAMA_ARG_CACHE_RAM=3072.prompt cache is enabled, size limit: 3072 MiB.qwen3.5-4b-q4→ 2048), then back. The value follows the profile and is otherwise kept. Deleting the key and then activating should write 3072.memory.stat anon, thecache state: … (limits: 3072.000 MiB …)line, MemAvailable and SwapFree.journalctl -k | grep "Out of memory"must show no new kills.found better promptrestores for the Pixel conversation, and that returning to it costs about 1 s rather than a 20 s re-prefill.🤖 Generated with Claude Code