[opticon454]

What

A custom llama.cpp model whose parallel slots leave too little context per request now gets an in-app warning that names the fix.

Why

llama-server runs several parallel slots (four by default on the build measured) that share one context pool. Claude Code sends concurrent requests: the main session plus every parallel subagent, each carrying about 40K tokens of its own system prompt and tools. Measured against a real llama-swap server, four simultaneous 40K-token requests against a 4-slot ~131K pool all failed with HTTP 500 "Context size has been exceeded"; the same requests against one slot all succeeded. The existing 40K floor check only looked at the total context, so it never warned.

How

• Discovery records each loaded model's slot count: from an explicit --parallel/-np in the launch command, or, when the command states none (the server chooses), from /props total_slots.
• When /props is probed its n_ctx is recorded separately and used only to divide the context across the slots. A launch command's --fit-ctx is just the auto-fitter's floor (a small model on a large card was measured at 262,144 against a floor of 65,536), so dividing the floor would warn about a server with plenty of room. modelContextLengths still comes from the launch command, unchanged.
• The warning explains the shared-slot case and the fix: --parallel 1, or a context large enough for every slot.
• Documented in the Custom Model Endpoints page and Troubleshooting.

Tests

test/custom-model-slot-awareness.test.ts (new), plus cases in custom-model-endpoint-rediscovery, custom-model-run-menu-ui and routes/session-custom-model. Changeset included.

Fixes #641

[Ark0N]

Thanks @opticon454 for this, and for the measurement in #641. This PR records each llama.cpp model's parallel slot count during discovery and warns before a Claude launch when the context split across those slots leaves less than 40K tokens per request.

The slot-count side is solid: --parallel/-np read from the launch command, total_slots as the fallback, an unknown count treated as one slot, and good tests around all of it.

The /props rule stays as it is. In #641 I asked that a /props value only ever raise the warning, never skip it. slotPoolContext() (src/web/routes/custom-model-routes.ts:77) does the opposite: when /props reports a larger n_ctx than the launch command, that figure becomes the pool and the warning is skipped. I see your point that llama.cpp documents --fit-ctx as the auto-fitter's floor, but the rule comes from a live measurement where /props reported 154,112 for a backend that was really running at 16,384 (docs/architecture-invariants.md:636), and I would rather show a cautious warning than skip one that was needed. So please drop slotPoolContext() and modelServerContexts, and divide only the context parsed from /running's cmd. /props keeps its two jobs: the context fallback when cmd states no context flag at all, and total_slots when there is no --parallel. That also removes the stale server context that applyDiscoveredModels (custom-model-routes.ts:666) carried across rounds, since nothing stores it any more.

This means a default llama-swap model (auto slots, --fit-ctx under 160K) shows the slot dialog when Claude launches on it. That is fine with me: the dialog names the fix (--parallel 1, or a larger -c), and the user can still launch anyway.

Also needed:

1. src/web/routes/custom-model-routes.ts:328: the 5-minute rediscovery sweep now calls /props?model= for every loaded model whose command has no --parallel, which is the default llama-swap setup. llama-swap routes /props through the model's process handler, and that handler refreshes the model's last-use time, so a model with a ttl above 300 seconds never unloads while Codeman is running. Please probe only when the slot count is unknown for that launch command, for example by storing the cmd the count was read from and reusing the stored count while /running reports the same cmd.
2. src/web/routes/custom-model-routes.ts:331: with an explicit --parallel N and no context flag, the context comes from the /props fallback, and llama.cpp reports that figure per slot when the slot count is explicit (the unified KV cache is only on by default when the count is auto). It is then divided by N a second time, so --parallel 4 with 65,536 per slot warns at 16,384. Please skip the division in that case.
3. Docs: please write the slot check into the places where the /props rule lives, and say there that /props never widens the pool: CLAUDE.md (Custom Model Endpoint Profiles, the llama-swap paragraph), docs/architecture-invariants.md:636 and :638 (the requiresContextWarning shape now also has slotCount and poolContext), docs/custom-model-endpoints.md (discovery around line 88, which now also reads total_slots, and the floor warning around line 440), docs/api-reference.md:988, and the fetchServerProps doc comment (custom-model-routes.ts:212). The new paragraph in docs/wiki/Custom-Model-Endpoints.md says the division uses the context the server reports rather than --fit-ctx; that sentence needs to go with the change.
4. test/routes/quick-start-custom-model.test.ts: the quick-start path (src/web/routes/session-routes.ts:4273) has the same new slot branch as the dedicated route, but only the dedicated route is tested. One warning case and one no-warning case in the existing context-floor block would cover it.

One wording point: with an explicit --parallel N, each slot has a hard limit of context / N even when it runs alone, so "share ... roughly N each when all are busy" (src/web/public/session-ui.js:1121) undersells it. "Split across N slots, about X each" covers both cases.

Once those are in, this is ready to merge.

[opticon454]

Thanks @Ark0N, all four items and the wording point are in the new commit (rebased on current master first).

The /props rule stays. slotPoolContext(), modelServerContexts and the poolContext response field are gone. The division uses only the context parsed from /running's cmd; /props keeps its two jobs (the context when cmd states none, total_slots when there is no --parallel). applyDiscoveredFields, the schema, the host type, both apply routes and the modal no longer carry a server context, and the response gains only slotCount.

1. No probe on every sweep. discoverModels goes through propsForLaunch(), which remembers the /props answer per host and model id together with the cmd it was read under and reuses it while /running reports the same cmd. A changed cmd probes again, a probe that did not answer is not remembered, and a non-string cmd is probed each time. It is held in memory, so a restart probes once more. New test: two sweeps probe once, a changed command probes a second time.
2. No second division. With an explicit --parallel N and no context flag, the /props context is stored as the model's context and no slot count is recorded, since that figure is already per slot. New test: --parallel 4 with n_ctx: 65536 stores 65,536 and no slot count.
3. Docs carry the slot check and say /props never widens the pool: CLAUDE.md (llama-swap paragraph), architecture-invariants.md (the /props paragraph, the floor paragraph with slotCount, and the once-per-command probe), custom-model-endpoints.md (discovery, with a new paragraph on slots, and the floor warning), api-reference.md, the fetchServerProps doc comment and the modelSlotCounts comment. The sentence in Custom-Model-Endpoints.md that said the division uses the server's reported context is replaced, and it now says a default auto-slot model shows the dialog.
4. Quick-start tests: one warning case and one no-warning case in the context-floor block of test/routes/quick-start-custom-model.test.ts.

Wording: the dialog now reads "'s N tokens of context are split across S parallel server slots, about X each."

The old /props-widening tests are rewritten for the new rule. Typecheck, lint and the custom-model, session and quick-start tests pass locally (275 tests).

[Ark0N]

Thanks @opticon454 for turning this round so quickly. One thing in the new commit (c5a4602f) has to come out: it adds node_modules to the repository, as a symlink to ../cfg/node_modules (I guess from a worktree that shares one install). Please drop it with git rm --cached node_modules, commit and push.

It got in because master's .gitignore had node_modules/, and the trailing slash only matches a directory, so a symlink with that name showed up as an untracked file. Master ignores it now (3907728), so after your next rebase it cannot slip in again.

The symlink also stopped my review checkout of that head, which is why the rework has not had its review yet. It runs once the new head is up.

[opticon454]

Thanks @Ark0N, and sorry about that: I share one install between my worktrees through a symlink and git add -A picked it up. The symlink is dropped in a new commit (git rm --cached node_modules) and the branch is rebased onto current master, so your .gitignore fix now covers it. The new head is up for the review.

[Ark0N]

Thanks @opticon454 for the quick rework, and for cleaning up the symlink. This PR records each llama.cpp model's parallel slot count during discovery and warns before a Claude launch when the context split across those slots leaves less than 40K tokens per request.

All four items and the wording point are in: /props never widens the pool, it is probed once per launch command, the quick-start path is tested, and the docs carry the rule. Typecheck, lint, format and the full test suite are green here. Two things still need a change before merge, plus a few small ones.

Please fix:

1. src/web/routes/custom-model-routes.ts:360: a slot count stored by an earlier round survives a round that deliberately records none. With an explicit --parallel and a /props context the branch sets slots = undefined, so the model is left out of slotCounts, and applyDiscoveredModels (:691) spreads { ...host.modelSlotCounts, ...slotCounts }, which keeps the old count. Two discovery rounds show it:

   • round 1: llama-server -m m.gguf, /props answers {n_ctx: 131072, total_slots: 4}: stores 4 slots and warns (correct)
   • round 2: the user follows the dialog and adds --parallel 1, /props answers {n_ctx: 131072, total_slots: 1}: the stored count stays 4, so the dialog still says "split across 4 parallel server slots, about 32,768 each"

   The same path undoes the "no second division" fix for a model first seen on auto slots and later given --parallel 2. The test at test/custom-model-endpoint-rediscovery.test.ts:371 starts from a fresh host, so it never reaches this. Please set slots = 1 in that branch instead of undefined (the stored context is already what one request gets, perSlotContext() leaves it whole, the routes omit slotCount at 1, and the merge overwrites the stale value), update that test to expect { m: 1 }, and add a two-round test like the one above.

2. src/web/routes/custom-model-routes.ts:223 and :298: a /props that answers non-2xx is not remembered, so a llama-swap model whose upstream does not serve /props and whose cmd has no --parallel is probed on every 5-minute sweep, which keeps its ttl from expiring (the fixture at test/custom-model-endpoint-rediscovery.test.ts:292 is this case). Please remember a non-2xx answer per cmd as an empty props object, and keep not caching network errors and timeouts.

Small ones (fine in the same commit, or I fix them at merge time):

3. docs/api-reference.md:993 says slotCount is present only when the per-slot split is the cause, but both routes include it whenever the count is above 1. The wording in docs/custom-model-endpoints.md ("plus slotCount when the model runs more than one parallel slot") is the accurate one.
4. docs/wiki/Troubleshooting.md:174: n_ctx is the shared context only when the slot count is automatic; with an explicit --parallel N it is per slot. On llama-swap the call is GET /props?model=<id>, for a loaded model.
5. src/web/routes/custom-model-routes.ts:274: the clearPropsProbeCache() comment names "a host whose server was replaced behind the same URL", but only tests call it. Either clear that host's entries from the endpoint PUT and DELETE routes, or trim the comment to tests.

Once 1 and 2 are in, this is ready to merge. I will squash it on merge, so the commit that briefly carried the node_modules symlink stays out of master's history.

[opticon454]

Thanks @Ark0N, both items and the small ones are in the new commit.

1. Stale slot count. With an explicit --parallel and a /props context, discovery now records 1 instead of nothing, so the merge overwrites an earlier round's count. test/custom-model-endpoint-rediscovery.test.ts:371 expects { m: 1 }, and a new three-round test runs your scenario: round 1 auto slots with 4 stored, round 2 --parallel 1 leaves { m: 1 }, round 3 --parallel 2 with a per-slot n_ctx of 65,536 stays at { m: 1 } with the context 65,536, so nothing brings a pool back. The routes already omit slotCount at 1.
2. Non-2xx /props. fetchServerProps answers an empty object for a non-2xx status, which propsForLaunch remembers per cmd; a network error, timeout or unreadable body still answers undefined and is retried. New test: a 404 /props is probed once across two sweeps, a thrown network error twice.

Small ones: api-reference.md now says slotCount is present whenever the model runs more than one slot (matching custom-model-endpoints.md); the Troubleshooting /props advice says n_ctx is the shared context only with automatic slots (per slot with an explicit --parallel N) and gives the ?model=<id> form for llama-swap; the clearPropsProbeCache() comment is trimmed to tests. The modelSlotCounts doc comment also says the explicit---parallel case is recorded as 1.

Typecheck, lint and the custom-model, session and quick-start tests pass locally (277).

[Ark0N]

Thanks @opticon454 for the third round, and for the measurements behind it. This PR records each llama.cpp model's parallel slot count during discovery and warns before a Claude launch when the context split across those slots leaves less than 40K tokens per request.

Both items from the last round are in and tested. An explicit --parallel with a /props context now records 1 slot, so it overwrites a count an earlier round stored (the three-round test at test/custom-model-endpoint-rediscovery.test.ts:387 runs exactly that scenario). A non-2xx /props is remembered per launch command, and a network error is retried (:417). The small ones are in too. Typecheck, lint, format, frontend syntax and the full test suite are green here.

One sentence went stale with that change, and I will fix it at merge time:

• docs/architecture-invariants.md:636 still ends with "a /props context beside it is stored with no slot count instead of being divided a second time". Since your last commit it is stored with a slot count of 1, which is what lets a later round overwrite an earlier count. I will reword it to say that.

One note for later, outside this PR: the injected CLAUDE_CODE_MAX_CONTEXT_TOKENS is still the total context (src/custom-model-injection.ts:128). So with an explicit --parallel N and a context stated in cmd, Claude Code compacts later than one slot allows. The slot counts this PR records are what a fix for that needs.

I will squash this on merge, so the node_modules commits stay out of master's history. Thanks again for turning every round around so quickly.

[Ark0N]

Merged and shipped in 1.44.0, squashed into one commit (33d697a) as promised, so the node_modules symlink commits stay out of master's history. Because of the squash GitHub shows this PR as closed rather than merged; the code is all in. Thank you @opticon454, three rounds, and the measurements behind the slot math made the review easy.

At merge I fixed the one stale sentence in docs/architecture-invariants.md: an explicit --parallel beside a /props context now records a slot count of 1, which is what lets a later round overwrite an earlier count.

The note from the review still stands as a follow-up if you want it: the injected CLAUDE_CODE_MAX_CONTEXT_TOKENS is still the total context, so with an explicit --parallel N Claude Code compacts later than one slot allows. The slot counts this PR records are what that fix needs. Closing #641 with this.