QVAC-23389 fix[api]: honor preload:false via lazy-load, keep DELETE reversible

[lauripiisang]

🎯 What problem does this PR solve?

qvac serve openai had a preload / lifecycle gap:

• preload: false never worked. Such models were registered but never loaded, and every request returned 503 model_not_ready forever — even though the docs promised lazy cold-start loading.
• DELETE was destructive. DELETE /v1/models/{id} removed the alias from the registry with no reload path, so the model stayed resolvable from config but permanently unavailable until a full server restart.

📝 How does it solve it?

• Lazy-load on first request. A model with preload: false now loads the first time a request names it. Loading is concurrency-safe: simultaneous first requests share a single load (per-registry in-flight dedup), and a failed cold start surfaces as 503 model_load_failed and retries on the next request.
• Reversible unload. DELETE /v1/models/{id} now resets the alias to IDLE (drops the SDK handle) instead of removing it, so the next request reloads it.
• OpenAI-shaped listing. GET /v1/models lists all configured models (loaded or not) and GET /v1/models/{id} returns any configured alias — loading is transparent to the client.
• All inference gates (require-model, audio speech, vector-stores embedding) go through one shared ensureReady helper.
• preload default asymmetry is unchanged (constant entries default true, explicit { src, type } default false); --model <alias> still forces a warm start.
• Docs aligned: packages/cli/docs/serve-openai.md, website http-server / configuration, and the CLI README.

Dedicated load endpoints and listing the full SDK model catalog (with config introspection) are intentionally left as a follow-up; the code is structured to keep that path open.

🧪 How was it tested?

• New unit test test/lifecycle.test.ts: lazy-load state machine, concurrent-load dedup (single SDK load), error-then-retry, and reload-after-unload.
• New / updated e2e (model tier): real lazy-load of a preload: false embedding on first request; GET /v1/models lists all configured; DELETE keeps the alias listed + reloads on the next request.
• Added a loadModelOverride test seam (mirrors transcribeOverride) so the model gate can be exercised without a heavy real load; migrated the video HTTP-layer tests to it.
• Added config.test.ts assertions locking the kept preload defaults.
• Local runs: unit 472/472, model-tier e2e 75/75, modelless e2e 120/122 (the 2 failures are live-network OpenAI-spec fetches returning HTTP 429, unrelated). Typecheck, lunte, and prettier clean.

🔌 API Changes

HTTP surface behavior (non-breaking — naming a configured model just works):

# preload:false model — first request lazy-loads it (blocks), later requests are fast
curl -X POST http://localhost:11434/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"my-llm","messages":[{"role":"user","content":"hi"}]}'

# unload frees resources but keeps the alias; the next request reloads it
curl -X DELETE http://localhost:11434/v1/models/my-llm

# lists every configured model, loaded or not
curl http://localhost:11434/v1/models

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: Team Lead: 1

Pending reviews: Needs 1 more from Management, Team Lead, or Member.

[github-actions]

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

• ./.github/actions/release-merge-guard
• ./docs/website
• ./packages/ggml-coload-smoke
• ./packages/fabric/test/integration
• ./packages/inference-addon-cpp/mobile
• ./packages/sdk/e2e
• ./packages/llm-llamacpp/benchmarks/performance
• ./packages/llm-llamacpp/benchmarks/server
• ./packages/vla-ggml/sim/server
• ./packages/embed-llamacpp/benchmarks/performance
• ./packages/embed-llamacpp/benchmarks/server
• ./packages/asr-ggml/benchmarks/server

[opaninakuffo]

lazy-load + reversible DELETE is the right fix, and the happy path is covered (unit + e2e). one hole: DELETE during an in-flight load is a no-op that still returns {deleted:true}, then setReady brings the model back.

🔴 requesting changes

[arun-mani-j]

The new in-flight map is keyed by alias, so it collapses concurrent requests for the same model but not across different ones. Two requests naming two different preload: false aliases each start a full load, and N aliases give N at once.

preloadModels just above, on line 55, still loads one at a time (for … await), so the same set of models that loads comfortably in sequence at startup can now land in parallel at request time. A config that is fine with preload: true could get tight on memory with preload: false under the same traffic.

Worth considering alongside this: these loads have no deadline, and nothing cancels one if the client disconnects. The server leaves Fastify's requestTimeout and connectionTimeout at the default 0 (serve/index.ts:125, unchanged here), which was harmless while no request path could start model I/O, but now means a single request can hold a socket for as long as the download takes.

[lauripiisang]

Addressed in e5ffbac with a new serve.load config block + mirrored CLI flags:

• concurrency (default 1, mirrors preload's sequential loading) — distinct-alias lazy loads no longer fan out to N parallel loads. --load-concurrency.
• timeoutMs (default null) — bounds a load; on expiry the SDK load is cancelled and the request gets 503 model_load_timeout. --load-timeout.
• cancelOnDisconnect (default true) — a client disconnecting mid-load cancels the load, but only once no other request is still waiting on that same (shared) load; refcounted in the new LoadManager. --no-cancel-load-on-disconnect.
• lazy (default true) — --no-lazy-load disables request-time loads entirely (unloaded → 503 model_not_loaded), the bounded-latency mode for operators who don't want in-request downloads.

Concurrency + refcounted cancel + timeout are covered in load-manager.test.ts.

[arun-mani-j]

The speech gate now goes through ensureReady, so a preload: false TTS model is served on first request — but GET /v1/audio/models further down at line 500 was left as registry.getReady().filter(entry => entry.endpointCategory === 'speech').

So after this change such a model shows up in /v1/models, works fine on /v1/audio/speech, and is missing from /v1/audio/models until something loads it. Explicit { src, type } entries default to preload: false, which is the usual shape for a TTS entry, so this may come up fairly often.

packages/cli/docs/serve-openai.md:724 still describes that endpoint as "the speech-capable subset of /v1/models" and names Open WebUI's TTS model selector as the consumer, so the selector would come up empty on a fresh server. Keying it off serveConfig.models, the same way line 58 of routes/models.ts now does, would line the two back up.

[lauripiisang]

Fixed — /v1/audio/models now lists configured speech models from serveConfig.models (same as /v1/models), so a preload:false TTS model shows up before its first use and the Open WebUI selector isn't empty on a fresh server.

[arun-mani-j]

Listing every configured alias here removes the last HTTP-visible signal of load state. Together with GET /v1/models/{id} no longer filtering on READY and DELETE keeping the alias, the listing is now identical before and after an unload — ModelObject on line 10 carries only {id, object, created, owned_by}.

The case this affects is the warm-up poll the old description invited ("List all currently-loaded (READY) models"): poll /v1/models until the alias appears, then send traffic. That loop now returns straight away, and the first real request carries the whole cold start. The new e2e test reads app.qvac.registry directly to assert the unload, which points at the same gap.

An extra non-standard state field on the model object would bring the signal back without upsetting OpenAI clients, since they ignore unknown fields. This may already be part of the follow-up mentioned in the description.

[lauripiisang]

Added — model objects on /v1/models and /v1/models/:id now carry a non-standard state field (idle/loading/ready/error), so the warm-up poll works again without loading the model. OpenAI clients ignore the unknown field. This is the listing seed for QVAC-23390.

[arun-mani-j]

markUnloaded takes over the one place that called remove, which leaves remove (line 173, declared on line 90) without callers anywhere in src or test. Might be worth dropping while it's fresh.

[lauripiisang]

Done — registry.remove is removed (interface, impl, and return).

[lauripiisang]

Thanks both — all review points addressed in e5ffbac.

Correctness

• DELETE during an in-flight load now waits for the load to settle before unloading, so it can't report success while the model reloads.

Load management (serve.load, new) — answering the concurrency/timeout/socket-hold concern with explicit, configurable controls:

• lazy (default true) — --no-lazy-load disables request-time loads; unloaded → 503 model_not_loaded.
• concurrency (default 1) — mirrors preload; distinct-alias lazy loads no longer fan out.
• timeoutMs (default null) — cancels the SDK load on expiry → 503 model_load_timeout.
• cancelOnDisconnect (default true) — refcounted: cancels only when no request still waits on the shared load.

All four have CLI flags. Logic lives in a new LoadManager with its own unit suite.

Listing

• /v1/models + /v1/models/:id now expose a non-standard state field (idle/loading/ready/error), restoring the warm-up signal.
• /v1/audio/models lists configured speech models (not just READY), matching /v1/models.

Cleanups

• swagger summary → "List configured models"; model_not_ready docs reworded (rare fallback) with model_not_loaded/model_load_timeout added; dropped the now-unused registry.remove.

Local: unit 474/474 (+ 1 unrelated live-network OpenAI-spec fetch), non-model e2e 125/125, model tier 75/75.

[lauripiisang]

Follow-up on two self-review items on top of the earlier fixes:

• Cancel/timeout state (d60246600): a client-disconnect cancel and a load timeout now reset the entry to idle and log the event (info for cancel, warn for timeout) instead of surfacing as state: error in the model listing. Genuine load failures still set error.
• loadManager certainty (d60246600): dropped the undefined as unknown as placeholder — buildServer now builds the manager first as a real required field; the load-fn resolves the test override via an accessor-backed ref so the post-build test seam still works.

Coverage added (ccd81a090): cancel/unload are now injectable in the manager, so there are unit tests for last-waiter-abort cancellation, timeout cancellation, and ensureReady cancelling on a real req.raw disconnect (and not cancelling when disabled); plus spawned-CLI tests for --no-lazy-load (→ 503 model_not_loaded) and --load-concurrency/--load-timeout parsing/validation.

Local: unit 478/478 (+1 unrelated live-network OpenAI-spec fetch), non-model e2e 128/128, model tier 75/75. All prior inline threads were addressed on e5ffbac.