Dual-model serving: 1.7B default + 0.6B, backward-compatible - #3
Merged
Conversation
Extend the serving layer to serve both fine-tunes from one process, selectable per request, without changing the default path. - api.py: model registry (1.7b -> committed-gguf, 0.6b -> committed-gguf-0.6b) as the source of truth; optional `model` field on /generate (defaults to "1.7b" so a no-model request is byte-for-byte v1 behavior); unknown model -> 400, not a crash; /health extended with models_loaded/models_available while keeping the model_loaded field the portfolio reads. Default loads eagerly at startup; 0.6b loads lazily on first request and is cached. Every instance gets n_threads=2/n_threads_batch=2 (the v1 CPU-Space thread fix). The default is built via the engine resolver so the COMMITTED_MODEL_* Space Variables still override it. - app.py (Gradio): model dropdown wired to the same shared registry/factory; default eager warm-up, 0.6b lazy; thread fix and NotADiffError guard carried. Verified with a TestClient + fake generator (no model download): backward-compat no-model->1.7b, explicit 0.6b lazy-loads, unknown->400, empty->400, health reports both. The 0.6b GGUF repo is still PRIVATE on the Hub — must be made public (or given a Space token) before the live deploy can pull it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the serving layer to serve both fine-tunes (1.7B + 0.6B) from one process, selectable per request, without changing the existing default path.
src/committed/serving/api.py— model registry (1.7b→committed-gguf,0.6b→committed-gguf-0.6b) as the source of truth.POST /generategains an optionalmodelfield defaulting to"1.7b", so a request with nomodelis byte-for-byte the v1 behavior (backward compatibility is a hard requirement — the live portfolio sends nomodel). Unknown model → 400, not a crash./healthextended withmodels_loaded/models_available, keeping the existingmodel_loadedfield the portfolio reads.n_threads=2, n_threads_batch=2(the v1 CPU-Space fix). The default is constructed via the engine resolver, so theCOMMITTED_MODEL_*Space Variables still override it.app/app.py(Gradio) — model dropdown wired to the same shared registry/factory; default eager warm-up, 0.6B lazy; thread fix +NotADiffErrorguard carried.Verification (local, no model download)
TestClient+ a fake generator exercised every path:model→ 200 served by 1.7b (backward-compat) ✅model: "0.6b"→ 200, and/healthmodels_loadedgrew['1.7b']→['1.7b','0.6b'](lazy load) ✅Deploy notes (Phase 2 — after merge)
The Spaces install
committedfrom@mainin a cached layer, so merge alone does nothing — each Space must be factory-rebuilt to pull new code. Two prerequisites before the 0.6B path works live:committed-gguf-0.6brepo is currently PRIVATE — must be made public (or a token added to the Space) or the runtime pull 401s.llama-cpp-python==0.3.30— couldn't verify the CPU wheel exists at the abetlen index from CI (network-restricted); if it's missing the first rebuild falls back to a (slower, cached-after) source compile. Pin intentionally not loosened.Does not touch the portfolio front-end or any eval/training/grammar/prompt code.
🤖 Generated with Claude Code
Generated by Claude Code