Skip to content

Default to 0.6B across API/CLI/demo + CLI --model picker (ADR 0051) - #5

Merged
marzoukbaig14 merged 1 commit into
mainfrom
serving-cli/default-0.6b
Jul 6, 2026
Merged

Default to 0.6B across API/CLI/demo + CLI --model picker (ADR 0051)#5
marzoukbaig14 merged 1 commit into
mainfrom
serving-cli/default-0.6b

Conversation

@marzoukbaig14

Copy link
Copy Markdown
Owner

What

Make 0.6B the default everywhere Committed downloads or serves; 1.7B stays fully selectable. Supersedes ADR 0048 (1.7B serving-of-record) → ADR 0051.

  • engine.py: DEFAULT_MODEL_REPO/FILE → the 0.6B GGUF, DEFAULT_TOKENIZERQwen/Qwen3-0.6B (identical vocab, v2-i1 Gate 0). Single source for the bare-download default.
  • serving/api.py: registry DEFAULT_MODEL"0.6b" (eager at startup); 1.7B now lazy on first model:"1.7b" request. A no-model /generate now returns 0.6B — an intentional default flip (coordinated with the portfolio reframing).
  • cli.py: --model {0.6b,1.7b} (default 0.6b) mapping to the two GGUF repos; --model-path + COMMITTED_MODEL_* overrides retained; model-aware download-size notice; --help updated.
  • Demo: dropdown default follows DEFAULT_MODEL → 0.6B on next rebuild.
  • Thread fix (n_threads=2) retained on every instance.

Facts for the model cards (observed, not guessed)

  • 0.6B GGUF marzoukbaig14/committed-gguf-0.6b/committed-0.6b-finetuned-Q4_K_M.gguf: 396,704,512 bytes ≈ 396.7 MB (378.3 MiB), quant Q4_K_M (general.file_type=15), arch qwen3. Hub-reported size matches the local build byte-for-byte.
  • Both repos public: committed-gguf-0.6b and committed-ggufprivate=False ("no token needed" is true).
  • Exact CLI commands (after this change):
    • default (0.6B): git diff | committed
    • explicit 1.7B: git diff | committed --model 1.7b
    • the pip install --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu committed @ git+... install line is unchanged and model-agnostic (the package pulls whichever public GGUF the default/--model selects).
  • Real 0.6B generation (divide-by-zero guard diff → git diff | committed): fix: divide by zero — valid Conventional Commit. The default-path download notice fired for committed-gguf-0.6b (~380 MB), confirming the CLI default resolves to 0.6B.

Verification

ruff check clean; committed --help shows --model {0.6b,1.7b} (default 0.6b); api.DEFAULT_MODEL == "0.6b"; engine default → the 0.6B GGUF; live 0.6B generation above.

After merge (Part 4 — I'll do it)

Factory-rebuild committed-api and committed-demo (merge alone doesn't pull @main), then verify live: /health shows 0.6B default/eager, no-model → 0.6B (~1s warm), model:"1.7b" lazy-loads, demo dropdown defaults to 0.6B. Note: once this deploys, the portfolio's no-model requests begin returning 0.6B (intended).

Does not touch the portfolio repo; does not write the Hub cards (human does, from the facts above).

🤖 Generated with Claude Code


Generated by Claude Code

Make 0.6B the default everywhere Committed downloads/serves; 1.7B stays
selectable. Supersedes ADR 0048's 1.7B serving-of-record.

- engine.py: DEFAULT_MODEL_REPO/FILE -> the 0.6B GGUF; DEFAULT_TOKENIZER ->
  Qwen/Qwen3-0.6B (identical vocab, v2-i1 Gate 0). Single source for the bare
  download default.
- serving/api.py: registry DEFAULT_MODEL -> "0.6b" (eager at startup); 1.7b now
  lazy. A no-model /generate resolves to 0.6b (intentional default flip).
- cli.py: add --model {0.6b,1.7b} (default 0.6b) mapping to the two GGUF repos;
  keep --model-path and COMMITTED_MODEL_* overrides; model-aware download-size
  notice; --help updated.
- Demo dropdown default follows DEFAULT_MODEL (0.6b) on the next rebuild.
- ADR 0051 + regenerated decision log/tree.

Thread fix (n_threads=2) retained on every instance.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CyxW8LpuJuQRCzdnfYp8Xo
@marzoukbaig14
marzoukbaig14 merged commit 091f4e7 into main Jul 6, 2026
1 check passed
@marzoukbaig14
marzoukbaig14 deleted the serving-cli/default-0.6b branch July 6, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants