feat(runtime): select Gemma 4 by hardware profile - #25
feat(runtime): select Gemma 4 by hardware profile#25kristoffersodersten wants to merge 1 commit into
Conversation
Use E2B QAT for low-memory hardware and E4B QAT for balanced Apple Silicon. Keep explicit model overrides authoritative and remove silent substitution of unrelated installed models.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a00b947479
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return self._resolve_model(self.config.model, GENERATION_MODEL_PREFERENCES, models) | ||
| # The configured hardware profile is authoritative. A different local | ||
| # model must never be selected implicitly when the requested one is absent. | ||
| return self.config.model |
There was a problem hiding this comment.
When the configured model exists only as an Ollama remote model, active_generation_model() now ignores the supplied local-only model list and returns that remote name. The /summarize, /extract, and /agent handlers do not gate execution on health, so direct requests pass the remote model to /api/generate and can send selected content off-device instead of reporting the model as missing. Reject generation before the request whenever _model_available_locally() is false.
Useful? React with 👍 / 👎.
| Automatic selection uses the smallest effective local profile: | ||
|
|
||
| - Fast: `gemma4:e2b-it-qat` for unknown or under-16-GB hardware and Intel Macs | ||
| - Balanced: `gemma4:e4b-it-qat` for Apple Silicon with 16 GB or more |
There was a problem hiding this comment.
For users following the README Quick Start, pnpm setup:local still invokes bootstrap-macos-local.sh --profile fast in package.json, bypassing the hardware recommendation entirely. Consequently, Apple Silicon machines with 16 GB or more install E2B rather than the newly documented automatic E4B profile; remove the forced profile from the standard setup path or avoid claiming that path selects automatically.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Updates SelectPilot’s runtime model selection to use Gemma 4 QAT variants by default, with hardware-profile-driven recommendations and stricter “configured model is authoritative” behavior in the Ollama client.
Changes:
- Switch Fast/Balanced runtime profiles to
gemma4:e2b-it-qatandgemma4:e4b-it-qat, respectively. - Remove implicit local-model fallback behavior in
OllamaClientso the configured model is used as-is. - Expand unit tests and README guidance around auto recommendations and explicit overrides.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/server/test_runtime_profiles.py | Adds coverage for new Gemma defaults, auto recommendations, and explicit model/profile behavior. |
| server/runtime_profiles.py | Updates Fast/Balanced profile model IDs and descriptions to Gemma 4 QAT variants. |
| server/ollama_client.py | Removes implicit model preference fallback and makes the configured model authoritative. |
| scripts/install-macos-local.sh | Updates the macOS LaunchAgent install default model to gemma4:e2b-it-qat. |
| README.md | Documents the new automatic profile/model selection behavior and override expectations. |
Suppressed comments (1)
server/ollama_client.py:94
- In
OllamaClient.__init__,recommend_runtime_profile()is called unconditionally, even whenCHROMEAI_RUNTIME_PROFILEis explicitly set to something other thanauto. Sincerecommend_runtime_profile()can run sysctl/subprocess detection, this is unnecessary work on startup. Consider only callingrecommend_runtime_profile()whenruntime_profile == "auto".
default_generation_model = "gemma4:e2b-it-qat"
default_embed_model = "nomic-embed-text-v2-moe:latest"
runtime_profile = os.environ.get("CHROMEAI_RUNTIME_PROFILE", "auto")
try:
from runtime_profiles import get_runtime_profile, recommend_runtime_profile
recommendation = recommend_runtime_profile()
resolved_profile = recommendation["recommended_profile"] if runtime_profile == "auto" else runtime_profile
profile = get_runtime_profile(resolved_profile)
default_generation_model = profile.generation_model
default_embed_model = profile.embedding_model
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| def active_generation_model(self, models: list[str] | None = None) -> str: | ||
| models = models if models is not None else self._model_names() | ||
| return self._resolve_model(self.config.model, GENERATION_MODEL_PREFERENCES, models) | ||
| # The configured hardware profile is authoritative. A different local | ||
| # model must never be selected implicitly when the requested one is absent. | ||
| return self.config.model |
| OLLAMA_BASE_URL="${CHROMEAI_OLLAMA_BASE_URL:-http://127.0.0.1:11434}" | ||
| OLLAMA_MODEL="${CHROMEAI_OLLAMA_MODEL:-qwen2.5:0.5b}" | ||
| OLLAMA_MODEL="${CHROMEAI_OLLAMA_MODEL:-gemma4:e2b-it-qat}" | ||
| OLLAMA_EMBED_MODEL="${CHROMEAI_OLLAMA_EMBED_MODEL:-nomic-embed-text-v2-moe:latest}" |
| - Fast: `gemma4:e2b-it-qat` for unknown or under-16-GB hardware and Intel Macs | ||
| - Balanced: `gemma4:e4b-it-qat` for Apple Silicon with 16 GB or more | ||
| - Advanced: explicit opt-in only; it is never selected automatically |
Summary
gemma4:e2b-it-qatfor Fast / low-memory hardwaregemma4:e4b-it-qatfor Balanced / 16 GB+ Apple Siliconllama3.2implicit defaultHardware correction
The previous Linear issue assumed
gemma4:e4bwas about 6 GB. The current official Ollama registry lists the default E4B artifact at 9.6 GB; the QAT tag is 6.1 GB, while E2B QAT is 4.3 GB. A universal E4B default would therefore violate the low-memory contract.Source: https://ollama.com/library/gemma4/tags
Stack boundary
This PR targets
codex/sod-795-novaforge-adoption. Its diff contains only the five SOD-395 files. It does not modify PR #22 or claim delivery tomain.Verification
Exact SHA:
a00b94747922ab69338cdb34b7ded7e8b8d131d9Hetzner:
Local bounded smoke:
Remaining promotion evidence
Real M4/M1 latency and memory measurements remain required before SOD-395 can be marked Done. This PR establishes the deterministic implementation and test contract without downloading models outside the existing consent path.