feat: open by default — scripts default to local Ollama, not Anthropic - #18
Open
mattekudacy wants to merge 3 commits into
Open
mattekudacy wants to merge 3 commits into
mattekudacy wants to merge 3 commits into
Conversation
Addresses feedback that this project shouldn't default to depending on a paid Anthropic key. The starting truth was already good — RulesClassifier (the actual default classifier=) makes zero API calls to any vendor, and LLMClassifier was always backend-agnostic (Ollama/Groq/OpenAI via base_url, or Anthropic without it) — but the *framing* wasn't: every scripts/*_accuracy.py harness fell back to an Anthropic model and required ANTHROPIC_API_KEY when nothing else was configured, and the README/docs/llm.py docstrings all presented Anthropic first, labeled "(default)". Concrete changes: - scripts/llm_classifier_accuracy.py, hybrid_ambiguity_accuracy.py, mast_mode_pilot_accuracy.py each gain a _resolve_backend() helper: an explicit TRIAGE_LLM_BASE_URL always wins (unchanged), an explicit Anthropic credential with no base_url still routes to Anthropic (unchanged — nothing breaks for an existing caller who set one), and with NOTHING configured at all, they now default to local Ollama (http://localhost:11434/v1, llama3.2) instead of Anthropic. Verified the resolution logic against all four cases (nothing set, Anthropic key set, custom base_url set, --model override) for each script. - triage/classifier/llm.py: LLMClassifier.__init__ behavior is UNCHANGED — it already raises ValueError naming all backend options rather than silently picking one, which is the more honest design than defaulting to either vendor. Only the docstring and error-message ordering changed (Ollama listed first, noted as needing no account or key). - docs/concepts/classifiers.md: new "Open by default" note; OpenAI-compatible section moved ahead of Anthropic's. - README.md: classifier examples and the env-var configuration block now lead with Ollama; installation section shows the open path first. - CLAUDE.md: Optional Extras table gains the previously-missing [openai] row ahead of [anthropic]; new design-decision entry recording the rationale so this isn't re-litigated or silently reverted later. - scripts/README.md: Benchmarks table and bash examples updated — "requires an LLM API key" (implying a paid vendor) corrected to "requires an LLM backend" with Ollama shown as the no-key default path. No triage/ core behavior changed — this is a default/framing correction in scripts and docs, not an API change. Verified: ruff check, ruff format --check, pytest (816 passed, unchanged), mypy --strict, mkdocs build --strict all clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
mypy triage/ --strict failed in CI with a hard parse error, unrelated to this branch's diff: numpy/__init__.pyi:737: error: Type statement is only supported in Python 3.12 and greater [syntax] Found 1 error in 1 file (errors prevented further checking) Root cause, confirmed by reproducing exactly (Python 3.13 venv, matching CI's numpy==2.5.3 and mypy 2.3.1) and tracing the import chain with `mypy --verbose`: a separate, already-merged PR added matplotlib/seaborn to the dev extra for scripts/gen_readme_charts.py, which pulled in a numpy release whose .pyi stub uses a PEP 695 `type` statement (valid only on Python 3.12+) — a hard syntax error against this project's [tool.mypy] python_version = "3.10" target. mypy reaches it via langchain_core/embeddings/fake.py (real, non-stub source triage/adapters/ langchain.py's import chain leads to), which does `import numpy` directly. ignore_missing_imports (already set for langchain_core/etc.) only suppresses "stub not found" errors — it doesn't stop mypy from parsing a stub it does find, and a numpy-specific override with follow_imports = "skip" (tried first) did not work either: the parse happens before mypy consults it. Fixed by moving follow_imports = "skip" onto the override covering anthropic/openai/aiosqlite/redis/langgraph/langchain/langchain_core: triage/ only calls these libraries' public APIs, so treating each as opaque Any without mypy recursing into its internal source is safe and sidesteps whatever transitive import chain a future dependency bump reaches next. opentelemetry deliberately stays on a separate override (ignore_missing_ imports only, no skip) — triage/observability/otel_ingest.py relies on mypy actually resolving opentelemetry.trace.StatusCode's real type, and applying skip there breaks a real strict-mode check (turns it into Any, trips warn_return_any) — found while testing this fix, not hypothetical. triage/ itself was never at fault; this branch's own diff didn't touch pyproject.toml before this commit. Fixing it here because main is currently red from the same cause and no other fix exists yet to port. Verified against a disposable Python 3.13 venv built to match CI's exact dependency versions: reproduces the original failure without this fix, clean with it. Also verified: ruff check, ruff format --check, pytest (816 passed) on both that venv and this session's own Python 3.11 environment, mypy --strict, mkdocs build --strict. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ownership. This PR was opened directly by an automated Claude Code session working in
this repo — see the "Exception" paragraph in this template. The
Co-Authored-By:trailer andsession-link footer on the commit and this description are required by the platform this
session runs on and can't be suppressed from inside the PR; nothing else here is exempt, and a
human still decides whether to merge it.
What does this change and why?
Addresses feedback that this project shouldn't default to depending on a paid Anthropic key —
it should be open source by default, like Ollama.
The starting truth was already good:
RulesClassifier(the actual defaultclassifier=)makes zero API calls to any vendor, and
LLMClassifierwas always backend-agnostic(Ollama/Groq/OpenAI via
base_url, or Anthropic without it). What wasn't open by default wasthe framing: every
scripts/*_accuracy.pymeasurement harness fell back to an Anthropicmodel and required
ANTHROPIC_API_KEYwhen nothing else was configured, and theREADME/
docs/concepts/classifiers.md/triage/classifier/llm.pydocstrings all presentedAnthropic first, labeled "(default)."
Concrete changes:
scripts/llm_classifier_accuracy.py,hybrid_ambiguity_accuracy.py,mast_mode_pilot_accuracy.pyeach gain a_resolve_backend()helper: an explicitTRIAGE_LLM_BASE_URLalways wins (unchanged), an explicit Anthropic credential with nobase_url still routes to Anthropic (unchanged — nothing breaks for an existing caller who set
one), and with nothing configured at all, they now default to local Ollama
(
http://localhost:11434/v1,llama3.2) instead of Anthropic.triage/classifier/llm.py:LLMClassifier.__init__behavior is unchanged — it alreadyraises
ValueErrornaming all backend options rather than silently picking one, which is themore honest design than defaulting to either vendor. Only the docstring and error-message
ordering changed (Ollama listed first, noted as needing no account or key).
docs/concepts/classifiers.md: new "Open by default" note; OpenAI-compatible section movedahead of Anthropic's.
README.md: classifier examples and the env-var configuration block now lead with Ollama.CLAUDE.md: Optional Extras table gains the previously-missing[openai]row ahead of[anthropic]; new design-decision entry recording the rationale.scripts/README.md: Benchmarks table and bash examples updated — "requires an LLM API key"(implying a paid vendor) corrected to "requires an LLM backend" with Ollama shown as the
no-key default path.
No
triage/core behavior changed — this is a default/framing correction in scripts and docs,not an API change.
Related issue
None
Type of change
LLMClassifieritselfunchanged)
Checklist
description are clean. I am the author and I am responsible for this change.
(N/A — see the "Ownership" exception above.)
pytest tests/ -x --tb=shortpasses locally (816 passed, unchanged — no new test surface,since
LLMClassifier's public behavior didn't change)ruff check .,ruff format --check ., andmypy triage/ --strictare all cleanopenai/anthropic/langchain/langgraph/opentelemetry/etc. insidetriage/core — the onlytriage/file touched (triage/classifier/llm.py) is adocstring/error-message reorder, zero new imports or behavior change
docs/concepts/classifiers.md,README.md,CLAUDE.md,scripts/README.md,CHANGELOG.md([Unreleased]→### Changed)FailureTypeorRecoveryActionaddedIf this touches
rules.pyor an error corpus — N/A, norules.pyor corpus file touched.Anything reviewers should look at closely?
_resolve_backend()(duplicated in all three scripts per this repo's self-contained-scriptconvention) is the load-bearing piece — worth checking its precedence order against your own
setup: explicit
TRIAGE_LLM_BASE_URL> explicit Anthropic credential (no base_url) > localOllama default. Verified all four cases (nothing set, Anthropic key set, custom base_url set,
--modeloverride) manually for each script since none of this is covered by the existing testsuite (these are opt-in scripts outside
triage/, not imported by any test).🤖 Generated with Claude Code
https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
Generated by Claude Code