Skip to content

feat: add Ollama support for local AI models#31

Open
poiley wants to merge 4 commits into
mainfrom
feat/ollama-support
Open

feat: add Ollama support for local AI models#31
poiley wants to merge 4 commits into
mainfrom
feat/ollama-support

Conversation

@poiley

@poiley poiley commented Aug 21, 2025

Copy link
Copy Markdown
Contributor

Summary

Features

  • Local AI Models: Support for llama3, qwen2, mistral, phi3, gemma, and other Ollama models
  • CLI Integration: --model flag for model selection, --server flag for server configuration
  • Smart Fallback: AdaptiveClient prefers Ollama but falls back to Anthropic Claude if needed
  • Cross-Platform: Works on Windows, macOS, and Linux
  • Robust Validation: Pre-analysis connectivity and model availability checking
  • Clear Error Messages: Helpful troubleshooting guidance for common issues

Usage Examples

# Use local llama3 model
rulectl start --model llama3

# Custom server address
rulectl start --model qwen2 --server localhost:11434

# Remote Ollama server
rulectl start --model mistral --server 192.168.1.100:11434

Technical Implementation

  • AdaptiveClient: BAML client with fallback strategy (Ollama → Claude)
  • Environment Variables: Dynamic configuration via OLLAMA_BASE_URL and OLLAMA_MODEL
  • OpenAI-Compatible: Uses openai-generic provider for Ollama integration
  • Connection Validation: Pre-flight checks for server connectivity and model availability
  • Error Recovery: Graceful fallback to cloud providers on Ollama failures

Documentation

  • OLLAMA_SUPPORT.md: Comprehensive setup and usage guide
  • Updated README.md: Basic usage examples and feature overview
  • Clear Default Behavior: Uses Anthropic Claude when no --model specified

Benefits

  • Cost Savings: No API costs for local models
  • Privacy: Code analysis stays on local machine
  • Offline Capability: Works without internet connection
  • Model Choice: Access to open-source models via Ollama

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

@poiley
poiley requested a review from etbyrd August 21, 2025 01:06
@poiley

poiley commented Aug 21, 2025

Copy link
Copy Markdown
Contributor Author

I recommended merging the following PRs, rebasing this branch, and moving documentation to docs/ before merging this PR.

@poiley
poiley force-pushed the feat/ollama-support branch 3 times, most recently from 202c853 to d1775a4 Compare September 2, 2025 21:51
poiley and others added 4 commits September 2, 2025 18:45
- Add --model and --server CLI flags for Ollama configuration
- Implement AdaptiveClient with fallback strategy (Ollama → Claude)
- Add comprehensive Ollama connectivity validation and error handling
- Support all popular Ollama models (llama3, qwen2, mistral, phi3, gemma)
- Add robust error messages and troubleshooting guidance
- Include cross-platform support (Windows, macOS, Linux)
- Update all BAML functions to use AdaptiveClient
- Add comprehensive documentation in OLLAMA_SUPPORT.md
- Update README.md with Ollama usage examples

Resolves #28

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
When using the --model flag with Ollama, skip API key prompts entirely and use
OllamaOnlyClient for purely local analysis without cloud fallback.

Changes:
- Add OllamaOnlyClient to BAML configuration for local-only analysis
- Modify ensure_api_keys() to skip prompts when use_ollama=True
- Update RepoAnalyzer to accept ollama_only parameter
- Add aiohttp dependency for Ollama connectivity
- Use appropriate client (OllamaOnlyClient vs AdaptiveClient) based on mode

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add 54 tests covering all Ollama integration features
- Test connection validation, environment setup, CLI flags
- Test BAML client configuration and fallback behavior
- Test API key handling and error scenarios
- Include test runners and documentation
- All tests pass with proper async mocking

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@poiley
poiley force-pushed the feat/ollama-support branch from 9d6c65e to acd0547 Compare September 3, 2025 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant