Description
Create a benchmark to measure and compare SQL generation accuracy across different LLM models (GPT-4o, Claude, Gemini, Ollama, etc.).
Requirements
Example usage
python -m pytest tests/test_benchmark.py --model gpt-4o
python -m pytest tests/test_benchmark.py --model claude-3-5-sonnet-20241022
Output
Model: gpt-4o
Accuracy: 18/20 (90%)
Failed: "Show monthly trend" - wrong GROUP BY
Description
Create a benchmark to measure and compare SQL generation accuracy across different LLM models (GPT-4o, Claude, Gemini, Ollama, etc.).
Requirements
Example usage
Output