Inspiration: Unsloth
Unsloth's Model Arena lets you compare any two models side-by-side with the same prompt. This is especially powerful after fine-tuning to compare base vs. fine-tuned outputs, or when choosing between models for a task.
What RustyClaw could do
Add a model comparison feature accessible via the TUI/desktop client:
- Side-by-side chat: Send the same prompt to two different models (or two different providers) simultaneously, display responses next to each other
- Diff mode: Highlight differences between responses
- Blind test mode: Randomize which is A/B, reveal only after user votes
- Model comparison tool: An agent tool (
model_arena) that lets agents themselves compare model outputs
- Latency/token tracking: Show side-by-side stats for each model
Scope
- TUI client: split-pane view using ratatui
- Desktop client: side-by-side panels in Dioxus
- Gateway: support multi-model concurrent queries (already partially supported via session spawning)
- Core: a
compare_models tool for agents
Why this matters for RustyClaw
RustyClaw is provider-agnostic and encourages model experimentation. A built-in comparison tool makes that experimentation concrete rather than requiring users to manually copy-paste between sessions.
Prior art
Inspiration: Unsloth
Unsloth's Model Arena lets you compare any two models side-by-side with the same prompt. This is especially powerful after fine-tuning to compare base vs. fine-tuned outputs, or when choosing between models for a task.
What RustyClaw could do
Add a model comparison feature accessible via the TUI/desktop client:
model_arena) that lets agents themselves compare model outputsScope
compare_modelstool for agentsWhy this matters for RustyClaw
RustyClaw is provider-agnostic and encourages model experimentation. A built-in comparison tool makes that experimentation concrete rather than requiring users to manually copy-paste between sessions.
Prior art