Skip to content

feat: Model Arena — side-by-side model comparison #463

Description

@rexlunae

Inspiration: Unsloth

Unsloth's Model Arena lets you compare any two models side-by-side with the same prompt. This is especially powerful after fine-tuning to compare base vs. fine-tuned outputs, or when choosing between models for a task.

What RustyClaw could do

Add a model comparison feature accessible via the TUI/desktop client:

  1. Side-by-side chat: Send the same prompt to two different models (or two different providers) simultaneously, display responses next to each other
  2. Diff mode: Highlight differences between responses
  3. Blind test mode: Randomize which is A/B, reveal only after user votes
  4. Model comparison tool: An agent tool (model_arena) that lets agents themselves compare model outputs
  5. Latency/token tracking: Show side-by-side stats for each model

Scope

  • TUI client: split-pane view using ratatui
  • Desktop client: side-by-side panels in Dioxus
  • Gateway: support multi-model concurrent queries (already partially supported via session spawning)
  • Core: a compare_models tool for agents

Why this matters for RustyClaw

RustyClaw is provider-agnostic and encourages model experimentation. A built-in comparison tool makes that experimentation concrete rather than requiring users to manually copy-paste between sessions.

Prior art

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions