feat: free-model speed + quality eval with tracked results

[cbangera2]

Adds a runnable eval across all live OpenRouter :free models plus the first (partial) results table. Full writeup in evals/free-models-results.md.

Results 2026-09-09 (partial — daily quota exhausted midway, see below)

model	pass	avg total	avg TTFT	failures
cohere/north-mini-code:free	7/7	11187ms	10652ms	—
dots-studio/dots-3-note-preview:free	7/7	15674ms	14908ms	—
nvidia/nemotron-3-super-120b-a12b:free	7/7	38631ms	38229ms	—
nex-agi/nex-n2.5-mini:free	5/7	6863ms	7498ms	1x rate_limited, 1x quality
nvidia/nemotron-3-ultra-550b-a55b:free	5/7	38235ms	43567ms	1x rate_limited, 1x quality
nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free	5/7	78045ms	72176ms	2x quality
nex-agi/nex-n2.5-pro:free	4/7	77632ms	27873ms	3x timeout
liquid/lfm-2.5-2.6b:free	3/7	6614ms	6849ms	1x rate_limited, 3x quality
inclusionai/ling-3.0-flash-sante:free	2/7	1829ms	1508ms	5x rate_limited
thinkingmachines/inkling-small:free	0/7	—	—	7x unsupported (403 agentic-only)
thinkingmachines/inkling:free	0/7	—	—	7x unsupported (403 agentic-only)
all others (9 models)	0/7	—	—	quota noise, untested

Caveats

• Fresh OpenRouter keys get ~50 free-model reqs/day; a full run needs 126, so the tail is quota noise, not quality. The script now classifies failures (rate_limited/unsupported/timeout/quality) and stops early on daily-quota errors — rerun after reset for a clean pass.
• TTFT ≈ total on every model (queue wait dominates); tokens/sec in raw JSON is the better speed signal.
• Flags for follow-up: both inkling models 403 on direct API yet sit on DEMO_MODEL_ALLOWLIST; demo default nemotron-3.5-lightning never got tested.

What's in this PR

• evals/free-models.eval.mjs (+ pnpm eval:free-models) — live :free discovery, 7 finance cases, TTFT/total/tok-s metrics, ranked table + JSON
• .env.example — OPENROUTER_KEY setup (wrangler secrets can't be read back via CLI)
• evals/free-models-results.md — tracked summary (raw JSON stays gitignored)

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: b02969f3-a84b-40b8-98f3-0c4650a68e08

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}