You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add more real-world questions to the retrieval eval #5
Kitty's tutor is checked by a retrieval eval: 52 real-world Git questions, each with the source files that should come back in the top 6 results (hit@6). More questions = better answers, because every miss points at a gap we can fix.
What to do
Add 5–10 new questions to kitty/ingest/eval/questions.json. Write them the way a real person would ask (short, casual, even with typos), on topics that aren't covered yet, e.g. git tag, git worktree, .gitattributes, git clean, sparse checkout, git log filters, signing commits.
For each one, list the source files that answer it in expected (source names are in kitty/server/data/index.json, or run the eval with --verbose to see what comes back).
Run:
cd kitty
npm ci
npm run eval -w ingest -- --verbose
Open a PR with the new questions. If one is a MISS, say so in the PR; finding a miss is a useful contribution too.
Activity
Rpann commentedon Nov 3, 2026
Can i work on it ??
Krish6115 commentedon Nov 3, 2026
@Rpann Sure u can work on it!!
Krish6115 commentedon Nov 3, 2026
Done in #10. Thanks @Rpann for the 10 new eval questions! They all pass and bring the retrieval eval to 62/62. 🐾