Skip to content

Add more real-world questions to the retrieval eval #5

Closed

Description

@Krish6115

Kitty's tutor is checked by a retrieval eval: 52 real-world Git questions, each with the source files that should come back in the top 6 results (hit@6). More questions = better answers, because every miss points at a gap we can fix.

What to do

  1. Add 5–10 new questions to kitty/ingest/eval/questions.json. Write them the way a real person would ask (short, casual, even with typos), on topics that aren't covered yet, e.g. git tag, git worktree, .gitattributes, git clean, sparse checkout, git log filters, signing commits.
  2. For each one, list the source files that answer it in expected (source names are in kitty/server/data/index.json, or run the eval with --verbose to see what comes back).
  3. Run:
    cd kitty
    npm ci
    npm run eval -w ingest -- --verbose
  4. Open a PR with the new questions. If one is a MISS, say so in the PR; finding a miss is a useful contribution too.

No API key is needed. The eval runs offline.

Activity

  1. Rpann commented on Nov 3, 2026

    @Rpann
    Contributor

    Can i work on it ??

  2. Krish6115 commented on Nov 3, 2026

    @Krish6115
    OwnerAuthor

    @Rpann Sure u can work on it!!

  3. Krish6115 commented on Nov 3, 2026

    @Krish6115
    OwnerAuthor

    Done in #10. Thanks @Rpann for the 10 new eval questions! They all pass and bring the retrieval eval to 62/62. 🐾

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions