Repository navigation
Calibration: Aligning rubric scores #1258
Description
Is your feature request related to a problem?
We lack trustworthy ground truth for rubric refinement as the team's human scores are not aligned. This misalignment prevents us from determining whether the agent performs close to human evaluations.
Describe the solution you'd like
- Use ~45–50 CSF submissions for calibration.
- Have Kartikeya, Pritam, and Prashant evaluate and align scores pairwise.
- Document reasoning/comments for each score as qualitative ground truth.
- Assess the agent's refined rubric using only numeric scores.
- Experiment with inputs both with and without program information.
- Incorporate explainability to ensure user understanding.
- Aim to complete experiments by mid-November and prepare a v1 release.
Original issue
Context
We don't yet have trustworthy ground truth for rubric refinement — on CSF the team's own human scores didn't align. Before judging whether the agent is "close to human," the team must calibrate among itself. The core product question is which inputs NGOs must supply (golden set, program info, comments) so we can tell them exactly what to provide in the console.
Plan / Acceptance criteria
- Take ~45–50 CSF submissions as a sample set.
- Kartikeya, Pritam and Prashant act as human evaluators and calibrate pairwise until aligned.
- Record the reasoning/comments behind each score — these become the qualitative ground truth (the "true" rubric).
- Give the agent only the numeric scores (blind to comments) and check whether its refined rubric converges on what the team wrote.
- Run variants with and without program information to learn which inputs the agent actually needs.
- Design explainability in: if users can't understand what the agent is doing, they won't use it.
- Target: finish experiments by mid-November, with a v1 release to share.
Activity
[-]Run blind CSF rubric-refinement experiment with human calibration ground truth[/-][+]Calibration: Aligning rubric scores[/+]