Calibration Analysis (Brier Score)

This part was my “wait… accuracy isn’t enough” moment.

Accuracy tells me how often the model gets ARC questions right. But it doesn’t tell me whether the model knows when it’s right—or worse, whether it confidently blurts out wrong answers. That second part is what calibration is about, and it matters a lot once you deploy an LLM into any real workflow where you need to decide:

• Should I trust this answer?
• Should I ask a human to review?
• Should I abstain and say “I’m not sure”?

Brier Score is a clean way to put a number on that.

---

What Brier Score measures (in this evaluation)

For each question, I compute:

• Outcome o:

  • o = 1 if the model’s predicted answer is correct
  • o = 0 if incorrect

• Confidence p:
  A probability-like value derived from the model’s first output token log probability: [ p = \exp(\log p_{\text{first token}}) ]

Then the Brier Score is:

[ BS = \frac{1}{N}\sum_{i=1}^{N} (p_i - o_i)^2 ]

Intuition

Brier Score is basically “mean squared error,” but for probability forecasts:

• If the model is very confident and correct (e.g., p ≈ 1, o = 1) → small penalty
• If the model is very confident and wrong (e.g., p ≈ 1, o = 0) → big penalty
• If the model is uncertain (e.g., p ≈ 0.5) → penalty is moderate, whether right or wrong

So Brier Score doesn’t just reward correctness—it rewards honest confidence.

---

Why this matters specifically for LLM evaluation

In LLM systems, calibration isn’t academic—it’s operational:

• Selective answering / abstention: If confidences are meaningful, you can set a threshold and only answer when the model is likely correct.
• Human-in-the-loop routing: Low confidence can be a trigger to escalate to a reviewer.
• Safer automation: Overconfident errors are often the most harmful class of failures.
• Model comparison beyond accuracy: Two models can have similar accuracy, but the one with better calibration is usually safer and easier to integrate into decision pipelines.

Brier Score captures exactly that: how much you should trust the model’s confidence as a decision signal.

---

My implementation (per spec)

I followed the instructions exactly:

1. Request token logprobs using the Responses API include parameter.
2. Extract the logprob of the first output token, regardless of whether that token is the answer letter.
3. Convert to confidence with exp(logprob).
4. Compute the binary outcome o from correctness.
5. Compute Brier Score and report it to 4 decimal places.

---

Result

• Brier Score: 0.0987
• Valid confidence values: 299/299

A lower Brier score indicates better calibration (i.e., the model’s confidence aligns more closely with correctness), especially by avoiding “confident but wrong” behavior.