Skip to content

llm-judge passes on unclamped scores, a 0-100 reply always passes #35

Description

@royalpinto007

Problem

A stub judge returning {"score": 85} on a wrong answer yields {score: 1, passed: true}: the raw number is compared to the threshold before clamping.

Where

src/scorers/llm-judge.ts (Number(obj.score), passed = score >= threshold; util.ts clamps only the reported score).

Acceptance

Add a failing test first asserting score 85 yields passed: false/parse-reject. Then clamp (or reject) before the pass check.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions