Skip to content

fairtask-eval skill: run the evaluation tooling from inside an agent session - #6

Closed
mnkprs wants to merge 1 commit into
mainfrom
eval-skill
Closed

mnkprs wants to merge 1 commit into
mainfrom
eval-skill

Conversation

@mnkprs

@mnkprs mnkprs commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Why

The screening skill (/fairtask) covers the product use, but everything else a reader might want to see — the evaluation set, scores, the comparison report, the evidence audit, one instance laid out — still required a terminal. Task authors and judges work inside agents; the evaluation should be reachable there too, with numbers that come from the scripts rather than from the model's memory.

What changed

  • skills/fairtask-eval/SKILL.md: maps a request to one of the repository's evaluation commands (data:eval-set, score, report, audit, code-check, data:annotations --check, show, trajectory), runs it from the repository root, prints the output verbatim, and adds one or two sentences on how to read it. Offline except the pinned data fetches; makes no model calls; refuses to start paid reproduction runs and points at REPRODUCE.md instead.
  • Plugin now carries two skills; CI's inventory check expects Skills (2); the manifest validator checks both skills' frontmatter.
  • README §2b mentions the skill; plugin description updated.

Verified by invoking the skill for "show the evaluation set": it ran npm run data:eval-set, printed the 30-row table, and git status stayed clean.

…n agent session

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mnkprs

mnkprs commented Aug 29, 2026

Copy link
Copy Markdown
Owner Author

Superseded by a clean branch: this one accidentally carried a 6-byte change to data/raw/ensembled_annotations_public.csv, which the provenance check in CI caught.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant