Skip to content

fairtask-eval skill: run the evaluation tooling from inside an agent session - #7

Merged
mnkprs merged 1 commit into
mainfrom
eval-skill-2
Aug 29, 2026
Merged

mnkprs merged 1 commit into
mainfrom
eval-skill-2

Conversation

@mnkprs

@mnkprs mnkprs commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Why

The screening skill (/fairtask) covers the product use, but everything else a reader might want to see — the evaluation set, scores, the comparison report, the evidence audit, one instance laid out — still required a terminal. Task authors and judges work inside agents; the evaluation should be reachable there too, with numbers that come from the scripts rather than from the model's memory.

What changed

  • skills/fairtask-eval/SKILL.md: maps a request to one of the repository's evaluation commands (data:eval-set, score, report, audit, code-check, data:annotations --check, show, trajectory), runs it from the repository root, prints the output verbatim, and adds one or two sentences on how to read it. Offline except the pinned data fetches; makes no model calls; refuses to start paid reproduction runs and points at REPRODUCE.md instead.
  • Plugin now carries two skills; CI's inventory check expects Skills (2); the manifest validator checks both skills' frontmatter.
  • README §2b mentions the skill; plugin description updated.

Verified by invoking the skill for "show the evaluation set": it ran npm run data:eval-set, printed the 30-row table, and git status stayed clean. (A first version of this branch, #6, accidentally carried a 6-byte edit to the annotation CSV; CI's provenance check caught it, which is what it is for.)

…n agent session

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mnkprs
mnkprs merged commit fab07f9 into main Aug 29, 2026
2 checks passed
@mnkprs
mnkprs deleted the eval-skill-2 branch August 29, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant