[vk-ai]

Review-only PR. Please don't merge yet.

What

This adds trigger precision/recall for skill descriptions. Today compare_skill / run_ab compute trigger_rate only over expect == "call" cases, so they measure recall. False triggers on prompts that should not load the skill are never measured, and each case runs once.

New skill_eval.trigger module:

• TriggerCase(tid, prompt, should_trigger, note=""). Use should_trigger=False for near-miss negatives: prompts that share words or a domain with the skill but belong elsewhere.
• run_triggers(cases, skill, agent, k=3, threshold=0.5, competing=())
  • Calls agent(case, offered, run_index) k times per case. offered is the skill under test followed by any competing skills.
  • The agent returns the loaded skill name, a collection of names, None, or a bool.
  • A case counts as triggered when it fires on at least threshold of its runs.
• TriggerReport gives:
  • precision, recall, false-trigger rate, F1 and accuracy, plus TP/FP/FN/TN counts
  • run_level raw fire rates for positive and negative cases
  • a flaky list (cases that fired on some runs but not all) and failures()
  • as_dict() and a markdown table() with the worst cases first
• from_observations(cases, skill_name, fired=[[...], ...]) scores runs you already logged.
• split_cases(cases, val_fraction=0.4, seed=0) gives a deterministic, stratified train/val split, so you can tune the description on train and report on val.

These are pure functions with no new deps. The existing A/B API is unchanged.

Why

Skill authors iterate on the description, and the failure mode is a description that fires everywhere. A recall-only metric rewards that.

• anthropics/skills skill-creator uses should-trigger and should-not-trigger queries with about 3 runs each. agentskills: optimizing descriptions recommends the same.
• anthropics/skills#1559: run_eval reports 0% trigger rate (fragile tooling)
• webmcp-skills#65: "Trigger eval can't measure description changes; add competing skills"
• modelcontextprotocol/inspector#2202: tune skill descriptions to a reliable trigger rate

How to try

pip install -e ".[dev]"
python examples/trigger_eval.py   # vague description: recall 1.00 but false-trigger 1.00; specific one: P=R=1.00
pytest -q

Tests

tests/test_trigger.py adds 7 tests:

• recall alone hides false triggers, and precision exposes them
• the k-repeat threshold, flaky detection and run-level rates
• competing skills are offered and resolved by name or set
• the table and dict list the worst cases first
• from_observations
• the stratified, deterministic split
• input validation

Locally: 13 passed (6 existing + 7 new) on Python 3.13. CI runs 3.10 and 3.12.

OSS / learning project; no production claims.