Skip to content

SCRUM-1066-issue 216 - #468

Open
emnarahmouni-exmergo wants to merge 3 commits into
mainfrom
SCRUM-1066-issue-216
Open

SCRUM-1066-issue 216#468
emnarahmouni-exmergo wants to merge 3 commits into
mainfrom
SCRUM-1066-issue-216

Conversation

@emnarahmouni-exmergo

Copy link
Copy Markdown
Contributor

Closes #216

  • Adds evals/corpus/ade_bench_triggering.json: 30 real analytics-engineering requests sourced from dbt-labs/ade-bench (Apache-2.0), hand-labeled with the skill each should fire (or none), with provenance and license recorded on the file itself.
  • Unlike each skill's own evals.json positive/negative list (written by whoever wrote the description it tests, checked with siblings disabled), this corpus runs every skill available at once and is scored as per-skill precision/recall/F1 (Corpus/CorpusCase/load_corpus in suite.py, Classifier protocol/run_corpus/SkillPrecisionRecall in runner.py, ClaudeCliClassifier in claude_agent.py, --corpus flag in __main__.py).
  • python -m evals --corpus evals/corpus/ade_bench_triggering.json always exits 0 — it's a measurement against externally authored prompts, not a release gate, since the initial pass rate is expected to be low.
  • Classifier.classify reports every skill marker a call finds via a ClassifyResult(fired_skills, error), not a single winner: a prompt two skills both claim shows up as real cross-skill contamination (a false positive for the unwanted one) instead of being silently hidden behind whichever name came first.
  • A per-call failure is caught by the classifier itself and recorded per case, excluded from precision/recall. A setup failure (the claude binary missing) is left to propagate out of run_corpus immediately instead, so it aborts the whole run once with the existing prerequisite message and exit code 2, rather than being recorded as N separate case errors while the command still reports a clean exit.
  • Adds a short mention of the corpus to references/evaluation.md's Tier 2 section.
  • Removes a stray changelog entry that had leaked into the already-released 1.9.1 section via an unrelated PR (SCRUM-1123-explore inventory: cap --rank's payload by default #379), claiming this feature shipped when none of its code had actually merged.

Test plan

  • uvx ruff check evals/ / uvx ruff format --check evals/ — clean
  • uvx pytest evals — 46 passed (16/16 in test_runner.py, including new regression tests for the cross-skill-contamination fix and the setup-failure-propagation fix); 6 pre-existing test_wrapper.py failures are a Windows/PosixPath environment quirk that also fails on a clean main, unrelated to this diff
  • Zero em-dashes across every touched file
  • Manually verified the corpus JSON's provenance/license fields and spot-checked several hand-labels against their source ADE-bench task prompts

…into SCRUM-1066-issue-216

# Conflicts:
#	CHANGELOG.md
#	evals/__main__.py
#	evals/claude_agent.py
#	evals/runner.py
#	evals/tests/test_runner.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

evals: add a real-world prompt corpus as a triggering fixture

1 participant