SCRUM-1066-issue 216 - #468
Open
emnarahmouni-exmergo wants to merge 3 commits into
Open
Conversation
…into SCRUM-1066-issue-216 # Conflicts: # CHANGELOG.md # evals/__main__.py # evals/claude_agent.py # evals/runner.py # evals/tests/test_runner.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #216
evals/corpus/ade_bench_triggering.json: 30 real analytics-engineering requests sourced from dbt-labs/ade-bench (Apache-2.0), hand-labeled with the skill each should fire (ornone), with provenance and license recorded on the file itself.evals.jsonpositive/negativelist (written by whoever wrote the description it tests, checked with siblings disabled), this corpus runs every skill available at once and is scored as per-skill precision/recall/F1 (Corpus/CorpusCase/load_corpusinsuite.py,Classifierprotocol/run_corpus/SkillPrecisionRecallinrunner.py,ClaudeCliClassifierinclaude_agent.py,--corpusflag in__main__.py).python -m evals --corpus evals/corpus/ade_bench_triggering.jsonalways exits 0 — it's a measurement against externally authored prompts, not a release gate, since the initial pass rate is expected to be low.Classifier.classifyreports every skill marker a call finds via aClassifyResult(fired_skills, error), not a single winner: a prompt two skills both claim shows up as real cross-skill contamination (a false positive for the unwanted one) instead of being silently hidden behind whichever name came first.claudebinary missing) is left to propagate out ofrun_corpusimmediately instead, so it aborts the whole run once with the existing prerequisite message and exit code 2, rather than being recorded as N separate case errors while the command still reports a clean exit.references/evaluation.md's Tier 2 section.1.9.1section via an unrelated PR (SCRUM-1123-explore inventory: cap --rank's payload by default #379), claiming this feature shipped when none of its code had actually merged.Test plan
uvx ruff check evals//uvx ruff format --check evals/— cleanuvx pytest evals— 46 passed (16/16 intest_runner.py, including new regression tests for the cross-skill-contamination fix and the setup-failure-propagation fix); 6 pre-existingtest_wrapper.pyfailures are a Windows/PosixPathenvironment quirk that also fails on a cleanmain, unrelated to this diff