test(metadata): stop conflating drift with the warning gate - #446
Merged
Conversation
The --no-ai case asserted on main()'s bare exit code. --check returns 1 for output drift as well as for skill warnings, so any PR that edits skill content failed this test for reasons unrelated to enrichment. PR #445 is the first to hit it: 70 changed files including SKILL.md at 62+/147-, which legitimately drifts the checked-in metadata. Neutralise diff_text for the duration of the run so the exit code is attributable to the warning gate alone. Drift is already covered by case 4. A first attempt simply excused drift with 'rc == 0 or drifted'. That was worse than the original bug: removing the --no-ai gate entirely still passed, because drift was present. Mutation-checked properly this time. Verified against PR #445's content: reproduces the failure without this change, passes with it, and still fails when the --no-ai gate is removed -- on both #445's tree and a clean one. Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
--no-aicase intest_generate_skill_metadata.pyasserted onmain()'s bare exit code:--checkreturns1for output drift as well as for skill warnings. The behaviour being pinned is the warning gate — unenriched skills must not fail PR CI when no inference key is available — but the assertion cannot tell the two causes apart.So any PR that edits skill content fails this test for reasons unrelated to enrichment. #445 is the first to hit it: 70 changed files including
SKILL.mdat 62+/147-, which legitimately drifts the checked-in metadata. My defect, currently blocking someone else's PR.The irony is that the test was added in
276b9bcto prevent exactly this class of problem — a change that red-lines every open PR — and its own commit message says "adds test cases pinning both halves so neither can regress."Fix
Neutralise
diff_textfor the duration of the run so the exit code is attributable to the warning gate alone. Drift is already covered by case 4.What I got wrong first
The first attempt excused drift instead of isolating it:
That was worse than the original bug. Mutation testing showed why: removing the
--no-aigate entirely — the exact regression the case exists to catch — still passed, because drift was present and the assertion excused it. A test that cannot fail is worse than no test.Verification
--no-aigate removed--no-aigate removedgenerate-skill-metadata.py --check --no-aiNote on the with-AI case
Added a comment recording that
unenrichable new skill with AI available -> fails the runasserts the outcome, not the route. In practicerc=1arrives there via schema validation before the warning gate is consulted, so it does not pin the gate — the--no-aicase is what does. Pre-existing, documented rather than redesigned.Scope
Test file only. The generator is untouched.
This does not guarantee #445 turns green — that PR still drifts
metadata.jsonlegitimately, socheckmay fail for a correct reason. This only removes my defect from the picture.