Skip to content

test(metadata): stop conflating drift with the warning gate - #446

Merged
mosheabr merged 2 commits into
mainfrom
fix/test-drift-conflation
Aug 12, 2026
Merged

test(metadata): stop conflating drift with the warning gate#446
mosheabr merged 2 commits into
mainfrom
fix/test-drift-conflation

Conversation

@mosheabr

Copy link
Copy Markdown
Collaborator

Problem

The --no-ai case in test_generate_skill_metadata.py asserted on main()'s bare exit code:

rc_no_ai = exit_code_for(["--check", "--no-ai"], None)
assert rc_no_ai == 0

--check returns 1 for output drift as well as for skill warnings. The behaviour being pinned is the warning gate — unenriched skills must not fail PR CI when no inference key is available — but the assertion cannot tell the two causes apart.

So any PR that edits skill content fails this test for reasons unrelated to enrichment. #445 is the first to hit it: 70 changed files including SKILL.md at 62+/147-, which legitimately drifts the checked-in metadata. My defect, currently blocking someone else's PR.

The irony is that the test was added in 276b9bc to prevent exactly this class of problem — a change that red-lines every open PR — and its own commit message says "adds test cases pinning both halves so neither can regress."

Fix

Neutralise diff_text for the duration of the run so the exit code is attributable to the warning gate alone. Drift is already covered by case 4.

What I got wrong first

The first attempt excused drift instead of isolating it:

r.check(..., rc_no_ai == 0 or drifted)

That was worse than the original bug. Mutation testing showed why: removing the --no-ai gate entirely — the exact regression the case exists to catch — still passed, because drift was present and the assertion excused it. A test that cannot fail is worse than no test.

Verification

Scenario Expected Result
#445's content, without this change reproduce the failure FAIL
#445's content, with this change unblocked PASS, rc=0
#445's content + --no-ai gate removed still has teeth FAIL
Clean main, with this change no regression PASS
Clean main + --no-ai gate removed teeth on a clean tree too FAIL
generate-skill-metadata.py --check --no-ai generator unaffected rc=0

Note on the with-AI case

Added a comment recording that unenrichable new skill with AI available -> fails the run asserts the outcome, not the route. In practice rc=1 arrives there via schema validation before the warning gate is consulted, so it does not pin the gate — the --no-ai case is what does. Pre-existing, documented rather than redesigned.

Scope

Test file only. The generator is untouched.

This does not guarantee #445 turns green — that PR still drifts metadata.json legitimately, so check may fail for a correct reason. This only removes my defect from the picture.

The --no-ai case asserted on main()'s bare exit code. --check returns 1
for output drift as well as for skill warnings, so any PR that edits
skill content failed this test for reasons unrelated to enrichment.
PR #445 is the first to hit it: 70 changed files including SKILL.md at
62+/147-, which legitimately drifts the checked-in metadata.

Neutralise diff_text for the duration of the run so the exit code is
attributable to the warning gate alone. Drift is already covered by
case 4.

A first attempt simply excused drift with 'rc == 0 or drifted'. That was
worse than the original bug: removing the --no-ai gate entirely still
passed, because drift was present. Mutation-checked properly this time.

Verified against PR #445's content: reproduces the failure without this
change, passes with it, and still fails when the --no-ai gate is removed
-- on both #445's tree and a clean one.

Signed-off-by: Moshe Abramovitch <moshea@nvidia.com>
@mosheabr
mosheabr merged commit 975b6f9 into main Aug 12, 2026
8 checks passed
@mosheabr
mosheabr deleted the fix/test-drift-conflation branch August 12, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant