Repository navigation
S4: Guard audit — mutation-test the new guards against both tests and code #89
Description
Epic: #85 · Order: Story 4 of 8 · Depends on: S2, S3
Background
S1–S3 add guards. This story asks the question those guards cannot ask about themselves: would
they actually catch anything?
The motivation is not hypothetical. This repo already had 248 passing tests, a golden byte-identity
suite, a pandas-free subprocess tripwire and three strict-xfail ratchets while every production
model was dead. A test count is not evidence. The specific failure was a guard whose fixture had
drifted from the contract it claimed to guard — and nothing in the suite could detect that.
Work
Run /falsify in guard mode against the guards added in S1–S3, plus the gate. Guard mode
mutates production code and asks which mutations leave the guards green.
- Mutate
views_baseline/model/catalog.pyand confirm each is caught:self.targets = list(config["regression_targets"])→config.get("regression_targets", [])
(does anything fail on an empty target list, or do we emit a zero-target frame?)→ config["classification_targets"]- drop the
list(...)(aliasing the config's list into the model — mutation leak) - swap two
_get_*builders'targets=for a hardcoded[]
- Mutate
views_baseline/infrastructure/reproducibility_gate.py:- remove
regression_targetsfromCORE_GENOME— S2's tests must fail - remove
level— likewise - reorder the four checks in
audit_manifestso the None check precedes the presence check - make
audit_manifestreturn early instead of raising
- remove
- Mutate the S3 conformance fixture — change
regression_targetstotargetsin the checked-in
config. The conformance test must fail; if it passes, the fixture is not being read the way the
test claims. - For every mutation that survives, do not patch it inline. Record it via
/register-risk
with the mutation as the trigger, and note it here. A surviving mutant is a finding about the
guard, and inventing a test to kill one specific mutant usually produces a worse suite. - Report the kill rate per file in this issue.
Acceptance criteria
- Every mutation listed above has a recorded verdict (killed / survived) with the failing test named
- Reverting S1 alone fails at least S3's conformance test
- Reverting S2 alone fails at least two gate tests
- Any surviving mutant is registered, not silently fixed
- Kill rates for
catalog.pyandreproducibility_gate.pyreported in a comment - No production code changed by this story unless a mutation exposed a genuine defect (say so if it did)
Validation
# per mutation, by hand or via /falsify guard mode:
conda run -n views_pipeline python -m pytest -q # must go RED for a killed mutant
git checkout views_baseline/ # restore between mutationsNotes — why this story exists
- Nygard, Release It Ch5: integration testing verifies only that the system works when its
dependencies work correctly; a mock constrained to the assumed interface cannot reveal that the
real interface moved. - Martin, Clean Architecture Ch28: the Fragile Tests Problem — tests coupled to structure
break on trivial change, so developers stop making changes. The inverse failure is this one:
tests coupled to a fixture that no longer matches reality never break at all. - Sculley et al. 2015, Hidden Technical Debt in ML Systems §4.4: configuration debt can exceed
code debt; each configuration line is a potential mistake, and configurations are "ephemeral and
less well tested." This outage was one config key.
Labels: story, testing
Activity
Polichinel commentedon Sep 9, 2026
Guard audit complete. 10 mutations, 8 killed, 1 real gap found and closed, 1 equivalent mutant.
Results
list(config["regression_targets"])→config.get("regression_targets", [])classification_targetsinsteadlist()copy (alias the config's list)targets=[]in two buildersregression_targetsfromCORE_GENOMElevelfromCORE_GENOMEaudit_manifestlogs but does not raise on missing coretargetskeyif k not in config and k != "level")The real finding: M5/M6
Deleting either newly-promoted key from
CORE_GENOMEleft all 277 tests green. S2 shippedwithout a guard on its own change: its tests assert that a config missing a key raises, but with
the key removed from the genome, every test config still has the key, so nothing fails. I had
verified the contract in a shell one-liner and never encoded it.
Closed with two tests (
tests/test_reproducibility_gate.py), and the shape matters:test_core_genome_declares_every_unconditionally_read_key— asserts the five keys by name,each annotated with where it is dereferenced. Deliberately not parametrized over
CORE_GENOME: atest that iterates the genome cannot catch a key being removed from it, because the
parametrize list just shrinks and the case stops running. That is precisely how M5/M6 hid.
test_each_core_key_is_enforced_not_merely_listed— deletes each key in turn and requires a loudrejection, so a key can't be listed-but-unaudited.
Both mutations now fail 2 tests each.
The equivalent mutant: M10
Exempting a key from check 1 leaves the suite green — but this is not a gap. Checks 1 and 4 of
audit_manifestoverlap: a missing key reaches check 4 asconfig.get(k) is None, so it is stillrejected, just with a different message:
The contract holds; only the diagnostic text changes. Recorded in the new test's docstring so the
next reader does not mistake that test for proof that check 1 works.
Worth noting as a design observation, not a defect: check 1 is largely redundant with check 4.
Its distinct value is the clearer message. Not changing it — that is exactly the kind of
consolidation that trades a real message for a theoretical tidiness.
Notes
git statusverified clean of mutation residue;views_baseline/__pycache__cleared and thesuite re-run to confirm no stale bytecode influenced a verdict.
283 passed, 1 xfailed,ruff check .clean.the outcome the story wanted. Nothing survived that needed carrying.
Polichinel commentedon Sep 9, 2026
Closed by #94, merged to
development(2b3d3b4). CI green:check (3.11),lint,build,check-branchall pass — the first green Lint and Test since 2026-08-02. Suite 248 → 295 passed.