Skip to content

Mutation gate: forbid Stryker suppression comments in source #2050

Open
Open
Mutation gate: forbid Stryker suppression comments in source#2050

Description

@easonLiangWorldedtech
easonLiangWorldedtech
opened on Oct 14, 2026

Problem

// Stryker disable next-line ... suppression annotations keep spreading through production source files. main currently carries 38 directives across 9 files (Task.ts, pathUtils.ts, lm-studio.ts, qwen-code.ts, openai.ts, openai-codex.ts, abort-signal.ts, webviewMessageHandler.ts, NativeToolCallParser.ts), and every contributor who runs Stryker locally adds more.

The root cause is a policy gap, not a missing tool:

  • The changed-code mutation gate (scripts/stryker-diff.mjs, run by the mutation-diff CI job on every PR) already exists.
  • Its directive policy allows suppressions and validates only their shape (named mutator + reason, no All).
  • Malformed directives are reported as advisories — the job still exits 0, so nothing blocks them.

The result is the worst of both worlds: annotations accumulate, each one permanently silences a mutant class on that line, and the codebase trends toward an annotation soup where every contributor's favorite static-analysis tool leaves its fingerprints in comments.

Decision

Forbid Stryker suppression comments in source. The mutation gate is the tool of record in this repository's CI; its exclusions belong in gate configuration (maintainer-owned), not scattered through source as contributor-owned comments.

Plan (two PRs)

  1. Enforcement PR (ci(mutation-gate): forbid Stryker suppression comments in source #2051): scripts/stryker-diff.mjs replaces the advisory hygiene check with a hard failure — any // Stryker disable/// Stryker enable comment in a touched mutation-tested file fails the mutation-diff job. CI summary guidance, workflow step name, AGENTS.md and CONTRIBUTING.md updated. This PR does NOT touch the existing directives.
  2. Cleanup issue/PR (Mutation gate: remove the 38 legacy Stryker directives on main with evidence-backed central exclusions #2060 / chore(mutation-gate): remove all Stryker directives with evidence-backed exclusions #2054): removes the 38 directives already on main, backed by a per-line mutation repro — the 38 protected lines are re-exposed to Stryker with no directives, and each survivor is resolved either by a kill test (where killable) or by a narrow gate-config exclusion with the equivalence argument recorded (where genuinely equivalent).

Contributors whose mutant is genuinely equivalent explain the equivalence in the PR body; maintainers decide on narrow gate-config exclusions (e.g. excludedPaths in PACKAGE_CONFIGS).

Activity

  1. easonLiangWorldedtech commented on Oct 22, 2026

    @easonLiangWorldedtech
    easonLiangWorldedtech
    on Oct 22, 2026
    ContributorAuthor

    #2051 closed by the author: a direction change, not a technical failure

    Recording the closure of step 1 of this plan, so the plan of record below matches reality.

    Closure: #2051 (ci(mutation-gate): forbid Stryker suppression comments in source) was closed unmerged by the author at 2026-10-22T09:57:41Z, head 44ca22216893a7ab544852cd892243bfa0fb68e9 (+140/−55 across 5 files).

    What blocked it: edelauna's CHANGES_REQUESTED review at 2026-10-18T03:49:28Z (review id 5561319525, on the same head). Key sentence, verbatim:

    I'm not sure I want to take this approach just yet - there are some valid reasons for disabling Stryker. If anything the approach would be to make the report generated by Stryker more accessible for agents, because as it stands agents might not be aware of the findings, unless they navigate into the annotations, and then agents might not even know they can download the report as an artifact on the ci stryker job - I'm leaning towards closing this PR and approaching this differently.

    The PR was green, so this is a direction change rather than a technical failure:

    • 6/6 required checks success on 44ca22216 — check-translations, platform-unit-test (ubuntu-latest), platform-unit-test (windows-latest), compile, knip, e2e-mock; mutation-diff was green too.
    • CodeRabbit APPROVED (2026-10-18T03:36:21Z, same head). Pre-merge checks: 7 passed / 1 warning, and 0 of 5 review threads open.
    • Diff was 195 a+d across 5 files — no lockfile, no changeset, no CHANGELOG churn, no suppression-count increase.

    Nothing from #2051 landed on main (verified against 05d01f33d today):

    • scripts/stryker-diff.mjs still exports validateDisableDirectives (advisory shape check); assertNoDisableDirectives is absent.
    • The .github/workflows/mutation-testing.yml step is still named Enforce executable-line scope and run advisory mutation testing.
    • AGENTS.md still permits named + justified Stryker disable comments.
    • 40 Stryker disable directives remain across 10 files: pathUtils.ts 11, Task.ts 7, lm-studio.ts 5, qwen-code.ts 5, gemini.ts 3, openai.ts 2, openai-codex.ts 2, NativeToolCallParser.ts 2, webviewMessageHandler.ts 2, abort-signal.ts 1.

    Direction now pursued instead — discoverability, not prohibition:

    Effect on this issue's two-PR plan: step 1 (enforcement) is withdrawn in favour of the discoverability work in #2149; step 2 (#2054) now stands on its own as a cleanup of already-dead suppressions. #2051 will not be reopened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions