Add the Flaky Test Detector: find flaky tests by reading them, fix what you approve

[Ali Bahaloo (alibahaloo)]

Summary

Spectra's sixth command, speckit.spectra.flaky-test-detector, and the first that edits code the user
wrote.

Finding a flaky test conventionally means instrumenting CI, collecting hundreds of runs, and computing a
score — pipeline changes, a results store, and weeks of waiting before the first answer. The causes are
already sitting in the source: an unconditional sleep before an assertion, an un-awaited async call, state
one test leaves behind for another, a live network call, an unseeded random value, an assertion against
the real clock. This agent reads the suite and names them on the day you install it. No CI integration, no
results store, no run history.

Because it edits tests, the limits matter more than the detection. Four of them, each independent, so no
single misjudgment is enough to cause damage:

Limit	Why it is there
Executes nothing — not the suite, not a build, not an install, and not to verify a fix it just applied	The exclusion an agent would grant itself. Given the ability to run the tests it just edited, it can iterate until green — and iterating until green is how a test ends up asserting nothing.
A fix removes the cause	Deleting an assertion, loosening one until it always passes, skipping, marking expected-to-fail, adding a retry, lengthening a sleep — all named and forbidden. An item it cannot fix that way stays open with a reason.
Edits stay in test and test-support files	It may create a helper where a fix needs one; a mock with nowhere to live is a fix that never lands. Never production source, never a new dependency. Where the real remedy is in the application, it records what would change and where.
Nothing is committed	The working-tree diff is the last checkpoint before any of it becomes permanent.

Two gates, with pruning in between. The run reports a ranked table — test, file, confidence, concrete
fix — and stops. On a go-ahead it writes .specify/memory/flaky-test-analysis.md: a run summary, one
[ ] row per candidate, the evidence behind each, and what it could not examine. The developer deletes
the rows they disagree with. On a second go-ahead it works what is left, one item at a time.

The list outlives the session. Every run reads that file first and branches on its state: unfinished
work resumes without re-analysing and without discarding the pruning, a completed list is not replaced
without asking, and a file it cannot parse is never overwritten silently. Progress is written after each
individual fix, so an interrupted session leaves a file that is exactly true rather than one claiming
nothing happened. Before a run scoped to one suite replaces a broader plan, it names the pending items
that would be dropped.

Confidence is High/Medium/Low over the strength of the evidence, not a failure rate — with no run
history there is no denominator, so no percentage, score, or flakiness index is ever emitted. The
consumer project's own constitution binds which fixes are allowed: where a guardrail rules out the only
remedy, the item is left open with that rule named.

No template is registered for the analysis file. Principle VIII shapes deliverables; Principle VII
already places a .specify/ write in the category of context for another command, and here that command
is this one's next run. The deciding argument is narrower than the category, though: the file's structure
is a parse contract, and VIII's honour-don't-repair rule would oblige the command to accept an
override that renamed ## Tasks — which would not restyle the output, it would make the file unreadable
to the run that has to resume from it. Reasoning recorded in specs/018-flaky-test-detector/research.md
(R-002) and the plan's Constitution Check.

Spec, plan, research, data model, contracts, quickstart, and tasks:
specs/018-flaky-test-detector/. Derived from brds/flaky-test-detector.md (BRD-008), which
deliberately scopes down the QE Practice reference document — that one specifies the execution-telemetry
version of this capability, and remains a separate, later product.

Testing

• python3 -m unittest discover -s tests → 752 tests, OK (was 696), including a new 56-assertion
  tests/test_flaky_test_detector_flow.py covering the refusals, both gates, the prohibited-remedy list,
  the write scope, the four state branches, the honesty rules, and the guardrail binding.
• Mutation check — the new assertions passed on the first run, so they were tested against real
  regressions: softening the prohibited-remedy list, removing the no-verification clause, and permitting
  production edits each produced a failure; reverting restored green.
• Two deliberate negative guards: no template is registered for this command, and the command file
  references neither gh nor git — unlike create-pr and review-pr, it gates on no binary at all.
• python3 tools/generate_agent_docs.py --check → 46 agents, 6 prose blocks, roster and manifest
  agree.
• python3 tools/build_package.py → idempotent (identical hash across consecutive builds); the packaged
  zip carries spectra/commands/flaky-test-detector.md and a manifest reading 1.11.0.
• Installed into a throwaway project with specify extension add --dev: registers as
  speckit-spectra-flaky-test-detector, and specify extension info spectra lists six commands.

Release

Extension 1.10.0 → 1.11.0 (minor: a command is added; none renamed or removed, and no existing
command's behaviour changes). Manifest, catalog.json (5 → 6 commands), changelog, generated regions,
landing page, and spectra.zip are in sync. The roster entry moves from under-development to available
with its command recorded, and carries a hand-authored prose block. The CLI channel is untouched — no
tag, VERSION unchanged (Principle VI).

Blocked / not covered

Nothing blocked. Not verified end-to-end by an agent run: the command is a prompt, so the enforceable
surface is its text plus the tests on it. The eleven behavioural scenarios in
specs/018-flaky-test-detector/quickstart.md still need a human pass in a real project — the four worth
the time are the diff-level safety check, checkpointing across an interruption, the stale-plan guard, and
the refusals, none of which a unit test can replace. A scratch suite with six planted patterns and two
deliberately stable control tests is described in that quickstart for the purpose.

Also worth a reviewer's attention: this is the first Spectra command that modifies files it did not
create, in a project it did not author. The constitution says nothing about that category today, and this
change deliberately does not invent a principle — the limits live in the spec. If a second such agent
appears, they should be lifted into the constitution rather than copied.

🤖 Generated with Claude Code