Test-count drift in story records: counts transcribed by the LLM, not computed

[Pawel-N-pl]

Problem

Story records quote exact test totals (e.g. "+23 tests, 241/943"). Like the File
List (fixed in #41), these are transcribed by the dev agent from memory and
drift — e.g. claimed 235/935 vs actual 241/943; claimed 253/1027 vs actual
260/1076. Same root cause as #41 (transcription vs computation), same recurring
MEDIUM review finding that three "be more careful" guidelines (AI-1 → AI-2.1 →
AI-3.1) failed to close.

Why it's harder than the File List

The test runner is project-specific (phpunit / pytest / jest / go / …). JUnit XML
is the lever: parsing it is stack-agnostic — one parser reads
tests/failures/errors/skipped from any runner (assertions is PHPUnit-only, so
optional). Producing it is stack-specific, so it belongs in per-project config —
no per-runner logic in the tool.

Proposed direction

1. New test-counts --repo --story [--write] subcommand (sibling of
   reconcile-story).
2. A single configurable test_command in the policy override
   (_bmad/bmm/story-automator.policy.json) that emits JUnit to a known path.
3. Capture-first, re-run as the deterministic floor:
   • Tier 1: parse a fresh JUnit artifact from the dev run (no re-run).
   • Tier 2: no artifact + test_command set → the orchestrator re-runs it and
     parses JUnit.
   • Tier 3: neither → skip with a logged warning (File List reconcile still runs).
4. Nudge (optimisation, not a dependency): extend the automator's own dev prompt
   (data/prompts/dev.md) to ask the agent to emit JUnit and never omit the test
   summary. Tier 2 guarantees correctness if the agent ignores it.
5. Record test_counts {tests, failures, errors, skipped, assertions?} in
   data/parse/dev.json so the orchestrator stores real numbers instead of a
   self-reported tests_passed.

Design decisions (settled)

• Single canonical test environment. Test count must not differ across
  environments — only skipped does (env-gated tests). If a project containerises,
  the container is the canonical, prod-like env and test_command is the
  containerised invocation. No test_command / test_command_container duality,
  no per-env merging.
• Counts live in one machine-owned place — a ### Test Counts block under Dev
  Agent Record, plus test_counts in dev.json. They are removed from the
  Change Log / Completion Notes prose. ⚠️ Potential breaking change for anyone
  relying on counts in the Change Log.

Out of scope (separate, upstream)

A suggestion to bmad-method's core bmad-dev-story to emit JUnit natively would
raise the Tier-1 hit rate and cut double-runs. Decoupled from this issue.

[coderabbitai]

🔗 Related PRs

#2 - Fix tmux shell compatibility and Claude runner completion [merged]
#16 - feat(agent-config): per-task model selection for claude/codex [merged]
#17 - ci: make verification meaningful [merged]
#18 - Resolve BMAD artifacts path from config [merged]
#23 - fix(story-keys): complete non-numeric epic support [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!