eval: the 14-task smoke gate at 95% makes any single nondeterministic task a merge blocker #3394

Symptom

Run skill smoke tests is a required check. Today it blocked BPMN PRs repeatedly — #3377 once, #3380 three times — each time on a single task, needing a rerun before an approved, verified PR could merge.

Why one failure is fatal

The suite runs 14 tasks against a 95% pass-rate gate (.github/workflows/smoke-skills.yml). One failure is 13/14 = 92.9%, so the gate cannot tolerate a single nondeterministic task. There is no threshold between "all 14 pass" and "red".

Both of today's blockers were real behavior, not infra

Pulled from the smoke artifacts:

• skill-bpmn-parallel-fork-join — the agent authored <bpmn:ParallelGateway> (PascalCase), so FAIL: missing bpmn:parallelGateway is correct. Its own summary claimed "Validation result: Valid" because uip wasn't on PATH in that sandbox and its fallback was a bare XML well-formedness parse.
• skill-bpmn-operate-diagnose-minimal-fault-triage — read the mocked CLI's backing files; judge scored 0.000.

Neither is a guidance gap: SKILL.md:337-340 already forbids PascalCase tags at the point of use, and references/diagnose/CAPABILITY.md rule 3 already forbids reading mock/fixture files.

The model asymmetry

The smoke suite resolves vars.CODEX_MODEL = gpt-5.6-luna. The nightly arms run claude-sonnet-5, gpt-5.6-terra, and gemini-3.8-flash. Both nightly models scored parallel-fork-join 1.0 on 2026-09-17 — so the required gate is policing every PR with a model no nightly arm uses, and PRs get blocked on failures that the shipping configurations don't exhibit.

Options (not a recommendation — needs an owner's call)

1. Widen the suite. More tasks makes one failure survivable (20 tasks → 95% tolerates one).
2. Change the statistic. Gate on "no task regressed versus main" rather than an absolute rate, so a pre-existing nondeterministic task can't block an unrelated PR.
3. Align the smoke model with a nightly arm, so the gate measures a configuration we actually ship.
4. Document a rerun policy for the known-nondeterministic tasks, accepting the current friction.

Option 2 is the only one that also fixes the general case of "someone else's flaky task blocks my PR".

Evidence links: #3380's comment thread has the artifact-level diagnosis; run 35261849777 is the third parallel-fork-join failure.

🤖 Generated with Claude Code