feat: add test orchestration

CLAUDE.md

@@ -44,7 +44,7 @@ src/core/            # pure library — typed returns, injected deps, never prin
   types.ts, urls.ts
 src/mcp/             # thin frontend
   server.ts              # buildServer: registerTool/registerResource wiring (testable over InMemoryTransport)
-  tools/                 # the 11 bcdev_* tool definitions (zod schemas + metadata + handlers)
+  tools/                 # the 23 bcdev_* tool definitions (zod schemas + metadata + handlers)
   skills.generated.ts    # GENERATED from skills/ by scripts/embed-skills.ts — do not edit
   state.ts               # debug session singleton, event queue, run lock
   index.ts               # stdio entry: builds deps, calls buildServer, connects transport

README.md

@@ -5,7 +5,7 @@ MCP server for Business Central AL development: run tests (with code coverage) a
 [![Bun](https://img.shields.io/badge/bun-1.x-black)](https://bun.sh)
 [![TypeScript](https://img.shields.io/badge/typescript-strict-blue)](https://typescriptlang.org)
 [![BC dev API](https://img.shields.io/badge/BC%20dev%20API-%E2%89%A57.0-purple)]()
-[![Tests](https://img.shields.io/badge/tests-622%20passing-green)]()
+[![Tests](https://img.shields.io/badge/tests-641%20passing-green)]()
 
 ## Overview
 
@@ -24,6 +24,7 @@ MCP server for Business Central AL development: run tests (with code coverage) a
 |---------|-------------|
 | **Agent-grade responses** | Every success returns schema-validated `structuredContent` plus contextual `nextSteps`; failures use stable, redacted JSON error bodies |
 | **Structured test runs** | Run-level pass/fail/abort counts, per-method duration, parsed AL call stacks, and local source mappings while retaining raw server output |
+| **Test orchestration** | Repeat one test selection, diff adjacent passed/failed sets, retain every run, and classify stable, flaky, inconsistent, or incomplete methods |
 | **Code coverage** | `procedure` mode, mapped back to local source files (`line` mode exists but is unproven against real BC) |
 | **Coverage gap analysis** | Cross a Git diff with exact procedure coverage to report which changed procedures the selected tests did and did not exercise |
 | **Multi-codeunit plans** | Sequential codeunit groups over one hub connection |
@@ -74,9 +75,10 @@ Running from source instead: `git clone` → `bun install && bun run build` →
 1. `bcdev_status` — verify reachability, auth, and that test running is supported.
 2. `bcdev_test_discover` — list test codeunits and `[Test]` methods from local `.al` files.
 3. `bcdev_test_run { codeunits: [{ id: 50100 }], coverage: "procedure" }` — structured results. Add `coverageAgainst: "origin/main"` for changed-procedure analysis. After publishing the current changed objects to the target, also pass `changesDeployed: true` to permit `covered`/`uncovered` classifications; without that explicit assertion, changed procedures remain `unknown`.
-4. `bcdev_debug_attach { breakOnError: true }` → trigger the workload → `bcdev_debug_wait` for `sessionBound` and breaks → inspect with `bcdev_debug_variables` / `bcdev_debug_eval` → `bcdev_debug_continue` → `bcdev_debug_detach`. For a debug-bound test run, trigger with `bcdev_debug_run_tests { codeunits: [{ id: 50100 }] }`.
-5. To find writers of one table without stepping through every stop: `bcdev_record_writes_start { tableId: 18 }` → trigger the matching workload → check `bcdev_record_writes_status` → call `bcdev_record_writes_finish` for grouped stacks.
-6. To use Business Central's own MCP catalog: `bcdev_native_list { company: "...", context: "business" }` → inspect the returned schemas → `bcdev_native_call` with one exact returned tool name.
+4. `bcdev_test_orchestrate { codeunits: [{ id: 50100 }], runs: 3 }` — repeat the same selection, retain every run, diff adjacent pass/fail sets, and flag flaky methods.
+5. `bcdev_debug_attach { breakOnError: true }` → trigger the workload → `bcdev_debug_wait` for `sessionBound` and breaks → inspect with `bcdev_debug_variables` / `bcdev_debug_eval` → `bcdev_debug_continue` → `bcdev_debug_detach`. For a debug-bound test run, trigger with `bcdev_debug_run_tests { codeunits: [{ id: 50100 }] }`.
+6. To find writers of one table without stepping through every stop: `bcdev_record_writes_start { tableId: 18 }` → trigger the matching workload → check `bcdev_record_writes_status` → call `bcdev_record_writes_finish` for grouped stacks.
+7. To use Business Central's own MCP catalog: `bcdev_native_list { company: "...", context: "business" }` → inspect the returned schemas → `bcdev_native_call` with one exact returned tool name.
 
 Debugger attach returns as soon as Business Central accepts the request; binding is asynchronous. With no selector it binds the next session of the `breakOnNext` client type. Pass `userId` to filter that next session by Business Central user, or pass a known positive `sessionId` to attach to an existing NST session. `sessionId` and `userId` are mutually exclusive; exact `sessionId` targeting takes precedence over `breakOnNext`. `bcdev_debug_wait` reports `{ kind: "sessionBound", sessionId, hostId }` after binding; `hostId` may be null when Business Central omits that optional field. If identity lookup fails, it reports a nonfatal warning-form `sessionBound` event and debugging remains active. If Business Central emits a fatal user-filter rejection before `sessionBound`, the debugger tears down without binding or delivering breaks and reports an actionable `fatal` event.
 
@@ -117,7 +119,7 @@ Azure access tokens are acquired with `az account get-access-token`, cached only
 MCP client (agent)
   |
   v  (stdio)
-src/mcp/server.ts ── tools/ (22 bcdev_* tools) ── state.ts (debug/test/profile ownership)
+src/mcp/server.ts ── tools/ (23 bcdev_* tools) ── state.ts (debug/test/profile ownership)
   |
   v
 src/core/  (pure library — typed returns, injected deps)
@@ -137,6 +139,7 @@ src/core/  (pure library — typed returns, injected deps)
 | `bcdev_status` | Preflight: reachability, auth, dev API version, feature gates |
 | `bcdev_test_discover` | List test codeunits and `[Test]` methods from local `.al` files |
 | `bcdev_test_run` | Run tests with a summary, parsed/source-mapped failures, optional coverage, and Git-aware changed-procedure gaps |
+| `bcdev_test_orchestrate` | Repeat one selection 2–20 times, retain every run, diff adjacent pass/fail sets, and flag flaky/incomplete methods |
 | `bcdev_debug_attach` | Arm next-session/user-filtered attach or target an existing NST session; optionally set breakpoints |
 | `bcdev_debug_run_tests` | Run tests with breakpoints live |
 | `bcdev_debug_wait` | Long-poll for session-bound / break / run-finished lifecycle events |
@@ -188,7 +191,8 @@ requires an exact `company` and one explicit context:
 - `business` lists the dynamic `bc_actions_*` catalog. Pass `configurationName` only when selecting
   a named Business Central MCP configuration.
 - `runtime` lists the BC28 AL runtime catalog. Native runtime calls share the same singleton
-  test-run lock as `bcdev_test_run` and `bcdev_debug_run_tests`.
+  test-run lock as `bcdev_test_run`, `bcdev_test_orchestrate`, and
+  `bcdev_debug_run_tests`.
 - `debugging` lists troubleshooting tools for the active manual debugger. Attach, trigger the
   workload, and wait for a `break` first; a merely bound or resumed session is rejected.
 
@@ -261,10 +265,27 @@ Test runs add `summary`; failed method rows add `failure.message`, `failure.pars
 `failure.callStack` frames. The original `output` and each frame's `raw` text are retained so an
 unrecognized or localized server format never loses evidence. Local source mapping is lazy and
 best-effort: passing runs without coverage do not scan the AL tree, indexes are reused per MCP
-server and project, and an unreadable/missing project returns complete server results plus a
+server and project, and an unreadable/missing project preserves the server evidence plus a
 nonfatal `sourceMappingWarning` that identifies whether call-stack or coverage file mapping was
 unavailable.
 
+Use `bcdev_test_orchestrate` when repeatability is the question. It claims the same singleton
+test-run slot for the entire sequence, executes 2–20 ordinary TestRunnerHub runs sequentially,
+retains each enriched attempt, and returns exact adjacent additions/removals for the passed and failed
+sets. A method is `flaky` only with positive pass-and-fail evidence. Missing or duplicate
+observations, an aborted run, or a run with no real methods forces `complete: false`; mixed
+passed/skipped or failed/skipped observations are `inconsistent`, not mislabeled as flaky.
+If an attempt cannot start, earlier evidence and a final aborted attempt remain in `runs[]`.
+After an aborted attempt, later requested runs are left as missing rather than started while
+the prior server-side run may still be active. Client cancellation is observed between attempts;
+it does not cancel an active Business Central run. `tests[].method` keeps the first requested or
+observed spelling, so its casing can differ from raw server rows. An all-skipped selection retains
+the existing `passed` outcome convention but warns that it contains no passing execution evidence.
+Overlapping codeunit/method selections are rejected, while disjoint method groups for one codeunit
+remain valid. Non-rolled-back test side effects repeat on every attempt.
+Orchestration intentionally does not aggregate coverage — use `bcdev_test_run` for coverage and
+changed-procedure analysis.
+
 Pass `coverageAgainst` to `bcdev_test_run` for roadmap item 5's coverage-gap analysis. The tool
 resolves the ref's merge base with `HEAD`, compares that commit with the current working tree
 (committed branch changes, staged and unstaged edits, plus nonignored untracked AL files), and
@@ -325,9 +346,10 @@ seconds, preventing a stale or hung preflight from holding that slot indefinitel
 |------|---------|
 | `src/mcp/index.ts` | stdio entry: builds deps, calls buildServer, connects transport |
 | `src/mcp/server.ts` | buildServer: registerTool/registerResource wiring (testable over InMemoryTransport) |
-| `src/mcp/tools/` | The 22 bcdev_* tool definitions (zod schemas + metadata + handlers) |
+| `src/mcp/tools/` | The 23 bcdev_* tool definitions (zod schemas + metadata + handlers) |
 | `src/mcp/state.ts` | Debug session singleton, event queue, run lock |
 | `src/core/hubs/test-runner-hub.ts` | TestRunnerHub client (Initialize/RunTests, coverage) |
+| `src/core/test-orchestration.ts` | Pure repeat-run identity, stability classification, and adjacent pass/fail diff analysis |
 | `src/core/hubs/debugger-hub.ts` | DebuggerHub client (attach, breakpoints, stepping, inspection) |
 | `src/core/hubs/signalr-base.ts` | Hub seam: auth query params, key normalization, `HubProxy` |
 | `src/core/authorization.ts` | Shared Basic/Azure CLI authorization provider and token cache |
@@ -347,7 +369,7 @@ Ordered by intent, not commitment:
 3. **CPU profiling** (`bcdev_profile_*`) — **shipped.** Capture a sampling CPU profile (`.alcpuprofile`, V8 format) of a live session and get back a ranked AL-hotspot summary, not just a blob. Four tools (status/start/poll/finish), validated end-to-end against a live BC28 — a real profile with AL call frames captured through the tools (`scripts/e2e-profile-results-2026-07-04.md`). (Full snapshot recording for VS Code replay shares the same core and stays deferred.)
 4. **Entra ID auth** — **shipped.** Azure CLI-backed cloud sandbox/production access alongside explicit on-prem UserPassword.
 5. **Coverage gap analysis** — **shipped.** Pass `coverageAgainst` to `bcdev_test_run` to cross the merge-base-to-working-tree Git diff with exact procedure coverage and identify covered, uncovered, or unresolved changed procedures.
-6. **Test orchestration** — repeat runs, diff pass/fail sets, flag flaky tests.
+6. **Test orchestration** — **shipped.** Repeat one selected test plan, retain every enriched run, diff adjacent passed/failed sets, and flag flaky or incomplete method outcomes.
 7. **Break-on-record-write triage** — **shipped.** Arm bounded write triage for one numeric table ID and automatically collect grouped exact writer stacks.
 8. **BC native MCP passthrough** — **shipped.** Dynamically list and invoke BC28 cloud business-action, AL-runtime, and active-debugger native tools through one agent-facing bridge while retaining complete upstream results.
 9. **On-demand source & symbols** — fetch a server object's source, or download a single dependency package, without leaving the agent.

scripts/e2e-test-orchestration-2026-07-29.md

@@ -0,0 +1,126 @@
+# Test orchestration — SaaS evidence
+
+Validated on 2026-07-29 against a Business Central 28 SaaS Sandbox through the
+real `bcdev_test_orchestrate` handler. The harness accepted only
+`environmentType: "Sandbox"` and made no Production or on-premises call.
+
+Tenant, environment, company, user, token, authorization, authenticated URL,
+raw server payloads, failure text, call-stack text, durations for individual
+methods, and source paths were not retained. The fixture object and method
+names are also omitted from this evidence; stable role labels identify the two
+selections.
+
+## Stable passing selection
+
+- The default repeat count requested and completed three runs.
+- The retained run numbers were exactly 1, 2, and 3.
+- Every raw run summary reported `passed`, and every run retained at least one
+  real result row.
+- The aggregate returned `complete: true`, `outcome: "passed"`,
+  `tests: 1`, `stablePassed: 1`, and zero stable failures, flakes,
+  inconsistencies, or incomplete identities.
+- Both adjacent diffs had no passed/failed additions or removals and no changed
+  observation.
+
+## Stable failing selection
+
+- An explicit repeat count requested and completed two runs.
+- The retained run numbers were exactly 1 and 2.
+- Every raw run summary reported `failed`.
+- Every raw run retained nonempty failure output plus a parsed call stack whose
+  frames retained raw evidence and used only null or string local file fields.
+  No output, frame text, or path was written to this evidence.
+- The aggregate returned `complete: true`, `outcome: "failed"`,
+  `tests: 1`, `stableFailed: 1`, and zero stable passes, flakes,
+  inconsistencies, or incomplete identities.
+- The adjacent diff had no passed/failed additions or removals and no changed
+  observation.
+
+## Lifecycle and exclusions
+
+- The shared singleton test-run slot was released after both orchestration
+  calls.
+- Runs were issued through the existing TestRunnerHub client; no new wire
+  format or endpoint was introduced.
+- No deliberately flaky server test was added or claimed live. Pass/fail flaky
+  classification, skipped inconsistency, missing/duplicate observations,
+  safe stop after an aborted attempt, missing later observations, mixed
+  flaky/inconsistent guidance, and exact adjacent set diffs are covered by
+  deterministic core and fake-hub tests. The abort path was not induced during
+  this initial run; see the expanded rerun below.
+- No coverage, debugger, native runtime, profiling, record-write, Production,
+  or on-premises call was made during this acceptance run.
+
+## Expanded post-review rerun
+
+The corrected branch was exercised again through a real MCP `Client` and
+`buildServer`, with the production SignalR and Azure CLI authorization
+implementations. The harness refused Production-like environment names and
+printed only role labels, counts, classifications, and stable error codes.
+
+### Surface and retained evidence
+
+- `tools/list` exposed the orchestration annotations, the 2–20 run bounds, all
+  stability states, ambiguous observations, and additive `nextSteps`.
+- Four invalid inputs (below-minimum, above-maximum, fractional runs, and an
+  empty selection) returned `INVALID_ARGUMENT`, no `structuredContent`, and
+  input-schema recovery guidance without claiming the test-run slot.
+- The default stable-pass case completed three attempts. Deterministic failure,
+  mixed pass/fail, whole-codeunit, case-insensitive method, and two-codeunit
+  selections each completed two attempts with the expected stable
+  classifications and empty adjacent diffs.
+- The maximum accepted count completed all 20 attempts, retained 20 passing
+  observations, produced exactly 19 empty adjacent diffs, and returned no
+  warning.
+- Every returned text payload matched its `structuredContent`. Aggregate
+  duration equalled the retained real-row durations. Orchestration returned no
+  coverage payload or coverage-gap analysis.
+- Deterministic failures retained nonempty raw output, parsed frames, raw frame
+  evidence, and at least one local source mapping. Stable failure guidance
+  named the `stableFailed` classification; completed passing runs returned no
+  unnecessary next step.
+
+### Live fail-closed paths
+
+- A nonexistent selected method completed two server attempts. Each retained
+  the server's synthetic row, while the explicitly requested method received
+  two `missing` observations and the aggregate returned `complete: false`,
+  `outcome: "incomplete"`, warnings, and cautious guidance.
+- A nonexistent codeunit returned three non-aborted, empty attempt envelopes.
+  The tool invented no method identity and returned an incomplete aggregate
+  with warnings.
+- Repeating the same codeunit group twice in one plan produced a useful live
+  abort: the Sandbox returned one passing result, then aborted that attempt
+  before a duplicate row arrived. The orchestration retained that attempted
+  run, did not start the second requested attempt, represented the identity as
+  `passed, missing`, returned `complete: false`, and warned that server-side
+  cancellation could not be confirmed. A clean orchestration immediately
+  afterward succeeded, confirming lock release.
+
+This live duplicate-group result remains historical server evidence. The
+post-review correction now rejects overlapping codeunit/method selections at
+the MCP boundary while continuing to accept disjoint method groups for one
+codeunit; deterministic tests cover both shapes. No additional server call was
+needed for that input-only correction.
+
+The post-review correction also retains a synthetic aborted attempt when a
+later setup or authorization call rejects, observes client cancellation between
+attempts without claiming to cancel the active server run, aggregates duplicate
+row warnings per run, and warns on an all-skipped selection. These lifecycle
+and analysis paths are deterministic tests; they were not induced against the
+Sandbox.
+
+### Live singleton contention
+
+- While one five-attempt orchestration held the shared slot, one direct
+  `bcdev_test_run` and a second `bcdev_test_orchestrate` both returned
+  `TEST_RUN_ACTIVE` plus wait guidance.
+- The owning orchestration completed all five attempts, released the slot, and
+  a subsequent two-attempt orchestration succeeded.
+
+No deliberately flaky or skipped Sandbox fixture was introduced. Their
+classifications, mixed flaky/inconsistent guidance, duplicate-row ambiguity,
+and the debug-bound/native-runtime contention variants remain deterministic
+tests. No Production, on-premises, debugger, native MCP, profiling,
+record-write, source-download, package-download, or coverage call was made in
+this rerun.