Bilingual Navigation: English (this document) · Versión en Español
Document Status: Draft Type: Test Strategy Satellite: Evolith Tracker Upstream: Evolith Core Date: 2026-06-07 Author: QA Agent + Architect Agent (BMAD) Resolves: FINDING-008 / GAP-005
This document defines the testing strategy for Evolith Tracker. It is the authoritative reference for what is tested, at which layer, with which tools, and against which thresholds — and it defines how the QA Phase Gate (Gate 4) is evaluated.
It derives its mandates from existing governance:
| Source | Mandate |
|---|---|
.harness/standards/tech-standards.md |
"Consumer-Driven Contract Tests (Pact) required for all core integrations"; "Strict TypeScript" |
| PRD EPIC-003 | Contract test execution, Change Failure Rate (CFR) < 2%, coverage heatmaps, Root Cleanliness |
| TAD | Hexagonal layers (domain/application/infrastructure/presentation), Unit of Work, Transactional Outbox |
| React Frontend Design | Vitest + React Testing Library + Playwright; microfrontend topology |
| Global Rules R-15 | Multi-tenancy: app-layer isolation primary, RLS secondary — both must be tested |
Out of scope (this document): load/performance testing methodology (covered in Hardening, Roadmap Phase 8) and security penetration testing (covered by the forthcoming Security Specification, GAP-006).
| # | Principle |
|---|---|
| P1 | Test behavior, not implementation. Domain tests assert business rules; they never assert ORM calls. |
| P2 | The domain layer is tested in pure isolation — zero NestJS, zero TypeORM, zero database (mirrors the TAD "Domain Layer Rule"). |
| P3 | Every acceptance criterion maps to at least one test. Traceability is mandatory (R-07). |
| P4 | Contracts are consumer-driven. Every cross-system integration (UMS, Core, .harness, frontend↔backend) is verified with Pact. |
| P5 | Tenant isolation is a first-class test target. Both application-layer scoping and RLS failsafe are tested (R-15). |
| P6 | Tests are deterministic and idempotent. No order dependence; no shared mutable state; UTC clocks injected. |
| P7 | The pyramid is respected. Many fast unit tests, fewer integration tests, few E2E tests. |
╱╲ E2E (Playwright) — critical SDLC journeys only
╱ ╲ ── few, slow, high-confidence
╱────╲ Contract (Pact) — every external boundary
╱ ╲ ── consumer-driven, run in both consumer & provider CI
╱────────╲ Integration (Jest + Testcontainers) — adapters, repos, RLS
╱ ╲ ── real PostgreSQL, real transactions
╱────────────╲ Unit (Vitest/Jest) — domain + application handlers
╱──────────────╲ ── many, fast, no I/O
| Level | Target | Tool | Speed | DB? |
|---|---|---|---|---|
| Unit — Domain | Aggregates, entities, value objects, domain services, policies | Vitest (or Jest) | ms | No |
| Unit — Application | Command/query handlers, Unit of Work orchestration (mocked ports) | Vitest/Jest + test doubles | ms | No |
| Integration | Repositories, ACL adapters, outbox, RLS, migrations | Jest + Testcontainers (PostgreSQL 15) | seconds | Yes (ephemeral) |
| Contract | UMS, Evolith Core, .harness, frontend↔backend REST |
Pact (consumer-driven) | seconds | No (provider state stubs) |
| Component (FE) | React components, hooks, permission-driven UI | Vitest + React Testing Library | ms | No |
| E2E | Cross-gate user journeys through the Shell Host + remotes | Playwright | minutes | Full stack |
No NestJS, no ORM, no DB. Construct aggregates directly and assert invariants and emitted domain events.
// modules/discovery/domain/aggregates/initiative.spec.ts
describe('Initiative', () => {
it('emits InitiativeApprovedEvent and transitions to APPROVED when the canvas gate passes', () => {
const initiative = Initiative.create({ /* valid canvas, ROI, KPIs */ });
initiative.submitForApproval();
const result = initiative.approve({ approver: poUser, justification: 'ROI validated' });
expect(result.isSuccess).toBe(true);
expect(initiative.status).toBe(InitiativeStatus.APPROVED);
expect(initiative.domainEvents).toContainEqual(
expect.objectContaining({ eventType: 'InitiativeApprovedEvent' }),
);
});
it('rejects approval when the RequirementChecklist is incomplete (BR-001)', () => {
const initiative = Initiative.create({ /* incomplete canvas */ });
const result = initiative.approve({ approver: poUser, justification: 'x' });
expect(result.isFailure).toBe(true);
expect(result.error).toBeInstanceOf(IncompleteChecklistError);
});
});Domain coverage targets: aggregate state transitions, every business rule (BR-001…BR-009), value object validation, domain service logic (e.g. DriftDetectionService, PhaseGateEvaluator).
Command/query handlers tested with in-memory repository fakes and a fake IUnitOfWork. Assert the orchestration contract: handler result, events persisted to outbox, transaction boundary respected.
// modules/discovery/application/commands/approve-initiative.handler.spec.ts
describe('ApproveInitiativeHandler', () => {
it('persists the initiative and writes InitiativeApprovedEvent to the outbox in one UoW', async () => {
const repo = new InMemoryInitiativeRepository([draftInitiative]);
const uow = new FakeUnitOfWork(); // captures work executed inside runInTransaction
const outbox = new FakeOutbox();
const handler = new ApproveInitiativeHandler(repo, uow, outbox);
const result = await handler.handle(new ApproveInitiativeCommand(initiativeId, poUser));
expect(result.isSuccess).toBe(true);
expect(uow.committed).toBe(true);
expect(outbox.events.map(e => e.eventType)).toContain('InitiativeApprovedEvent');
});
it('rolls back the UoW and writes no outbox event when approval fails', async () => {
// ... assert uow.rolledBack === true && outbox.events.length === 0
});
});Run against an ephemeral PostgreSQL via Testcontainers. Cover repositories, the outbox, migrations, and multi-tenancy.
// modules/discovery/infrastructure/persistence/typeorm-initiative.repository.int-spec.ts
describe('TypeOrmInitiativeRepository (integration)', () => {
it('round-trips an aggregate with optimistic concurrency (xmin)', async () => { /* ... */ });
it('enforces tenant isolation via RLS — a query under tenant B cannot read tenant A rows', async () => {
await withTenant(tenantA, () => repo.save(initiativeA));
const leaked = await withTenant(tenantB, () => repo.findById(initiativeA.id));
expect(leaked).toBeNull(); // RLS failsafe (R-15 secondary layer)
});
it('OutboxProcessor delivers pending events and marks processedAt (at-least-once)', async () => { /* ... */ });
});Mandatory integration scenarios:
| Scenario | Asserts |
|---|---|
| Cross-schema isolation | No FK between tracker_* schemas; UUID references only |
| RLS tenant isolation | Tenant B cannot read/write Tenant A rows (R-15 secondary) |
| Application-layer tenant scoping | TenantContextMiddleware sets app.current_tenant_id (R-15 primary) |
| Unit of Work atomicity | Handler failure rolls back BOTH aggregate writes AND outbox inserts |
| Outbox at-least-once | Events survive a simulated crash between publish and processedAt update |
| Migration up/down | Each migration is reversible; schema matches the data design |
Use @nestjs/testing + supertest to exercise controllers, guards, pipes, and interceptors with the UMS adapter stubbed.
// modules/discovery/presentation/initiative.controller.e2e-spec.ts
it('returns 403 when the caller lacks tracker:initiative:approve (fail-closed)', async () => {
umsAdapter.setPermissions(['tracker:initiative:read']);
await request(app.getHttpServer())
.post(`/api/v1/initiatives/${id}/approve`)
.expect(403);
});Per tech-standards.md, every core integration boundary is covered by a consumer-driven Pact contract. The consumer defines expectations; the provider verifies them in its own CI.
| Consumer | Provider | Contract covers |
|---|---|---|
| Tracker backend | UMS | Token validation, authorization graph resolution, effective permissions |
| Tracker backend | Evolith Core | Ruleset fetch, artifact definitions, taxonomy, compliance validation |
| Tracker backend | .harness |
Pipeline trigger, test-result ingestion, quality-gate signal |
| Tracker frontend (each remote) | Tracker backend | REST endpoints consumed by each microfrontend (OpenAPI-aligned) |
| Tracker backend (webhook consumer) | GitHub / Jira ACLs | Inbound webhook payload shape → canonical event mapping |
Pact workflow:
- Consumer test generates a pact file (expected request/response per interaction).
- Pact file is published to a Pact Broker (or shared artifact in CI).
- Provider CI runs
pact-verifyagainst the consumer's expectations using provider states. - A provider change that breaks a consumer contract fails the provider build — surfacing drift before deploy.
Why Pact and not just OpenAPI validation: OpenAPI validates shape; Pact validates the actual interactions a consumer depends on, catching breaking changes the schema alone would miss (e.g. a field a consumer relies on being removed from a specific response).
| Level | Tool | Target |
|---|---|---|
| Unit/Component | Vitest + React Testing Library | Components, hooks, usePermission gating, form validation (Zod) |
| Contract | Pact (consumer side) | Each remote's REST calls to the backend |
| Module Federation integration | Vitest + MF test harness | Shell Host loads each remote; error boundary isolates a crashing remote |
| E2E | Playwright | Full journeys through the composed Shell + remotes |
Microfrontend-specific test mandates (per T-002):
| Mandate | Test |
|---|---|
| Fault isolation | A thrown error in mfe-qa is caught by the Shell error boundary; other remotes stay interactive |
| Shared singletons | react, react-dom, @tanstack/react-query resolve to a single instance across remotes |
| Permission-driven UI | A remote renders no privileged action when the shared auth context denies the permission |
| Design System consistency | Remotes consume ui-kit primitives only (lint + visual checks) |
// packages/auth — permission-driven UI component test
it('hides the Approve button when the user lacks tracker:initiative:approve', () => {
renderWithPermissions(<ApproveButton initiativeId="x" />, { permissions: [] });
expect(screen.queryByRole('button', { name: /approve/i })).toBeNull();
});E2E tests cover critical cross-gate journeys only — not exhaustive UI coverage. Each maps to a PRD use case.
| E2E Journey | Use Case | Gates traversed |
|---|---|---|
| Submit → approve Discovery Canvas → backlog generated | UC-001 | Gate 1 |
| Approved initiative → contract + ADR submitted → blueprint approved | UC-002 | Gate 2 |
| Construction task → drift alert raised on deviation | UC-003 | Gate 3 |
| QA cycle → contract tests run → quality gate verdict | UC-004 | Gate 4 |
| Release planned → authorized → deployment recorded | UC-005 | Gate 5 |
| Full AI-Native run via CLI/MCP (agent assignment → gate pass) | UC-006, UC-008 | All |
E2E runs against a seeded multi-tenant environment with UMS and .harness in stub/sandbox mode.
Coverage is enforced per layer in CI. Thresholds reflect risk: domain logic is held to the highest bar; generated/presentation code lower.
| Layer | Line | Branch | Rationale |
|---|---|---|---|
| Domain | ≥ 95% | ≥ 90% | Business rules are the product; near-total coverage |
| Application (handlers) | ≥ 90% | ≥ 85% | Orchestration + UoW correctness |
| Infrastructure (adapters) | ≥ 80% | ≥ 70% | Integration-tested; some glue is low-risk |
| Presentation (controllers) | ≥ 80% | ≥ 70% | Guard/permission paths must be covered |
| Frontend components | ≥ 80% | ≥ 70% | Permission gating fully covered |
Coverage is a floor, not a goal. A green coverage number with weak assertions still fails review (P1).
| Concern | Approach |
|---|---|
| Aggregate builders | Test data builders (anInitiative().approved().build()) — no raw object literals |
| Tenant fixtures | At least two tenants seeded in every integration suite to prove isolation |
| Clock | IClock port injected; tests use a fixed UTC clock for determinism |
| IDs | Deterministic UUIDs in tests; gen_random_uuid() only in production paths |
| External systems | UMS / Core / .harness replaced by Pact stubs (contract) or fakes (unit) |
PR pipeline (every commit):
lint (strict TS, no unused imports) → unit (domain+application) → component (FE)
→ integration (Testcontainers PG) → contract (Pact consumer) → build
main / provider pipeline:
pact-verify (provider) → e2e (Playwright, seeded) → coverage gate → Root Cleanliness
Gate 4 (QA Phase Gate) pass criteria — maps to PRD success metrics:
| Criterion | Threshold | Source |
|---|---|---|
| Change Failure Rate (CFR) | < 2% | PRD §9, BR-003 |
| All Pact provider verifications | pass | tech-standards.md |
| Coverage floors (§8) | met per layer | this document |
| Critical E2E journeys | green | §7 |
| Root Cleanliness | zero unauthorized root files | PRD EPIC-003 |
| No open Critical/High defect | enforced | PRD EPIC-003 |
A failing Gate 4 blocks Release authorization (BR-003 — non-negotiable system lock).
The .harness external CI system is the execution surface for contract testing and regression. The Tracker integrates via an ACL (tracker-integration context):
| Direction | Interaction |
|---|---|
Tracker → .harness |
Triggers contract-test + regression runs on QA cycle start |
.harness → Tracker |
Posts results (pass/fail, CFR, coverage) via webhook → TestExecutionCompletedEvent |
| Tracker → Gate | TestCycle aggregate computes pass rate; PhaseGateEvaluator emits QualityGatePassed/FailedEvent |
The .harness payload contract is itself Pact-verified (§5) so a .harness format change cannot silently break gate evaluation.
Per R-07, every test traces to an acceptance criterion, and every business rule traces to a test.
| Business Rule | Verifying Test Level | Example |
|---|---|---|
| BR-001 (no Design without cleared Canvas) | Domain + E2E | Initiative.approve checklist test; UC-001 E2E |
| BR-002 (no Construction without Blueprint) | Application + E2E | Design gate handler; UC-002 E2E |
| BR-003 (no deploy without QA gate) | Integration + Gate 4 | PhaseGateEvaluator; CFR threshold |
| BR-004 (drift triggers alert) | Domain | DriftDetectionService unit test |
| BR-006 (tenant isolation absolute) | Integration | RLS + app-layer scoping tests (§4.3) |
| BR-007 (agent deliverables = human) | Application | Deliverable validation handler |
| BR-008 (CLI/MCP = Web parity) | Contract + E2E | Same backend contracts exercised by all surfaces |
| BR-009 (agent actions audited) | Integration | Audit trail append assertion |
A pending Story breakdown (GAP-019, BMAD SM Agent) will produce story-level acceptance criteria; each criterion must be linked to a test ID at that point.
- Tracker Target Architecture (TAD) — layers, Unit of Work (§12.1), Outbox (§13)
- PostgreSQL Data Design — schemas, RLS, migrations
- React Frontend Design — microfrontend topology
- PRD — EPIC-003, success metrics
- Tech Standards — Pact mandate
- Global Rules — R-07, R-15
- Pact — Consumer-Driven Contracts