[Documentation] Close canonical architecture coverage and live traceability gaps

[seonghobae]

Why this issue exists

PR #149 establishes the first coherent canonical mightyETL documentation spine, but the documentation graph must remain code-current and must not become a one-time snapshot that immediately drifts while other commercial-readiness work continues.

The current protected baseline is develop@622e5e6c3d534f230c390f10e3832efadfc01825. PR #149 supplies or rewrites the core families: PRD, TRD, Architecture, Security, ADR index + detailed ADRs, UML, ERD, API contract, Threat Model, Test Strategy, Operability, Traceability, Documentation Assessment, AGENTS/CLAUDE/CHANGELOG, plus machine-checkable documentation tests.

That is structurally much stronger than protected develop, but it is not yet sufficient as a durable acquisition-grade source of truth because live repository work continues to expand while #149 remains Draft.

Live drift discovered after #149 was created

The canonical graph must reconcile all material work below without promoting it to shipped truth before protected integration:

• fix(data): retire legacy auth tables from clean bootstrap #155 — retire abandoned local-auth bootstrap objects from new default installations while preserving explicit compatibility handling;
• fix(connectors): remove Qlik scaffold from production registry #156 — remove the misleading Qlik row-write scaffold from production registry/catalog/configuration binding;
• feat(api): add machine-readable HTTP and CDC contracts #157 — add checked-in machine-readable OpenAPI/AsyncAPI contracts without promoting active-PR behavior to shipped truth;
• fix(cdc): remove MySQL scaffold from production discovery #158 — remove the nonfunctional MySQL Debezium scaffold from automatic Spring production discovery;
• fix(security): move Jackson to fixed 2.21.5 baseline #160 — shared Jackson 2.21.5 security baseline fixing inherited jackson-databind advisories exposed by fix(cdc): remove MySQL scaffold from production discovery #158;
• [Product/Security Gap] Replace direct ETL HTTP Basic trust boundary #161 — known_gap / architecture decision for replacing the direct etl-service HTTP Basic trust boundary rather than hiding it from OpenAPI;
• [Quality gate] Make JaCoCo fail closed when production class selection is empty #162 and fix(coverage): make JaCoCo production gate non-vacuous #164 — shared quality-gate defect and active repair for JaCoCo's vacuous zero-production-class coverage pass;
• fix(cdc): remove SQL Server scaffold from production discovery #163 — remove the nonfunctional SQL Server Debezium scaffold from automatic Spring production discovery;
• [Product gap] Eliminate production scaffold-only connector implementations #153 — canonical connector-support lifecycle issue tying Qlik/MySQL/SQL Server active retirements to the still-planned Databricks/Snowflake terminal decisions.

All open PRs above are active_pr; issues describing unresolved product/security/control work are planned or known_gap as appropriate. None is implemented_on_develop until the corresponding exact code and evidence integrate into protected history.

Fresh writer / source-identity evidence

PR #149 is actively being modified by another writer. Its PR body still names historical head 5f1c10891c32c5c56f1ab97ae8eea529d0466423, while GitHub's live ref is 74c24a3a8950009fdffc2d5f660f40ee3250b863 on unchanged base develop@622e5e6c3d534f230c390f10e3832efadfc01825. This mismatch is direct evidence that exact-head/run facts embedded in the body are already stale and that this loop must not race that source branch.

PR #157 shows the same pattern: its body names historical head d43c257c044ed3006980317bd8932cb5b22a32ab, while its live ref is aa558f25363bfb833e49ed79b49cfdbea39c8eea. PR #164 is also under an independent live writer and currently sits at e9cec632d0b2336040a3e86363dc0348e0cab059; its hosted workflows are queued and therefore not passing evidence.

These SHAs are dated execution evidence only. Canonical architecture must describe capability/lifecycle state, not freeze unstable pull-request head identities into timeless design prose.

Sufficiency decision

Core architecture families — present in #149

The following are considered the minimum canonical spine and are represented by #149:

• Product Requirements (PRD.md)
• Technical Requirements (TRD.md)
• System Architecture (ARCHITECTURE.md)
• ADR index and detailed status-bearing ADRs (docs/adr/)
• UML / component / sequence / state / deployment views (docs/UML.md)
• ERD / logical data model with persisted-vs-planned distinction (docs/ERD.md)
• API/error/event contract entry point (docs/API_CONTRACT.md)
• Security and Threat Model
• Test Strategy
• Operability
• Requirement/decision/code/test/PR traceability
• Documentation completeness assessment
• AGENTS / CLAUDE / CHANGELOG alignment

This is necessary but not sufficient by file count alone.

Remaining cross-cutting coverage that must be explicit and discoverable

For each category below, either maintain a dedicated canonical document or explicitly index the existing authoritative section(s). Do not create duplicate prose merely to satisfy a filename checklist.

1. Roadmap / lifecycle status — dependency order, accepted issues, active PRs, planned work, superseded paths, release milestones and exit criteria.
2. Data governance / privacy / retention — data classification, payload retention, principal/tenant authority, purpose limitation, deletion/retention, audit evidence, PII handling without blanket masking, and known gaps.
3. Migration / rollback / compatibility — database migrations, connector/API compatibility, stack replacement rules, rollback/forward-recovery, downgrade constraints and operator evidence.
4. Release / provenance — release acceptance, versioning, CHANGELOG, SBOM/provenance, reproducibility, artifact/license/NOTICE requirements, rollback and protected-release proof.
5. Deployment / failure domains — standalone and modular MSA deployment profiles, dependencies that are optional vs required, degraded modes, queue/broker/database failures and recovery.
6. Research / standards doctoring — one discoverable index from product/ADR decisions to authoritative standards and APA 7 references where material; feature-specific doctoring may remain distributed if indexed.
7. Connector support matrix — production / scaffold / removed-from-discovery / planned status, transport/auth prerequisites, target/source semantics, support evidence and non-goals. This must reconcile [Product gap] Eliminate production scaffold-only connector implementations #153/fix(connectors): remove Qlik scaffold from production registry #156/fix(cdc): remove MySQL scaffold from production discovery #158/fix(cdc): remove SQL Server scaffold from production discovery #163 and must not advertise reference-only scaffolds as supported connectors.
8. Operational SLI/SLO evidence — distinguish target requirements from actually measured attainment; never invent SLA/SLO success from prose.
9. Acquisition-diligence controls — licensing/copyright status ([Acquisition diligence] Establish explicit repository licensing and copyright policy #151), dependency/license obligations, security review, tenant/data authority, backups/recovery, supply chain, and residual known risks.
10. Identity / trust-boundary authority — gateway JWT (fix(gateway): replace placeholder token handling with JWT resource server #142) and downstream ETL authentication ([Product/Security Gap] Replace direct ETL HTTP Basic trust boundary #161) are separate boundaries; canonical Security/Threat/Deployment/API material must not imply that an edge JWT control automatically solves direct/east-west service authentication.
11. Quality-gate evidence semantics — document that synthetic-merge-only source execution and vacuous zero-class coverage are not accepted release evidence; bind ci: schedule NVIDIA OpenCode maintenance agent #121 and [Quality gate] Make JaCoCo fail closed when production class selection is empty #162/fix(coverage): make JaCoCo production gate non-vacuous #164 to the exact controls they repair.

Required live traceability reconciliation

After #149 stabilizes or after a successor documentation branch is created from protected develop:

• add fix(data): retire legacy auth tables from clean bootstrap #155, fix(connectors): remove Qlik scaffold from production registry #156, feat(api): add machine-readable HTTP and CDC contracts #157, fix(cdc): remove MySQL scaffold from production discovery #158, fix(security): move Jackson to fixed 2.21.5 baseline #160, [Product/Security Gap] Replace direct ETL HTTP Basic trust boundary #161, [Quality gate] Make JaCoCo fail closed when production class selection is empty #162/fix(coverage): make JaCoCo production gate non-vacuous #164, fix(cdc): remove SQL Server scaffold from production discovery #163, [Product gap] Eliminate production scaffold-only connector implementations #153 and any subsequently opened material PR/issue to the canonical status/traceability graph;
• preserve the exact status vocabulary (implemented_on_develop, active_pr, planned, superseded, out_of_scope, known_gap);
• bind each material status to actual source/migration/API/security/quality evidence and relevant tests;
• explicitly distinguish shared prerequisites (fix(security): move Jackson to fixed 2.21.5 baseline #160 Jackson, fix(coverage): make JaCoCo production gate non-vacuous #164 coverage, ci: schedule NVIDIA OpenCode maintenance agent #121 source-identity controls) from feature-local behavior so checks from one PR are never treated as transferring to another;
• when one of those PRs merges, change the canonical status in the same integration path or an immediately following bounded documentation reconciliation PR;
• close/relabel superseded issue/PR references only when exact ancestry/diff proves unique work has transferred;
• never reuse checks/reviews/approvals from a predecessor head or base.

Documentation-as-code acceptance

Add or extend tests so they fail when material documentation drifts from live repository contracts. At minimum verify where practical:

• canonical documentation/index files resolve;
• ADR index, ADR IDs, statuses and supersession links are consistent;
• Mermaid/code fences parse or pass structural checks;
• public service/API names and lifecycle status vocabulary match source contracts;
• protected baseline cannot advertise active_pr functionality as shipped;
• newly removed scaffold connectors are not still documented as production-capable;
• machine-readable OpenAPI/AsyncAPI artifacts, once feat(api): add machine-readable HTTP and CDC contracts #157 integrates, are linked and versioned from the canonical API/event documentation;
• database object names/statuses in the ERD remain synchronized with actual migrations and compatibility artifacts;
• release, migration, rollback, data-governance and licensing authority locations are discoverable from the documentation index;
• security architecture distinguishes gateway authentication from direct/east-west ETL authentication;
• test strategy cannot call zero analyzed production classes 100% coverage and identifies the fail-closed non-vacuity invariant;
• source/provenance docs distinguish literal PR source from GitHub synthetic merge revisions;
• stale literal exact-head/run IDs are restricted to dated evidence, not timeless architecture assertions.

Writer/stack safety

Do not race an active writer on #149. The live-vs-body head mismatch above proves the branch is currently moving. Reconcile it read-only and perform source documentation updates only after that branch is stable or from a new direct-develop successor after #149 merges/closes. A documentation branch conflict blocks only that branch; continue other non-conflicting mightyETL work.

The same branch-local freeze applies to #157 and #164 for the remainder of the invocation because their source heads moved under another writer. Metadata/issue truth can still be maintained without mutating those source refs.

Do not deepen or rewrite the #121→#143→#148 stack solely to make documentation current. Document active stack state truthfully and repair source ancestry only for independent integration value.

Completion criteria

This issue can close only when:

• the canonical documentation spine is integrated on protected develop;
• the cross-cutting categories above each have one clearly discoverable authoritative location or an explicit documented non-applicability decision;
• the live material PR/issue set is represented by source-backed status/traceability without stale shipped claims;
• connector support, trust boundaries, source identity, and quality-gate evidence semantics agree with protected code/workflows;
• documentation consistency tests pass on the exact integrated head;
• security/data/operability/migration/release/standards links are coherent;
• no maintainer or buyer must reconstruct the current product architecture from chat history or PR bodies.

Documentation completion is not a commercial-loop terminal state. After this issue or #149 is green/merged/waiting, the autonomous loop must immediately return to the live implementation/security/product queue and continue safe work.

[seonghobae]

Concrete follow-up findings from the current #149 documentation diff:

1. docs/adr/README.md uses Accepted direction / implementation pending for ADR-0005, but the same index defines only Proposed, Accepted, Accepted with known gaps, Superseded, and Rejected. Either define that additional ADR status formally or normalize ADR-0005 to a defined status. This is ADR-status vocabulary drift, distinct from product capability status.
2. docs/TRACEABILITY.md defines product status as exactly implemented_on_develop, active_pr, planned, superseded, out_of_scope, known_gap, but the central-scanner row currently uses the compound status text ``read_only_dependencyrepresented asplanned` from mightyETL perspective`. The Status cell should contain one canonical product status (likely `planned`), with dependency-ownership detail moved to Source/Evidence/Notes.
3. CanonicalDocumentationContractTest.adrIndexCarriesCoreCrossCuttingDecisions() currently proves ADR IDs and broad Accepted/known gaps tokens exist, but does not validate every ADR index row against the ADR-status vocabulary. Extend the documentation-as-code gate so undefined compound ADR statuses fail deterministically.
4. The canonical product-status tests have improved to bind several capabilities to exact rows, but they do not yet bind the newly opened fix(data): retire legacy auth tables from clean bootstrap #155–fix(cdc): remove MySQL scaffold from production discovery #158 work or ensure future material PRs are represented. Once those branches stabilize/integrate, add source-backed rows and assertions rather than generic token-presence checks.
5. docs/UML.md, docs/ERD.md, and docs/OPERABILITY.md materially satisfy component/sequence/state/deployment, persisted-vs-active persistence, and recovery/SLO-objective coverage. Do not split those into duplicate files solely for naming consistency; index them as the authoritative equivalents.

These are follow-through defects, not justification to race the currently moving #149 branch. Fix after branch stability or in a safe successor, then return to the broader PR/product queue.

[seonghobae]

Live traceability expansion after this issue's last body refresh: new issue #165 ([Release/Supply chain] Establish reproducible exact-head artifacts and provenance) is now the canonical planned owner for the release/provenance/reproducibility gap already identified here. It was opened only after a fresh protected-repository search found no dedicated release/provenance workflow, attestation path, or equivalent existing issue. Do not represent #165 as implemented_on_develop; implementation is intentionally deferred behind #121 source-identity controls, #162/#164 non-vacuous coverage, #151 licensing posture, and the remaining exact protected release prerequisites. When #149 stabilizes or a successor documentation branch becomes safe, add #165 to the canonical status/traceability graph and index its release/provenance authority rather than duplicating the issue prose.

[seonghobae]

Dated live-traceability correction (2026-08-09, after another executable sweep): several exact identities in this issue's body are now historical and should stay evidence-only rather than be copied into timeless architecture.

• fix(coverage): make JaCoCo production gate non-vacuous #164 live source is f8fa8504f0cceb2f072befda09eef3d13169b5b4 on unchanged develop@622e5e6c3d534f230c390f10e3832efadfc01825. Its non-vacuous JaCoCo selector now actually analyzes 8 production classes. CI 31317634945 correctly fails on genuine residual coverage (7 missed instructions, 2 missed lines, 2 missed branches); Dependency Review, SBOM, SAST and Security are green. Canonical Test Strategy must therefore describe fix(coverage): make JaCoCo production gate non-vacuous #164 as active_pr, not as a completed 100% proof and not as the earlier stale-policy-test state.
• fix(cdc): remove SQL Server scaffold from production discovery #163 live source is d96d1c815511dffe8d48fa9d8b6eb0c92b353304, not the head recorded in its body. Its five aggregate workflows are now green and its diff includes an explicit factory-level contract proving configured sqlserver-debezium remains unregistered / unknown_source_type; it is still active_pr, not shipped truth.
• feat(api): add machine-readable HTTP and CDC contracts #157 live source is aa558f25363bfb833e49ed79b49cfdbea39c8eea. Its truthful OpenAPI Basic-auth scheme has a valid unresolved GHAS/Semgrep finding (yaml.openapi.security.use-of-basic-authentication.use-of-basic-authentication). The finding has been explicitly dispositioned in the live review thread as requiring runtime remediation through [Product/Security Gap] Replace direct ETL HTTP Basic trust boundary #161 rather than contract falsification or suppression. Canonical Security/API/Threat material should keep ETL direct/east-west Basic authentication as known_gap / planned remediation until the runtime trust boundary changes.

The current documentation implication is semantic, not SHA-centric: (1) source identity and coverage non-vacuity are separate release-evidence controls; (2) SQL Server retirement is active but unshipped; (3) machine-readable contracts may intentionally surface a real protected-runtime security gap and should not be edited to make scanners green. Keep exact SHAs/run IDs in dated evidence only. #149 is still a moving writer-owned branch, so this loop is not mutating its source.

[seonghobae]

Fresh read-only reconciliation of the current canonical-docs branch (#149 live head 74c24a3a8950009fdffc2d5f660f40ee3250b863 over protected develop@622e5e6c3d534f230c390f10e3832efadfc01825) found concrete live traceability drift that should be repaired by the active #149 writer or a safe successor; this loop will not race that moving source branch.

Current exact-head aggregate workflows for #149 are now terminal-success: CI 31315551730, Dependency Review 31315551721, SBOM 31315551733, SAST 31315551751, and Security Scan 31315551738. Those greens do not turn the branch into protected truth and do not solve current source-identity / coverage / review-governance constraints.

Exact docs/TRACEABILITY.md at 74c24a3... still says the non-vacuous JaCoCo repair is “sequenced after overlapping #157 POM work stabilizes” / “must be repaired after overlapping POM work stabilizes.” That dependency assumption is now disproven by repository state: #164 explicitly revalidated that #157 does not modify etl-service/pom.xml, #164 is already the active direct-develop repair, and its current live head is f8fa8504f0cceb2f072befda09eef3d13169b5b4. Hosted CI on that head reaches a real non-empty selected bundle of 8 production classes and now fails correctly on genuine residual debt (7 missed instructions, 2 missed lines, 2 missed branches). The canonical row should therefore become active_pr #164 (with issue #162 as the underlying known_gap) and must not retain the falsified #157 sequencing dependency.

The same current traceability file also omits several material live lanes that issue #159 already requires:

• fix(cdc): remove SQL Server scaffold from production discovery #163 SQL Server CDC scaffold retirement is absent from the core and conversation connector rows. Its live head is now d96d1c815511dffe8d48fa9d8b6eb0c92b353304; current aggregate CI/dependency/SBOM/SAST/security are green, CodeRabbit's executable unknown_source_type/MD058 findings are addressed, but literal-source, non-vacuous shared coverage and independent approval still block shipped status.
• [Product/Security Gap] Replace direct ETL HTTP Basic trust boundary #161 direct etl-service HTTP Basic trust-boundary replacement is not represented as a known_gap. Protected default Compose publishes ETL directly on port 8000 and protected ETL independently authenticates /api/** with HTTP Basic, so gateway JWT fix(gateway): replace placeholder token handling with JWT resource server #142 must not be documented as solving this separate service boundary.
• [Release/Supply chain] Establish reproducible exact-head artifacts and provenance #165 reproducible exact-head artifacts/provenance is not represented as planned, despite being the repository's explicit release/supply-chain authority gap.
• The connector reconciliation row still says only active_pr #156/#158 plus issue #153, so it does not reflect the live SQL Server retirement fix(cdc): remove SQL Server scaffold from production discovery #163.

Required safe docs remedy: update source-backed rows/tests to bind #162=known_gap + #164=active_pr, add #163/#161/#165 with the canonical taxonomy, remove the disproven #157-POM blocker text, and keep every active PR unshipped until protected integration. Do not copy unstable SHAs/run IDs into timeless architecture prose; these identities belong only in dated execution evidence. Source mutation remains branch-locally frozen here because #149 moved under another writer earlier in this invocation.

[seonghobae]

2026-08-10 canonical traceability drift found on current #149 head

Fresh read-only inspection of #149 live head 74c24a3a8950009fdffc2d5f660f40ee3250b863 found a concrete machine-checkable documentation gap. The branch is still source-frozen here because another writer moved it earlier in this invocation, so this records the exact correction boundary without racing that ref.

docs/TRACEABILITY.md on the live head still stops its live product/control rows at #160/#162 and currently omits material newer work that the repository itself now treats as acquisition-relevant:

• [Product/Security Gap] Replace direct ETL HTTP Basic trust boundary #161 — direct/east-west etl-service HTTP Basic trust boundary is a known_gap/planned security architecture item;
• fix(cdc): remove SQL Server scaffold from production discovery #163 — SQL Server CDC scaffold retirement is an active_pr and must remain unshipped truth;
• fix(coverage): make JaCoCo production gate non-vacuous #164 — the JaCoCo remediation itself is an active_pr; issue [Quality gate] Make JaCoCo fail closed when production class selection is empty #162 remains the protected known_gap, but the current matrix incorrectly says repair is sequenced only after overlapping feat(api): add machine-readable HTTP and CDC contracts #157 POM work stabilizes. That premise has already been falsified and fix(coverage): make JaCoCo production gate non-vacuous #164 is active. Its latest hosted run proves 8 production classes are analyzed and exposes real remaining zero-missed deficits (7 instructions, 2 lines, 2 branches), so the matrix should distinguish protected gap vs active repair without claiming success;
• [Release/Supply chain] Establish reproducible exact-head artifacts and provenance #165 — release/reproducible-artifact/provenance authority is a new planned supply-chain gap and is absent from the current matrix.

The current LiveCommercialTraceabilityTest has only two live-work tests: #160 Jackson and #162 vacuous coverage. It therefore cannot fail when #161/#163/#164/#165 disappear from canonical traceability, nor can it detect the now-stale sentence that #162 repair waits for #157 POM stabilization.

Smallest safe follow-through once the #149 writer is stable: RED first by extending LiveCommercialTraceabilityTest to require the exact canonical statuses for #161/#163/#164/#165 and to reject the obsolete #157-sequencing claim; then update only the traceability/assessment/indexed authority needed for GREEN. Keep exact SHAs/run IDs in dated evidence, not timeless architecture. This is a documentation-control defect, not permission to promote any active PR to implemented_on_develop.