diff --git a/docs/documentation_coverage.md b/docs/documentation_coverage.md index d528eca08..d99dbeab1 100644 --- a/docs/documentation_coverage.md +++ b/docs/documentation_coverage.md @@ -1,112 +1,153 @@ # Architecture documentation completeness and maintenance matrix Status: **Authoritative maintenance audit** -Last reviewed: 2026-08-09 +Last reviewed: 2026-08-12 -This matrix answers whether the repository has enough durable documentation to reconstruct current product intent, technical boundaries, architecture, decisions, logical data relationships, threat model, standards status, verification/validation obligations, scientific evidence and release obligations without mining chat history or PR bodies. +This matrix answers whether GitHub can reconstruct the current `fast-mlsirm` product, technical, scientific, security, operability, and release intent without relying on chat history or stale pull-request prose. File existence is not sufficient: document availability and product-capability maturity are evaluated separately against protected-main code, tests, workflows, accepted decisions, and live work. ## Status vocabulary -- **IMPLEMENTED** — canonical document exists on this branch and describes protected-main behavior/policy without relying on unmerged code. -- **ACTIVE PR** — durable requirement/decision is known, but the corresponding runtime/scientific feature is still under open PR; documentation must not call it released. -- **PLANNED** — accepted/proposed direction with incomplete implementation/evidence. -- **DOWNSTREAM** — owned by Psychometrics Commons or another service; fast-mlsirm documents only the versioned boundary/handoff. -- **REJECTED/SUPERSEDED** — considered or historical design that must not silently return as current authority. +### Documentation-family state + +- **PRESENT_CURRENT** — canonical artifact exists on protected main and is consistent with current protected-main ownership and policy. +- **PRESENT_STALE** — artifact exists but materially contradicts current protected-main behavior, ownership, version, or accepted policy. +- **PARTIAL** — artifact exists and is directionally correct but lacks material coverage needed for the affected release or buyer claim. +- **MISSING** — a required canonical artifact has no discoverable equivalent. +- **NOT_APPLICABLE** — the family is intentionally not owned here and the ownership boundary is documented. +- **SUPERSEDED** — historical artifact is retained only for compatibility/history and is not authoritative. +- **OWNED_BY_ACTIVE_PR** — a non-protected-main branch currently owns a coherent update; the branch must not be described as shipped truth. + +### Capability maturity + +- **IMPLEMENTED_ON_PROTECTED_MAIN** — accepted source/tests/contracts are ancestral to protected main. +- **IMPLEMENTED_ON_ACTIVE_PR** — implementation exists only on a current open PR. +- **PARTIAL** — useful protected-main primitives exist but the declared end-to-end capability is incomplete. +- **ACCEPTED_ARCHITECTURE** — durable design is accepted while implementation remains incomplete. +- **PLANNED** — desired work is known but not accepted as implemented. +- **RESEARCH_ONLY** — evidence or exploration exists without a production contract. +- **DOWNSTREAM** — owned by `ContextualWisdomLab/psychometrics-commons` or another explicit host/service. +- **SUPERSEDED** — a previous capability/implementation path has been replaced. +- **REJECTED** — reviewed and intentionally excluded. +- **OUT_OF_SCOPE** — outside the reusable measurement-core boundary. ## Canonical documentation coverage -| Documentation capability | Before canonical baseline | Current target | Status | Maintenance / remaining gap | -|---|---|---|---|---| -| Product requirements | narrow/stale early-MVP summary plus feature plans | `docs/PRD.md` | IMPLEMENTED | update when buyer workflow/non-goal changes | -| Technical requirements | scattered agent/doctoring/spec rules | `docs/TRD.md` | IMPLEMENTED | update on numerical/runtime/security/resource/release policy changes | -| Root architecture | no root architecture authority | `ARCHITECTURE.md` | IMPLEMENTED | keep current vs proposed explicit | -| Documentation authority/index | feature folders only | `docs/README.md` | IMPLEMENTED | new canonical categories must be linked here | -| Architecture decision log | decisions scattered across AGENTS/plans/PRs | `docs/adr/README.md`, ADRs | IMPLEMENTED | material decision needs status-bearing ADR/supersession | -| Standards status/watch | standards mixed into doctoring without one normative/watch registry | `docs/standards_watch.md` | IMPLEMENTED | verify official edition/publication status before material release claims; drafts stay watch-only until adopted | -| Verification and validation plan | method-specific checks without one evidence hierarchy | `docs/verification_validation_plan.md` | IMPLEMENTED | update when a new estimator/scorer/generalization claim changes recovery, anti-leakage, resource, security or release evidence | -| Component/UML views | no coherent canonical set | `docs/uml/*.puml` | IMPLEMENTED | update when module/dependency/lifecycle changes | -| Logical ERD | absent | `docs/erd/domain-model.puml` | IMPLEMENTED | remains logical/persistence-neutral; physical hosted DB is downstream | -| Requirements traceability | absent | `docs/traceability/requirements-matrix.md` | IMPLEMENTED | update maturity/evidence with material feature PRs | -| Scientific/standards basis | strong but scattered doctoring | `docs/traceability/research-basis.md` + doctoring | IMPLEMENTED | keep APA 7, primary/current standards and preprint status honest | -| Reusable-core threat model | implicit in security feature docs | `docs/security/threat-model.md` | IMPLEMENTED | update on new trust/native/provider/artifact boundaries | -| Documentation contract CI | absent | `tests/test_architecture_documentation_contract.py` | ACTIVE PR | this PR enforces the complete ADR-template/UML/index/source-hygiene set; protected-main integration is required before calling the baseline IMPLEMENTED | -| Release/changelog evidence | managed changelog and release docs already exist | changelog fragment + existing release controls | IMPLEMENTED | render fragment before Ready/merge according to repo policy | -| Operational runbook for hosted product | intentionally not owned here | Psychometrics Commons/operator docs | DOWNSTREAM | link only when a versioned integration requires it | -| Physical DB schema/migrations | intentionally not owned here | Psychometrics Commons/owning host | DOWNSTREAM | do not manufacture ORM from logical ERD | -| Tenant/RBAC/SSO/SCIM/UI/billing | not a reusable core concern | hosted product/services | DOWNSTREAM | retain dependency direction only | - -## Conversation-wide scientific/product coverage - -| Work family | Documentation maturity | Runtime/evidence maturity | State | +| Documentation capability | Canonical target | State | Current fitness / maintenance rule | |---|---|---|---| -| Fallible human/LLM raters and many-facet calibration | PRD/TRD + ADR-0005 + traceability + V&V | baseline facets/scoring exists; generalized discrimination/range/drift remains incremental | IMPLEMENTED / PLANNED extensions | -| Correlation is not parameter recovery/agreement | ADR-0008 + PRD/TRD + research basis + V&V | recovery/simulation evidence exists across model families | IMPLEMENTED governance | -| Reference-free RAG measurement | PRD/TRD + traceability + V&V | no single canonical end-to-end RAG observation adapter/bank workflow on protected main | PLANNED | -| Dynamic evidence-grounded rubric generation | PRD/TRD + ADR-0003/0004 + item UML/ERD + V&V | strong rubric/generation/audit/pilot primitives exist | IMPLEMENTED primitives / PLANNED closed loop | -| Governed item-bank lifecycle | ADR-0004 + state diagram + V&V | pilot/admission/lifecycle pieces exist; unified approve/active/link/drift/exposure/retire workflow remains incomplete | PLANNED/partial | -| Bifactor / higher-order / testlet / two-tier / many-facet relation | ADR-0006 + model-selection UML/research basis + V&V | family-specific features/evidence vary | IMPLEMENTED policy / partial family coverage | -| Latent-space residual interaction | architecture/PRD/TRD + V&V | supported model family exists, but must follow substantive dimension/testlet/facet diagnosis | IMPLEMENTED with interpretation gate | -| Formal non-nested distinguishability | ADR-0006 + traceability + V&V | fail-closed comparison exists where formal family inputs are incomplete; full score/information metadata still needed | PLANNED extension | -| Adaptive rotation criterion selection | ADR-0009 + PRD/TRD + protected-main adaptive-rotation doctoring + V&V | protected main exposes the Rust-backed criterion registry, deterministic multi-start optimizer, criterion-neutral selector/report surfaces and public Python API; GPU batching, additional criteria and broader recovery evidence remain future increments | IMPLEMENTED / PLANNED extensions | -| Multilevel/multiple-membership/cross-classified contracts | ADR-0007 + UML/ERD/PRD/TRD + V&V | active PR exists; dedicated namespace not accepted until protected-main evidence | ACTIVE PR | -| Temporal/longitudinal/drift models | ADR-0007 + PRD/TRD + V&V | design/primitives exist; continuous-time estimator claims require separate recovery | PLANNED/partial | -| Automated essay scoring calibration/validation | PRD/TRD + ADR-0005 + V&V | governed essay contracts/calibration/validation/reporting exist; rater-range/discrimination/drift extensions remain active | IMPLEMENTED baseline / ACTIVE extensions | -| Enterprise issue measurement | PRD/TRD + traceability + V&V | reusable evidence/calibration adapters exist | IMPLEMENTED measurement; causal intervention utility DOWNSTREAM/policy | -| Factor retention | PRD/TRD + ADR-0006 + V&V | diagnostics exist; unified evidence API remains a gap | PLANNED extension | -| Rust-first numerical core / CPU+GPU parity | ADR-0002 + TRD + V&V | current model-specific support/evidence varies by kernel | IMPLEMENTED architecture, feature-specific evidence required | -| Canonical PyO3/public-export registry | ADR-0011 | current exports work, but future feature PRs must converge rather than creating competing initialization schemes | PLANNED hardening | -| PII/purpose limitation | ADR-0012 + threat model + PRD/TRD + V&V abuse cases | source-free/digest-based contracts exist where possible; hosted access/retention is downstream | IMPLEMENTED reusable policy / DOWNSTREAM operations | -| LLM orchestration/model credentials | ADR-0010 + V&V | repository/org automation policy exists | IMPLEMENTED governance | -| Continuous execution and canonical docs ownership | ADR-0013 + documentation contract | repository process contract is proposed by the canonical docs PR; runtime scheduler state is external to shipped package capability | PROPOSED governance | +| Product requirements | `docs/PRD.md` | PRESENT_CURRENT | update when users/JTBD, product scope, non-goals, buyer acceptance, or downstream ownership changes | +| Technical requirements | `docs/TRD.md` | PRESENT_CURRENT | update on numerical authority, public-contract, runtime, resource, security, privacy, interoperability, or release-rule changes | +| Root architecture | `ARCHITECTURE.md` | PRESENT_CURRENT | protected-main architecture baseline is integrated; keep protected-main vs active-PR/planned behavior explicit | +| Documentation authority/index | `docs/README.md` | PRESENT_CURRENT | all canonical families and strong equivalents must remain discoverable from one graph | +| Architecture decisions | `docs/adr/README.md`, status-bearing ADRs | PRESENT_CURRENT | durable cross-cutting decisions require explicit status and supersession rather than silent prose edits | +| Continuous execution governance | ADR-0013 + architecture/TRD links | PRESENT_CURRENT | protected-main governance describes RCA, feasibility, work conservation, writer leases, evidence freshness, and final-sweep behavior; scheduler runtime remains external to package capability | +| Standards status/watch | `docs/standards_watch.md` | PRESENT_CURRENT | published editions remain normative; drafts/revision projects remain watch items and never imply certification | +| Verification / test strategy | `docs/verification_validation_plan.md` | PRESENT_CURRENT | serves the Test Strategy/V&V role; update for new estimators, scorers, generalization claims, recovery evidence, security boundaries, or release evidence | +| UML/component and behavioral views | `docs/uml/*.puml` | PARTIAL | current component/deployment/scoring/model-selection/item-lifecycle views are useful; maintain a complete indexed inventory and add authority/recovery/public-contract class views when materially needed | +| Logical ERD / evidence model | `docs/erd/domain-model.puml` | PARTIAL | remains logical and persistence-neutral; cardinalities and immutable-revision/provenance relationships must track public contracts and must not invent a hosted DB | +| Requirements traceability | `docs/traceability/requirements-matrix.md` | PARTIAL | useful baseline exists; refresh after material protected merges and active-PR state changes so no active work is presented as shipped | +| Scientific / standards basis | `docs/traceability/research-basis.md` + doctoring | PRESENT_CURRENT | use APA 7, primary sources, stable links, scope/equation traceability, and conservative interpretation boundaries | +| Reusable-core threat model | `docs/security/threat-model.md` | PRESENT_CURRENT | update on native/provider/artifact/serialization/secret/resource/trust-boundary changes | +| Public interface/version/serialization/fingerprint contract | indexed public-contract docs + ADR-0003/0011 | PARTIAL | make canonicalization, schema/version, compatibility/deprecation, fingerprint preimage, cross-language vectors, and cross-repository handoffs explicitly discoverable | +| Reusable-core operability/recovery | resource/failure/release docs + TRD/V&V | PARTIAL | package-owned bounded resources, cancellation, deterministic failure evidence, fallback/degraded behavior, wheel/reinstall recovery, and diagnostics need one discoverable operating index | +| Security/data-governance index | threat model + ADR-0012 + security docs | PARTIAL | purpose limitation, authorization, retention/export responsibility, provider/secret/raw-content boundaries, supply-chain ownership, and responsible disclosure must stay discoverable | +| Release/migration/rollback/provenance/licensing index | release acceptance + packaging/SBOM/provenance/licensing docs | PARTIAL | existing controls are substantial but need one canonical index across compatibility, rollback, artifact hashes, SBOM/provenance, reproducibility, NOTICE/licensing, and release acceptance | +| Documentation contract CI | architecture/documentation contract tests | PRESENT_CURRENT | documentation-as-code checks are protected-main behavior; extend them when status vocabulary, canonical inventory, links, UML/ERD source hygiene, or ownership rules change | +| Root README / AGENTS / CLAUDE / CHANGELOG alignment | root authority files | PARTIAL | fail documentation fitness when obsolete product names, authoritative NumPy-first claims, stale version support, or contradictions with PRD/TRD/Architecture reappear | +| Hosted product operational runbook | Psychometrics Commons/operator docs | NOT_APPLICABLE | hosted tenant/session/consent/database/UI/deployment operations stay downstream | +| Physical product DB schema/migrations | Psychometrics Commons/owning host | NOT_APPLICABLE | do not manufacture ORM/DDL merely to satisfy an ERD request | +| Tenant/RBAC/SSO/SCIM/UI/billing | hosted product/services | NOT_APPLICABLE | retain only explicit versioned reusable-core handoff boundaries | + +## Protected-main scientific and product maturity + +The table below records product truth, not documentation-file presence. “Implemented” means ancestral to protected main, not merely discussed in an issue or present on an open branch. + +| Work family | Current maturity | Evidence / remaining boundary | +|---|---|---| +| Fallible human/AI/LLM raters and many-facet scoring | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | governed scoring/facet primitives exist; broader discrimination/range/drift extensions remain incremental | +| Correlation is not recovery/agreement/validity | IMPLEMENTED_ON_PROTECTED_MAIN | recovery and interpretation governance is protected-main policy | +| Reference-free RAG request/provenance boundary | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | governed privacy-preserving RAG scoring-request adapter is integrated; full calibration/validation/bank workflow remains broader work | +| Dynamic evidence-grounded rubric generation | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | rubric/blueprint/generation/audit/pilot primitives exist; closed-loop governed bank evolution remains broader than generation | +| Governed post-pilot item-bank lifecycle | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | immutable post-pilot lifecycle/evidence gates are integrated; linking/exposure/drift/assembly/release integration continues incrementally | +| Bifactor / higher-order / testlet / two-tier / many-facet relation governance | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | relation-safe policy is established; family-specific estimator/validation evidence varies | +| Latent-space residual interaction | IMPLEMENTED_ON_PROTECTED_MAIN | interpretation remains gated on substantive dimension/testlet/facet diagnosis | +| Formal non-nested distinguishability/model comparison | PARTIAL | fail-closed relation-aware comparison exists; additional family-specific evidence and metadata remain incremental | +| Adaptive rotation criterion selection | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | Rust-backed criterion registry/multi-start selector/report surfaces are integrated; additional criteria/GPU/recovery remain incremental | +| Multilevel / cross-classified / multiple-membership contracts | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | contextual and longitudinal contracts are integrated; estimator identification/recovery remains separate work | +| Temporal/longitudinal/drift estimators | PARTIAL | governed contracts/design primitives exist; continuous-time or richer estimator claims require separate recovery evidence | +| Automated essay scoring calibration/validation | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | governed essay contracts/validation/reporting exist; generalized rater discrimination/range/drift remains incremental | +| Paired automated-vs-reference rating-range evidence | IMPLEMENTED_ON_PROTECTED_MAIN | Rust-owned paired range/compression diagnostic is integrated | +| Enterprise issue measurement | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | reusable evidence/calibration adapters exist; causal intervention utility remains downstream/policy-bound | +| Factor retention evidence contract | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | governed factor-retention evidence contract is integrated; structural model selection remains a distinct subsequent decision | +| Rust-first numerical ownership | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | fixed-anchor linking, CAT/ATA, covariance, observed-information Hessian assembly, second-order diagnostics, and JMLE Adam/L-BFGS optimizer sequencing are protected-main Rust/PyO3 paths; remaining kernel-specific migrations stay explicit | +| Fixed-anchor parameter linking arithmetic | IMPLEMENTED_ON_PROTECTED_MAIN | protected main owns scale/shift estimation and theta/alpha/b transformation in Rust/PyO3 | +| Observed-information Hessian and second-order diagnostics | IMPLEMENTED_ON_PROTECTED_MAIN | protected main owns finite-difference coefficients/symmetric Hessian assembly and eigenvalue/positive-definiteness diagnostics in Rust/PyO3; Python only evaluates objective samples and transports results | +| JMLE Adam/L-BFGS optimizer arithmetic | IMPLEMENTED_ON_PROTECTED_MAIN | PR #760 is ancestral to current protected main; `backend="rust"` delegates Adam/L-BFGS/combined optimizer control to compiled Rust while recovery evidence remains governed separately by issue #626 | +| Parallel-analysis public control/resource hardening | IMPLEMENTED_ON_ACTIVE_PR | current fail-first/implementation PR owns strict integer/control and bounded-workspace hardening; it remains non-shipped until exact-head integration | +| Hourly review-repair caller | IMPLEMENTED_ON_PROTECTED_MAIN / PARTIAL | PR #763 integrated the product-side bounded caller; operational scheduler/control-plane acceptance remains external evidence rather than a library capability | +| LLM-judge raw JSON depth hardening | IMPLEMENTED_ON_PROTECTED_MAIN | PR #764 is ancestral to current protected main and bounds recursive JSON nesting before parser materialization | +| Essay-report native dark-mode status accents | IMPLEMENTED_ON_ACTIVE_PR | current accessibility PR owns the CSS-variable/media-query change; do not treat it as protected-main until integration | +| Canonical PyO3/public-export governance | ACCEPTED_ARCHITECTURE / PARTIAL | ADR-0011 governs convergence; feature-by-feature hardening continues | +| Purpose-limited sensitive-data handling | IMPLEMENTED_ON_PROTECTED_MAIN / DOWNSTREAM | reusable contracts prefer purpose limitation/minimization/separated identities; hosted authorization/retention execution remains downstream | +| LLM orchestration/model credentials | IMPLEMENTED_ON_PROTECTED_MAIN | provider execution and independent reviewer identity/credential boundaries are governed; provider calls remain outside psychometric numerical core | +| Continuous execution and canonical documentation ownership | IMPLEMENTED_ON_PROTECTED_MAIN | ADR-0013 is integrated documentation/process authority; runtime scheduler state is external and is not a shipped library capability | + +## Current active-PR boundary + +At this review, material open work includes: + +- strict/bounded public controls and Rust allocation preflight for parallel analysis; +- the documentation-fitness refresh itself, which may describe current protected truth but is not authoritative until merged; and +- native dark-mode report status accents. + +These remain active-PR evidence, not protected-main capability. Their source heads, checks, reviews, writer leases, and mergeability are operational evidence and must be re-fetched rather than copied into timeless architecture prose. ## P0 documentation gaps -A P0 gap blocks treating the architecture package as complete: +A P0 documentation defect blocks calling the affected architecture/release story complete when any of the following is true: -- the canonical documentation baseline or its complete contract test is still only on an open PR rather than protected main; -- missing canonical PRD or TRD; -- missing root architecture boundary; -- missing ADR index/status for a material cross-cutting decision; -- missing standards status registry when a release or buyer claim relies on standards; -- missing V&V plan for a new scientific/scoring/generalization claim; -- missing UML/ERD or persistence-neutral domain/public-contract view for a major public-contract or lifecycle change; -- missing reusable-core threat model after a new trust boundary; -- traceability that falsely marks an active/planned feature as protected-main implemented; -- a stale historical summary that competes with the canonical requirements source; or -- architecture claims that move hosted product DB/HTTP/tenant/RBAC ownership into fast-mlsirm without a superseding ADR. +- missing or materially stale canonical PRD, TRD, root architecture, ADR authority, standards registry, V&V/Test Strategy, threat model, logical data/evidence model, or requirement traceability; +- an active PR, issue, target diagram, research result, or scheduler behavior is promoted to protected-main product truth; +- public interface/version/serialization/fingerprint behavior cannot be reconstructed without source archaeology; +- UML/ERD names, ownership, state, cardinality, or conceptual-vs-persisted semantics contradict the public contract; +- an obsolete early-MVP/NumPy-first/product-name/version claim competes with the current Rust-first protected-main authority; +- hosted product database/HTTP/tenant/RBAC ownership is moved into `fast-mlsirm` without a superseding accepted decision; or +- release, migration, rollback, provenance, licensing, operability, or security ownership for the affected capability cannot be discovered from the canonical graph. ## P1 documentation gaps -P1 gaps do not automatically block unrelated development, but must be repaired before release of the affected capability: +P1 gaps do not automatically block unrelated development, but must be closed before releasing or making the affected claim: -- missing method-specific doctoring/primary source; -- missing recovery/scoreability interpretation boundary; -- missing V&V evidence class or resampling/generalization unit for the changed claim; +- missing method-specific doctoring or primary-source traceability; +- missing recovery/scoreability/identification interpretation boundary; +- missing resampling/generalization unit in V&V; - missing failure/recovery/rollback instructions for a changed public artifact; -- missing privacy/security abuse case for new provider/native/artifact surfaces; or -- missing changelog/release evidence for a user-visible accepted capability. +- missing privacy/security abuse case for a new provider/native/artifact surface; +- missing changelog/release/provenance evidence for an accepted user-visible capability; or +- missing machine-check for a high-risk documentation invariant that has already drifted in practice. ## P2 improvements -- richer rendered architecture diagrams/site navigation; -- downstream hosted-workbench Figma/UX links; -- automated link/PlantUML rendering checks beyond the current source contract; -- generated traceability views from contract metadata; and -- buyer/operator views that consume these artifacts without becoming a second source of truth. +- richer rendered architecture/site navigation generated from the canonical graph; +- stronger PlantUML renderability/link checking in normal CI where supported; +- generated traceability views from contract metadata; +- downstream buyer/operator views that consume, rather than duplicate, canonical artifacts; and +- optional downstream Figma/UX links for hosted workbench experiences. ## Maintenance gate Every material PR should answer: -1. Did product requirements or non-goals change? -2. Did a technical invariant/trust/resource/release rule change? -3. Did a durable architecture/scientific decision change or need supersession? +1. Did product requirements, users/JTBD, or non-goals change? +2. Did a technical invariant, public contract, trust/resource/privacy/release rule, or ownership boundary change? +3. Did a durable architecture/scientific decision change or require supersession? 4. Did an applicable published standard edition/status or watch item change? -5. Did the required software/numerical/scientific V&V evidence or generalization unit change? -6. Did component/data/lifecycle/deployment/ERD views change? -7. Did the threat model gain a new asset/actor/abuse case? -8. Did an implementation maturity state change (PLANNED/ACTIVE -> protected-main IMPLEMENTED)? -9. Did source/test evidence change enough that traceability is stale? -10. Is the changelog/release evidence synchronized? - -If yes, update the corresponding canonical document in the same PR or record a precise downstream/no-change justification. Documentation drift is treated as a repository defect rather than post-release cleanup. +5. Did software/numerical/scientific V&V evidence, identification, recovery, scoreability, or generalization unit change? +6. Did component/data/lifecycle/deployment/authority/recovery/UML/ERD views change? +7. Did the threat model or data-governance boundary gain an asset, actor, abuse case, provider, secret, native boundary, or retention/export responsibility? +8. Did capability maturity change between planned, active-PR, protected-main, downstream, or superseded states? +9. Did source/test/evidence move enough that requirements/research traceability is stale? +10. Are public interface/version/serialization/fingerprint compatibility and deprecation rules still discoverable? +11. Are operability, migration/rollback, SBOM/provenance/reproducibility/licensing and release evidence synchronized? +12. Do README, AGENTS, CLAUDE, Architecture, PRD/TRD and CHANGELOG tell one consistent current story? + +If any answer is yes, update the corresponding canonical artifact in the same coherent workstream or record a precise `NOT_APPLICABLE`/downstream/no-change justification. Documentation drift is a repository defect, not post-release cleanup. diff --git a/tests/test_documentation_coverage_fitness.py b/tests/test_documentation_coverage_fitness.py new file mode 100644 index 000000000..5b943a2a7 --- /dev/null +++ b/tests/test_documentation_coverage_fitness.py @@ -0,0 +1,84 @@ +"""Regression contracts for the canonical documentation fitness matrix.""" + +from pathlib import Path + + +_MATRIX = Path(__file__).parents[1] / "docs" / "documentation_coverage.md" + + +def _matrix() -> str: + """Return the canonical documentation-fitness matrix source.""" + return _MATRIX.read_text(encoding="utf-8") + + +def _row(source: str, capability: str) -> str: + """Return one capability row from the protected-main maturity table.""" + return next(line for line in source.splitlines() if f"| {capability} |" in line) + + +def test_document_and_capability_states_are_not_conflated() -> None: + """The matrix must expose separate finite vocabularies for docs and runtime maturity.""" + source = _matrix() + for state in ( + "PRESENT_CURRENT", + "PRESENT_STALE", + "PARTIAL", + "MISSING", + "NOT_APPLICABLE", + "SUPERSEDED", + "OWNED_BY_ACTIVE_PR", + ): + assert f"**{state}**" in source + for state in ( + "IMPLEMENTED_ON_PROTECTED_MAIN", + "IMPLEMENTED_ON_ACTIVE_PR", + "ACCEPTED_ARCHITECTURE", + "PLANNED", + "RESEARCH_ONLY", + "DOWNSTREAM", + "REJECTED", + "OUT_OF_SCOPE", + ): + assert f"**{state}**" in source + + +def test_protected_main_docs_are_not_described_as_open_pr_only() -> None: + """Integrated architecture docs must not regress to the former open-PR-only narrative.""" + source = _matrix() + stale_claim = "canonical documentation baseline or its complete contract test is still only on an open PR" + assert stale_claim not in source + assert "Documentation contract CI" in source + assert "PRESENT_CURRENT" in source + + +def test_recently_integrated_capabilities_are_not_left_as_active_pr_only() -> None: + """Recent protected-main contracts and numerical migrations remain shipped truth.""" + source = _matrix() + for capability in ( + "Reference-free RAG request/provenance boundary", + "Governed post-pilot item-bank lifecycle", + "Multilevel / cross-classified / multiple-membership contracts", + "Factor retention evidence contract", + "Fixed-anchor parameter linking arithmetic", + "Observed-information Hessian and second-order diagnostics", + "JMLE Adam/L-BFGS optimizer arithmetic", + "Hourly review-repair caller", + "LLM-judge raw JSON depth hardening", + ): + assert "IMPLEMENTED_ON_PROTECTED_MAIN" in _row(source, capability) + + +def test_parallel_hardening_is_not_promoted_to_protected_main() -> None: + """The active parallel-analysis hardening lane cannot be called shipped before merge.""" + source = _matrix() + row = _row(source, "Parallel-analysis public control/resource hardening") + assert "IMPLEMENTED_ON_ACTIVE_PR" in row + assert "IMPLEMENTED_ON_PROTECTED_MAIN" not in row + + +def test_dark_mode_report_change_is_not_promoted_before_merge() -> None: + """The current accessibility PR remains active-PR evidence until integrated.""" + source = _matrix() + row = _row(source, "Essay-report native dark-mode status accents") + assert "IMPLEMENTED_ON_ACTIVE_PR" in row + assert "IMPLEMENTED_ON_PROTECTED_MAIN" not in row