docs: rebuild canonical product and technical contracts

docs/DOCUMENTATION_FITNESS.md

@@ -0,0 +1,63 @@
+# Documentation Fitness
+
+## Authority and status model
+
+This inventory evaluates the documentation graph against the exact protected-main tree `d2f1e32271910a6db98a0757d67194ddadca4566`. It does not promote pull-request content into shipped truth. The canonical status vocabulary is shared with `docs/product/PRD.md` and `docs/product/TRD.md`: **IMPLEMENTED-ON-PROTECTED-MAIN**, **ACTIVE-PR**, **PARTIAL**, **PLANNED**, and **SUPERSEDED**.
+
+A document is *fit* only when it states the correct authority boundary, is consistent with the protected-main code/schema/tests it describes, avoids transferring predecessor or active-branch evidence into shipped claims, and gives an operator or reviewer enough information to use, verify, recover, or reject the relevant behavior safely.
+
+The tenant lifecycle material in this canonical cohort is a reconstruction of already integrated protected-main behavior, not a new runtime tenant/RLS/migration contract. Companion-document verification against protected main confirms the same contract is already present in the public `README.md`, root `ARCHITECTURE.md`, topic operator guide `docs/remote-batch-lifecycle.md`, accepted ADR 0002, `docs/doctoring/tenant-scoped-lifecycle.md`, and `CHANGELOG.md`. Those authorities already cover trusted tenant identity, `NOSUPERUSER NOBYPASSRLS`, transaction-local/forced RLS, direct-SQL limits, legacy-to-`standalone` migration, and rollback constraints. The canonical PRD/TRD therefore may cite that existing evidence without rewriting those protected-main files or racing PR #184's separate legacy-extension-retirement documentation lane.
+
+Protected main now also contains the bounded recovery-evidence primitives integrated by #205, #206, and #207: content-free PostgreSQL recovery receipts, descriptor-pinned backup-artifact hash/size evidence, and packaged-schema hash/size evidence. These are evidence primitives only. They do not make executable logical backup/restore, isolated restore acceptance, WAL/PITR, key/config custody, or any RPO/RTO/HA/DR objective shipped. #208 remains the active logical-backup candidate. Direct-restore predecessor #209 is not mergeable as a product contract because its EOF-consumption postcondition conflicts with seekable PostgreSQL custom archives; Draft #212 is the active successor and Issue #204 remains the end-to-end recovery authority.
+
+## Current fitness matrix
+
+| Documentation surface | Status | Fitness assessment | Required next action |
+| --- | --- | --- | --- |
+| `README.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Useful public entry point; protected-main tenant lifecycle guidance already reflects trusted tenant scope and standalone compatibility. It is not the sole product/architecture authority and must not absorb transient PR/check state. The newly integrated recovery-evidence primitives are not yet a complete operator recovery procedure. | Keep beginner-readable; add shipped recovery semantics only when adjacent writer ownership permits and do not present evidence primitives as a restorable-backup guarantee. |
+| `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, direct-SQL trust limits, rollback considerations, interoperability, and verification, but does not by itself form a complete system/UML/data model. Recovery evidence has advanced on protected main without establishing end-to-end recovery semantics. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file, and update recovery architecture only through a non-conflicting governed documentation lane. |
+| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states and cites deterministic protected-main tenant/standalone acceptance authority. | Reflect the integrated recovery-evidence primitives while keeping executable backup/restore and end-to-end recovery PARTIAL/ACTIVE-PR; treat #209 as an unsafe predecessor and #212 as the Draft direct-restore successor until protected integration. |
+| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries, tenant pre-effect validation, standalone recorder compatibility, and release/testing invariants are explicit. | Keep the bounded recovery-evidence technical boundary without promoting #208/#212; remove predecessor #209 from any implied merge path before canonical integration. |
+| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. ADR 0002 already records trusted tenant identity, explicit `standalone` compatibility, RLS/direct-SQL role boundaries, migration, and rollback. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. No protected-main ADR is inferred for the newly integrated recovery-evidence primitives merely because code exists. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. Draft #212 proposes ADR 0016 for custom-format restore seek semantics, but that record is not protected-main authority until integrated. |
+| `docs/adr/README.md` | ACTIVE-PR | This canonical branch supplies the missing protected-main ADR navigation/status index, explains numbering gaps, separates ADR decision status from implementation status, and forbids implicit supersession. | Keep the referenced protected-main tree current; do not list #212's proposed ADR 0016 as a protected-main decision before integration. |
+| `docs/result-streaming.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Describes bounded provider result streaming/checkpoint behavior; checkpoint persistence has a separate accepted ADR. | Update only when the integrated result-application contract changes the operator-facing boundary. |
+| `docs/remote-batch-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Serves as the topic-specific operator authority for durable lifecycle tenancy. It already specifies trusted scope selection, pre-effect validation, four-argument standalone recorder compatibility, tenant-qualified identity/conflict/read behavior, forced RLS/application-role limits, migration, direct-SQL impact, rollback, recovery, and deterministic verification. | Keep synchronized with tenant/RLS and lifecycle migrations; do not duplicate this contract into a competing operator file. |
+| `docs/doctoring/tenant-scoped-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Supporting evidence independently records pre-effect validation, explicit standalone compatibility, database role/RLS limits, migration and rollback, plus current primary references. | Keep supporting references current; do not promote doctoring into the sole product authority. |
+| `CHANGELOG.md` tenant lifecycle entry | IMPLEMENTED-ON-PROTECTED-MAIN | The protected-main Unreleased section already records tenant-scoped durable lifecycle, forced default-deny RLS, standalone compatibility, and the atomic tenant migration fix. | Change only when integrated shipped behavior changes; do not log active-PR claims as shipped. |
+| Topic-specific doctoring/reference docs | IMPLEMENTED-ON-PROTECTED-MAIN | Useful deep evidence exists, but doctoring material is supporting evidence, not the canonical product-status authority. | Keep primary-reference and APA-style citations current where material. |
+| PostgreSQL recovery evidence | IMPLEMENTED-ON-PROTECTED-MAIN / end-to-end PARTIAL | `postgres_recovery_receipt.py`, `postgres_backup_evidence.py`, and `postgres_schema_evidence.py` are integrated protected-main primitives with focused tests. They produce bounded content-free metadata/hash/size evidence without executing SQL or claiming cluster parity. | Keep the evidence/guarantee distinction explicit. Do not claim restorable backup, isolated restore, PITR, RPO/RTO/HA/DR, or compliance readiness from these primitives alone. |
+| PostgreSQL logical backup/restore execution | ACTIVE-PR | #208 proposes bounded `pg_dump` execution. #209's direct-restore predecessor has a verified false-failure defect because a seekable custom-format `pg_restore` need not leave the shared descriptor at EOF after successful transactional restore. Draft #212 is the active successor: it removes that invalid EOF postcondition while preserving metadata-fingerprint verification and owns the permanent README/architecture/ADR/doctoring/CHANGELOG restore-contract documentation. None is protected-main truth. | Freeze #209 rather than merge or patch it in parallel; do not race #208/#212 active writers. Promote only the unchanged successor after exact-head gates, valid findings, required documentation, qualifying approval, and protected-main integration. |
+| Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation for retiring legacy `http` / `pg_cron`; it is a different contract from the already-shipped tenant lifecycle migration and is not protected-main truth. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. |
+| OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. |
+| Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. |
+| Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay; FR-4/end-to-end result application stays `PARTIAL` on protected main, and neither this row nor the TRD content-fidelity invariant promotes #194 into shipped truth. | Keep FR-4 and `docs/TRACEABILITY.md` as the validation/status authority until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. |
+| Runtime config/schema provisioning separation | ACTIVE-PR | PR #193 is a current-main least-privilege candidate: runtime constructors move away from DDL/default seeding, validate the search-path-resolved base-table shape and current-role read capability through bounded catalog probes, and retain provisioning/default seeding at the explicit schema boundary. It is not shipped until normal governance integrates it. | Preserve the trusted-search-path/non-ownership caveat and reacquire exact-head checks/review after every push; do not transfer predecessor or earlier-head evidence. |
+| Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map and distinguishes integrated recovery evidence from active executable candidates and unsafe predecessor heads. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs; record #212 only as an active overlay until integration. |
+| Threat model | PLANNED | No canonical protected-main `docs/THREAT_MODEL.md` is present. Security rules are distributed across AGENTS, architecture, ADRs, tests, and issue/PR evidence. | Add a threat model that distinguishes assets, trust boundaries, attacker capabilities, mitigations, residual risk, and non-guarantees without certification claims. |
+| Data governance | PLANNED | No single canonical data-governance document currently maps data classes, retention/ownership, tenant authority, privacy exposure, backup/restore, and deletion limits. | Add a protected-main-grounded data-governance contract. |
+| UML/component/sequence views | PLANNED | Root architecture is prose-first; no canonical UML authority is present. | Add textual Mermaid/PlantUML-compatible component and critical sequence diagrams after active source contracts stabilize. |
+| ERD / schema model | PLANNED | SQL and schema tests are authoritative, but no canonical ERD summarizes package-owned identities, tenancy, checkpoint state, and relationships. | Generate/maintain an ERD from protected-main schema; label migration-only/active-PR objects separately. |
+| General standalone operator guide | PARTIAL | Protected main has strong operational guidance across `README.md` and topic authorities such as `docs/remote-batch-lifecycle.md`; there is no single omnibus protected-main `docs/OPERABILITY.md`. This is a navigation/consolidation gap, not absence of tenant lifecycle operator guidance. | Establish one general operator index/authority only after checking adjacent writers; do not race #184 or #212 documentation surfaces. |
+| Release governance | PARTIAL | Release-evidence ADRs/tests/workflows are strong, but the canonical graph lacks one concise protected-main release/operator authority tying versioning, SBOM, provenance, artifact identity, rollback/recovery, and publication verification together. | Add a stable release contract after checking active release writers; do not copy workflow-run IDs. |
+| Licensing / third-party notices | PARTIAL | Apache-2.0 source headers and repository licensing exist, but acquisition diligence should explicitly trace package license, dependency/SBOM evidence, and third-party notice process. | Evaluate a concise licensing/compliance evidence index without claiming legal certification. |
+
+## Non-negotiable documentation invariants
+
+- Protected-main behavior is the shipped authority. Active PRs and historical branches are overlays, not proof of implementation.
+- Exact contributor heads, generated merge commits, check IDs, and transient queue state belong in PR/review evidence, not durable architecture/product documents.
+- Standalone operation and modular MSA embedding must be documented as co-equal supported boundaries; no ContextualWisdomLab host is a hidden runtime requirement.
+- Tenant scope is selected only by a trusted authenticated/authorized host boundary and, for the tenant durable client, validates before observation reservation, credential lookup, provider I/O, or lifecycle database I/O. Lifecycle identities, conflict targets, exact reads, and operational status indexes remain tenant-qualified where tenancy is enabled.
+- `pg_llm_batch.tenant_scope` is transaction-local routing context, not a credential or authenticated identity. Any database role capable of arbitrary SQL can call `set_config` with an arbitrary tenant scope; the trusted application boundary must therefore prevent generic tenant-controlled SQL, SQL injection, and incorrect identity mapping from selecting tenant authority. RLS does not provide those guarantees.
+- The standalone `DurableBatchAPIClient` retains its four-argument lifecycle-recorder seam and explicit `standalone` persistence/read scope unless a separately reviewed compatibility change is integrated.
+- PostgreSQL RLS is defense in depth, not host authentication/authorization, SQL-injection prevention, or correct identity mapping; production application roles for the tenant lifecycle boundary are `NOSUPERUSER NOBYPASSRLS`.
+- Provider/model content never becomes tenant, credential, endpoint, filesystem, or database authority.
+- Content-fidelity invariants constrain any result-application path that exists, but they do not prove that end-to-end result application is shipped; FR-4 and `docs/TRACEABILITY.md` remain the capability-status authority while protected main is `PARTIAL`.
+- Bounded recovery receipts, artifact hashes, and packaged-schema hashes are evidence primitives, not proof that a backup is restorable or that a stated recovery objective is met.
+- A direct logical restore contract must keep caller-owned source trust, target isolation, allowed libpq environment, transaction rollback behavior, and post-restore acceptance explicit. Its archive-completion checks must match PostgreSQL custom-format random-access semantics: final descriptor offset is not proof of complete restore. #209 is therefore an unsafe predecessor, while #212 remains only an ACTIVE-PR successor until integrated.
+- Distributed exactly-once processing is never implied by PostgreSQL checkpoint atomicity alone.
+- Security, privacy, SOC 2, and CSAP material is evidence-readiness documentation only unless an external certification actually exists.
+- Database object names, migration/rollback behavior, exact owned coverage requirements, Python-version acceptance, packaging/SBOM/provenance expectations, and bounded diagnostics must remain synchronized with repository tests and live governance.
+
+## Fitness gate for future canonical changes
+
+Before changing a canonical surface, refetch protected main, open PRs, non-default branches, the exact affected source/schema/test authorities, and any adjacent documentation writer. A documentation repair must not race a source-affecting writer whose behavior is still unsettled. When a canonical change merely reconstructs an already integrated contract, verify the existing permanent companion set before widening the change; when runtime behavior actually changes, update the required permanent companion surfaces in that same governed change or explicitly coordinate their writer ownership. After a product PR merges, update the canonical status classification from `ACTIVE-PR` or `PARTIAL` only when the protected-main tree contains the capability and the integrated evidence contract remains valid.