diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md new file mode 100644 index 000000000..2ef97bca1 --- /dev/null +++ b/docs/DOCUMENTATION_FITNESS.md @@ -0,0 +1,63 @@ +# Documentation Fitness + +## Authority and status model + +This inventory evaluates the documentation graph against the exact protected-main tree `d2f1e32271910a6db98a0757d67194ddadca4566`. It does not promote pull-request content into shipped truth. The canonical status vocabulary is shared with `docs/product/PRD.md` and `docs/product/TRD.md`: **IMPLEMENTED-ON-PROTECTED-MAIN**, **ACTIVE-PR**, **PARTIAL**, **PLANNED**, and **SUPERSEDED**. + +A document is *fit* only when it states the correct authority boundary, is consistent with the protected-main code/schema/tests it describes, avoids transferring predecessor or active-branch evidence into shipped claims, and gives an operator or reviewer enough information to use, verify, recover, or reject the relevant behavior safely. + +The tenant lifecycle material in this canonical cohort is a reconstruction of already integrated protected-main behavior, not a new runtime tenant/RLS/migration contract. Companion-document verification against protected main confirms the same contract is already present in the public `README.md`, root `ARCHITECTURE.md`, topic operator guide `docs/remote-batch-lifecycle.md`, accepted ADR 0002, `docs/doctoring/tenant-scoped-lifecycle.md`, and `CHANGELOG.md`. Those authorities already cover trusted tenant identity, `NOSUPERUSER NOBYPASSRLS`, transaction-local/forced RLS, direct-SQL limits, legacy-to-`standalone` migration, and rollback constraints. The canonical PRD/TRD therefore may cite that existing evidence without rewriting those protected-main files or racing PR #184's separate legacy-extension-retirement documentation lane. + +Protected main now also contains the bounded recovery-evidence primitives integrated by #205, #206, and #207: content-free PostgreSQL recovery receipts, descriptor-pinned backup-artifact hash/size evidence, and packaged-schema hash/size evidence. These are evidence primitives only. They do not make executable logical backup/restore, isolated restore acceptance, WAL/PITR, key/config custody, or any RPO/RTO/HA/DR objective shipped. #208 remains the active logical-backup candidate. Direct-restore predecessor #209 is not mergeable as a product contract because its EOF-consumption postcondition conflicts with seekable PostgreSQL custom archives; Draft #212 is the active successor and Issue #204 remains the end-to-end recovery authority. + +## Current fitness matrix + +| Documentation surface | Status | Fitness assessment | Required next action | +| --- | --- | --- | --- | +| `README.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Useful public entry point; protected-main tenant lifecycle guidance already reflects trusted tenant scope and standalone compatibility. It is not the sole product/architecture authority and must not absorb transient PR/check state. The newly integrated recovery-evidence primitives are not yet a complete operator recovery procedure. | Keep beginner-readable; add shipped recovery semantics only when adjacent writer ownership permits and do not present evidence primitives as a restorable-backup guarantee. | +| `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, direct-SQL trust limits, rollback considerations, interoperability, and verification, but does not by itself form a complete system/UML/data model. Recovery evidence has advanced on protected main without establishing end-to-end recovery semantics. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file, and update recovery architecture only through a non-conflicting governed documentation lane. | +| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states and cites deterministic protected-main tenant/standalone acceptance authority. | Reflect the integrated recovery-evidence primitives while keeping executable backup/restore and end-to-end recovery PARTIAL/ACTIVE-PR; treat #209 as an unsafe predecessor and #212 as the Draft direct-restore successor until protected integration. | +| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries, tenant pre-effect validation, standalone recorder compatibility, and release/testing invariants are explicit. | Keep the bounded recovery-evidence technical boundary without promoting #208/#212; remove predecessor #209 from any implied merge path before canonical integration. | +| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. ADR 0002 already records trusted tenant identity, explicit `standalone` compatibility, RLS/direct-SQL role boundaries, migration, and rollback. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. No protected-main ADR is inferred for the newly integrated recovery-evidence primitives merely because code exists. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. Draft #212 proposes ADR 0016 for custom-format restore seek semantics, but that record is not protected-main authority until integrated. | +| `docs/adr/README.md` | ACTIVE-PR | This canonical branch supplies the missing protected-main ADR navigation/status index, explains numbering gaps, separates ADR decision status from implementation status, and forbids implicit supersession. | Keep the referenced protected-main tree current; do not list #212's proposed ADR 0016 as a protected-main decision before integration. | +| `docs/result-streaming.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Describes bounded provider result streaming/checkpoint behavior; checkpoint persistence has a separate accepted ADR. | Update only when the integrated result-application contract changes the operator-facing boundary. | +| `docs/remote-batch-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Serves as the topic-specific operator authority for durable lifecycle tenancy. It already specifies trusted scope selection, pre-effect validation, four-argument standalone recorder compatibility, tenant-qualified identity/conflict/read behavior, forced RLS/application-role limits, migration, direct-SQL impact, rollback, recovery, and deterministic verification. | Keep synchronized with tenant/RLS and lifecycle migrations; do not duplicate this contract into a competing operator file. | +| `docs/doctoring/tenant-scoped-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Supporting evidence independently records pre-effect validation, explicit standalone compatibility, database role/RLS limits, migration and rollback, plus current primary references. | Keep supporting references current; do not promote doctoring into the sole product authority. | +| `CHANGELOG.md` tenant lifecycle entry | IMPLEMENTED-ON-PROTECTED-MAIN | The protected-main Unreleased section already records tenant-scoped durable lifecycle, forced default-deny RLS, standalone compatibility, and the atomic tenant migration fix. | Change only when integrated shipped behavior changes; do not log active-PR claims as shipped. | +| Topic-specific doctoring/reference docs | IMPLEMENTED-ON-PROTECTED-MAIN | Useful deep evidence exists, but doctoring material is supporting evidence, not the canonical product-status authority. | Keep primary-reference and APA-style citations current where material. | +| PostgreSQL recovery evidence | IMPLEMENTED-ON-PROTECTED-MAIN / end-to-end PARTIAL | `postgres_recovery_receipt.py`, `postgres_backup_evidence.py`, and `postgres_schema_evidence.py` are integrated protected-main primitives with focused tests. They produce bounded content-free metadata/hash/size evidence without executing SQL or claiming cluster parity. | Keep the evidence/guarantee distinction explicit. Do not claim restorable backup, isolated restore, PITR, RPO/RTO/HA/DR, or compliance readiness from these primitives alone. | +| PostgreSQL logical backup/restore execution | ACTIVE-PR | #208 proposes bounded `pg_dump` execution. #209's direct-restore predecessor has a verified false-failure defect because a seekable custom-format `pg_restore` need not leave the shared descriptor at EOF after successful transactional restore. Draft #212 is the active successor: it removes that invalid EOF postcondition while preserving metadata-fingerprint verification and owns the permanent README/architecture/ADR/doctoring/CHANGELOG restore-contract documentation. None is protected-main truth. | Freeze #209 rather than merge or patch it in parallel; do not race #208/#212 active writers. Promote only the unchanged successor after exact-head gates, valid findings, required documentation, qualifying approval, and protected-main integration. | +| Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation for retiring legacy `http` / `pg_cron`; it is a different contract from the already-shipped tenant lifecycle migration and is not protected-main truth. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. | +| OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. | +| Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. | +| Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay; FR-4/end-to-end result application stays `PARTIAL` on protected main, and neither this row nor the TRD content-fidelity invariant promotes #194 into shipped truth. | Keep FR-4 and `docs/TRACEABILITY.md` as the validation/status authority until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. | +| Runtime config/schema provisioning separation | ACTIVE-PR | PR #193 is a current-main least-privilege candidate: runtime constructors move away from DDL/default seeding, validate the search-path-resolved base-table shape and current-role read capability through bounded catalog probes, and retain provisioning/default seeding at the explicit schema boundary. It is not shipped until normal governance integrates it. | Preserve the trusted-search-path/non-ownership caveat and reacquire exact-head checks/review after every push; do not transfer predecessor or earlier-head evidence. | +| Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map and distinguishes integrated recovery evidence from active executable candidates and unsafe predecessor heads. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs; record #212 only as an active overlay until integration. | +| Threat model | PLANNED | No canonical protected-main `docs/THREAT_MODEL.md` is present. Security rules are distributed across AGENTS, architecture, ADRs, tests, and issue/PR evidence. | Add a threat model that distinguishes assets, trust boundaries, attacker capabilities, mitigations, residual risk, and non-guarantees without certification claims. | +| Data governance | PLANNED | No single canonical data-governance document currently maps data classes, retention/ownership, tenant authority, privacy exposure, backup/restore, and deletion limits. | Add a protected-main-grounded data-governance contract. | +| UML/component/sequence views | PLANNED | Root architecture is prose-first; no canonical UML authority is present. | Add textual Mermaid/PlantUML-compatible component and critical sequence diagrams after active source contracts stabilize. | +| ERD / schema model | PLANNED | SQL and schema tests are authoritative, but no canonical ERD summarizes package-owned identities, tenancy, checkpoint state, and relationships. | Generate/maintain an ERD from protected-main schema; label migration-only/active-PR objects separately. | +| General standalone operator guide | PARTIAL | Protected main has strong operational guidance across `README.md` and topic authorities such as `docs/remote-batch-lifecycle.md`; there is no single omnibus protected-main `docs/OPERABILITY.md`. This is a navigation/consolidation gap, not absence of tenant lifecycle operator guidance. | Establish one general operator index/authority only after checking adjacent writers; do not race #184 or #212 documentation surfaces. | +| Release governance | PARTIAL | Release-evidence ADRs/tests/workflows are strong, but the canonical graph lacks one concise protected-main release/operator authority tying versioning, SBOM, provenance, artifact identity, rollback/recovery, and publication verification together. | Add a stable release contract after checking active release writers; do not copy workflow-run IDs. | +| Licensing / third-party notices | PARTIAL | Apache-2.0 source headers and repository licensing exist, but acquisition diligence should explicitly trace package license, dependency/SBOM evidence, and third-party notice process. | Evaluate a concise licensing/compliance evidence index without claiming legal certification. | + +## Non-negotiable documentation invariants + +- Protected-main behavior is the shipped authority. Active PRs and historical branches are overlays, not proof of implementation. +- Exact contributor heads, generated merge commits, check IDs, and transient queue state belong in PR/review evidence, not durable architecture/product documents. +- Standalone operation and modular MSA embedding must be documented as co-equal supported boundaries; no ContextualWisdomLab host is a hidden runtime requirement. +- Tenant scope is selected only by a trusted authenticated/authorized host boundary and, for the tenant durable client, validates before observation reservation, credential lookup, provider I/O, or lifecycle database I/O. Lifecycle identities, conflict targets, exact reads, and operational status indexes remain tenant-qualified where tenancy is enabled. +- `pg_llm_batch.tenant_scope` is transaction-local routing context, not a credential or authenticated identity. Any database role capable of arbitrary SQL can call `set_config` with an arbitrary tenant scope; the trusted application boundary must therefore prevent generic tenant-controlled SQL, SQL injection, and incorrect identity mapping from selecting tenant authority. RLS does not provide those guarantees. +- The standalone `DurableBatchAPIClient` retains its four-argument lifecycle-recorder seam and explicit `standalone` persistence/read scope unless a separately reviewed compatibility change is integrated. +- PostgreSQL RLS is defense in depth, not host authentication/authorization, SQL-injection prevention, or correct identity mapping; production application roles for the tenant lifecycle boundary are `NOSUPERUSER NOBYPASSRLS`. +- Provider/model content never becomes tenant, credential, endpoint, filesystem, or database authority. +- Content-fidelity invariants constrain any result-application path that exists, but they do not prove that end-to-end result application is shipped; FR-4 and `docs/TRACEABILITY.md` remain the capability-status authority while protected main is `PARTIAL`. +- Bounded recovery receipts, artifact hashes, and packaged-schema hashes are evidence primitives, not proof that a backup is restorable or that a stated recovery objective is met. +- A direct logical restore contract must keep caller-owned source trust, target isolation, allowed libpq environment, transaction rollback behavior, and post-restore acceptance explicit. Its archive-completion checks must match PostgreSQL custom-format random-access semantics: final descriptor offset is not proof of complete restore. #209 is therefore an unsafe predecessor, while #212 remains only an ACTIVE-PR successor until integrated. +- Distributed exactly-once processing is never implied by PostgreSQL checkpoint atomicity alone. +- Security, privacy, SOC 2, and CSAP material is evidence-readiness documentation only unless an external certification actually exists. +- Database object names, migration/rollback behavior, exact owned coverage requirements, Python-version acceptance, packaging/SBOM/provenance expectations, and bounded diagnostics must remain synchronized with repository tests and live governance. + +## Fitness gate for future canonical changes + +Before changing a canonical surface, refetch protected main, open PRs, non-default branches, the exact affected source/schema/test authorities, and any adjacent documentation writer. A documentation repair must not race a source-affecting writer whose behavior is still unsettled. When a canonical change merely reconstructs an already integrated contract, verify the existing permanent companion set before widening the change; when runtime behavior actually changes, update the required permanent companion surfaces in that same governed change or explicitly coordinate their writer ownership. After a product PR merges, update the canonical status classification from `ACTIVE-PR` or `PARTIAL` only when the protected-main tree contains the capability and the integrated evidence contract remains valid. diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md new file mode 100644 index 000000000..1a67bb9c7 --- /dev/null +++ b/docs/TRACEABILITY.md @@ -0,0 +1,85 @@ +# Requirements Traceability + +## Purpose and authority + +This map ties the canonical PRD/TRD requirements to stable protected-main implementation, test, ADR, and operator evidence. It intentionally avoids workflow-run IDs, predecessor SHAs, generated merge commits, and review comments because those are transient verification records rather than durable requirements authority. + +The reference protected-main tree is `d2f1e32271910a6db98a0757d67194ddadca4566`. Rows marked **ACTIVE-PR** or **PARTIAL** are not shipped implementation claims. + +## Product-to-technical traceability + +| Requirement | Status | Primary protected-main implementation authority | Durable verification/documentation authority | Known gap or active overlay | +| --- | --- | --- | --- | --- | +| FR-1 deterministic bounded batch preparation | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/orchestrator.py`, `pg_llm_batch/token_counter.py`, package schema | preparation/token tests; `docs/idempotent-preparation.md`; `docs/schema-integrity.md` | No gap claimed by the canonical product contract. | +| FR-2 validated bounded provider interaction | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/batch_api_client.py` | provider URL/resource/retry/response-budget tests; `docs/batch-endpoints.md`; `docs/resource-identifiers.md`; ADR 0015 | Provider-specific widening requires a separately reviewed contract. | +| FR-3 standalone + tenant-qualified durable lifecycle | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/durable_client.py`, `pg_llm_batch/db.py`, schema/RLS objects | tenant lifecycle/integration tests; `ARCHITECTURE.md`; `docs/remote-batch-lifecycle.md`; ADR 0002 | Arbitrary SQL, superuser, and BYPASSRLS remain outside the isolation guarantee. | +| FR-4 scheduler-independent bounded reconciliation | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/reconciliation.py` | reconciliation tests and protected-main release gates | Discovery #190 and single-flight #191 remain ACTIVE-PR; autonomous worker semantics remain PARTIAL. | +| FR-4 durable reconciliation candidate discovery | ACTIVE-PR | none on protected main beyond existing lifecycle/read primitives | PR #190 exact-head evidence only | Not shipped; current review/infrastructure gates must not be transferred. | +| FR-4 tenant-qualified cross-process single-flight | ACTIVE-PR | none on protected main beyond existing DB primitives | PR #191 exact-head evidence only | Not shipped; qualifying independent approval remains a live-governance concern. | +| FR-4 durable result application + checkpoint coupling | PARTIAL | `pg_llm_batch/result_streaming.py`, `pg_llm_batch/checkpoint_store.py` provide streaming/checkpoint primitives | ADR 0006; ADR 0007; checkpoint/result-streaming tests; `docs/result-streaming.md` | PR #194 is ACTIVE-PR test-first work that adds a same-transaction local result-effect/checkpoint seam; protected main still does not claim end-to-end or distributed exactly-once application. | +| FR-5 package persistence integrity | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/db.py`, `pg_llm_batch/schema.sql`, Docker schema mirror | schema-integrity, payload, lifecycle, checkpoint migration tests | Existing-volume legacy-extension retirement is separately ACTIVE-PR #184. | +| FR-5 bounded PostgreSQL recovery evidence primitives | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/postgres_recovery_receipt.py`, `pg_llm_batch/postgres_backup_evidence.py`, `pg_llm_batch/postgres_schema_evidence.py` | focused receipt/artifact/schema evidence tests; merged PRs #205, #206, #207 are historical integration evidence | Protected main can derive bounded content-free receipt, backup-artifact hash/size, and packaged-schema hash/size evidence. These primitives do not prove that a backup command ran successfully, that an artifact is restorable, that a live cluster matches packaged schema, or that PITR/RPO/RTO/HA/DR objectives are satisfied. | +| FR-5 executable PostgreSQL logical backup | ACTIVE-PR | none on protected main beyond the recovery-evidence primitives | PR #208 exact-head evidence only | The `pg_dump` execution candidate is not shipped. Its service selector is not tenant authorization, and current exact-head review/check evidence must be reacquired before integration. | +| FR-5 executable PostgreSQL logical restore | ACTIVE-PR | none on protected main beyond the recovery-evidence primitives | PR #212 exact-head evidence only; PR #209 is predecessor defect evidence | Draft #212 is the active direct-`pg_restore` successor. PR #209 must not merge because its EOF-consumption postcondition conflicts with seekable PostgreSQL custom archives and can report failure after transactional restore has already succeeded. The successor's caller-owned source-superuser trust, target-isolation responsibility, libpq allowlist, single-transaction failure boundary, metadata-integrity checks, and permanent documentation remain unshipped until integration. | +| FR-5 end-to-end PostgreSQL recovery readiness | PARTIAL | bounded receipt/artifact/schema evidence primitives are integrated | Issue #204 acceptance authority plus protected-main recovery evidence tests | No protected-main end-to-end isolated restore drill yet proves schema/RLS/constraint/extension parity, migration compatibility, external key/config custody, physical/WAL/PITR recovery, or a stated RPO/RTO/HA/DR objective. | +| FR-5 legacy `http` / `pg_cron` authority retirement on existing volumes | ACTIVE-PR | protected main does not yet contain the retirement migration contract | PR #184 migration/smoke/operator evidence only | Must remain active until unchanged exact head satisfies migration/security/release/review gates and merges. | +| FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Protected main deliberately retains `SecretStore(require_encryption=False)` as the compatibility default; Fernet support is therefore not an encryption-required production claim. PR #193 proposes an encryption-required default plus explicit local/development opt-out while also separating runtime construction from provisioning. | +| FR-6 production secret-at-rest policy lifecycle | PARTIAL | Protected main can enforce Fernet when callers explicitly select `require_encryption=True`, and fails before database access when required encryption lacks usable configuration. | config/secret/bootstrap tests; security issue acceptance criteria | No protected-main contract yet detects and atomically migrates existing `is_encrypted = FALSE` rows, proves bounded key rotation/recovery, or exposes the selected deployment policy through redacted readiness/operator evidence. PR #193 addresses the default-policy boundary only while it remains ACTIVE-PR. | +| FR-7 bounded diagnostics/readiness | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/health.py`, bounded error surfaces | health/confidentiality tests and architecture requirements | Broader generic validation rejected-value confidentiality is not protected-main truth; PR #202 is the ACTIVE-PR compatibility-aware repair and must not be inferred as shipped before integration. | +| FR-7 generic validation rejected-value confidentiality | ACTIVE-PR | protected-main `ValidationError` still renders and retains arbitrary rejected values by default | PR #202 candidate privacy/compatibility tests and review evidence only | PR #202 is Draft because its current `safe_value` guard accepts `str` subclasses via `isinstance`, which can execute caller-controlled special methods while validation or rendering handles supposedly safe evidence. The candidate must require an exact built-in `str` and prove a hostile-subclass regression before it can become Ready; the intended fixed-redaction default and bounded `safe_value` opt-in remain unshipped. | +| FR-7 opt-in OpenTelemetry | IMPLEMENTED-ON-PROTECTED-MAIN / packaging PARTIAL | `pg_llm_batch/observability.py` | observability tests | First-class locked installation extra remains ACTIVE-PR #175. | +| FR-8 standalone + modular MSA deployment | IMPLEMENTED-ON-PROTECTED-MAIN | package/CLI/container composition; injectable host seams | `README.md`, `ARCHITECTURE.md`, package/container tests | CWL host repositories are optional integrations, not package runtime dependencies. | +| Exact owned 100% statement/branch coverage | IMPLEMENTED-ON-PROTECTED-MAIN governance contract | repository CI configuration and owned production code | coverage gate/tests | Must be re-proven on every changed exact head; predecessor evidence never transfers. | +| Python 3.10/3.12/3.14 validation | IMPLEMENTED-ON-PROTECTED-MAIN governance contract | package metadata/workflow matrix | exact-head repository CI | Requires exact-head terminal success; queued/skipped/infrastructure-failed jobs are not proof. | +| Reproducible release evidence | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/release_evidence.py` and release workflows | ADR 0003; ADR 0004; release-evidence/artifact-identity tests | Release publication itself still occurs only from a fully accepted integrated protected head. | +| SBOM/provenance/artifact identity | IMPLEMENTED-ON-PROTECTED-MAIN governance contract | release workflows/evidence helpers | release acceptance and artifact verification tests | No certification claim follows from repository evidence alone. | +| SOC 2 / CSAP evidence readiness | PARTIAL | security, tenancy, logging, release and governance controls | PRD/TRD/security tests/ADRs | Evidence readiness only; no external certification is claimed. | + +## Security and privacy traceability + +| Control objective | Protected-main authority | Verification evidence | Residual boundary | +| --- | --- | --- | --- | +| Trusted tenant selection | `AGENTS.md`, `ARCHITECTURE.md`, tenant validation in DB/durable-client paths | tenant-scope/RLS tests | Host authentication/authorization remains external. | +| RLS defense in depth | tenant-qualified schema and transaction-local scope binding | live PostgreSQL isolation/migration tests | Superuser/BYPASSRLS/arbitrary SQL are administrative bypasses. | +| Provider destination validation | `batch_api_client.py` | endpoint/URL tests | Host/network infrastructure TLS policy is external. | +| Bounded provider input | provider client + result streaming | response/download/JSONL/resource-budget tests | Provider authenticity is not established by payload validation. | +| Secret/config boundary | `config.py`, bootstrap contract | config/secret/bootstrap tests | Protected main supports Fernet but does not require it by default; the compatibility path is not a production confidentiality claim. PR #193's stricter default is ACTIVE-PR, while enterprise secret-manager choice remains host-owned. | +| Diagnostic confidentiality | health/error/logging contracts | traceback/health/redaction tests | Generic `ValidationError` rejected-value confidentiality remains incomplete on protected main; PR #202 is ACTIVE-PR and its evidence does not transfer until merge. | +| Checkpoint concurrency/integrity | `checkpoint_store.py` | CAS/concurrency/RLS/rollback tests | PostgreSQL atomicity does not extend to external systems. | +| Recovery evidence confidentiality/integrity | `postgres_recovery_receipt.py`, `postgres_backup_evidence.py`, `postgres_schema_evidence.py` | recovery receipt/artifact/schema evidence regressions | Evidence is bounded and content-free but does not authenticate an operator, prove backup provenance beyond caller-supplied receipt inputs, or prove restore semantics. | +| Release artifact integrity | `release_evidence.py` + release contracts | descriptor/dirfd/reproducibility tests | Publication credentials and external registry availability are operational dependencies. | + +## Data and persistence traceability + +| Data family | Durable identity / authority | Principal protected-main documents | Recovery / non-guarantee | +| --- | --- | --- | --- | +| Tenant lifecycle state | `(tenant_scope, endpoint_alias, remote_batch_id)` | `ARCHITECTURE.md`, ADR 0002, `docs/remote-batch-lifecycle.md` | Tenant scope must come from trusted host authorization; direct SQL bypass is out of scope. | +| Result checkpoints | `(tenant_scope, checkpoint_consumer_name, endpoint_alias, remote_batch_id)` | ADR 0006, ADR 0007, `docs/result-streaming.md` | Prefix checkpoint is not provider authentication or whole-stream immutability; cross-system exactly-once is not claimed. | +| Package JSONL/payload state | package-owned schema identities and virtual payload references | PRD/TRD, schema-integrity and payload docs/tests | Persisted package data is canonical, not a disposable cache. | +| Configuration/secrets | `com_config`, `com_secrets` | PRD/TRD and config tests | Shipped compatibility mode can still persist `is_encrypted = FALSE`; runtime/provisioning least-privilege separation, required-encryption default, legacy-row migration, key rotation/recovery, and redacted policy-readiness evidence are not all protected-main behavior. | +| PostgreSQL recovery evidence | bounded receipt metadata plus backup/schema SHA-256 and byte-size evidence | protected-main recovery evidence modules/tests; canonical PRD/TRD/this traceability map once integrated | The evidence does not itself persist a backup, execute backup/restore, prove isolated target parity, or establish PITR/RPO/RTO/HA/DR. Backup storage, encryption, retention, key custody, and external copies remain deployment/operator responsibilities unless a separately integrated contract says otherwise. | +| Release evidence | descriptor/artifact identity contracts | ADR 0003, ADR 0004 | Evidence proves the reviewed artifact path, not organizational certification. | + +## Active overlay register + +The following open pull requests are intentionally represented only as overlays on this traceability map: + +- **#175** — OpenTelemetry packaging extra; dependency-lock/materialization and final package evidence remain outside protected-main truth. +- **#184** — existing-volume legacy PostgreSQL extension retirement; migration/operator behavior remains active until merged. +- **#190** — durable reconciliation candidate discovery; not protected-main truth and currently subject to live review/tooling evidence. +- **#191** — tenant-qualified reconciliation single-flight; not protected-main truth until live approval/gates and merge. +- **#192** — this canonical PRD/TRD/fitness/traceability reconstruction itself. +- **#193** — runtime-store/schema-provisioning separation plus an encryption-required `SecretStore` default with explicit local/development compatibility opt-out; both remain active overlays, and this PR does not by itself establish legacy-row migration, key rotation/recovery, or readiness-policy lifecycle completion. +- **#194** — atomic local result-effect/checkpoint application; a current-main-compatible implementation is under review but remains an active overlay and is not shipped. +- **#202** — compatibility-aware `ValidationError` rejected-value confidentiality hardening. It is Draft because the current `safe_value` guard accepts `str` subclasses via `isinstance`; the branch must first require an exact built-in `str` and pass a hostile-subclass regression. Protected main still discloses arbitrary rejected values by default, and the intended safe-default redaction/bounded `safe_value` contract remains ACTIVE-PR until integrated. +- **#208** — bounded logical PostgreSQL backup executor candidate using `pg_dump`; active source only and not evidence that protected main can create a restorable backup. +- **#209** — unsafe direct logical-restore predecessor. Its EOF-consumption postcondition is incompatible with seekable PostgreSQL custom archives and can produce a false post-commit failure. It must not merge in this state and is retained only as defect/history context while the successor is active. +- **#212** — Draft direct logical-restore successor. It replaces the invalid EOF postcondition with metadata-fingerprint verification and owns the accompanying README/architecture/ADR/doctoring/CHANGELOG restore-contract documentation. It remains ACTIVE-PR and is not evidence of isolated restore acceptance or end-to-end recovery readiness. + +Merged recovery PRs #205, #206, and #207 are no longer active overlays; their bounded receipt, backup-artifact, and packaged-schema evidence primitives are present on the referenced protected-main tree. Their historical PR descriptions remain integration evidence, not durable runtime authority. + +This register is descriptive, not a substitute for refetching GitHub. Before changing a status, verify the PR still exists, its exact contributor head, live protected-main base, ancestry, current reviews/threads, exact-head gates, and resulting protected-main integration. + +## Change-control rule + +When a capability merges, update the PRD/TRD status and this traceability map in a canonical documentation change only after the new protected-main tree is read. When a capability is abandoned or superseded, mark the overlay accordingly rather than silently deleting its historical decision context. New requirements must identify at least one intended implementation authority and one deterministic verification authority before they can be called acquisition-ready. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 000000000..d9f677522 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,51 @@ +# Architecture Decision Record Index + +## Authority + +This index is a navigation and status aid for architecture decision records that exist on protected `main`. The decision record itself remains the normative source for its context, decision, consequences, security boundary, and supersession rules. + +The protected-main tree used to reconstruct this index is `d2f1e32271910a6db98a0757d67194ddadca4566`. A pull request, historical branch, generated merge commit, workflow run, review comment, or proposed file that is not on protected main is not silently promoted into this index. + +## Protected-main decisions + +| ADR | Decision | Document status | Protected-main applicability | +| --- | --- | --- | --- | +| [0002](0002-tenant-scoped-lifecycle.md) | Tenant-scoped durable lifecycle state | Accepted | Defines trusted host-selected `tenant_scope`, tenant-qualified durable lifecycle identity, transaction-local RLS binding, and the standalone compatibility scope. | +| [0003](0003-reproducible-release-evidence.md) | Reproducible release evidence before publication | Proposed | Documents the reproducibility/evidence design present in the repository. The ADR's own `Proposed` status is retained and must not be rewritten as architectural acceptance merely because related release-evidence code or workflows exist. | +| [0004](0004-descriptor-pinned-release-artifact-verification.md) | Descriptor-pinned release artifact verification | Proposed | Documents the descriptor-pinned TOCTOU-hardening design. Its `Proposed` decision status remains authoritative until explicitly changed through reviewed documentation governance. | +| [0006](0006-resumable-result-checkpoints.md) | Resumable provider-result checkpoints | Accepted | Defines immutable prefix checkpoint evidence and its explicit prefix-only, non-authentication, non-whole-stream assurance boundary. | +| [0007](0007-durable-result-checkpoint-store.md) | Durable tenant-isolated result checkpoint store | Accepted | Adds optional PostgreSQL persistence, tenant isolation, compare-and-swap concurrency, and a caller-owned transaction seam without claiming distributed exactly-once delivery. Depends on ADR 0006. | +| [0015](0015-http-425-too-early-retry.md) | HTTP 425 retry for bounded idempotent GETs | Accepted for the bounded retry slice | Keeps the default GET retry-status set closed, side-effecting POSTs single-attempt, and TLS/certificate/fingerprint failures outside automatic retry. | + +Protected main also contains bounded PostgreSQL recovery-evidence primitives in `postgres_recovery_receipt.py`, `postgres_backup_evidence.py`, and `postgres_schema_evidence.py`. Their integration does **not** create an implicit ADR. They are implementation/evidence contracts recorded by the PRD/TRD/traceability map, while executable logical backup/restore and end-to-end recovery remain active/partial work. If a future restore executor introduces a durable architectural decision about direct SQL authority, target isolation, source-superuser trust, libpq credential/environment handling, or rollback/acceptance semantics, that decision must be added or explicitly amended through normal ADR governance rather than inferred from the implementation. + +## Numbering and missing identifiers + +ADR numbers are stable identifiers, not a promise of contiguous numbering. The absence of `0001`, `0005`, or `0008` through `0014` from the protected-main directory does **not** mean those decisions are missing, rejected, accepted elsewhere, or safe to reconstruct from old branches. Gaps may reflect historical work, superseded proposals, unmerged branches, or reserved identifiers. Only a reviewed protected-main document may establish the status of a missing number. + +New ADRs should use the next repository-approved stable identifier rather than renumbering existing records. Renaming an integrated ADR changes external references and should be treated as an architecture-governance migration, not cleanup. + +## Decision status versus implementation status + +ADR status answers whether an architectural decision has been accepted, proposed, deprecated, or superseded. It is not interchangeable with product implementation status. + +The canonical product/technical documents use `IMPLEMENTED-ON-PROTECTED-MAIN`, `ACTIVE-PR`, `PARTIAL`, `PLANNED`, and `SUPERSEDED` for implementation truth. A `Proposed` ADR can coexist with related code on protected main, and an `Accepted` ADR can describe a bounded contract whose larger product capability is still only partial. Do not infer one status system from the other. + +When an ADR and protected-main implementation appear inconsistent, treat the inconsistency as a defect to reconcile explicitly. Do not silently edit the index to make the conflict disappear. + +## Supersession and amendments + +A decision is superseded only when a reviewed record or amendment says so explicitly. A later implementation, issue, pull request, or historical branch does not implicitly supersede an ADR. + +For a material architecture change: + +1. refetch protected main and all source/documentation writers touching the decision boundary; +2. identify the existing ADRs and product/technical requirements that constrain the change; +3. record the new decision or explicit amendment, including rejected alternatives and operational/security consequences where material; +4. keep active/unmerged behavior classified as an overlay rather than protected-main truth; +5. update traceability after the capability actually reaches protected main; and +6. preserve standalone operation and modular MSA embedding unless a separately accepted decision changes that product contract. + +## Non-guarantees that must remain visible + +The current ADR set does not establish authentication from PostgreSQL RLS alone, provider authenticity from result checkpointing, full-stream immutability from a prefix checkpoint, distributed exactly-once processing, retry permission for arbitrary HTTP failures, backup restorability or PITR/RPO/RTO/HA/DR from recovery receipts/hashes alone, or organizational security/compliance certification. Those boundaries must not be weakened by summaries, operator docs, marketing material, or future ADR titles. diff --git a/docs/product/PRD.md b/docs/product/PRD.md new file mode 100644 index 000000000..47ef86feb --- /dev/null +++ b/docs/product/PRD.md @@ -0,0 +1,148 @@ +# Product Requirements Document + +## Document authority + +This PRD defines the product contract for `pg-llm-batch`. Repository behavior is authoritative only after it is integrated into the protected default branch. Pull requests, issue plans, historical branches, generated merge commits, workflow artifacts, and review commentary are evidence or work-in-progress, not shipped product truth. + +Use these status terms consistently: + +- **IMPLEMENTED-ON-PROTECTED-MAIN** — present in the protected default-branch tree and covered by its repository contract. +- **ACTIVE-PR** — implemented or being repaired in an open pull request; not shipped. +- **PARTIAL** — a protected-main primitive exists, but an end-to-end product capability still has an explicit gap. +- **PLANNED** — accepted requirement without an integrated implementation. +- **SUPERSEDED** — historical implementation or proposal that is not current product authority. + +## Product purpose + +`pg-llm-batch` is a standalone and embeddable PostgreSQL-centered engine for preparing, submitting, observing, and retrieving bounded LLM Batch API workloads. It is intended for operators and host applications that need durable batch state, deterministic token/resource accounting, explicit tenant boundaries, bounded provider I/O, and acquisition-grade operational evidence without coupling the package to one gateway, scheduler, web application, or ContextualWisdomLab host service. + +## Primary users + +1. **Platform operators** running the package as an independently deployable service or Compose component. +2. **Application/platform engineers** embedding the Python package and PostgreSQL schema in a larger product. +3. **Multi-tenant host services** that authenticate and authorize callers before supplying a trusted `tenant_scope`. +4. **Reliability and security reviewers** who need deterministic failure boundaries, migrations, rollback, audit evidence, security gates, SBOM/provenance evidence, and bounded diagnostics. + +## Product outcomes + +The product must let a qualified host: + +- count model tokens through the reviewed PostgreSQL tokenizer boundary; +- prepare JSONL batch payloads under finite token, byte, and record limits; +- preserve package-owned payloads and lifecycle state durably in PostgreSQL; +- submit, poll, wait, cancel, and retrieve through validated OpenAI-compatible Batch API clients; +- operate in exact `standalone` mode or with a trusted tenant-qualified lifecycle identity; +- recover or reconcile provider lifecycle state without introducing a second database-side networking authority; +- derive bounded content-free PostgreSQL recovery evidence for backup artifacts and the packaged schema without treating evidence as proof of restorability; and +- prove package quality, security, reproducibility, release-artifact identity, and rollback assumptions through repository evidence. + +## Protected-main capability contract + +| Capability | Status | Product requirement | +| --- | --- | --- | +| PostgreSQL token counting and bounded batch preparation | IMPLEMENTED-ON-PROTECTED-MAIN | Token/resource accounting and batch partitioning remain deterministic and finite. | +| Disk-free package payload persistence | IMPLEMENTED-ON-PROTECTED-MAIN | Package-owned JSONL payloads persist in PostgreSQL and are validated before credential/provider effects. | +| OpenAI-compatible upload/create/poll/wait/cancel/retrieve | IMPLEMENTED-ON-PROTECTED-MAIN | Provider destinations, resource identifiers, control responses, downloads, retries, and timeouts remain bounded and validated. | +| Standalone durable lifecycle | IMPLEMENTED-ON-PROTECTED-MAIN | `DurableBatchAPIClient` preserves the existing four-argument lifecycle-recorder seam `(postgres_dsn, endpoint_alias, provider_batch, observation_order)` and its default recorder stores lifecycle state under the exact `standalone` scope. | +| Tenant-qualified durable lifecycle with forced RLS | IMPLEMENTED-ON-PROTECTED-MAIN | Tenant scope comes only from a trusted host authorization boundary, is validated before any tenant-client observation reservation, credential resolution, provider I/O, or lifecycle database I/O, and qualifies lifecycle identities, conflict targets, reads, and operational status indexing; direct arbitrary SQL remains outside the isolation guarantee. | +| Durable resumable result checkpoint/CAS storage | IMPLEMENTED-ON-PROTECTED-MAIN | PostgreSQL supplies tenant-qualified durable checkpoint authority and conflict detection; this is not a distributed exactly-once claim. | +| Redacted readiness reporting | IMPLEMENTED-ON-PROTECTED-MAIN | Public readiness evidence does not disclose arbitrary lower-layer diagnostic content. | +| Scheduler-independent bounded provider reconciliation primitive | IMPLEMENTED-ON-PROTECTED-MAIN | A host can submit a finite, validated candidate set for polling/retrieval through the existing bounded provider client; candidate discovery, scheduling, and cross-process lease ownership remain outside this primitive. | +| Bounded PostgreSQL recovery receipt | IMPLEMENTED-ON-PROTECTED-MAIN | The package can encode/decode deterministic bounded content-free metadata that identifies package/source/PostgreSQL/schema/backup evidence without carrying credentials, DSNs, SQL, business payloads, or arbitrary diagnostics. | +| Bounded PostgreSQL backup-artifact integrity evidence | IMPLEMENTED-ON-PROTECTED-MAIN | The package can derive SHA-256 and byte-size evidence from one private regular backup artifact under descriptor-pinned, no-follow, finite-work constraints without executing backup or restore. | +| Bounded packaged PostgreSQL schema evidence | IMPLEMENTED-ON-PROTECTED-MAIN | The package can derive SHA-256 and byte-size evidence from the exact distributed `schema.sql` resource under a finite package-owned work budget without executing SQL or asserting live-cluster parity. | +| PostgreSQL logical backup execution | ACTIVE-PR | A `pg_dump` candidate exists in #208 but is not shipped; protected main must not be described as creating a restorable backup from the evidence primitives alone. | +| PostgreSQL logical restore execution | ACTIVE-PR | A direct `pg_restore` candidate exists in #209 but is not shipped; its caller-owned source trust, target-isolation responsibility, libpq allowlist, transactional failure boundary, and archive-integrity contract remain active-PR semantics. | +| End-to-end PostgreSQL recovery readiness | PARTIAL | Integrated evidence primitives do not yet prove an isolated restore with schema/RLS/constraint/extension parity, migration compatibility, external key/config custody, physical/WAL/PITR recovery, or a stated RPO/RTO/HA/DR objective. | +| Durable reconciliation candidate discovery | ACTIVE-PR | Discovery must be tenant-qualified, bounded, deterministic, and database-authoritative before it can become product truth. | +| Cross-process reconciliation single-flight | ACTIVE-PR | Concurrent workers must not race the same tenant/provider identity; merge eligibility remains governed by live repository policy. | +| Existing-volume legacy `http` / `pg_cron` retirement | ACTIVE-PR | Existing deployments need fail-closed, reversible migration evidence before compatibility packages can be removed. | +| First-class OpenTelemetry installation extra | ACTIVE-PR | Ordinary installs must remain telemetry-dependency-free; the optional package graph must be locked and reproducible. | +| Autonomous package-owned reconciliation worker with crash/restart completion semantics | PARTIAL | Protected main has reconciliation and durable state primitives, but does not yet claim a complete package scheduler/worker control plane. | +| Durable result application coupled to checkpoint advancement | PARTIAL | Existing checkpoint and retrieval primitives must not be described as end-to-end exactly-once result application until a reviewed coupling contract is integrated. | + +## Functional requirements + +### FR-1: Batch preparation + +The engine shall resolve an existing batch identity, select eligible requests, count tokens through the package tokenizer boundary, partition work under explicit provider/resource limits, and persist payload/file/line/request assignment state atomically for one preparation operation. Re-running a supported preparation path must not silently duplicate assignment or corrupt package-owned state. + +### FR-2: Provider interaction + +All provider operations shall use validated endpoint configuration and finite response/download budgets. Automatic retries are restricted to reviewed idempotent GET behavior. Side-effecting provider POST operations shall not gain implicit retry authority. Credentials and provider content shall not be copied into ordinary diagnostics. + +### FR-3: Durable lifecycle and tenancy + +The durable business identity is tenant-qualified where tenancy is enabled. `TenantDurableBatchAPIClient` shall validate its trusted host-selected `tenant_scope` synchronously at construction, before observation reservation, credential resolution, provider I/O, or lifecycle database I/O can occur. The tenant-qualified lifecycle key is `(tenant_scope, endpoint_alias, remote_batch_id)`; lifecycle persistence conflict targets, exact-row lookups, and operational status indexes shall retain `tenant_scope`, and package reads/writes shall bind that validated scope through parameterized transaction-local PostgreSQL context with forced row-level security for application roles. Provider/model content, endpoint aliases, remote identifiers, and transport data never select tenant authority. + +The transaction-local `pg_llm_batch.tenant_scope` custom setting is routing context, not a credential or authenticated identity. Only package code acting on a trusted authenticated/authorized host selection may set it for tenant-owned operations; arbitrary SQL must not select or override tenant authority by calling `set_config`. PostgreSQL RLS is defense in depth and does not replace host authentication/authorization, SQL-injection prevention, or correct identity mapping. PostgreSQL superuser/BYPASSRLS and arbitrary SQL access remain administrative escape hatches outside the tenant isolation guarantee. + +The deployment credential provider remains a separate host/configuration authority: validating tenant scope before credential resolution does not make credentials tenant-keyed and does not authorize a tenant to select secrets. + +Standalone source compatibility is also normative. `DurableBatchAPIClient` retains its four-argument lifecycle-recorder interface `(postgres_dsn, endpoint_alias, provider_batch, observation_order)`, and the default persistence/read helpers use the explicit `standalone` database scope rather than silently introducing a required tenant argument. + +Protected-main acceptance authority for these invariants is deterministic: `tests/test_tenant_durable_client.py` proves construction-time pre-effect tenant validation and the four-argument standalone recorder seam; `tests/test_tenant_lifecycle_persistence.py` proves explicit `standalone` delegation, tenant-qualified conflict targets and reads, malformed-scope rejection before database access, and distinct tenant identities; `pg_llm_batch/schema.sql` supplies the tenant-qualified unique key, forced RLS policy, and tenant-qualified operational status index. `docs/remote-batch-lifecycle.md`, ADR 0002, and `docs/doctoring/tenant-scoped-lifecycle.md` describe the same protected-main contract. + +### FR-4: Reconciliation and provider-effect recovery + +Reconciliation shall be finite, deterministic, payload-free in its operational evidence, and use the same validated provider client boundary as normal operations. Candidate discovery, concurrency control, scheduling, and result-application semantics must be explicit capabilities rather than inferred from polling code. Provider-success/database-failure cases must remain observable recovery states rather than being rewritten as if the provider effect never occurred. + +### FR-5: Persistence integrity and PostgreSQL recovery evidence + +Package-owned database rows shall use descriptive two-or-more-word `snake_case` object names where applicable, explicit durable identities, parameterized SQL, and migrations that are idempotent or have a documented one-way boundary. Schema copies maintained for package and Docker initialization shall remain synchronized where the repository contract requires it. Malformed durable payload/state shall fail closed before downstream credential or provider effects when correctness depends on that state. + +Protected main shall provide bounded, content-free recovery evidence without overstating its guarantee. `PostgresRecoveryReceipt` identifies one evidence set using package version, exact source commit, PostgreSQL major version, schema SHA-256, a reviewed backup-method vocabulary, backup-artifact SHA-256 and size, and bounded start/completion epochs. Backup-artifact inspection shall pin directory/file identity, reject symlink traversal and unsafe file shapes, remain within a caller-visible finite hashing budget, and return only hash/size evidence. Packaged-schema inspection shall stream the exact distributed `schema.sql` under a finite package-owned budget and return only hash/size evidence. Malformed, ambiguous, duplicate-member, hostile-subclass, oversized, mutating, unreadable, or cleanup-failing evidence shall fail closed through fixed package diagnostics. + +These primitives do not execute `pg_dump` or `pg_restore`, persist backup bytes, prove backup provenance or restorability, prove a live-cluster schema matches the package, authorize a tenant/operator, manage encryption keys, manage WAL, establish PITR, or prove RPO/RTO/HA/DR/compliance. Executable logical backup/restore and isolated-restore acceptance remain separate governed capabilities. A future direct restore must make caller-owned source-superuser trust, target isolation, allowed libpq credential/service environment, transaction rollback behavior, and post-restore acceptance explicit before it can become protected-main truth. + +### FR-6: Configuration and secrets + +PostgreSQL-backed configuration and encrypted-secret support remain available for standalone use. Environment variables are bootstrap transport only where explicitly documented. Embedding hosts may inject credential providers without changing provider protocol semantics. The package shall not invent an external secret-management product dependency. + +### FR-7: Observability and diagnostics + +Operational telemetry is opt-in. Bounded operation/outcome vocabularies may be emitted; prompts, provider response bodies, credentials, arbitrary endpoint aliases, resource IDs, and arbitrary exception text are not telemetry attributes. Public readiness and ordinary errors shall use bounded categories when lower-layer text could contain sensitive data. + +### FR-8: Standalone and modular deployment + +The package shall remain usable without `contextual-orchestrator`, `naruon`, or another CWL repository. Host services may provide authentication, tenant routing, secret resolution, gateway/model routing, OpenTelemetry export, or scheduling, but those integrations shall not become hidden standalone requirements. + +## Non-functional requirements + +### Quality + +Owned production Python must maintain exact 100% statement and branch coverage and complete public docstrings under the repository's configured gates. Supported validation includes Python 3.10, 3.12, and 3.14 plus realistic PostgreSQL/container integration where behavior depends on PostgreSQL. + +### Security and privacy + +Security-sensitive validation fails closed. Secrets, DSNs, prompt/provider content, unvalidated identifiers, and arbitrary lower-layer exception text must not be retained in logs or review evidence merely for debugging. Repository controls should support SOC 2 / CSAP evidence preparation without claiming certification. + +The package must not apply blanket masking or lossy transformation to authorized business payloads merely because they contain PII: changing prompt, request, or result content can invalidate business meaning, token accounting, provider behavior, auditability, or downstream decisions. Authorized content fidelity is therefore a product requirement. Confidentiality controls belong at explicit boundaries instead: trusted host authentication/authorization and tenant selection, least-privilege database/service access, transport protection, deployment/storage protection where provided, purpose-limited retention and deletion, and package-owned logs/telemetry/errors that omit content-bearing values. Any content transformation must be an explicit host/business policy with provenance and acceptance evidence, not a hidden package default or a claim that redacted diagnostics mean persisted business data was masked. + +### Reliability + +Network, response, retry, wait, candidate-scan, payload, recovery-evidence, and release-evidence operations must be explicitly bounded. Recovery receipts and hashes are integrity/identity evidence, not restoration success. Recovery and rollback must be documented before migrations or release changes are considered complete. Queued, skipped, cancelled, absent, stale, predecessor-head, synthetic-merge-only, or infrastructure-failed evidence is not success for an exact source head. + +### Interoperability + +Provider interaction stays OpenAI-Batch-compatible behind a validated Python client seam. Embedding hosts can provide credentials and control-plane context without changing the package's durable/provider semantics. PostgreSQL backup/restore executors, when integrated, must remain caller-targeted infrastructure seams rather than hidden dependencies on a specific ContextualWisdomLab host. + +### Packaging and release + +Dependencies must be locked/reproducible according to repository policy. Release acceptance must include required quality, security, package/container, SBOM, provenance, artifact-identity, rollback/recovery, and governance evidence on the exact integrated protected head. A release is not implied by a version string or a successful pull request. + +## Explicit non-goals + +- Replacing host authentication or authorization. +- Treating PostgreSQL RLS as a credential or as SQL-injection prevention. +- Making provider payload/model output an authority for tenant or endpoint selection. +- Providing an unbounded general-purpose HTTP proxy. +- Reintroducing provider networking or independent scheduling inside PostgreSQL. +- Claiming a backup is restorable, a live cluster matches packaged schema, or a recovery objective is met from receipt/hash evidence alone. +- Claiming distributed exactly-once processing without an integrated transaction/recovery contract spanning every external effect. +- Requiring a specific ContextualWisdomLab host service for standalone operation. +- Claiming SOC 2, CSAP, or other certification solely from repository controls. + +## Product acceptance boundary + +A product capability moves to **IMPLEMENTED-ON-PROTECTED-MAIN** only after its unchanged source has satisfied the live ruleset, required exact-head CI/security/coverage/package/provenance/release checks, valid review findings are resolved, and the resulting tree is integrated into the protected default branch. Documentation must then be updated to move the capability out of `ACTIVE-PR`, `PARTIAL`, or `PLANNED`; historical evidence must not be transferred as if it were proof for a different head. diff --git a/docs/product/TRD.md b/docs/product/TRD.md new file mode 100644 index 000000000..3449661f1 --- /dev/null +++ b/docs/product/TRD.md @@ -0,0 +1,194 @@ +# Technical Requirements Document + +## Document authority + +This TRD defines technical invariants for `pg-llm-batch`. It distinguishes protected-main behavior from work that is only present in an active pull request. The repository's live code, schema, tests, ruleset, and exact-head evidence remain stronger authority than historical branches, stale PR prose, predecessor checks, or generated merge commits. + +Status vocabulary is shared with the PRD: **IMPLEMENTED-ON-PROTECTED-MAIN**, **ACTIVE-PR**, **PARTIAL**, **PLANNED**, and **SUPERSEDED**. + +## System boundary + +`pg-llm-batch` is a Python package plus PostgreSQL schema and container assets. It may run standalone or be embedded by another service. The package owns batch preparation, package persistence, validated provider Batch API access, durable lifecycle projection, resumable checkpoint storage, bounded reconciliation primitives, and bounded content-free PostgreSQL recovery-evidence primitives. It does not own host authentication, business authorization, ingress/WAF, infrastructure TLS policy, external secret-manager choice, global OpenTelemetry configuration, backup-storage infrastructure, WAL/archive infrastructure, or a cross-system distributed transaction. + +## Component contract + +| Component | Protected-main responsibility | Prohibited authority | +| --- | --- | --- | +| `token_counter.py` | resolve reviewed tokenizer metadata and call `pg_tiktoken` for token accounting | provider credentials, tenant authorization, provider I/O | +| `orchestrator.py` | select queued requests, partition under limits, persist package payload/file/line/request assignments | provider protocol and retry policy | +| `batch_api_client.py` | validate provider destination and remote identifiers; upload/create/poll/wait/cancel/retrieve under finite budgets | tenant authentication, scheduler ownership | +| `durable_client.py` | compose provider operations with durable lifecycle ordering/persistence; preserve `DurableBatchAPIClient` source compatibility, its four-argument `LifecycleRecorder(postgres_dsn, endpoint_alias, provider_batch, observation_order)` seam, and default `standalone` database scope; provide a distinct tenant-qualified recorder seam | authenticating or authorizing host tenants, deriving tenant authority from provider data | +| `db.py` | schema application and parameterized persistence/read helpers, including tenant-qualified lifecycle context, conflict identity, and exact-row lookup | database-side provider networking, authenticating tenant callers | +| `checkpoint_store.py` | tenant-qualified PostgreSQL checkpoint/CAS operations and conflict semantics | distributed exactly-once claims | +| `reconciliation.py` | finite host-selected polling/retrieval pass using the existing validated client surface | candidate discovery, scheduling, cross-process leasing | +| `postgres_recovery_receipt.py` | encode/decode one deterministic bounded content-free PostgreSQL recovery evidence receipt | proving backup success/restorability, authenticating operators, carrying DSNs/credentials/business content | +| `postgres_backup_evidence.py` | derive SHA-256 and byte-size evidence from one private regular backup artifact through descriptor-pinned no-follow traversal and finite work | executing `pg_dump`/`pg_restore`, persisting backup bytes, proving restore semantics | +| `postgres_schema_evidence.py` | derive SHA-256 and byte-size evidence from the exact distributed `pg_llm_batch/schema.sql` resource under a finite package-owned budget | executing SQL, proving live-cluster parity or migration currency | +| `config.py` | PostgreSQL-backed configuration and encrypted secret storage for standalone composition | prescribing an embedding host's external secret manager | +| `observability.py` | opt-in bounded traces/metrics around reviewed operations | configuring global SDK/exporter/resource policy | +| `health.py` | readiness aggregation and redacted public health response | general-purpose web serving or arbitrary diagnostic reflection | +| `cli.py` | standalone operator composition and bounded input surfaces | higher-level workflow orchestration | + +The protected-main validation authority for the `durable_client.py` compatibility row is `pg_llm_batch/durable_client.py`, `tests/test_tenant_durable_client.py`, and `tests/test_tenant_lifecycle_persistence.py`: the tests prove construction-time tenant rejection before downstream effects, exact four-argument standalone recorder invocation, explicit `standalone` persistence/read delegation, and tenant-qualified lifecycle persistence/read identities. + +The protected-main validation authority for recovery evidence is the three recovery modules plus their focused test suites. Historical merged PRs #205, #206, and #207 are integration evidence, not runtime authority. `docs/TRACEABILITY.md` remains the canonical status map for the distinction between integrated evidence primitives and active backup/restore execution candidates. + +## Runtime architecture + +### Batch preparation + +1. Resolve a supported package batch identity. +2. Read eligible unassigned requests from PostgreSQL. +3. Count model tokens through the reviewed PostgreSQL tokenizer boundary. +4. Partition requests in memory under explicit batch token, byte, record, and provider limits. +5. Under one package preparation transaction, persist virtual payloads, batch-file rows, JSONL line rows, request assignments, and aggregate totals in deterministic order. +6. On failure before commit, roll back the package preparation transaction rather than exposing a partially committed preparation from that invocation. + +Package-generated provider payloads are represented by `memory://` references and are reconstructed from PostgreSQL rather than written to a package-owned local payload file. + +### Provider I/O + +Provider gateway URLs and endpoint aliases are configuration inputs but are untrusted until validated by the applicable package boundary. Production gateway destinations require HTTPS; only explicitly reviewed loopback development HTTP destinations are accepted. Userinfo, query, fragment, whitespace, malformed port, or other forbidden URL forms must fail before credentials are used. + +Control-plane responses are consumed through a finite decoded-byte budget and strict UTF-8/JSON parsing. Provider output/error files are streamed in finite chunks with an independent decoded-byte ceiling before JSONL parsing. The client must fail closed when an adapter cannot provide the bounded streaming interface required by the operation. + +Automatic provider retry is limited to reviewed idempotent GET operations and the exact default status set `{408, 425, 429, 502, 503, 504}`. TLS handshake, certificate, and fingerprint failures are not automatically retried. Upload/create/cancel POST operations remain single-attempt unless a separately reviewed provider-specific contract is introduced. + +### Durable lifecycle tenancy + +Standalone lifecycle data uses the exact `standalone` tenant scope. `DurableBatchAPIClient` preserves its original four-argument lifecycle-recorder interface `(postgres_dsn, endpoint_alias, provider_batch, observation_order)`; its default persistence path and `get_remote_batch_state(...)` compatibility helper resolve through the explicit `standalone` scope. Tenant-aware hosts instead use `TenantDurableBatchAPIClient` with a distinct tenant-qualified recorder seam so tenant identity cannot be silently dropped. + +A tenant-aware client validates the trusted host-selected `tenant_scope` synchronously during construction, before any observation reservation, credential-provider lookup, provider I/O, or lifecycle database I/O can occur. Provider metadata, request payloads, model output, transport headers, endpoint aliases, provider resource identifiers, and credential data never choose tenant authority. Credential resolution remains a separate deployment/host concern: ordering tenant validation before it does not make the credential store tenant-keyed. + +The durable lifecycle identity is: + +```text +(tenant_scope, endpoint_alias, remote_batch_id) +``` + +Every tenant-aware lifecycle persistence conflict target and exact-row lookup includes the full identity. The protected schema's lifecycle operational status index begins with `tenant_scope`, and package reads/writes bind the validated scope with parameterized transaction-local `set_config`. The schema enables and forces PostgreSQL row-level security for tenant-qualified lifecycle state. Production application roles must be `NOSUPERUSER NOBYPASSRLS`. A role that can execute arbitrary SQL can choose arbitrary custom setting values; therefore generic arbitrary SQL access is explicitly outside the package isolation guarantee. + +These invariants are deterministically verified on protected main: `tests/test_tenant_durable_client.py` proves malformed scope fails before reservation or credentials and proves the unchanged four-argument standalone recorder seam; `tests/test_tenant_lifecycle_persistence.py` proves malformed scope fails before database access, the upsert conflict target is `(tenant_scope, endpoint_alias, remote_batch_id)`, exact reads bind the same full identity, and standalone helpers delegate to `standalone`; `pg_llm_batch/schema.sql` supplies the tenant-qualified unique constraint, forced RLS policy, and `idx_llm_remote_batch_jobs_tenant_status_observed` index. `docs/remote-batch-lifecycle.md`, ADR 0002, and `docs/doctoring/tenant-scoped-lifecycle.md` document the same migration, direct-SQL/RLS, role, and rollback boundaries. + +### Durable result checkpoints + +Protected main contains tenant-qualified durable result checkpoint storage with compare-and-swap/conflict semantics. The checkpoint store can participate in a caller-owned PostgreSQL transaction where the caller's durable result application is in the same database transaction. The mere existence of this store does not prove exactly-once application across provider/network/database boundaries. + +Checkpoint counters, offsets, and identities must validate before mutation. Conflicting writes fail explicitly rather than silently overwriting a newer checkpoint. Rollback/recovery tests and schema parity are part of the storage contract. + +### Reconciliation + +Protected main supplies a scheduler-independent `reconcile_batch_candidates(...)` primitive. A host supplies candidate identities and a finite `max_jobs` budget. The primitive validates candidates, bounds scanning/work, polls through the validated provider client, retrieves completed jobs through that same client, and returns payload-free finite outcome/error categories. + +The host still owns candidate discovery, tenant authorization, scheduling, and cross-process concurrency. Durable candidate discovery and tenant-qualified advisory single-flight are **ACTIVE-PR** surfaces; neither is treated as shipped until protected-main integration. Package-owned autonomous scheduling, crash/restart completion semantics, terminal-work retirement after durable result application, and an end-to-end exactly-once worker remain **PARTIAL** or **PLANNED** capabilities. + +### PostgreSQL recovery evidence + +Protected main supplies three deliberately non-executing recovery-evidence primitives. + +1. `PostgresRecoveryReceipt` binds exact built-in primitive metadata for package version, source commit, PostgreSQL major, packaged-schema SHA-256, reviewed backup-method vocabulary (`logical`, `physical`, or `pitr`), backup artifact SHA-256/size, and bounded timestamps. Its JSON representation is deterministic and size-bounded, rejects duplicate/unknown fields and hostile subclasses, and maps ordinary malformed input/decoder failures to fixed content-free diagnostics. +2. `inspect_postgres_backup_artifact(...)` traverses path components through pinned directory descriptors with no-follow semantics, rejects `..`, symlinked parents/final components, non-regular/empty/oversized files and unsafe link counts/permissions as defined by the implementation contract, hashes under an explicit finite maximum-size work budget, bounds each read request by remaining budget, compares descriptor identity/metadata before and after hashing, and treats cleanup failures as bounded evidence without masking an already-selected primary error. +3. `inspect_postgres_schema()` streams the exact distributed `pg_llm_batch/schema.sql` resource through SHA-256 under a finite package-owned work budget and returns only SHA-256 plus byte size. Missing, unreadable, empty, oversized, malformed-chunk, hostile-subclass, or cleanup-failing resources fail closed through fixed content-free diagnostics. + +These primitives do not execute SQL or database mutation. They do not prove the backup command succeeded, prove backup provenance beyond caller-controlled receipt fields, prove restorability, prove a live database matches the packaged schema, provide target isolation, manage keys/secrets, manage physical/WAL/PITR infrastructure, or establish RPO/RTO/HA/DR/compliance. Those are separate acceptance domains. + +Logical `pg_dump` execution in #208 and direct `pg_restore` execution in #209 remain **ACTIVE-PR**. Until integration, no protected-main technical contract may rely on those executors. The direct-restore candidate additionally requires permanent operator/architecture/ADR/doctoring/CHANGELOG coverage for caller-owned source-superuser trust, the non-authorizing service selector, permitted inherited libpq variables, single-transaction rollback behavior, target isolation, and post-restore acceptance before it may be represented as shipped. + +## Persistence requirements + +### Naming and schema ownership + +New package-owned database objects use descriptive two-or-more-word `snake_case` names where applicable. SQL is parameterized; identifiers are not constructed from unvalidated user/provider text. Packaged schema and Docker initialization copies that represent the same contract must remain synchronized by regression tests. + +### Migration behavior + +Migrations must have an explicit compatibility and rollback boundary. Tenant migrations preserve legacy data under `standalone`, avoid a committed intermediate RLS-bypass state, restore forced RLS atomically, and remain idempotent where documented. Existing-volume retirement of legacy `http` / `pg_cron` authority is **ACTIVE-PR** and may not be described as shipped merely because fresh initialization no longer depends on SQL-side provider networking. + +### Data integrity + +Package-owned persisted virtual JSONL is canonical state, not a best-effort cache. Malformed shape, line count, framing, duplicate JSON members, non-finite numeric forms, or invalid record type fail closed through bounded package errors. Local payload integrity validation occurs before provider credentials/provider I/O where the provider effect relies on that payload. + +Durable provider/resource identifiers and tenant identities are validated before they become persistence or authorization inputs. Database/query failures must not be silently reclassified as an authoritative no-row result when correctness depends on distinguishing those cases. + +Recovery evidence is identity/integrity metadata, not a second persistence authority. The package does not own backup storage, replica lifecycle, object-store retention, WAL archives, encryption-at-rest infrastructure, or backup deletion merely because it can hash an operator-selected artifact. Operators/deployments must keep those responsibilities explicit. + +## Security and privacy requirements + +### Secrets + +Standalone provider configuration and encrypted secrets are PostgreSQL-backed. Environment variables are limited to explicitly documented bootstrap transport such as the database DSN and optional encryption key. CLI secret entry uses no-echo prompting or bounded standard input rather than plaintext process arguments. Embedding hosts may supply another credential provider through the supported seam. + +Recovery evidence must not carry DSNs, passwords, Fernet keys, prompts/results, ciphertext, arbitrary SQL, provider payloads, paths where not required by the callable interface, dynamic exception names, or reflected lower-layer diagnostics. A future backup/restore executor must isolate credentials from process arguments and ambient environment according to its separately reviewed contract. + +### Authorized content fidelity + +The package does not gain authority to mask, tokenize away, truncate for privacy, or otherwise alter an authorized prompt, request, JSONL record, or provider result merely because the content may contain PII. Silent transformation would change token counts, provider semantics, persisted evidence, replay behavior, and downstream business meaning. Serialization, token accounting, persistence, upload, and retrieval paths implemented on protected main therefore preserve authorized business content unless an explicit reviewed feature contract says otherwise. The same content-fidelity invariant constrains any result-application path that exists, but end-to-end result application remains **PARTIAL** under FR-4 and `docs/TRACEABILITY.md`; PR #194 is an **ACTIVE-PR** transaction-seam candidate, not protected-main proof of a completed result-application capability. + +Confidentiality for content-bearing data is enforced through boundary controls rather than a blanket masking default: the embedding host authenticates and authorizes the caller and selects tenant scope; package/database/service identities remain least-privilege; and transport uses the reviewed secure destination policy. Protected main does not define a universal business-data retention duration or a general destructive deletion workflow. The embedding host owns business purpose, retention period, deletion authorization/trigger, and evidence that its policy was executed; the deployment owner separately owns PostgreSQL backup/replica, log/telemetry, and infrastructure retention/deletion controls; provider-side retention/deletion remains a provider/account-policy responsibility unless an explicit reviewed package adapter contract implements and verifies it. Package-owned errors, logs, telemetry, readiness, CI/review evidence, and other operational surfaces omit content-bearing values. A host that intentionally transforms content must do so through an explicit business-policy boundary with provenance and acceptance tests. Redacted operational evidence must never be represented as proof that persisted or provider-bound business content was masked or deleted. + +### Diagnostic confidentiality + +Errors, logs, telemetry, check evidence, and public readiness must avoid DSNs, credentials, prompts, provider bodies, arbitrary SQL/provider exception text, unvalidated identifiers, and dynamic exception-class names where those values are not required for operation. Failure categories intended for public/operational evidence use bounded vocabularies. + +### Tenant boundary + +RLS augments a trusted host authorization boundary; it is not itself authentication, a credential, or SQL-injection prevention. Administrative database identities are outside the ordinary tenant guarantee. Pooling code must not leak transaction-local tenant context between logical operations. + +### Provider boundary + +Provider URLs, statuses, IDs, headers, JSON, JSONL, retry guidance, and metadata are untrusted external input. Validation occurs before the downstream effect that relies on that value. Model/provider output never grants tenant, endpoint, credential, or filesystem authority. + +### Recovery authority boundary + +A backup artifact, receipt, service selector, schema hash, or backup method string is not authorization to read, write, restore, or replace a database. Authentication, authorization, target isolation, backup custody, key custody, destructive-operation approval, and recovery-objective ownership remain external until a separately integrated contract explicitly supplies and verifies them. Recovery evidence must never be used to infer those authorities. + +## Observability requirements + +OpenTelemetry support is opt-in. Base installations must not require OpenTelemetry packages solely to use normal batch functionality. Operation names, outcomes, and error categories are finite. Telemetry attributes exclude endpoint aliases, provider URLs, resource identifiers, credentials, metadata, prompts, and provider response bodies. + +A first-class packaging extra for OpenTelemetry is **ACTIVE-PR** until its exact generated dependency lock is committed normally, temporary materialization machinery is removed, and final package/install/release gates succeed. + +## Health and operability + +Readiness covers the required PostgreSQL/tokenizer/configuration boundary and returns redacted public evidence. A failing dependency must not cause arbitrary lower-layer text to be reflected to a caller. Docker/container health and package health semantics must remain aligned. + +Operational migrations require backup/preflight/acceptance/recovery documentation before release. A recovery instruction must preserve package and operator-owned state; destructive `CASCADE`, hidden history rewrite, or deletion of unknown operator objects is not an acceptable shortcut. + +A recovery drill must distinguish artifact identity from restore acceptance. At minimum, acceptance criteria for a future end-to-end logical restore must address exact schema/package identity, required schema/RLS/constraint/extension behavior, migration compatibility, intended target isolation, credential/key availability, and rollback/recovery behavior. Physical/WAL/PITR drills additionally require their own timeline/target and infrastructure acceptance criteria. No repository evidence should claim universal RPO/RTO/HA/DR without an explicit measured deployment objective. + +## Concurrency requirements + +Batch preparation uses database coordination/transactionality appropriate to its package-owned state. Durable lifecycle writes use explicit ordering/conflict semantics. Any cross-process reconciliation exclusion must be tenant-qualified, non-blocking or finitely bounded, exception-safe, and clear about whether it is transient session state or durable lease state. Session advisory locking must never be promoted into a durable lease or distributed exactly-once claim. + +Recovery-evidence file inspection must remain finite and fail closed under concurrent mutation. A successful hash/size result is valid only for the descriptor identity/metadata contract that was revalidated by the implementation; it is not a lock or lease over external backup infrastructure. + +## Testing requirements + +Every source defect follows realistic RED → narrow fix → GREEN → focused/full validation. Required repository evidence includes: + +- Python 3.10, 3.12, and 3.14 where configured; +- exact 100% owned production statement and branch coverage; +- public docstring coverage; +- lint/static checks; +- realistic PostgreSQL integration for SQL/RLS/migration/concurrency behavior; +- package and container installation/health validation; +- migration idempotency and rollback/recovery coverage where applicable; +- recovery-evidence tests for exact primitive types, duplicate/unknown metadata, finite work, descriptor/path boundaries, concurrent mutation, cleanup failure, and content-free diagnostics; +- confidentiality regressions that inspect full exception/traceback surfaces when relevant; +- security scanning and SAST; +- dependency-lock and packaging reproducibility; +- release acceptance, artifact identity, SBOM, and provenance evidence required by the live repository contract. + +Queued, pending, skipped, cancelled, absent, neutral, stale, predecessor-head, synthetic-merge-only, status-only, infrastructure-failed, or rate-limited evidence is not exact-head success. + +## Release requirements + +A release may originate only from the exact integrated protected head after all live required quality, security, review, migration, rollback/recovery, operational, packaging, provenance, and release-acceptance gates are terminal-success. Versioning and CHANGELOG updates are followed by publication and artifact verification. Repository control evidence may support SOC 2 / CSAP readiness but must not be represented as certification. + +The presence of recovery-evidence primitives does not make a release recovery-ready for a deployment. A release or operator contract that claims restore/PITR/RPO/RTO/HA/DR readiness must cite the exact integrated executor/drill/acceptance evidence for that deployment objective rather than extrapolating from hash/receipt modules. + +## Documentation requirements + +Canonical product/technical/architecture/ADR/UML/ERD/security/operability/release/data-governance/traceability documents must distinguish shipped protected-main state from active or planned work. Durable documents should describe contracts rather than transient check-run IDs. When an active capability merges, the canonical graph is updated in the same governance model rather than relying on stale PR body claims. + +Direct-SQL or rollback authority introduced by a future backup/restore executor requires coordinated permanent README, operator, architecture, ADR (or explicit accepted amendment/no-new-decision rationale), doctoring/reference, and CHANGELOG coverage. Active PR source may document its own callable boundary, but canonical docs must not represent it as shipped before integration.