From e724e0cae06b04f1225fdcab5f97346728c4771e Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 14 Aug 2026 20:10:56 +0900 Subject: [PATCH 01/29] docs: establish protected-main product requirements --- docs/product/PRD.md | 126 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 docs/product/PRD.md diff --git a/docs/product/PRD.md b/docs/product/PRD.md new file mode 100644 index 000000000..905a3906c --- /dev/null +++ b/docs/product/PRD.md @@ -0,0 +1,126 @@ +# Product Requirements Document + +## Document authority + +This PRD defines the product contract for `pg-llm-batch`. Repository behavior is authoritative only after it is integrated into the protected default branch. Pull requests, issue plans, historical branches, generated merge commits, workflow artifacts, and review commentary are evidence or work-in-progress, not shipped product truth. + +Use these status terms consistently: + +- **IMPLEMENTED-ON-PROTECTED-MAIN** — present in the protected default-branch tree and covered by its repository contract. +- **ACTIVE-PR** — implemented or being repaired in an open pull request; not shipped. +- **PARTIAL** — a protected-main primitive exists, but an end-to-end product capability still has an explicit gap. +- **PLANNED** — accepted requirement without an integrated implementation. +- **SUPERSEDED** — historical implementation or proposal that is not current product authority. + +## Product purpose + +`pg-llm-batch` is a standalone and embeddable PostgreSQL-centered engine for preparing, submitting, observing, and retrieving bounded LLM Batch API workloads. It is intended for operators and host applications that need durable batch state, deterministic token/resource accounting, explicit tenant boundaries, bounded provider I/O, and acquisition-grade operational evidence without coupling the package to one gateway, scheduler, web application, or ContextualWisdomLab host service. + +## Primary users + +1. **Platform operators** running the package as an independently deployable service or Compose component. +2. **Application/platform engineers** embedding the Python package and PostgreSQL schema in a larger product. +3. **Multi-tenant host services** that authenticate and authorize callers before supplying a trusted `tenant_scope`. +4. **Reliability and security reviewers** who need deterministic failure boundaries, migrations, rollback, audit evidence, security gates, SBOM/provenance evidence, and bounded diagnostics. + +## Product outcomes + +The product must let a qualified host: + +- count model tokens through the reviewed PostgreSQL tokenizer boundary; +- prepare JSONL batch payloads under finite token, byte, and record limits; +- preserve package-owned payloads and lifecycle state durably in PostgreSQL; +- submit, poll, wait, cancel, and retrieve through validated OpenAI-compatible Batch API clients; +- operate in exact `standalone` mode or with a trusted tenant-qualified lifecycle identity; +- recover or reconcile provider lifecycle state without introducing a second database-side networking authority; +- prove package quality, security, reproducibility, release-artifact identity, and rollback assumptions through repository evidence. + +## Protected-main capability contract + +| Capability | Status | Product requirement | +| --- | --- | --- | +| PostgreSQL token counting and bounded batch preparation | IMPLEMENTED-ON-PROTECTED-MAIN | Token/resource accounting and batch partitioning remain deterministic and finite. | +| Disk-free package payload persistence | IMPLEMENTED-ON-PROTECTED-MAIN | Package-owned JSONL payloads persist in PostgreSQL and are validated before credential/provider effects. | +| OpenAI-compatible upload/create/poll/wait/cancel/retrieve | IMPLEMENTED-ON-PROTECTED-MAIN | Provider destinations, resource identifiers, control responses, downloads, retries, and timeouts remain bounded and validated. | +| Standalone durable lifecycle | IMPLEMENTED-ON-PROTECTED-MAIN | Existing standalone users retain source-compatible durable lifecycle behavior. | +| Tenant-qualified durable lifecycle with forced RLS | IMPLEMENTED-ON-PROTECTED-MAIN | Tenant scope comes only from a trusted host authorization boundary; direct arbitrary SQL remains outside the isolation guarantee. | +| Durable resumable result checkpoint/CAS storage | IMPLEMENTED-ON-PROTECTED-MAIN | PostgreSQL supplies tenant-qualified durable checkpoint authority and conflict detection; this is not a distributed exactly-once claim. | +| Redacted readiness reporting | IMPLEMENTED-ON-PROTECTED-MAIN | Public readiness evidence does not disclose arbitrary lower-layer diagnostic content. | +| Scheduler-independent bounded provider reconciliation primitive | IMPLEMENTED-ON-PROTECTED-MAIN | A host can submit a finite, validated candidate set for polling/retrieval through the existing bounded provider client; candidate discovery, scheduling, and cross-process lease ownership remain outside this primitive. | +| Durable reconciliation candidate discovery | ACTIVE-PR | Discovery must be tenant-qualified, bounded, deterministic, and database-authoritative before it can become product truth. | +| Cross-process reconciliation single-flight | ACTIVE-PR | Concurrent workers must not race the same tenant/provider identity; merge eligibility remains governed by live repository policy. | +| Existing-volume legacy `http` / `pg_cron` retirement | ACTIVE-PR | Existing deployments need fail-closed, reversible migration evidence before compatibility packages can be removed. | +| First-class OpenTelemetry installation extra | ACTIVE-PR | Ordinary installs must remain telemetry-dependency-free; the optional package graph must be locked and reproducible. | +| Autonomous package-owned reconciliation worker with crash/restart completion semantics | PARTIAL | Protected main has reconciliation and durable state primitives, but does not yet claim a complete package scheduler/worker control plane. | +| Durable result application coupled to checkpoint advancement | PARTIAL | Existing checkpoint and retrieval primitives must not be described as end-to-end exactly-once result application until a reviewed coupling contract is integrated. | + +## Functional requirements + +### FR-1: Batch preparation + +The engine shall resolve an existing batch identity, select eligible requests, count tokens through the package tokenizer boundary, partition work under explicit provider/resource limits, and persist payload/file/line/request assignment state atomically for one preparation operation. Re-running a supported preparation path must not silently duplicate assignment or corrupt package-owned state. + +### FR-2: Provider interaction + +All provider operations shall use validated endpoint configuration and finite response/download budgets. Automatic retries are restricted to reviewed idempotent GET behavior. Side-effecting provider POST operations shall not gain implicit retry authority. Credentials and provider content shall not be copied into ordinary diagnostics. + +### FR-3: Durable lifecycle and tenancy + +The durable business identity is tenant-qualified where tenancy is enabled. The package shall validate tenant scope before persistence/provider effects that rely on it, bind scope through parameterized transaction-local PostgreSQL context, and keep forced row-level security enabled for application roles. PostgreSQL superuser/BYPASSRLS and arbitrary SQL access are administrative escape hatches and are not part of the tenant isolation guarantee. + +### FR-4: Reconciliation and recovery + +Reconciliation shall be finite, deterministic, payload-free in its operational evidence, and use the same validated provider client boundary as normal operations. Candidate discovery, concurrency control, scheduling, and result-application semantics must be explicit capabilities rather than inferred from polling code. Provider-success/database-failure cases must remain observable recovery states rather than being rewritten as if the provider effect never occurred. + +### FR-5: Persistence integrity + +Package-owned database rows shall use descriptive two-or-more-word `snake_case` object names where applicable, explicit durable identities, parameterized SQL, and migrations that are idempotent or have a documented one-way boundary. Schema copies maintained for package and Docker initialization shall remain synchronized where the repository contract requires it. Malformed durable payload/state shall fail closed before downstream credential or provider effects when correctness depends on that state. + +### FR-6: Configuration and secrets + +PostgreSQL-backed configuration and encrypted-secret support remain available for standalone use. Environment variables are bootstrap transport only where explicitly documented. Embedding hosts may inject credential providers without changing provider protocol semantics. The package shall not invent an external secret-management product dependency. + +### FR-7: Observability and diagnostics + +Operational telemetry is opt-in. Bounded operation/outcome vocabularies may be emitted; prompts, provider response bodies, credentials, arbitrary endpoint aliases, resource IDs, and arbitrary exception text are not telemetry attributes. Public readiness and ordinary errors shall use bounded categories when lower-layer text could contain sensitive data. + +### FR-8: Standalone and modular deployment + +The package shall remain usable without `contextual-orchestrator`, `naruon`, or another CWL repository. Host services may provide authentication, tenant routing, secret resolution, gateway/model routing, OpenTelemetry export, or scheduling, but those integrations shall not become hidden standalone requirements. + +## Non-functional requirements + +### Quality + +Owned production Python must maintain exact 100% statement and branch coverage and complete public docstrings under the repository's configured gates. Supported validation includes Python 3.10, 3.12, and 3.14 plus realistic PostgreSQL/container integration where behavior depends on PostgreSQL. + +### Security and privacy + +Security-sensitive validation fails closed. Secrets, DSNs, prompt/provider content, unvalidated identifiers, and arbitrary lower-layer exception text must not be retained in logs or review evidence merely for debugging. Repository controls should support SOC 2 / CSAP evidence preparation without claiming certification. + +### Reliability + +Network, response, retry, wait, candidate-scan, payload, and release-evidence operations must be explicitly bounded. Recovery and rollback must be documented before migrations or release changes are considered complete. Queued, skipped, cancelled, absent, stale, predecessor-head, synthetic-merge-only, or infrastructure-failed evidence is not success for an exact source head. + +### Interoperability + +Provider interaction stays OpenAI-Batch-compatible behind a validated Python client seam. Embedding hosts can provide credentials and control-plane context without changing the package's durable/provider semantics. + +### Packaging and release + +Dependencies must be locked/reproducible according to repository policy. Release acceptance must include required quality, security, package/container, SBOM, provenance, artifact-identity, rollback/recovery, and governance evidence on the exact integrated protected head. A release is not implied by a version string or a successful pull request. + +## Explicit non-goals + +- Replacing host authentication or authorization. +- Treating PostgreSQL RLS as a credential or as SQL-injection prevention. +- Making provider payload/model output an authority for tenant or endpoint selection. +- Providing an unbounded general-purpose HTTP proxy. +- Reintroducing provider networking or independent scheduling inside PostgreSQL. +- Claiming distributed exactly-once processing without an integrated transaction/recovery contract spanning every external effect. +- Requiring a specific ContextualWisdomLab host service for standalone operation. +- Claiming SOC 2, CSAP, or other certification solely from repository controls. + +## Product acceptance boundary + +A product capability moves to **IMPLEMENTED-ON-PROTECTED-MAIN** only after its unchanged source has satisfied the live ruleset, required exact-head CI/security/coverage/package/provenance/release checks, valid review findings are resolved, and the resulting tree is integrated into the protected default branch. Documentation must then be updated to move the capability out of `ACTIVE-PR`, `PARTIAL`, or `PLANNED`; historical evidence must not be transferred as if it were proof for a different head. \ No newline at end of file From 25d7eacf4c40e3390c5f6223e223daed60e1ec25 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 14 Aug 2026 20:12:38 +0900 Subject: [PATCH 02/29] docs: establish protected-main technical requirements --- docs/product/TRD.md | 148 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 docs/product/TRD.md diff --git a/docs/product/TRD.md b/docs/product/TRD.md new file mode 100644 index 000000000..4a4249ab0 --- /dev/null +++ b/docs/product/TRD.md @@ -0,0 +1,148 @@ +# Technical Requirements Document + +## Document authority + +This TRD defines technical invariants for `pg-llm-batch`. It distinguishes protected-main behavior from work that is only present in an active pull request. The repository's live code, schema, tests, ruleset, and exact-head evidence remain stronger authority than historical branches, stale PR prose, predecessor checks, or generated merge commits. + +Status vocabulary is shared with the PRD: **IMPLEMENTED-ON-PROTECTED-MAIN**, **ACTIVE-PR**, **PARTIAL**, **PLANNED**, and **SUPERSEDED**. + +## System boundary + +`pg-llm-batch` is a Python package plus PostgreSQL schema and container assets. It may run standalone or be embedded by another service. The package owns batch preparation, package persistence, validated provider Batch API access, durable lifecycle projection, resumable checkpoint storage, and bounded reconciliation primitives. It does not own host authentication, business authorization, ingress/WAF, infrastructure TLS policy, external secret-manager choice, global OpenTelemetry configuration, or a cross-system distributed transaction. + +## Component contract + +| Component | Protected-main responsibility | Prohibited authority | +| --- | --- | --- | +| `token_counter.py` | resolve reviewed tokenizer metadata and call `pg_tiktoken` for token accounting | provider credentials, tenant authorization, provider I/O | +| `orchestrator.py` | select queued requests, partition under limits, persist package payload/file/line/request assignments | provider protocol and retry policy | +| `batch_api_client.py` | validate provider destination and remote identifiers; upload/create/poll/wait/cancel/retrieve under finite budgets | tenant authentication, scheduler ownership | +| `durable_client.py` | compose provider operations with durable lifecycle ordering/persistence | authenticating or authorizing host tenants | +| `db.py` | schema application and parameterized persistence/read helpers | database-side provider networking | +| `checkpoint_store.py` | tenant-qualified PostgreSQL checkpoint/CAS operations and conflict semantics | distributed exactly-once claims | +| `reconciliation.py` | finite host-selected polling/retrieval pass using the existing validated client surface | candidate discovery, scheduling, cross-process leasing | +| `config.py` | PostgreSQL-backed configuration and encrypted secret storage for standalone composition | prescribing an embedding host's external secret manager | +| `observability.py` | opt-in bounded traces/metrics around reviewed operations | configuring global SDK/exporter/resource policy | +| `health.py` | readiness aggregation and redacted public health response | general-purpose web serving or arbitrary diagnostic reflection | +| `cli.py` | standalone operator composition and bounded input surfaces | higher-level workflow orchestration | + +## Runtime architecture + +### Batch preparation + +1. Resolve a supported package batch identity. +2. Read eligible unassigned requests from PostgreSQL. +3. Count model tokens through the reviewed PostgreSQL tokenizer boundary. +4. Partition requests in memory under explicit batch token, byte, record, and provider limits. +5. Under one package preparation transaction, persist virtual payloads, batch-file rows, JSONL line rows, request assignments, and aggregate totals in deterministic order. +6. On failure before commit, roll back the package preparation transaction rather than exposing a partially committed preparation from that invocation. + +Package-generated provider payloads are represented by `memory://` references and are reconstructed from PostgreSQL rather than written to a package-owned local payload file. + +### Provider I/O + +Provider gateway URLs and endpoint aliases are configuration inputs but are untrusted until validated by the applicable package boundary. Production gateway destinations require HTTPS; only explicitly reviewed loopback development HTTP destinations are accepted. Userinfo, query, fragment, whitespace, malformed port, or other forbidden URL forms must fail before credentials are used. + +Control-plane responses are consumed through a finite decoded-byte budget and strict UTF-8/JSON parsing. Provider output/error files are streamed in finite chunks with an independent decoded-byte ceiling before JSONL parsing. The client must fail closed when an adapter cannot provide the bounded streaming interface required by the operation. + +Automatic provider retry is limited to reviewed idempotent GET operations and the exact default status set `{408, 425, 429, 502, 503, 504}`. TLS handshake, certificate, and fingerprint failures are not automatically retried. Upload/create/cancel POST operations remain single-attempt unless a separately reviewed provider-specific contract is introduced. + +### Durable lifecycle tenancy + +Standalone lifecycle data uses the exact `standalone` tenant scope. Tenant-aware hosts use a trusted host-selected scope after authentication and authorization. Provider metadata, request payloads, model output, transport headers, endpoint aliases, and provider resource identifiers never choose tenant authority. + +The durable lifecycle identity is: + +```text +(tenant_scope, endpoint_alias, remote_batch_id) +``` + +Package reads/writes bind tenant scope with parameterized transaction-local `set_config`. The schema enables and forces PostgreSQL row-level security for tenant-qualified lifecycle state. Production application roles must be `NOSUPERUSER NOBYPASSRLS`. A role that can execute arbitrary SQL can choose arbitrary custom setting values; therefore generic arbitrary SQL access is explicitly outside the package isolation guarantee. + +### Durable result checkpoints + +Protected main contains tenant-qualified durable result checkpoint storage with compare-and-swap/conflict semantics. The checkpoint store can participate in a caller-owned PostgreSQL transaction where the caller's durable result application is in the same database transaction. The mere existence of this store does not prove exactly-once application across provider/network/database boundaries. + +Checkpoint counters, offsets, and identities must validate before mutation. Conflicting writes fail explicitly rather than silently overwriting a newer checkpoint. Rollback/recovery tests and schema parity are part of the storage contract. + +### Reconciliation + +Protected main supplies a scheduler-independent `reconcile_batch_candidates(...)` primitive. A host supplies candidate identities and a finite `max_jobs` budget. The primitive validates candidates, bounds scanning/work, polls through the validated provider client, retrieves completed jobs through that same client, and returns payload-free finite outcome/error categories. + +The host still owns candidate discovery, tenant authorization, scheduling, and cross-process concurrency. Durable candidate discovery and tenant-qualified advisory single-flight are **ACTIVE-PR** surfaces; neither is treated as shipped until protected-main integration. Package-owned autonomous scheduling, crash/restart completion semantics, terminal-work retirement after durable result application, and an end-to-end exactly-once worker remain **PARTIAL** or **PLANNED** capabilities. + +## Persistence requirements + +### Naming and schema ownership + +New package-owned database objects use descriptive two-or-more-word `snake_case` names where applicable. SQL is parameterized; identifiers are not constructed from unvalidated user/provider text. Packaged schema and Docker initialization copies that represent the same contract must remain synchronized by regression tests. + +### Migration behavior + +Migrations must have an explicit compatibility and rollback boundary. Tenant migrations preserve legacy data under `standalone`, avoid a committed intermediate RLS-bypass state, restore forced RLS atomically, and remain idempotent where documented. Existing-volume retirement of legacy `http` / `pg_cron` authority is **ACTIVE-PR** and may not be described as shipped merely because fresh initialization no longer depends on SQL-side provider networking. + +### Data integrity + +Package-owned persisted virtual JSONL is canonical state, not a best-effort cache. Malformed shape, line count, framing, duplicate JSON members, non-finite numeric forms, or invalid record type fail closed through bounded package errors. Local payload integrity validation occurs before provider credentials/provider I/O where the provider effect relies on that payload. + +Durable provider/resource identifiers and tenant identities are validated before they become persistence or authorization inputs. Database/query failures must not be silently reclassified as an authoritative no-row result when correctness depends on distinguishing those cases. + +## Security and privacy requirements + +### Secrets + +Standalone provider configuration and encrypted secrets are PostgreSQL-backed. Environment variables are limited to explicitly documented bootstrap transport such as the database DSN and optional encryption key. CLI secret entry uses no-echo prompting or bounded standard input rather than plaintext process arguments. Embedding hosts may supply another credential provider through the supported seam. + +### Diagnostic confidentiality + +Errors, logs, telemetry, check evidence, and public readiness must avoid DSNs, credentials, prompts, provider bodies, arbitrary SQL/provider exception text, unvalidated identifiers, and dynamic exception-class names where those values are not required for operation. Failure categories intended for public/operational evidence use bounded vocabularies. + +### Tenant boundary + +RLS augments a trusted host authorization boundary; it is not itself authentication, a credential, or SQL-injection prevention. Administrative database identities are outside the ordinary tenant guarantee. Pooling code must not leak transaction-local tenant context between logical operations. + +### Provider boundary + +Provider URLs, statuses, IDs, headers, JSON, JSONL, retry guidance, and metadata are untrusted external input. Validation occurs before the downstream effect that relies on that value. Model/provider output never grants tenant, endpoint, credential, or filesystem authority. + +## Observability requirements + +OpenTelemetry support is opt-in. Base installations must not require OpenTelemetry packages solely to use normal batch functionality. Operation names, outcomes, and error categories are finite. Telemetry attributes exclude endpoint aliases, provider URLs, resource identifiers, credentials, metadata, prompts, and provider response bodies. + +A first-class packaging extra for OpenTelemetry is **ACTIVE-PR** until its exact generated dependency lock is committed normally, temporary materialization machinery is removed, and final package/install/release gates succeed. + +## Health and operability + +Readiness covers the required PostgreSQL/tokenizer/configuration boundary and returns redacted public evidence. A failing dependency must not cause arbitrary lower-layer text to be reflected to a caller. Docker/container health and package health semantics must remain aligned. + +Operational migrations require backup/preflight/acceptance/recovery documentation before release. A recovery instruction must preserve package and operator-owned state; destructive `CASCADE`, hidden history rewrite, or deletion of unknown operator objects is not an acceptable shortcut. + +## Concurrency requirements + +Batch preparation uses database coordination/transactionality appropriate to its package-owned state. Durable lifecycle writes use explicit ordering/conflict semantics. Any cross-process reconciliation exclusion must be tenant-qualified, non-blocking or finitely bounded, exception-safe, and clear about whether it is transient session state or durable lease state. Session advisory locking must never be promoted into a durable lease or distributed exactly-once claim. + +## Testing requirements + +Every source defect follows realistic RED → narrow fix → GREEN → focused/full validation. Required repository evidence includes: + +- Python 3.10, 3.12, and 3.14 where configured; +- exact 100% owned production statement and branch coverage; +- public docstring coverage; +- lint/static checks; +- realistic PostgreSQL integration for SQL/RLS/migration/concurrency behavior; +- package and container installation/health validation; +- migration idempotency and rollback/recovery coverage where applicable; +- confidentiality regressions that inspect full exception/traceback surfaces when relevant; +- security scanning and SAST; +- dependency-lock and packaging reproducibility; +- release acceptance, artifact identity, SBOM, and provenance evidence required by the live repository contract. + +Queued, pending, skipped, cancelled, absent, neutral, stale, predecessor-head, synthetic-merge-only, status-only, infrastructure-failed, or rate-limited evidence is not exact-head success. + +## Release requirements + +A release may originate only from the exact integrated protected head after all live required quality, security, review, migration, rollback/recovery, operational, packaging, provenance, and release-acceptance gates are terminal-success. Versioning and CHANGELOG updates are followed by publication and artifact verification. Repository control evidence may support SOC 2 / CSAP readiness but must not be represented as certification. + +## Documentation requirements + +Canonical product/technical/architecture/ADR/UML/ERD/security/operability/release/data-governance/traceability documents must distinguish shipped protected-main state from active or planned work. Durable documents should describe contracts rather than transient check-run IDs. When an active capability merges, the canonical graph is updated in the same governance model rather than relying on stale PR body claims. \ No newline at end of file From 6f8e6c951b81694bd2bec8048897489943c5ae53 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 14 Aug 2026 20:38:13 +0900 Subject: [PATCH 03/29] docs: add canonical documentation fitness inventory --- docs/DOCUMENTATION_FITNESS.md | 48 +++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 docs/DOCUMENTATION_FITNESS.md diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md new file mode 100644 index 000000000..c7306649d --- /dev/null +++ b/docs/DOCUMENTATION_FITNESS.md @@ -0,0 +1,48 @@ +# Documentation Fitness + +## Authority and status model + +This inventory evaluates the documentation graph against the exact protected-main tree `d0a4b30be1f46536e352443309f3a35533156767`. It does not promote pull-request content into shipped truth. The canonical status vocabulary is shared with `docs/product/PRD.md` and `docs/product/TRD.md`: **IMPLEMENTED-ON-PROTECTED-MAIN**, **ACTIVE-PR**, **PARTIAL**, **PLANNED**, and **SUPERSEDED**. + +A document is *fit* only when it states the correct authority boundary, is consistent with the protected-main code/schema/tests it describes, avoids transferring predecessor or active-branch evidence into shipped claims, and gives an operator or reviewer enough information to use, verify, recover, or reject the relevant behavior safely. + +## Current fitness matrix + +| Documentation surface | Status | Fitness assessment | Required next action | +| --- | --- | --- | --- | +| `README.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Useful public entry point, but it is not the sole product/architecture authority and must not absorb transient PR/check state. | Keep beginner-readable; update bounded shipped capability changes after merge. | +| `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, interoperability, and verification, but does not by itself form a complete system/UML/data model. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file. | +| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states. | Validate exact-head docs gates and integrate only through normal PR governance. | +| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries and release/testing invariants are explicit. | Validate exact-head docs gates and integrate only through normal PR governance. | +| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist, but there is no canonical ADR index describing numbering gaps, status, supersession, and protected-main applicability. | Add an ADR index on this canonical branch after checking active ADR writers. | +| `docs/result-streaming.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Describes bounded provider result streaming/checkpoint behavior; checkpoint persistence has a separate accepted ADR. | Update only when the integrated result-application contract changes the operator-facing boundary. | +| `docs/remote-batch-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Durable lifecycle/tenant behavior has detailed protected-main documentation. | Keep synchronized with tenant/RLS and lifecycle migrations. | +| Topic-specific doctoring/reference docs | IMPLEMENTED-ON-PROTECTED-MAIN | Useful deep evidence exists, but doctoring material is supporting evidence, not the canonical product-status authority. | Keep primary-reference and APA-style citations current where material. | +| Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation, but it is not shipped and its exact-head acceptance remains separate. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. | +| OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. | +| Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. | +| Atomic durable result application | ACTIVE-PR | Draft #194 is RED-only at its initial head and is not product truth. Protected main still has a PARTIAL end-to-end result-application capability. | Keep PARTIAL until test-first implementation, exact-head gates, review, and merge complete. | +| Runtime config/schema provisioning separation | ACTIVE-PR / BLOCKED | Draft #193 is RED-only and currently frozen because a source-affecting no-PR config branch remains live writer evidence. | Do not describe the least-privilege runtime-store repair as shipped or proceed through competing source ownership. | +| Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs. | +| Threat model | PLANNED | No canonical protected-main `docs/THREAT_MODEL.md` is present. Security rules are distributed across AGENTS, architecture, ADRs, tests, and issue/PR evidence. | Add a threat model that distinguishes assets, trust boundaries, attacker capabilities, mitigations, residual risk, and non-guarantees without certification claims. | +| Data governance | PLANNED | No single canonical data-governance document currently maps data classes, retention/ownership, tenant authority, privacy exposure, backup/restore, and deletion limits. | Add a protected-main-grounded data-governance contract. | +| UML/component/sequence views | PLANNED | Root architecture is prose-first; no canonical UML authority is present. | Add textual Mermaid/PlantUML-compatible component and critical sequence diagrams after active source contracts stabilize. | +| ERD / schema model | PLANNED | SQL and schema tests are authoritative, but no canonical ERD summarizes package-owned identities, tenancy, checkpoint state, and relationships. | Generate/maintain an ERD from protected-main schema; label migration-only/active-PR objects separately. | +| Standalone operator guide | PARTIAL | Operational instructions exist across README and topic docs, while #184 is adding migration-specific operability. There is no protected-main general `docs/OPERABILITY.md`. | Establish a general operator authority without racing #184's bounded migration changes. | +| Release governance | PARTIAL | Release-evidence ADRs/tests/workflows are strong, but the canonical graph lacks one concise protected-main release/operator authority tying versioning, SBOM, provenance, artifact identity, rollback/recovery, and publication verification together. | Add a stable release contract after checking active release writers; do not copy workflow-run IDs. | +| Licensing / third-party notices | PARTIAL | Apache-2.0 source headers and repository licensing exist, but acquisition diligence should explicitly trace package license, dependency/SBOM evidence, and third-party notice process. | Evaluate a concise licensing/compliance evidence index without claiming legal certification. | + +## Non-negotiable documentation invariants + +- Protected-main behavior is the shipped authority. Active PRs and historical branches are overlays, not proof of implementation. +- Exact contributor heads, generated merge commits, check IDs, and transient queue state belong in PR/review evidence, not durable architecture/product documents. +- Standalone operation and modular MSA embedding must be documented as co-equal supported boundaries; no ContextualWisdomLab host is a hidden runtime requirement. +- Tenant scope is selected only by a trusted authenticated/authorized host boundary. PostgreSQL RLS is defense in depth, not authentication or SQL-injection prevention. +- Provider/model content never becomes tenant, credential, endpoint, filesystem, or database authority. +- Distributed exactly-once processing is never implied by PostgreSQL checkpoint atomicity alone. +- Security, privacy, SOC 2, and CSAP material is evidence-readiness documentation only unless an external certification actually exists. +- Database object names, migration/rollback behavior, exact owned coverage requirements, Python-version acceptance, packaging/SBOM/provenance expectations, and bounded diagnostics must remain synchronized with repository tests and live governance. + +## Fitness gate for future canonical changes + +Before changing a canonical surface, refetch protected main, open PRs, non-default branches, the exact affected source/schema/test authorities, and any adjacent documentation writer. A documentation repair must not race a source-affecting writer whose behavior is still unsettled. After a product PR merges, update the canonical status classification from `ACTIVE-PR` or `PARTIAL` only when the protected-main tree contains the capability and the integrated evidence contract remains valid. \ No newline at end of file From 7a17e5fd61a333e7b3c18aa4602c743157952f37 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 14 Aug 2026 20:38:57 +0900 Subject: [PATCH 04/29] docs: add protected-main traceability map --- docs/TRACEABILITY.md | 71 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 docs/TRACEABILITY.md diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md new file mode 100644 index 000000000..f8e5f3ea4 --- /dev/null +++ b/docs/TRACEABILITY.md @@ -0,0 +1,71 @@ +# Requirements Traceability + +## Purpose and authority + +This map ties the canonical PRD/TRD requirements to stable protected-main implementation, test, ADR, and operator evidence. It intentionally avoids workflow-run IDs, predecessor SHAs, generated merge commits, and review comments because those are transient verification records rather than durable requirements authority. + +The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. Rows marked **ACTIVE-PR** or **PARTIAL** are not shipped implementation claims. + +## Product-to-technical traceability + +| Requirement | Status | Primary protected-main implementation authority | Durable verification/documentation authority | Known gap or active overlay | +| --- | --- | --- | --- | --- | +| FR-1 deterministic bounded batch preparation | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/orchestrator.py`, `pg_llm_batch/token_counter.py`, package schema | preparation/token tests; `docs/idempotent-preparation.md`; `docs/schema-integrity.md` | No gap claimed by the canonical product contract. | +| FR-2 validated bounded provider interaction | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/batch_api_client.py` | provider URL/resource/retry/response-budget tests; `docs/batch-endpoints.md`; `docs/resource-identifiers.md`; ADR 0015 | Provider-specific widening requires a separately reviewed contract. | +| FR-3 standalone + tenant-qualified durable lifecycle | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/durable_client.py`, `pg_llm_batch/db.py`, schema/RLS objects | tenant lifecycle/integration tests; `ARCHITECTURE.md`; `docs/remote-batch-lifecycle.md`; ADR 0002 | Arbitrary SQL, superuser, and BYPASSRLS remain outside the isolation guarantee. | +| FR-4 scheduler-independent bounded reconciliation | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/reconciliation.py` | reconciliation tests and protected-main release gates | Discovery #190 and single-flight #191 remain ACTIVE-PR; autonomous worker semantics remain PARTIAL. | +| FR-4 durable reconciliation candidate discovery | ACTIVE-PR | none on protected main beyond existing lifecycle/read primitives | PR #190 exact-head evidence only | Not shipped; current review/infrastructure gates must not be transferred. | +| FR-4 tenant-qualified cross-process single-flight | ACTIVE-PR | none on protected main beyond existing DB primitives | PR #191 exact-head evidence only | Not shipped; qualifying independent approval remains a live-governance concern. | +| FR-4 durable result application + checkpoint coupling | PARTIAL | `pg_llm_batch/result_streaming.py`, `pg_llm_batch/checkpoint_store.py` provide streaming/checkpoint primitives | ADR 0006; ADR 0007; checkpoint/result-streaming tests; `docs/result-streaming.md` | Draft #194 is test-first ACTIVE-PR work; protected main does not claim end-to-end exactly-once application. | +| FR-5 package persistence integrity | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/db.py`, `pg_llm_batch/schema.sql`, Docker schema mirror | schema-integrity, payload, lifecycle, checkpoint migration tests | Existing-volume legacy-extension retirement is separately ACTIVE-PR #184. | +| FR-5 legacy `http` / `pg_cron` authority retirement on existing volumes | ACTIVE-PR | protected main does not yet contain the retirement migration contract | PR #184 migration/smoke/operator evidence only | Must remain active until unchanged exact head satisfies migration/security/release/review gates and merges. | +| FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Least-privilege separation of runtime store construction from schema provisioning is ACTIVE-PR #193 and currently writer-blocked. | +| FR-7 bounded diagnostics/readiness | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/health.py`, bounded error surfaces | health/confidentiality tests and architecture requirements | Broader `ValidationError` value-confidentiality work must not be inferred as solved from readiness alone. | +| FR-7 opt-in OpenTelemetry | IMPLEMENTED-ON-PROTECTED-MAIN / packaging PARTIAL | `pg_llm_batch/observability.py` | observability tests | First-class locked installation extra remains ACTIVE-PR #175. | +| FR-8 standalone + modular MSA deployment | IMPLEMENTED-ON-PROTECTED-MAIN | package/CLI/container composition; injectable host seams | `README.md`, `ARCHITECTURE.md`, package/container tests | CWL host repositories are optional integrations, not package runtime dependencies. | +| Exact owned 100% statement/branch coverage | IMPLEMENTED-ON-PROTECTED-MAIN governance contract | repository CI configuration and owned production code | coverage gate/tests | Must be re-proven on every changed exact head; predecessor evidence never transfers. | +| Python 3.10/3.12/3.14 validation | IMPLEMENTED-ON-PROTECTED-MAIN governance contract | package metadata/workflow matrix | exact-head repository CI | Requires exact-head terminal success; queued/skipped/infrastructure-failed jobs are not proof. | +| Reproducible release evidence | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/release_evidence.py` and release workflows | ADR 0003; ADR 0004; release-evidence/artifact-identity tests | Release publication itself still occurs only from a fully accepted integrated protected head. | +| SBOM/provenance/artifact identity | IMPLEMENTED-ON-PROTECTED-MAIN governance contract | release workflows/evidence helpers | release acceptance and artifact verification tests | No certification claim follows from repository evidence alone. | +| SOC 2 / CSAP evidence readiness | PARTIAL | security, tenancy, logging, release and governance controls | PRD/TRD/security tests/ADRs | Evidence readiness only; no external certification is claimed. | + +## Security and privacy traceability + +| Control objective | Protected-main authority | Verification evidence | Residual boundary | +| --- | --- | --- | --- | +| Trusted tenant selection | `AGENTS.md`, `ARCHITECTURE.md`, tenant validation in DB/durable-client paths | tenant-scope/RLS tests | Host authentication/authorization remains external. | +| RLS defense in depth | tenant-qualified schema and transaction-local scope binding | live PostgreSQL isolation/migration tests | Superuser/BYPASSRLS/arbitrary SQL are administrative bypasses. | +| Provider destination validation | `batch_api_client.py` | endpoint/URL tests | Host/network infrastructure TLS policy is external. | +| Bounded provider input | provider client + result streaming | response/download/JSONL/resource-budget tests | Provider authenticity is not established by payload validation. | +| Secret/config boundary | `config.py`, bootstrap contract | config/secret/bootstrap tests | External enterprise secret manager choice remains host-owned. | +| Diagnostic confidentiality | health/error/logging contracts | traceback/health/redaction tests | Generic validation value confidentiality remains an explicit separate gap where applicable. | +| Checkpoint concurrency/integrity | `checkpoint_store.py` | CAS/concurrency/RLS/rollback tests | PostgreSQL atomicity does not extend to external systems. | +| Release artifact integrity | `release_evidence.py` + release contracts | descriptor/dirfd/reproducibility tests | Publication credentials and external registry availability are operational dependencies. | + +## Data and persistence traceability + +| Data family | Durable identity / authority | Principal protected-main documents | Recovery / non-guarantee | +| --- | --- | --- | --- | +| Tenant lifecycle state | `(tenant_scope, endpoint_alias, remote_batch_id)` | `ARCHITECTURE.md`, ADR 0002, `docs/remote-batch-lifecycle.md` | Tenant scope must come from trusted host authorization; direct SQL bypass is out of scope. | +| Result checkpoints | `(tenant_scope, checkpoint_consumer_name, endpoint_alias, remote_batch_id)` | ADR 0006, ADR 0007, `docs/result-streaming.md` | Prefix checkpoint is not provider authentication or whole-stream immutability; cross-system exactly-once is not claimed. | +| Package JSONL/payload state | package-owned schema identities and virtual payload references | PRD/TRD, schema-integrity and payload docs/tests | Persisted package data is canonical, not a disposable cache. | +| Configuration/secrets | `com_config`, `com_secrets` | PRD/TRD and config tests | Runtime/provisioning least-privilege separation is not yet shipped. | +| Release evidence | descriptor/artifact identity contracts | ADR 0003, ADR 0004 | Evidence proves the reviewed artifact path, not organizational certification. | + +## Active overlay register + +The following open pull requests are intentionally represented only as overlays on this traceability map: + +- **#175** — OpenTelemetry packaging extra; dependency-lock/materialization and final package evidence remain outside protected-main truth. +- **#184** — existing-volume legacy PostgreSQL extension retirement; migration/operator behavior remains active until merged. +- **#190** — durable reconciliation candidate discovery; not protected-main truth and currently subject to live review/tooling evidence. +- **#191** — tenant-qualified reconciliation single-flight; not protected-main truth until live approval/gates and merge. +- **#192** — this canonical PRD/TRD/fitness/traceability reconstruction itself. +- **#193** — runtime-store/schema-provisioning separation; RED-only and writer-blocked in the current inventory. +- **#194** — atomic local result-effect/checkpoint application; RED-only at its initial test head and not shipped. + +This register is descriptive, not a substitute for refetching GitHub. Before changing a status, verify the PR still exists, its exact contributor head, live protected-main base, ancestry, current reviews/threads, exact-head gates, and resulting protected-main integration. + +## Change-control rule + +When a capability merges, update the PRD/TRD status and this traceability map in a canonical documentation change only after the new protected-main tree is read. When a capability is abandoned or superseded, mark the overlay accordingly rather than silently deleting its historical decision context. New requirements must identify at least one intended implementation authority and one deterministic verification authority before they can be called acquisition-ready. \ No newline at end of file From 1cbc9ea0ffb1eb1809be8a284b210d5eba2efb12 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 14 Aug 2026 20:51:47 +0900 Subject: [PATCH 05/29] docs: index protected-main architecture decisions --- docs/adr/README.md | 49 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 docs/adr/README.md diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 000000000..d169ab9a0 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,49 @@ +# Architecture Decision Record Index + +## Authority + +This index is a navigation and status aid for architecture decision records that exist on protected `main`. The decision record itself remains the normative source for its context, decision, consequences, security boundary, and supersession rules. + +The protected-main tree used to reconstruct this index is `d0a4b30be1f46536e352443309f3a35533156767`. A pull request, historical branch, generated merge commit, workflow run, review comment, or proposed file that is not on protected main is not silently promoted into this index. + +## Protected-main decisions + +| ADR | Decision | Document status | Protected-main applicability | +| --- | --- | --- | --- | +| [0002](0002-tenant-scoped-lifecycle.md) | Tenant-scoped durable lifecycle state | Accepted | Defines trusted host-selected `tenant_scope`, tenant-qualified durable lifecycle identity, transaction-local RLS binding, and the standalone compatibility scope. | +| [0003](0003-reproducible-release-evidence.md) | Reproducible release evidence before publication | Proposed | Documents the reproducibility/evidence design present in the repository. The ADR's own `Proposed` status is retained and must not be rewritten as architectural acceptance merely because related release-evidence code or workflows exist. | +| [0004](0004-descriptor-pinned-release-artifact-verification.md) | Descriptor-pinned release artifact verification | Proposed | Documents the descriptor-pinned TOCTOU-hardening design. Its `Proposed` decision status remains authoritative until explicitly changed through reviewed documentation governance. | +| [0006](0006-resumable-result-checkpoints.md) | Resumable provider-result checkpoints | Accepted | Defines immutable prefix checkpoint evidence and its explicit prefix-only, non-authentication, non-whole-stream assurance boundary. | +| [0007](0007-durable-result-checkpoint-store.md) | Durable tenant-isolated result checkpoint store | Accepted | Adds optional PostgreSQL persistence, tenant isolation, compare-and-swap concurrency, and a caller-owned transaction seam without claiming distributed exactly-once delivery. Depends on ADR 0006. | +| [0015](0015-http-425-too-early-retry.md) | HTTP 425 retry for bounded idempotent GETs | Accepted for the bounded retry slice | Keeps the default GET retry-status set closed, side-effecting POSTs single-attempt, and TLS/certificate/fingerprint failures outside automatic retry. | + +## Numbering and missing identifiers + +ADR numbers are stable identifiers, not a promise of contiguous numbering. The absence of `0001`, `0005`, or `0008` through `0014` from the protected-main directory does **not** mean those decisions are missing, rejected, accepted elsewhere, or safe to reconstruct from old branches. Gaps may reflect historical work, superseded proposals, unmerged branches, or reserved identifiers. Only a reviewed protected-main document may establish the status of a missing number. + +New ADRs should use the next repository-approved stable identifier rather than renumbering existing records. Renaming an integrated ADR changes external references and should be treated as an architecture-governance migration, not cleanup. + +## Decision status versus implementation status + +ADR status answers whether an architectural decision has been accepted, proposed, deprecated, or superseded. It is not interchangeable with product implementation status. + +The canonical product/technical documents use `IMPLEMENTED-ON-PROTECTED-MAIN`, `ACTIVE-PR`, `PARTIAL`, `PLANNED`, and `SUPERSEDED` for implementation truth. A `Proposed` ADR can coexist with related code on protected main, and an `Accepted` ADR can describe a bounded contract whose larger product capability is still only partial. Do not infer one status system from the other. + +When an ADR and protected-main implementation appear inconsistent, treat the inconsistency as a defect to reconcile explicitly. Do not silently edit the index to make the conflict disappear. + +## Supersession and amendments + +A decision is superseded only when a reviewed record or amendment says so explicitly. A later implementation, issue, pull request, or historical branch does not implicitly supersede an ADR. + +For a material architecture change: + +1. refetch protected main and all source/documentation writers touching the decision boundary; +2. identify the existing ADRs and product/technical requirements that constrain the change; +3. record the new decision or explicit amendment, including rejected alternatives and operational/security consequences where material; +4. keep active/unmerged behavior classified as an overlay rather than protected-main truth; +5. update traceability after the capability actually reaches protected main; and +6. preserve standalone operation and modular MSA embedding unless a separately accepted decision changes that product contract. + +## Non-guarantees that must remain visible + +The current ADR set does not establish authentication from PostgreSQL RLS alone, provider authenticity from result checkpointing, full-stream immutability from a prefix checkpoint, distributed exactly-once processing, retry permission for arbitrary HTTP failures, or organizational security/compliance certification. Those boundaries must not be weakened by summaries, operator docs, marketing material, or future ADR titles. \ No newline at end of file From 05623306c705d3e79d2127f46c3c423f953d8edb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 14 Aug 2026 20:52:58 +0900 Subject: [PATCH 06/29] docs: record active ADR index reconstruction --- docs/DOCUMENTATION_FITNESS.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index c7306649d..124057e89 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -14,7 +14,8 @@ A document is *fit* only when it states the correct authority boundary, is consi | `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, interoperability, and verification, but does not by itself form a complete system/UML/data model. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file. | | `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states. | Validate exact-head docs gates and integrate only through normal PR governance. | | `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries and release/testing invariants are explicit. | Validate exact-head docs gates and integrate only through normal PR governance. | -| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist, but there is no canonical ADR index describing numbering gaps, status, supersession, and protected-main applicability. | Add an ADR index on this canonical branch after checking active ADR writers. | +| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. | +| `docs/adr/README.md` | ACTIVE-PR | This canonical branch now supplies the missing protected-main ADR navigation/status index, explains numbering gaps, separates ADR decision status from implementation status, and forbids implicit supersession. | Validate exact-head documentation gates and integrate only through normal PR governance. | | `docs/result-streaming.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Describes bounded provider result streaming/checkpoint behavior; checkpoint persistence has a separate accepted ADR. | Update only when the integrated result-application contract changes the operator-facing boundary. | | `docs/remote-batch-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Durable lifecycle/tenant behavior has detailed protected-main documentation. | Keep synchronized with tenant/RLS and lifecycle migrations. | | Topic-specific doctoring/reference docs | IMPLEMENTED-ON-PROTECTED-MAIN | Useful deep evidence exists, but doctoring material is supporting evidence, not the canonical product-status authority. | Keep primary-reference and APA-style citations current where material. | From b708dbbbf7af9be9c3e9add34718949bcc34099b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 14 Aug 2026 21:39:23 +0900 Subject: [PATCH 07/29] docs: refresh atomic result application status --- docs/DOCUMENTATION_FITNESS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index 124057e89..f0094a0e3 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -22,7 +22,7 @@ A document is *fit* only when it states the correct authority boundary, is consi | Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation, but it is not shipped and its exact-head acceptance remains separate. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. | | OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. | | Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. | -| Atomic durable result application | ACTIVE-PR | Draft #194 is RED-only at its initial head and is not product truth. Protected main still has a PARTIAL end-to-end result-application capability. | Keep PARTIAL until test-first implementation, exact-head gates, review, and merge complete. | +| Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay and does not make end-to-end result application or distributed exactly-once behavior protected-main truth. | Keep the protected-main end-to-end capability PARTIAL until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. | | Runtime config/schema provisioning separation | ACTIVE-PR / BLOCKED | Draft #193 is RED-only and currently frozen because a source-affecting no-PR config branch remains live writer evidence. | Do not describe the least-privilege runtime-store repair as shipped or proceed through competing source ownership. | | Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs. | | Threat model | PLANNED | No canonical protected-main `docs/THREAT_MODEL.md` is present. Security rules are distributed across AGENTS, architecture, ADRs, tests, and issue/PR evidence. | Add a threat model that distinguishes assets, trust boundaries, attacker capabilities, mitigations, residual risk, and non-guarantees without certification claims. | @@ -46,4 +46,4 @@ A document is *fit* only when it states the correct authority boundary, is consi ## Fitness gate for future canonical changes -Before changing a canonical surface, refetch protected main, open PRs, non-default branches, the exact affected source/schema/test authorities, and any adjacent documentation writer. A documentation repair must not race a source-affecting writer whose behavior is still unsettled. After a product PR merges, update the canonical status classification from `ACTIVE-PR` or `PARTIAL` only when the protected-main tree contains the capability and the integrated evidence contract remains valid. \ No newline at end of file +Before changing a canonical surface, refetch protected main, open PRs, non-default branches, the exact affected source/schema/test authorities, and any adjacent documentation writer. A documentation repair must not race a source-affecting writer whose behavior is still unsettled. After a product PR merges, update the canonical status classification from `ACTIVE-PR` or `PARTIAL` only when the protected-main tree contains the capability and the integrated evidence contract remains valid. From 65338dfa04ef942864bb0a250503de1344a1b996 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Fri, 14 Aug 2026 21:40:16 +0900 Subject: [PATCH 08/29] docs: align traceability with active result application --- docs/TRACEABILITY.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index f8e5f3ea4..89b33c0e1 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -16,7 +16,7 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | FR-4 scheduler-independent bounded reconciliation | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/reconciliation.py` | reconciliation tests and protected-main release gates | Discovery #190 and single-flight #191 remain ACTIVE-PR; autonomous worker semantics remain PARTIAL. | | FR-4 durable reconciliation candidate discovery | ACTIVE-PR | none on protected main beyond existing lifecycle/read primitives | PR #190 exact-head evidence only | Not shipped; current review/infrastructure gates must not be transferred. | | FR-4 tenant-qualified cross-process single-flight | ACTIVE-PR | none on protected main beyond existing DB primitives | PR #191 exact-head evidence only | Not shipped; qualifying independent approval remains a live-governance concern. | -| FR-4 durable result application + checkpoint coupling | PARTIAL | `pg_llm_batch/result_streaming.py`, `pg_llm_batch/checkpoint_store.py` provide streaming/checkpoint primitives | ADR 0006; ADR 0007; checkpoint/result-streaming tests; `docs/result-streaming.md` | Draft #194 is test-first ACTIVE-PR work; protected main does not claim end-to-end exactly-once application. | +| FR-4 durable result application + checkpoint coupling | PARTIAL | `pg_llm_batch/result_streaming.py`, `pg_llm_batch/checkpoint_store.py` provide streaming/checkpoint primitives | ADR 0006; ADR 0007; checkpoint/result-streaming tests; `docs/result-streaming.md` | PR #194 is ACTIVE-PR test-first work that adds a same-transaction local result-effect/checkpoint seam; protected main still does not claim end-to-end or distributed exactly-once application. | | FR-5 package persistence integrity | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/db.py`, `pg_llm_batch/schema.sql`, Docker schema mirror | schema-integrity, payload, lifecycle, checkpoint migration tests | Existing-volume legacy-extension retirement is separately ACTIVE-PR #184. | | FR-5 legacy `http` / `pg_cron` authority retirement on existing volumes | ACTIVE-PR | protected main does not yet contain the retirement migration contract | PR #184 migration/smoke/operator evidence only | Must remain active until unchanged exact head satisfies migration/security/release/review gates and merges. | | FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Least-privilege separation of runtime store construction from schema provisioning is ACTIVE-PR #193 and currently writer-blocked. | @@ -62,10 +62,10 @@ The following open pull requests are intentionally represented only as overlays - **#191** — tenant-qualified reconciliation single-flight; not protected-main truth until live approval/gates and merge. - **#192** — this canonical PRD/TRD/fitness/traceability reconstruction itself. - **#193** — runtime-store/schema-provisioning separation; RED-only and writer-blocked in the current inventory. -- **#194** — atomic local result-effect/checkpoint application; RED-only at its initial test head and not shipped. +- **#194** — atomic local result-effect/checkpoint application; a current-main-compatible implementation is under review but remains an active overlay and is not shipped. This register is descriptive, not a substitute for refetching GitHub. Before changing a status, verify the PR still exists, its exact contributor head, live protected-main base, ancestry, current reviews/threads, exact-head gates, and resulting protected-main integration. ## Change-control rule -When a capability merges, update the PRD/TRD status and this traceability map in a canonical documentation change only after the new protected-main tree is read. When a capability is abandoned or superseded, mark the overlay accordingly rather than silently deleting its historical decision context. New requirements must identify at least one intended implementation authority and one deterministic verification authority before they can be called acquisition-ready. \ No newline at end of file +When a capability merges, update the PRD/TRD status and this traceability map in a canonical documentation change only after the new protected-main tree is read. When a capability is abandoned or superseded, mark the overlay accordingly rather than silently deleting its historical decision context. New requirements must identify at least one intended implementation authority and one deterministic verification authority before they can be called acquisition-ready. From b6ce4d0107de9c7e867e0532de9268f937d03a0d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 01:21:07 +0900 Subject: [PATCH 09/29] docs: refresh runtime provisioning overlay status --- docs/DOCUMENTATION_FITNESS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index f0094a0e3..22b522a0c 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -23,7 +23,7 @@ A document is *fit* only when it states the correct authority boundary, is consi | OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. | | Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. | | Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay and does not make end-to-end result application or distributed exactly-once behavior protected-main truth. | Keep the protected-main end-to-end capability PARTIAL until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. | -| Runtime config/schema provisioning separation | ACTIVE-PR / BLOCKED | Draft #193 is RED-only and currently frozen because a source-affecting no-PR config branch remains live writer evidence. | Do not describe the least-privilege runtime-store repair as shipped or proceed through competing source ownership. | +| Runtime config/schema provisioning separation | ACTIVE-PR | PR #193 is a current-main least-privilege candidate: runtime constructors move away from DDL/default seeding, validate the search-path-resolved base-table shape and current-role read capability through bounded catalog probes, and retain provisioning/default seeding at the explicit schema boundary. It is not shipped until normal governance integrates it. | Preserve the trusted-search-path/non-ownership caveat and reacquire exact-head checks/review after every push; do not transfer predecessor or earlier-head evidence. | | Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs. | | Threat model | PLANNED | No canonical protected-main `docs/THREAT_MODEL.md` is present. Security rules are distributed across AGENTS, architecture, ADRs, tests, and issue/PR evidence. | Add a threat model that distinguishes assets, trust boundaries, attacker capabilities, mitigations, residual risk, and non-guarantees without certification claims. | | Data governance | PLANNED | No single canonical data-governance document currently maps data classes, retention/ownership, tenant authority, privacy exposure, backup/restore, and deletion limits. | Add a protected-main-grounded data-governance contract. | From 7f1dbb45d3f6bf7bdeeabe9e26954e1096d6eba3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 01:21:58 +0900 Subject: [PATCH 10/29] docs: refresh active runtime-store traceability --- docs/TRACEABILITY.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 89b33c0e1..9e1a81411 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -19,7 +19,7 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | FR-4 durable result application + checkpoint coupling | PARTIAL | `pg_llm_batch/result_streaming.py`, `pg_llm_batch/checkpoint_store.py` provide streaming/checkpoint primitives | ADR 0006; ADR 0007; checkpoint/result-streaming tests; `docs/result-streaming.md` | PR #194 is ACTIVE-PR test-first work that adds a same-transaction local result-effect/checkpoint seam; protected main still does not claim end-to-end or distributed exactly-once application. | | FR-5 package persistence integrity | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/db.py`, `pg_llm_batch/schema.sql`, Docker schema mirror | schema-integrity, payload, lifecycle, checkpoint migration tests | Existing-volume legacy-extension retirement is separately ACTIVE-PR #184. | | FR-5 legacy `http` / `pg_cron` authority retirement on existing volumes | ACTIVE-PR | protected main does not yet contain the retirement migration contract | PR #184 migration/smoke/operator evidence only | Must remain active until unchanged exact head satisfies migration/security/release/review gates and merges. | -| FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Least-privilege separation of runtime store construction from schema provisioning is ACTIVE-PR #193 and currently writer-blocked. | +| FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Least-privilege separation of runtime store construction from schema provisioning is ACTIVE-PR #193. Its current candidate moves DDL/default seeding to explicit provisioning and adds bounded catalog relation/type/read-capability probes, but none of that is shipped until normal governance integrates it. | | FR-7 bounded diagnostics/readiness | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/health.py`, bounded error surfaces | health/confidentiality tests and architecture requirements | Broader `ValidationError` value-confidentiality work must not be inferred as solved from readiness alone. | | FR-7 opt-in OpenTelemetry | IMPLEMENTED-ON-PROTECTED-MAIN / packaging PARTIAL | `pg_llm_batch/observability.py` | observability tests | First-class locked installation extra remains ACTIVE-PR #175. | | FR-8 standalone + modular MSA deployment | IMPLEMENTED-ON-PROTECTED-MAIN | package/CLI/container composition; injectable host seams | `README.md`, `ARCHITECTURE.md`, package/container tests | CWL host repositories are optional integrations, not package runtime dependencies. | @@ -61,7 +61,7 @@ The following open pull requests are intentionally represented only as overlays - **#190** — durable reconciliation candidate discovery; not protected-main truth and currently subject to live review/tooling evidence. - **#191** — tenant-qualified reconciliation single-flight; not protected-main truth until live approval/gates and merge. - **#192** — this canonical PRD/TRD/fitness/traceability reconstruction itself. -- **#193** — runtime-store/schema-provisioning separation; RED-only and writer-blocked in the current inventory. +- **#193** — runtime-store/schema-provisioning separation; a current-main least-privilege implementation is under review, but remains an active overlay rather than shipped behavior. - **#194** — atomic local result-effect/checkpoint application; a current-main-compatible implementation is under review but remains an active overlay and is not shipped. This register is descriptive, not a substitute for refetching GitHub. Before changing a status, verify the PR still exists, its exact contributor head, live protected-main base, ancestry, current reviews/threads, exact-head gates, and resulting protected-main integration. From fecc977ff9c359ab05e31b9af4decfa2cd9d9c3f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 20:10:01 +0900 Subject: [PATCH 11/29] docs: trace secret encryption policy status --- docs/TRACEABILITY.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 9e1a81411..bd24a9133 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -19,7 +19,8 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | FR-4 durable result application + checkpoint coupling | PARTIAL | `pg_llm_batch/result_streaming.py`, `pg_llm_batch/checkpoint_store.py` provide streaming/checkpoint primitives | ADR 0006; ADR 0007; checkpoint/result-streaming tests; `docs/result-streaming.md` | PR #194 is ACTIVE-PR test-first work that adds a same-transaction local result-effect/checkpoint seam; protected main still does not claim end-to-end or distributed exactly-once application. | | FR-5 package persistence integrity | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/db.py`, `pg_llm_batch/schema.sql`, Docker schema mirror | schema-integrity, payload, lifecycle, checkpoint migration tests | Existing-volume legacy-extension retirement is separately ACTIVE-PR #184. | | FR-5 legacy `http` / `pg_cron` authority retirement on existing volumes | ACTIVE-PR | protected main does not yet contain the retirement migration contract | PR #184 migration/smoke/operator evidence only | Must remain active until unchanged exact head satisfies migration/security/release/review gates and merges. | -| FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Least-privilege separation of runtime store construction from schema provisioning is ACTIVE-PR #193. Its current candidate moves DDL/default seeding to explicit provisioning and adds bounded catalog relation/type/read-capability probes, but none of that is shipped until normal governance integrates it. | +| FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Protected main deliberately retains `SecretStore(require_encryption=False)` as the compatibility default; Fernet support is therefore not an encryption-required production claim. PR #193 proposes an encryption-required default plus explicit local/development opt-out while also separating runtime construction from provisioning. | +| FR-6 production secret-at-rest policy lifecycle | PARTIAL | Protected main can enforce Fernet when callers explicitly select `require_encryption=True`, and fails before database access when required encryption lacks usable configuration. | config/secret/bootstrap tests; security issue acceptance criteria | No protected-main contract yet detects and atomically migrates existing `is_encrypted = FALSE` rows, proves bounded key rotation/recovery, or exposes the selected deployment policy through redacted readiness/operator evidence. PR #193 addresses the default-policy boundary only while it remains ACTIVE-PR. | | FR-7 bounded diagnostics/readiness | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/health.py`, bounded error surfaces | health/confidentiality tests and architecture requirements | Broader `ValidationError` value-confidentiality work must not be inferred as solved from readiness alone. | | FR-7 opt-in OpenTelemetry | IMPLEMENTED-ON-PROTECTED-MAIN / packaging PARTIAL | `pg_llm_batch/observability.py` | observability tests | First-class locked installation extra remains ACTIVE-PR #175. | | FR-8 standalone + modular MSA deployment | IMPLEMENTED-ON-PROTECTED-MAIN | package/CLI/container composition; injectable host seams | `README.md`, `ARCHITECTURE.md`, package/container tests | CWL host repositories are optional integrations, not package runtime dependencies. | @@ -37,7 +38,7 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | RLS defense in depth | tenant-qualified schema and transaction-local scope binding | live PostgreSQL isolation/migration tests | Superuser/BYPASSRLS/arbitrary SQL are administrative bypasses. | | Provider destination validation | `batch_api_client.py` | endpoint/URL tests | Host/network infrastructure TLS policy is external. | | Bounded provider input | provider client + result streaming | response/download/JSONL/resource-budget tests | Provider authenticity is not established by payload validation. | -| Secret/config boundary | `config.py`, bootstrap contract | config/secret/bootstrap tests | External enterprise secret manager choice remains host-owned. | +| Secret/config boundary | `config.py`, bootstrap contract | config/secret/bootstrap tests | Protected main supports Fernet but does not require it by default; the compatibility path is not a production confidentiality claim. PR #193's stricter default is ACTIVE-PR, while enterprise secret-manager choice remains host-owned. | | Diagnostic confidentiality | health/error/logging contracts | traceback/health/redaction tests | Generic validation value confidentiality remains an explicit separate gap where applicable. | | Checkpoint concurrency/integrity | `checkpoint_store.py` | CAS/concurrency/RLS/rollback tests | PostgreSQL atomicity does not extend to external systems. | | Release artifact integrity | `release_evidence.py` + release contracts | descriptor/dirfd/reproducibility tests | Publication credentials and external registry availability are operational dependencies. | @@ -49,7 +50,7 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | Tenant lifecycle state | `(tenant_scope, endpoint_alias, remote_batch_id)` | `ARCHITECTURE.md`, ADR 0002, `docs/remote-batch-lifecycle.md` | Tenant scope must come from trusted host authorization; direct SQL bypass is out of scope. | | Result checkpoints | `(tenant_scope, checkpoint_consumer_name, endpoint_alias, remote_batch_id)` | ADR 0006, ADR 0007, `docs/result-streaming.md` | Prefix checkpoint is not provider authentication or whole-stream immutability; cross-system exactly-once is not claimed. | | Package JSONL/payload state | package-owned schema identities and virtual payload references | PRD/TRD, schema-integrity and payload docs/tests | Persisted package data is canonical, not a disposable cache. | -| Configuration/secrets | `com_config`, `com_secrets` | PRD/TRD and config tests | Runtime/provisioning least-privilege separation is not yet shipped. | +| Configuration/secrets | `com_config`, `com_secrets` | PRD/TRD and config tests | Shipped compatibility mode can still persist `is_encrypted = FALSE`; runtime/provisioning least-privilege separation, required-encryption default, legacy-row migration, key rotation/recovery, and redacted policy-readiness evidence are not all protected-main behavior. | | Release evidence | descriptor/artifact identity contracts | ADR 0003, ADR 0004 | Evidence proves the reviewed artifact path, not organizational certification. | ## Active overlay register @@ -61,7 +62,7 @@ The following open pull requests are intentionally represented only as overlays - **#190** — durable reconciliation candidate discovery; not protected-main truth and currently subject to live review/tooling evidence. - **#191** — tenant-qualified reconciliation single-flight; not protected-main truth until live approval/gates and merge. - **#192** — this canonical PRD/TRD/fitness/traceability reconstruction itself. -- **#193** — runtime-store/schema-provisioning separation; a current-main least-privilege implementation is under review, but remains an active overlay rather than shipped behavior. +- **#193** — runtime-store/schema-provisioning separation plus an encryption-required `SecretStore` default with explicit local/development compatibility opt-out; both remain active overlays, and this PR does not by itself establish legacy-row migration, key rotation/recovery, or readiness-policy lifecycle completion. - **#194** — atomic local result-effect/checkpoint application; a current-main-compatible implementation is under review but remains an active overlay and is not shipped. This register is descriptive, not a substitute for refetching GitHub. Before changing a status, verify the PR still exists, its exact contributor head, live protected-main base, ancestry, current reviews/threads, exact-head gates, and resulting protected-main integration. From b2ea32278e1d4c491312117e7629bbc233785f94 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 20:18:50 +0900 Subject: [PATCH 12/29] docs: make lifecycle compatibility contract explicit --- docs/product/PRD.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/product/PRD.md b/docs/product/PRD.md index 905a3906c..f2fd8e282 100644 --- a/docs/product/PRD.md +++ b/docs/product/PRD.md @@ -42,8 +42,8 @@ The product must let a qualified host: | PostgreSQL token counting and bounded batch preparation | IMPLEMENTED-ON-PROTECTED-MAIN | Token/resource accounting and batch partitioning remain deterministic and finite. | | Disk-free package payload persistence | IMPLEMENTED-ON-PROTECTED-MAIN | Package-owned JSONL payloads persist in PostgreSQL and are validated before credential/provider effects. | | OpenAI-compatible upload/create/poll/wait/cancel/retrieve | IMPLEMENTED-ON-PROTECTED-MAIN | Provider destinations, resource identifiers, control responses, downloads, retries, and timeouts remain bounded and validated. | -| Standalone durable lifecycle | IMPLEMENTED-ON-PROTECTED-MAIN | Existing standalone users retain source-compatible durable lifecycle behavior. | -| Tenant-qualified durable lifecycle with forced RLS | IMPLEMENTED-ON-PROTECTED-MAIN | Tenant scope comes only from a trusted host authorization boundary; direct arbitrary SQL remains outside the isolation guarantee. | +| Standalone durable lifecycle | IMPLEMENTED-ON-PROTECTED-MAIN | `DurableBatchAPIClient` preserves the existing four-argument lifecycle-recorder seam `(postgres_dsn, endpoint_alias, provider_batch, observation_order)` and its default recorder stores lifecycle state under the exact `standalone` scope. | +| Tenant-qualified durable lifecycle with forced RLS | IMPLEMENTED-ON-PROTECTED-MAIN | Tenant scope comes only from a trusted host authorization boundary, is validated before any tenant-client observation reservation, credential resolution, provider I/O, or lifecycle database I/O, and qualifies lifecycle identities, conflict targets, reads, and operational status indexing; direct arbitrary SQL remains outside the isolation guarantee. | | Durable resumable result checkpoint/CAS storage | IMPLEMENTED-ON-PROTECTED-MAIN | PostgreSQL supplies tenant-qualified durable checkpoint authority and conflict detection; this is not a distributed exactly-once claim. | | Redacted readiness reporting | IMPLEMENTED-ON-PROTECTED-MAIN | Public readiness evidence does not disclose arbitrary lower-layer diagnostic content. | | Scheduler-independent bounded provider reconciliation primitive | IMPLEMENTED-ON-PROTECTED-MAIN | A host can submit a finite, validated candidate set for polling/retrieval through the existing bounded provider client; candidate discovery, scheduling, and cross-process lease ownership remain outside this primitive. | @@ -66,7 +66,13 @@ All provider operations shall use validated endpoint configuration and finite re ### FR-3: Durable lifecycle and tenancy -The durable business identity is tenant-qualified where tenancy is enabled. The package shall validate tenant scope before persistence/provider effects that rely on it, bind scope through parameterized transaction-local PostgreSQL context, and keep forced row-level security enabled for application roles. PostgreSQL superuser/BYPASSRLS and arbitrary SQL access are administrative escape hatches and are not part of the tenant isolation guarantee. +The durable business identity is tenant-qualified where tenancy is enabled. `TenantDurableBatchAPIClient` shall validate its trusted host-selected `tenant_scope` synchronously at construction, before observation reservation, credential resolution, provider I/O, or lifecycle database I/O can occur. The tenant-qualified lifecycle key is `(tenant_scope, endpoint_alias, remote_batch_id)`; lifecycle persistence conflict targets, exact-row lookups, and operational status indexes shall retain `tenant_scope`, and package reads/writes shall bind that validated scope through parameterized transaction-local PostgreSQL context with forced row-level security for application roles. Provider/model content, endpoint aliases, remote identifiers, and transport data never select tenant authority. + +The deployment credential provider remains a separate host/configuration authority: validating tenant scope before credential resolution does not make credentials tenant-keyed and does not authorize a tenant to select secrets. PostgreSQL superuser/BYPASSRLS and arbitrary SQL access are administrative escape hatches and are not part of the tenant isolation guarantee. + +Standalone source compatibility is also normative. `DurableBatchAPIClient` retains its four-argument lifecycle-recorder interface `(postgres_dsn, endpoint_alias, provider_batch, observation_order)`, and the default persistence/read helpers use the explicit `standalone` database scope rather than silently introducing a required tenant argument. + +Protected-main acceptance authority for these invariants is deterministic: `tests/test_tenant_durable_client.py` proves construction-time pre-effect tenant validation and the four-argument standalone recorder seam; `tests/test_tenant_lifecycle_persistence.py` proves explicit `standalone` delegation, tenant-qualified conflict targets and reads, malformed-scope rejection before database access, and distinct tenant identities; `pg_llm_batch/schema.sql` supplies the tenant-qualified unique key, forced RLS policy, and tenant-qualified operational status index. `docs/remote-batch-lifecycle.md`, ADR 0002, and `docs/doctoring/tenant-scoped-lifecycle.md` describe the same protected-main contract. ### FR-4: Reconciliation and recovery From f0cf1f73c9e129fa0d4edb7a604dd51e7cf2b23f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 20:19:48 +0900 Subject: [PATCH 13/29] docs: pin tenant lifecycle technical invariants --- docs/product/TRD.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/product/TRD.md b/docs/product/TRD.md index 4a4249ab0..1d0133da8 100644 --- a/docs/product/TRD.md +++ b/docs/product/TRD.md @@ -17,8 +17,8 @@ Status vocabulary is shared with the PRD: **IMPLEMENTED-ON-PROTECTED-MAIN**, **A | `token_counter.py` | resolve reviewed tokenizer metadata and call `pg_tiktoken` for token accounting | provider credentials, tenant authorization, provider I/O | | `orchestrator.py` | select queued requests, partition under limits, persist package payload/file/line/request assignments | provider protocol and retry policy | | `batch_api_client.py` | validate provider destination and remote identifiers; upload/create/poll/wait/cancel/retrieve under finite budgets | tenant authentication, scheduler ownership | -| `durable_client.py` | compose provider operations with durable lifecycle ordering/persistence | authenticating or authorizing host tenants | -| `db.py` | schema application and parameterized persistence/read helpers | database-side provider networking | +| `durable_client.py` | compose provider operations with durable lifecycle ordering/persistence; preserve `DurableBatchAPIClient` source compatibility, its four-argument `LifecycleRecorder(postgres_dsn, endpoint_alias, provider_batch, observation_order)` seam, and default `standalone` database scope; provide a distinct tenant-qualified recorder seam | authenticating or authorizing host tenants, deriving tenant authority from provider data | +| `db.py` | schema application and parameterized persistence/read helpers, including tenant-qualified lifecycle context, conflict identity, and exact-row lookup | database-side provider networking, authenticating tenant callers | | `checkpoint_store.py` | tenant-qualified PostgreSQL checkpoint/CAS operations and conflict semantics | distributed exactly-once claims | | `reconciliation.py` | finite host-selected polling/retrieval pass using the existing validated client surface | candidate discovery, scheduling, cross-process leasing | | `config.py` | PostgreSQL-backed configuration and encrypted secret storage for standalone composition | prescribing an embedding host's external secret manager | @@ -26,6 +26,8 @@ Status vocabulary is shared with the PRD: **IMPLEMENTED-ON-PROTECTED-MAIN**, **A | `health.py` | readiness aggregation and redacted public health response | general-purpose web serving or arbitrary diagnostic reflection | | `cli.py` | standalone operator composition and bounded input surfaces | higher-level workflow orchestration | +The protected-main validation authority for the `durable_client.py` compatibility row is `pg_llm_batch/durable_client.py`, `tests/test_tenant_durable_client.py`, and `tests/test_tenant_lifecycle_persistence.py`: the tests prove construction-time tenant rejection before downstream effects, exact four-argument standalone recorder invocation, explicit `standalone` persistence/read delegation, and tenant-qualified lifecycle persistence/read identities. + ## Runtime architecture ### Batch preparation @@ -49,7 +51,9 @@ Automatic provider retry is limited to reviewed idempotent GET operations and th ### Durable lifecycle tenancy -Standalone lifecycle data uses the exact `standalone` tenant scope. Tenant-aware hosts use a trusted host-selected scope after authentication and authorization. Provider metadata, request payloads, model output, transport headers, endpoint aliases, and provider resource identifiers never choose tenant authority. +Standalone lifecycle data uses the exact `standalone` tenant scope. `DurableBatchAPIClient` preserves its original four-argument lifecycle-recorder interface `(postgres_dsn, endpoint_alias, provider_batch, observation_order)`; its default persistence path and `get_remote_batch_state(...)` compatibility helper resolve through the explicit `standalone` scope. Tenant-aware hosts instead use `TenantDurableBatchAPIClient` with a distinct tenant-qualified recorder seam so tenant identity cannot be silently dropped. + +A tenant-aware client validates the trusted host-selected `tenant_scope` synchronously during construction, before any observation reservation, credential-provider lookup, provider I/O, or lifecycle database I/O can occur. Provider metadata, request payloads, model output, transport headers, endpoint aliases, provider resource identifiers, and credential data never choose tenant authority. Credential resolution remains a separate deployment/host concern: ordering tenant validation before it does not make the credential store tenant-keyed. The durable lifecycle identity is: @@ -57,7 +61,9 @@ The durable lifecycle identity is: (tenant_scope, endpoint_alias, remote_batch_id) ``` -Package reads/writes bind tenant scope with parameterized transaction-local `set_config`. The schema enables and forces PostgreSQL row-level security for tenant-qualified lifecycle state. Production application roles must be `NOSUPERUSER NOBYPASSRLS`. A role that can execute arbitrary SQL can choose arbitrary custom setting values; therefore generic arbitrary SQL access is explicitly outside the package isolation guarantee. +Every tenant-aware lifecycle persistence conflict target and exact-row lookup includes the full identity. The protected schema's lifecycle operational status index begins with `tenant_scope`, and package reads/writes bind the validated scope with parameterized transaction-local `set_config`. The schema enables and forces PostgreSQL row-level security for tenant-qualified lifecycle state. Production application roles must be `NOSUPERUSER NOBYPASSRLS`. A role that can execute arbitrary SQL can choose arbitrary custom setting values; therefore generic arbitrary SQL access is explicitly outside the package isolation guarantee. + +These invariants are deterministically verified on protected main: `tests/test_tenant_durable_client.py` proves malformed scope fails before reservation or credentials and proves the unchanged four-argument standalone recorder seam; `tests/test_tenant_lifecycle_persistence.py` proves malformed scope fails before database access, the upsert conflict target is `(tenant_scope, endpoint_alias, remote_batch_id)`, exact reads bind the same full identity, and standalone helpers delegate to `standalone`; `pg_llm_batch/schema.sql` supplies the tenant-qualified unique constraint, forced RLS policy, and `idx_llm_remote_batch_jobs_tenant_status_observed` index. `docs/remote-batch-lifecycle.md`, ADR 0002, and `docs/doctoring/tenant-scoped-lifecycle.md` document the same migration, direct-SQL/RLS, role, and rollback boundaries. ### Durable result checkpoints From b9f15c0ed62dbd35d2ef971d4885aefd770125bc Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 20:20:59 +0900 Subject: [PATCH 14/29] docs: verify tenant lifecycle companion authority --- docs/DOCUMENTATION_FITNESS.md | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index 22b522a0c..f62c180e8 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -6,20 +6,24 @@ This inventory evaluates the documentation graph against the exact protected-mai A document is *fit* only when it states the correct authority boundary, is consistent with the protected-main code/schema/tests it describes, avoids transferring predecessor or active-branch evidence into shipped claims, and gives an operator or reviewer enough information to use, verify, recover, or reject the relevant behavior safely. +The tenant lifecycle material in this canonical cohort is a reconstruction of already integrated protected-main behavior, not a new runtime tenant/RLS/migration contract. Companion-document verification against protected main confirms the same contract is already present in the public `README.md`, root `ARCHITECTURE.md`, topic operator guide `docs/remote-batch-lifecycle.md`, accepted ADR 0002, `docs/doctoring/tenant-scoped-lifecycle.md`, and `CHANGELOG.md`. Those authorities already cover trusted tenant identity, `NOSUPERUSER NOBYPASSRLS`, transaction-local/forced RLS, direct-SQL limits, legacy-to-`standalone` migration, and rollback constraints. The canonical PRD/TRD therefore may cite that existing evidence without rewriting those protected-main files or racing PR #184's separate legacy-extension-retirement documentation lane. + ## Current fitness matrix | Documentation surface | Status | Fitness assessment | Required next action | | --- | --- | --- | --- | -| `README.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Useful public entry point, but it is not the sole product/architecture authority and must not absorb transient PR/check state. | Keep beginner-readable; update bounded shipped capability changes after merge. | -| `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, interoperability, and verification, but does not by itself form a complete system/UML/data model. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file. | -| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states. | Validate exact-head docs gates and integrate only through normal PR governance. | -| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries and release/testing invariants are explicit. | Validate exact-head docs gates and integrate only through normal PR governance. | -| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. | +| `README.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Useful public entry point; protected-main tenant lifecycle guidance already reflects trusted tenant scope and standalone compatibility. It is not the sole product/architecture authority and must not absorb transient PR/check state. | Keep beginner-readable; update only when a bounded shipped capability actually changes after merge. | +| `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, direct-SQL trust limits, rollback considerations, interoperability, and verification, but does not by itself form a complete system/UML/data model. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file. | +| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states and cites deterministic protected-main tenant/standalone acceptance authority. | Validate exact-head docs gates and integrate only through normal PR governance. | +| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries, tenant pre-effect validation, standalone recorder compatibility, and release/testing invariants are explicit. | Validate exact-head docs gates and integrate only through normal PR governance. | +| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. ADR 0002 already records trusted tenant identity, explicit `standalone` compatibility, RLS/direct-SQL role boundaries, migration, and rollback. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. | | `docs/adr/README.md` | ACTIVE-PR | This canonical branch now supplies the missing protected-main ADR navigation/status index, explains numbering gaps, separates ADR decision status from implementation status, and forbids implicit supersession. | Validate exact-head documentation gates and integrate only through normal PR governance. | | `docs/result-streaming.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Describes bounded provider result streaming/checkpoint behavior; checkpoint persistence has a separate accepted ADR. | Update only when the integrated result-application contract changes the operator-facing boundary. | -| `docs/remote-batch-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Durable lifecycle/tenant behavior has detailed protected-main documentation. | Keep synchronized with tenant/RLS and lifecycle migrations. | +| `docs/remote-batch-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Serves as the topic-specific operator authority for durable lifecycle tenancy. It already specifies trusted scope selection, pre-effect validation, four-argument standalone recorder compatibility, tenant-qualified identity/conflict/read behavior, forced RLS/application-role limits, migration, direct-SQL impact, rollback, recovery, and deterministic verification. | Keep synchronized with tenant/RLS and lifecycle migrations; do not duplicate this contract into a competing operator file. | +| `docs/doctoring/tenant-scoped-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Supporting evidence independently records pre-effect validation, explicit standalone compatibility, database role/RLS limits, migration and rollback, plus current primary references. | Keep supporting references current; do not promote doctoring into the sole product authority. | +| `CHANGELOG.md` tenant lifecycle entry | IMPLEMENTED-ON-PROTECTED-MAIN | The protected-main Unreleased section already records tenant-scoped durable lifecycle, forced default-deny RLS, standalone compatibility, and the atomic tenant migration fix. | Change only when integrated shipped behavior changes; do not log active-PR claims as shipped. | | Topic-specific doctoring/reference docs | IMPLEMENTED-ON-PROTECTED-MAIN | Useful deep evidence exists, but doctoring material is supporting evidence, not the canonical product-status authority. | Keep primary-reference and APA-style citations current where material. | -| Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation, but it is not shipped and its exact-head acceptance remains separate. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. | +| Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation for retiring legacy `http` / `pg_cron`; it is a different contract from the already-shipped tenant lifecycle migration and is not protected-main truth. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. | | OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. | | Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. | | Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay and does not make end-to-end result application or distributed exactly-once behavior protected-main truth. | Keep the protected-main end-to-end capability PARTIAL until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. | @@ -29,7 +33,7 @@ A document is *fit* only when it states the correct authority boundary, is consi | Data governance | PLANNED | No single canonical data-governance document currently maps data classes, retention/ownership, tenant authority, privacy exposure, backup/restore, and deletion limits. | Add a protected-main-grounded data-governance contract. | | UML/component/sequence views | PLANNED | Root architecture is prose-first; no canonical UML authority is present. | Add textual Mermaid/PlantUML-compatible component and critical sequence diagrams after active source contracts stabilize. | | ERD / schema model | PLANNED | SQL and schema tests are authoritative, but no canonical ERD summarizes package-owned identities, tenancy, checkpoint state, and relationships. | Generate/maintain an ERD from protected-main schema; label migration-only/active-PR objects separately. | -| Standalone operator guide | PARTIAL | Operational instructions exist across README and topic docs, while #184 is adding migration-specific operability. There is no protected-main general `docs/OPERABILITY.md`. | Establish a general operator authority without racing #184's bounded migration changes. | +| General standalone operator guide | PARTIAL | Protected main has strong operational guidance across `README.md` and topic authorities such as `docs/remote-batch-lifecycle.md`; there is no single omnibus protected-main `docs/OPERABILITY.md`. This is a navigation/consolidation gap, not absence of tenant lifecycle operator guidance. | Establish one general operator index/authority only after checking adjacent writers; do not race #184's active migration-specific operability work. | | Release governance | PARTIAL | Release-evidence ADRs/tests/workflows are strong, but the canonical graph lacks one concise protected-main release/operator authority tying versioning, SBOM, provenance, artifact identity, rollback/recovery, and publication verification together. | Add a stable release contract after checking active release writers; do not copy workflow-run IDs. | | Licensing / third-party notices | PARTIAL | Apache-2.0 source headers and repository licensing exist, but acquisition diligence should explicitly trace package license, dependency/SBOM evidence, and third-party notice process. | Evaluate a concise licensing/compliance evidence index without claiming legal certification. | @@ -38,7 +42,9 @@ A document is *fit* only when it states the correct authority boundary, is consi - Protected-main behavior is the shipped authority. Active PRs and historical branches are overlays, not proof of implementation. - Exact contributor heads, generated merge commits, check IDs, and transient queue state belong in PR/review evidence, not durable architecture/product documents. - Standalone operation and modular MSA embedding must be documented as co-equal supported boundaries; no ContextualWisdomLab host is a hidden runtime requirement. -- Tenant scope is selected only by a trusted authenticated/authorized host boundary. PostgreSQL RLS is defense in depth, not authentication or SQL-injection prevention. +- Tenant scope is selected only by a trusted authenticated/authorized host boundary and, for the tenant durable client, validates before observation reservation, credential lookup, provider I/O, or lifecycle database I/O. Lifecycle identities, conflict targets, exact reads, and operational status indexes remain tenant-qualified where tenancy is enabled. +- The standalone `DurableBatchAPIClient` retains its four-argument lifecycle-recorder seam and explicit `standalone` persistence/read scope unless a separately reviewed compatibility change is integrated. +- PostgreSQL RLS is defense in depth, not authentication or SQL-injection prevention; production application roles for the tenant lifecycle boundary are `NOSUPERUSER NOBYPASSRLS`. - Provider/model content never becomes tenant, credential, endpoint, filesystem, or database authority. - Distributed exactly-once processing is never implied by PostgreSQL checkpoint atomicity alone. - Security, privacy, SOC 2, and CSAP material is evidence-readiness documentation only unless an external certification actually exists. @@ -46,4 +52,4 @@ A document is *fit* only when it states the correct authority boundary, is consi ## Fitness gate for future canonical changes -Before changing a canonical surface, refetch protected main, open PRs, non-default branches, the exact affected source/schema/test authorities, and any adjacent documentation writer. A documentation repair must not race a source-affecting writer whose behavior is still unsettled. After a product PR merges, update the canonical status classification from `ACTIVE-PR` or `PARTIAL` only when the protected-main tree contains the capability and the integrated evidence contract remains valid. +Before changing a canonical surface, refetch protected main, open PRs, non-default branches, the exact affected source/schema/test authorities, and any adjacent documentation writer. A documentation repair must not race a source-affecting writer whose behavior is still unsettled. When a canonical change merely reconstructs an already integrated contract, verify the existing permanent companion set before widening the change; when runtime behavior actually changes, update the required permanent companion surfaces in that same governed change or explicitly coordinate their writer ownership. After a product PR merges, update the canonical status classification from `ACTIVE-PR` or `PARTIAL` only when the protected-main tree contains the capability and the integrated evidence contract remains valid. From 2a540ba812a36c0b1bbd30a308d4a537b18b9735 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 21:10:09 +0900 Subject: [PATCH 15/29] docs: define utility-preserving PII controls --- docs/product/PRD.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/product/PRD.md b/docs/product/PRD.md index f2fd8e282..0e9ca63fa 100644 --- a/docs/product/PRD.md +++ b/docs/product/PRD.md @@ -104,6 +104,8 @@ Owned production Python must maintain exact 100% statement and branch coverage a Security-sensitive validation fails closed. Secrets, DSNs, prompt/provider content, unvalidated identifiers, and arbitrary lower-layer exception text must not be retained in logs or review evidence merely for debugging. Repository controls should support SOC 2 / CSAP evidence preparation without claiming certification. +The package must not apply blanket masking or lossy transformation to authorized business payloads merely because they contain PII: changing prompt, request, or result content can invalidate business meaning, token accounting, provider behavior, auditability, or downstream decisions. Authorized content fidelity is therefore a product requirement. Confidentiality controls belong at explicit boundaries instead: trusted host authentication/authorization and tenant selection, least-privilege database/service access, transport protection, deployment/storage protection where provided, purpose-limited retention and deletion, and package-owned logs/telemetry/errors that omit content-bearing values. Any content transformation must be an explicit host/business policy with provenance and acceptance evidence, not a hidden package default or a claim that redacted diagnostics mean persisted business data was masked. + ### Reliability Network, response, retry, wait, candidate-scan, payload, and release-evidence operations must be explicitly bounded. Recovery and rollback must be documented before migrations or release changes are considered complete. Queued, skipped, cancelled, absent, stale, predecessor-head, synthetic-merge-only, or infrastructure-failed evidence is not success for an exact source head. From 6279c63b5e85900de3227b9231029f9d4e1d0219 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 21:11:01 +0900 Subject: [PATCH 16/29] docs: specify content-fidelity privacy boundary --- docs/product/TRD.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/product/TRD.md b/docs/product/TRD.md index 1d0133da8..c689e6b93 100644 --- a/docs/product/TRD.md +++ b/docs/product/TRD.md @@ -99,6 +99,12 @@ Durable provider/resource identifiers and tenant identities are validated before Standalone provider configuration and encrypted secrets are PostgreSQL-backed. Environment variables are limited to explicitly documented bootstrap transport such as the database DSN and optional encryption key. CLI secret entry uses no-echo prompting or bounded standard input rather than plaintext process arguments. Embedding hosts may supply another credential provider through the supported seam. +### Authorized content fidelity + +The package does not gain authority to mask, tokenize away, truncate for privacy, or otherwise alter an authorized prompt, request, JSONL record, or provider result merely because the content may contain PII. Silent transformation would change token counts, provider semantics, persisted evidence, replay behavior, and downstream business meaning. Package serialization, token accounting, persistence, upload, retrieval, and result application therefore preserve authorized business content unless an explicit reviewed feature contract says otherwise. + +Confidentiality for content-bearing data is enforced through boundary controls rather than a blanket masking default: the embedding host authenticates and authorizes the caller and selects tenant scope; package/database/service identities remain least-privilege; transport uses the reviewed secure destination policy; deployment/storage protections are owned and evidenced at the applicable layer; retention and deletion are purpose-limited and explicit; and package-owned errors, logs, telemetry, readiness, CI/review evidence, and other operational surfaces omit content-bearing values. A host that intentionally transforms content must do so through an explicit business-policy boundary with provenance and acceptance tests. Redacted operational evidence must never be represented as proof that persisted or provider-bound business content was masked. + ### Diagnostic confidentiality Errors, logs, telemetry, check evidence, and public readiness must avoid DSNs, credentials, prompts, provider bodies, arbitrary SQL/provider exception text, unvalidated identifiers, and dynamic exception-class names where those values are not required for operation. Failure categories intended for public/operational evidence use bounded vocabularies. From 8c01d37a452369774964a03f8aff45a620368c33 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 21:45:23 +0900 Subject: [PATCH 17/29] docs(security): clarify tenant setting trust boundary --- docs/product/PRD.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/product/PRD.md b/docs/product/PRD.md index 0e9ca63fa..5fac473bb 100644 --- a/docs/product/PRD.md +++ b/docs/product/PRD.md @@ -68,7 +68,9 @@ All provider operations shall use validated endpoint configuration and finite re The durable business identity is tenant-qualified where tenancy is enabled. `TenantDurableBatchAPIClient` shall validate its trusted host-selected `tenant_scope` synchronously at construction, before observation reservation, credential resolution, provider I/O, or lifecycle database I/O can occur. The tenant-qualified lifecycle key is `(tenant_scope, endpoint_alias, remote_batch_id)`; lifecycle persistence conflict targets, exact-row lookups, and operational status indexes shall retain `tenant_scope`, and package reads/writes shall bind that validated scope through parameterized transaction-local PostgreSQL context with forced row-level security for application roles. Provider/model content, endpoint aliases, remote identifiers, and transport data never select tenant authority. -The deployment credential provider remains a separate host/configuration authority: validating tenant scope before credential resolution does not make credentials tenant-keyed and does not authorize a tenant to select secrets. PostgreSQL superuser/BYPASSRLS and arbitrary SQL access are administrative escape hatches and are not part of the tenant isolation guarantee. +The transaction-local `pg_llm_batch.tenant_scope` custom setting is routing context, not a credential or authenticated identity. Only package code acting on a trusted authenticated/authorized host selection may set it for tenant-owned operations; arbitrary SQL must not select or override tenant authority by calling `set_config`. PostgreSQL RLS is defense in depth and does not replace host authentication/authorization, SQL-injection prevention, or correct identity mapping. PostgreSQL superuser/BYPASSRLS and arbitrary SQL access remain administrative escape hatches outside the tenant isolation guarantee. + +The deployment credential provider remains a separate host/configuration authority: validating tenant scope before credential resolution does not make credentials tenant-keyed and does not authorize a tenant to select secrets. Standalone source compatibility is also normative. `DurableBatchAPIClient` retains its four-argument lifecycle-recorder interface `(postgres_dsn, endpoint_alias, provider_batch, observation_order)`, and the default persistence/read helpers use the explicit `standalone` database scope rather than silently introducing a required tenant argument. From d55b21efd08b800441537c762b646af4c1a39aa3 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 15 Aug 2026 21:46:28 +0900 Subject: [PATCH 18/29] docs(security): mirror tenant setting authority invariant --- docs/DOCUMENTATION_FITNESS.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index f62c180e8..d67ab301e 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -43,8 +43,9 @@ The tenant lifecycle material in this canonical cohort is a reconstruction of al - Exact contributor heads, generated merge commits, check IDs, and transient queue state belong in PR/review evidence, not durable architecture/product documents. - Standalone operation and modular MSA embedding must be documented as co-equal supported boundaries; no ContextualWisdomLab host is a hidden runtime requirement. - Tenant scope is selected only by a trusted authenticated/authorized host boundary and, for the tenant durable client, validates before observation reservation, credential lookup, provider I/O, or lifecycle database I/O. Lifecycle identities, conflict targets, exact reads, and operational status indexes remain tenant-qualified where tenancy is enabled. +- `pg_llm_batch.tenant_scope` is transaction-local routing context, not a credential or authenticated identity. Only package code acting on trusted host selection may set it for tenant-owned operations; arbitrary SQL must not select or override tenant authority through `set_config`. - The standalone `DurableBatchAPIClient` retains its four-argument lifecycle-recorder seam and explicit `standalone` persistence/read scope unless a separately reviewed compatibility change is integrated. -- PostgreSQL RLS is defense in depth, not authentication or SQL-injection prevention; production application roles for the tenant lifecycle boundary are `NOSUPERUSER NOBYPASSRLS`. +- PostgreSQL RLS is defense in depth, not host authentication/authorization, SQL-injection prevention, or correct identity mapping; production application roles for the tenant lifecycle boundary are `NOSUPERUSER NOBYPASSRLS`. - Provider/model content never becomes tenant, credential, endpoint, filesystem, or database authority. - Distributed exactly-once processing is never implied by PostgreSQL checkpoint atomicity alone. - Security, privacy, SOC 2, and CSAP material is evidence-readiness documentation only unless an external certification actually exists. From 6b363857c49f0cc1cadf2eec359135fcbf697c0c Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 16 Aug 2026 07:12:43 +0900 Subject: [PATCH 19/29] docs(traceability): register active validation privacy repair --- docs/TRACEABILITY.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index bd24a9133..8b3d140a0 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -21,7 +21,8 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | FR-5 legacy `http` / `pg_cron` authority retirement on existing volumes | ACTIVE-PR | protected main does not yet contain the retirement migration contract | PR #184 migration/smoke/operator evidence only | Must remain active until unchanged exact head satisfies migration/security/release/review gates and merges. | | FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Protected main deliberately retains `SecretStore(require_encryption=False)` as the compatibility default; Fernet support is therefore not an encryption-required production claim. PR #193 proposes an encryption-required default plus explicit local/development opt-out while also separating runtime construction from provisioning. | | FR-6 production secret-at-rest policy lifecycle | PARTIAL | Protected main can enforce Fernet when callers explicitly select `require_encryption=True`, and fails before database access when required encryption lacks usable configuration. | config/secret/bootstrap tests; security issue acceptance criteria | No protected-main contract yet detects and atomically migrates existing `is_encrypted = FALSE` rows, proves bounded key rotation/recovery, or exposes the selected deployment policy through redacted readiness/operator evidence. PR #193 addresses the default-policy boundary only while it remains ACTIVE-PR. | -| FR-7 bounded diagnostics/readiness | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/health.py`, bounded error surfaces | health/confidentiality tests and architecture requirements | Broader `ValidationError` value-confidentiality work must not be inferred as solved from readiness alone. | +| FR-7 bounded diagnostics/readiness | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/health.py`, bounded error surfaces | health/confidentiality tests and architecture requirements | Broader generic validation rejected-value confidentiality is not protected-main truth; PR #202 is the ACTIVE-PR compatibility-aware repair and must not be inferred as shipped before integration. | +| FR-7 generic validation rejected-value confidentiality | ACTIVE-PR | protected-main `ValidationError` still renders and retains arbitrary rejected values by default | PR #202 exact-head privacy/compatibility tests and review evidence only | The candidate safe default uses fixed redaction unless a reviewed call site explicitly opts into bounded non-sensitive `safe_value` evidence; this remains unshipped until normal governance integrates it. | | FR-7 opt-in OpenTelemetry | IMPLEMENTED-ON-PROTECTED-MAIN / packaging PARTIAL | `pg_llm_batch/observability.py` | observability tests | First-class locked installation extra remains ACTIVE-PR #175. | | FR-8 standalone + modular MSA deployment | IMPLEMENTED-ON-PROTECTED-MAIN | package/CLI/container composition; injectable host seams | `README.md`, `ARCHITECTURE.md`, package/container tests | CWL host repositories are optional integrations, not package runtime dependencies. | | Exact owned 100% statement/branch coverage | IMPLEMENTED-ON-PROTECTED-MAIN governance contract | repository CI configuration and owned production code | coverage gate/tests | Must be re-proven on every changed exact head; predecessor evidence never transfers. | @@ -39,7 +40,7 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | Provider destination validation | `batch_api_client.py` | endpoint/URL tests | Host/network infrastructure TLS policy is external. | | Bounded provider input | provider client + result streaming | response/download/JSONL/resource-budget tests | Provider authenticity is not established by payload validation. | | Secret/config boundary | `config.py`, bootstrap contract | config/secret/bootstrap tests | Protected main supports Fernet but does not require it by default; the compatibility path is not a production confidentiality claim. PR #193's stricter default is ACTIVE-PR, while enterprise secret-manager choice remains host-owned. | -| Diagnostic confidentiality | health/error/logging contracts | traceback/health/redaction tests | Generic validation value confidentiality remains an explicit separate gap where applicable. | +| Diagnostic confidentiality | health/error/logging contracts | traceback/health/redaction tests | Generic `ValidationError` rejected-value confidentiality remains incomplete on protected main; PR #202 is ACTIVE-PR and its evidence does not transfer until merge. | | Checkpoint concurrency/integrity | `checkpoint_store.py` | CAS/concurrency/RLS/rollback tests | PostgreSQL atomicity does not extend to external systems. | | Release artifact integrity | `release_evidence.py` + release contracts | descriptor/dirfd/reproducibility tests | Publication credentials and external registry availability are operational dependencies. | @@ -64,6 +65,7 @@ The following open pull requests are intentionally represented only as overlays - **#192** — this canonical PRD/TRD/fitness/traceability reconstruction itself. - **#193** — runtime-store/schema-provisioning separation plus an encryption-required `SecretStore` default with explicit local/development compatibility opt-out; both remain active overlays, and this PR does not by itself establish legacy-row migration, key rotation/recovery, or readiness-policy lifecycle completion. - **#194** — atomic local result-effect/checkpoint application; a current-main-compatible implementation is under review but remains an active overlay and is not shipped. +- **#202** — compatibility-aware `ValidationError` rejected-value confidentiality hardening; protected main still discloses arbitrary rejected values by default, so the candidate safe-default redaction and bounded `safe_value` opt-in remain ACTIVE-PR until integrated. This register is descriptive, not a substitute for refetching GitHub. Before changing a status, verify the PR still exists, its exact contributor head, live protected-main base, ancestry, current reviews/threads, exact-head gates, and resulting protected-main integration. From 3d5f79fef04f034f5e930717758992bfa173e86b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 16 Aug 2026 13:11:15 +0900 Subject: [PATCH 20/29] docs(traceability): record current PR 202 blocker --- docs/TRACEABILITY.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 8b3d140a0..49409b9f0 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -22,7 +22,7 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Protected main deliberately retains `SecretStore(require_encryption=False)` as the compatibility default; Fernet support is therefore not an encryption-required production claim. PR #193 proposes an encryption-required default plus explicit local/development opt-out while also separating runtime construction from provisioning. | | FR-6 production secret-at-rest policy lifecycle | PARTIAL | Protected main can enforce Fernet when callers explicitly select `require_encryption=True`, and fails before database access when required encryption lacks usable configuration. | config/secret/bootstrap tests; security issue acceptance criteria | No protected-main contract yet detects and atomically migrates existing `is_encrypted = FALSE` rows, proves bounded key rotation/recovery, or exposes the selected deployment policy through redacted readiness/operator evidence. PR #193 addresses the default-policy boundary only while it remains ACTIVE-PR. | | FR-7 bounded diagnostics/readiness | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/health.py`, bounded error surfaces | health/confidentiality tests and architecture requirements | Broader generic validation rejected-value confidentiality is not protected-main truth; PR #202 is the ACTIVE-PR compatibility-aware repair and must not be inferred as shipped before integration. | -| FR-7 generic validation rejected-value confidentiality | ACTIVE-PR | protected-main `ValidationError` still renders and retains arbitrary rejected values by default | PR #202 exact-head privacy/compatibility tests and review evidence only | The candidate safe default uses fixed redaction unless a reviewed call site explicitly opts into bounded non-sensitive `safe_value` evidence; this remains unshipped until normal governance integrates it. | +| FR-7 generic validation rejected-value confidentiality | ACTIVE-PR | protected-main `ValidationError` still renders and retains arbitrary rejected values by default | PR #202 candidate privacy/compatibility tests and review evidence only | PR #202 is Draft because its current `safe_value` guard accepts `str` subclasses via `isinstance`, which can execute caller-controlled special methods while validation or rendering handles supposedly safe evidence. The candidate must require an exact built-in `str` and prove a hostile-subclass regression before it can become Ready; the intended fixed-redaction default and bounded `safe_value` opt-in remain unshipped. | | FR-7 opt-in OpenTelemetry | IMPLEMENTED-ON-PROTECTED-MAIN / packaging PARTIAL | `pg_llm_batch/observability.py` | observability tests | First-class locked installation extra remains ACTIVE-PR #175. | | FR-8 standalone + modular MSA deployment | IMPLEMENTED-ON-PROTECTED-MAIN | package/CLI/container composition; injectable host seams | `README.md`, `ARCHITECTURE.md`, package/container tests | CWL host repositories are optional integrations, not package runtime dependencies. | | Exact owned 100% statement/branch coverage | IMPLEMENTED-ON-PROTECTED-MAIN governance contract | repository CI configuration and owned production code | coverage gate/tests | Must be re-proven on every changed exact head; predecessor evidence never transfers. | @@ -65,7 +65,7 @@ The following open pull requests are intentionally represented only as overlays - **#192** — this canonical PRD/TRD/fitness/traceability reconstruction itself. - **#193** — runtime-store/schema-provisioning separation plus an encryption-required `SecretStore` default with explicit local/development compatibility opt-out; both remain active overlays, and this PR does not by itself establish legacy-row migration, key rotation/recovery, or readiness-policy lifecycle completion. - **#194** — atomic local result-effect/checkpoint application; a current-main-compatible implementation is under review but remains an active overlay and is not shipped. -- **#202** — compatibility-aware `ValidationError` rejected-value confidentiality hardening; protected main still discloses arbitrary rejected values by default, so the candidate safe-default redaction and bounded `safe_value` opt-in remain ACTIVE-PR until integrated. +- **#202** — compatibility-aware `ValidationError` rejected-value confidentiality hardening. It is Draft because the current `safe_value` guard accepts `str` subclasses via `isinstance`; the branch must first require an exact built-in `str` and pass a hostile-subclass regression. Protected main still discloses arbitrary rejected values by default, and the intended safe-default redaction/bounded `safe_value` contract remains ACTIVE-PR until integrated. This register is descriptive, not a substitute for refetching GitHub. Before changing a status, verify the PR still exists, its exact contributor head, live protected-main base, ancestry, current reviews/threads, exact-head gates, and resulting protected-main integration. From 3525ab6b5e7f6d98001b2284ed76d6deb300ec62 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 16 Aug 2026 14:05:58 +0900 Subject: [PATCH 21/29] docs(security): clarify tenant and result authority --- docs/DOCUMENTATION_FITNESS.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index d67ab301e..94e538004 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -26,7 +26,7 @@ The tenant lifecycle material in this canonical cohort is a reconstruction of al | Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation for retiring legacy `http` / `pg_cron`; it is a different contract from the already-shipped tenant lifecycle migration and is not protected-main truth. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. | | OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. | | Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. | -| Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay and does not make end-to-end result application or distributed exactly-once behavior protected-main truth. | Keep the protected-main end-to-end capability PARTIAL until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. | +| Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay; FR-4/end-to-end result application stays `PARTIAL` on protected main, and neither this row nor the TRD content-fidelity invariant promotes #194 into shipped truth. | Keep FR-4 and `docs/TRACEABILITY.md` as the validation/status authority until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. | | Runtime config/schema provisioning separation | ACTIVE-PR | PR #193 is a current-main least-privilege candidate: runtime constructors move away from DDL/default seeding, validate the search-path-resolved base-table shape and current-role read capability through bounded catalog probes, and retain provisioning/default seeding at the explicit schema boundary. It is not shipped until normal governance integrates it. | Preserve the trusted-search-path/non-ownership caveat and reacquire exact-head checks/review after every push; do not transfer predecessor or earlier-head evidence. | | Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs. | | Threat model | PLANNED | No canonical protected-main `docs/THREAT_MODEL.md` is present. Security rules are distributed across AGENTS, architecture, ADRs, tests, and issue/PR evidence. | Add a threat model that distinguishes assets, trust boundaries, attacker capabilities, mitigations, residual risk, and non-guarantees without certification claims. | @@ -43,10 +43,11 @@ The tenant lifecycle material in this canonical cohort is a reconstruction of al - Exact contributor heads, generated merge commits, check IDs, and transient queue state belong in PR/review evidence, not durable architecture/product documents. - Standalone operation and modular MSA embedding must be documented as co-equal supported boundaries; no ContextualWisdomLab host is a hidden runtime requirement. - Tenant scope is selected only by a trusted authenticated/authorized host boundary and, for the tenant durable client, validates before observation reservation, credential lookup, provider I/O, or lifecycle database I/O. Lifecycle identities, conflict targets, exact reads, and operational status indexes remain tenant-qualified where tenancy is enabled. -- `pg_llm_batch.tenant_scope` is transaction-local routing context, not a credential or authenticated identity. Only package code acting on trusted host selection may set it for tenant-owned operations; arbitrary SQL must not select or override tenant authority through `set_config`. +- `pg_llm_batch.tenant_scope` is transaction-local routing context, not a credential or authenticated identity. Any database role capable of arbitrary SQL can call `set_config` with an arbitrary tenant scope; the trusted application boundary must therefore prevent generic tenant-controlled SQL, SQL injection, and incorrect identity mapping from selecting tenant authority. RLS does not provide those guarantees. - The standalone `DurableBatchAPIClient` retains its four-argument lifecycle-recorder seam and explicit `standalone` persistence/read scope unless a separately reviewed compatibility change is integrated. - PostgreSQL RLS is defense in depth, not host authentication/authorization, SQL-injection prevention, or correct identity mapping; production application roles for the tenant lifecycle boundary are `NOSUPERUSER NOBYPASSRLS`. - Provider/model content never becomes tenant, credential, endpoint, filesystem, or database authority. +- Content-fidelity invariants constrain any result-application path that exists, but they do not prove that end-to-end result application is shipped; FR-4 and `docs/TRACEABILITY.md` remain the capability-status authority while protected main is `PARTIAL`. - Distributed exactly-once processing is never implied by PostgreSQL checkpoint atomicity alone. - Security, privacy, SOC 2, and CSAP material is evidence-readiness documentation only unless an external certification actually exists. - Database object names, migration/rollback behavior, exact owned coverage requirements, Python-version acceptance, packaging/SBOM/provenance expectations, and bounded diagnostics must remain synchronized with repository tests and live governance. From 20b5362fbfdf1036ec891bfcf202865dd7000636 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 16 Aug 2026 14:06:55 +0900 Subject: [PATCH 22/29] docs(privacy): bound result and retention contracts --- docs/product/TRD.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/product/TRD.md b/docs/product/TRD.md index c689e6b93..918ec518a 100644 --- a/docs/product/TRD.md +++ b/docs/product/TRD.md @@ -101,9 +101,9 @@ Standalone provider configuration and encrypted secrets are PostgreSQL-backed. E ### Authorized content fidelity -The package does not gain authority to mask, tokenize away, truncate for privacy, or otherwise alter an authorized prompt, request, JSONL record, or provider result merely because the content may contain PII. Silent transformation would change token counts, provider semantics, persisted evidence, replay behavior, and downstream business meaning. Package serialization, token accounting, persistence, upload, retrieval, and result application therefore preserve authorized business content unless an explicit reviewed feature contract says otherwise. +The package does not gain authority to mask, tokenize away, truncate for privacy, or otherwise alter an authorized prompt, request, JSONL record, or provider result merely because the content may contain PII. Silent transformation would change token counts, provider semantics, persisted evidence, replay behavior, and downstream business meaning. Serialization, token accounting, persistence, upload, and retrieval paths implemented on protected main therefore preserve authorized business content unless an explicit reviewed feature contract says otherwise. The same content-fidelity invariant constrains any result-application path that exists, but end-to-end result application remains **PARTIAL** under FR-4 and `docs/TRACEABILITY.md`; PR #194 is an **ACTIVE-PR** transaction-seam candidate, not protected-main proof of a completed result-application capability. -Confidentiality for content-bearing data is enforced through boundary controls rather than a blanket masking default: the embedding host authenticates and authorizes the caller and selects tenant scope; package/database/service identities remain least-privilege; transport uses the reviewed secure destination policy; deployment/storage protections are owned and evidenced at the applicable layer; retention and deletion are purpose-limited and explicit; and package-owned errors, logs, telemetry, readiness, CI/review evidence, and other operational surfaces omit content-bearing values. A host that intentionally transforms content must do so through an explicit business-policy boundary with provenance and acceptance tests. Redacted operational evidence must never be represented as proof that persisted or provider-bound business content was masked. +Confidentiality for content-bearing data is enforced through boundary controls rather than a blanket masking default: the embedding host authenticates and authorizes the caller and selects tenant scope; package/database/service identities remain least-privilege; and transport uses the reviewed secure destination policy. Protected main does not define a universal business-data retention duration or a general destructive deletion workflow. The embedding host owns business purpose, retention period, deletion authorization/trigger, and evidence that its policy was executed; the deployment owner separately owns PostgreSQL backup/replica, log/telemetry, and infrastructure retention/deletion controls; provider-side retention/deletion remains a provider/account-policy responsibility unless an explicit reviewed package adapter contract implements and verifies it. Package-owned errors, logs, telemetry, readiness, CI/review evidence, and other operational surfaces omit content-bearing values. A host that intentionally transforms content must do so through an explicit business-policy boundary with provenance and acceptance tests. Redacted operational evidence must never be represented as proof that persisted or provider-bound business content was masked or deleted. ### Diagnostic confidentiality @@ -157,4 +157,4 @@ A release may originate only from the exact integrated protected head after all ## Documentation requirements -Canonical product/technical/architecture/ADR/UML/ERD/security/operability/release/data-governance/traceability documents must distinguish shipped protected-main state from active or planned work. Durable documents should describe contracts rather than transient check-run IDs. When an active capability merges, the canonical graph is updated in the same governance model rather than relying on stale PR body claims. \ No newline at end of file +Canonical product/technical/architecture/ADR/UML/ERD/security/operability/release/data-governance/traceability documents must distinguish shipped protected-main state from active or planned work. Durable documents should describe contracts rather than transient check-run IDs. When an active capability merges, the canonical graph is updated in the same governance model rather than relying on stale PR body claims. From 008aa9868602f5e0a886f2ddc537c312db70d184 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 17 Aug 2026 00:02:06 +0900 Subject: [PATCH 23/29] docs(recovery): trace integrated evidence and active executors --- docs/TRACEABILITY.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 49409b9f0..36df331a9 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -4,7 +4,7 @@ This map ties the canonical PRD/TRD requirements to stable protected-main implementation, test, ADR, and operator evidence. It intentionally avoids workflow-run IDs, predecessor SHAs, generated merge commits, and review comments because those are transient verification records rather than durable requirements authority. -The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. Rows marked **ACTIVE-PR** or **PARTIAL** are not shipped implementation claims. +The reference protected-main tree is `d2f1e32271910a6db98a0757d67194ddadca4566`. Rows marked **ACTIVE-PR** or **PARTIAL** are not shipped implementation claims. ## Product-to-technical traceability @@ -18,6 +18,10 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | FR-4 tenant-qualified cross-process single-flight | ACTIVE-PR | none on protected main beyond existing DB primitives | PR #191 exact-head evidence only | Not shipped; qualifying independent approval remains a live-governance concern. | | FR-4 durable result application + checkpoint coupling | PARTIAL | `pg_llm_batch/result_streaming.py`, `pg_llm_batch/checkpoint_store.py` provide streaming/checkpoint primitives | ADR 0006; ADR 0007; checkpoint/result-streaming tests; `docs/result-streaming.md` | PR #194 is ACTIVE-PR test-first work that adds a same-transaction local result-effect/checkpoint seam; protected main still does not claim end-to-end or distributed exactly-once application. | | FR-5 package persistence integrity | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/db.py`, `pg_llm_batch/schema.sql`, Docker schema mirror | schema-integrity, payload, lifecycle, checkpoint migration tests | Existing-volume legacy-extension retirement is separately ACTIVE-PR #184. | +| FR-5 bounded PostgreSQL recovery evidence primitives | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/postgres_recovery_receipt.py`, `pg_llm_batch/postgres_backup_evidence.py`, `pg_llm_batch/postgres_schema_evidence.py` | focused receipt/artifact/schema evidence tests; merged PRs #205, #206, #207 are historical integration evidence | Protected main can derive bounded content-free receipt, backup-artifact hash/size, and packaged-schema hash/size evidence. These primitives do not prove that a backup command ran successfully, that an artifact is restorable, that a live cluster matches packaged schema, or that PITR/RPO/RTO/HA/DR objectives are satisfied. | +| FR-5 executable PostgreSQL logical backup | ACTIVE-PR | none on protected main beyond the recovery-evidence primitives | PR #208 exact-head evidence only | The `pg_dump` execution candidate is not shipped. Its service selector is not tenant authorization, and current exact-head review/check evidence must be reacquired before integration. | +| FR-5 executable PostgreSQL logical restore | ACTIVE-PR | none on protected main beyond the recovery-evidence primitives | PR #209 exact-head evidence only | The direct `pg_restore` candidate is not shipped. Its caller-owned source-superuser trust assertion, target-isolation responsibility, libpq allowlist, single-transaction failure boundary, and archive-integrity checks remain active-PR contracts until integration. | +| FR-5 end-to-end PostgreSQL recovery readiness | PARTIAL | bounded receipt/artifact/schema evidence primitives are integrated | Issue #204 acceptance authority plus protected-main recovery evidence tests | No protected-main end-to-end isolated restore drill yet proves schema/RLS/constraint/extension parity, migration compatibility, external key/config custody, physical/WAL/PITR recovery, or a stated RPO/RTO/HA/DR objective. | | FR-5 legacy `http` / `pg_cron` authority retirement on existing volumes | ACTIVE-PR | protected main does not yet contain the retirement migration contract | PR #184 migration/smoke/operator evidence only | Must remain active until unchanged exact head satisfies migration/security/release/review gates and merges. | | FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Protected main deliberately retains `SecretStore(require_encryption=False)` as the compatibility default; Fernet support is therefore not an encryption-required production claim. PR #193 proposes an encryption-required default plus explicit local/development opt-out while also separating runtime construction from provisioning. | | FR-6 production secret-at-rest policy lifecycle | PARTIAL | Protected main can enforce Fernet when callers explicitly select `require_encryption=True`, and fails before database access when required encryption lacks usable configuration. | config/secret/bootstrap tests; security issue acceptance criteria | No protected-main contract yet detects and atomically migrates existing `is_encrypted = FALSE` rows, proves bounded key rotation/recovery, or exposes the selected deployment policy through redacted readiness/operator evidence. PR #193 addresses the default-policy boundary only while it remains ACTIVE-PR. | @@ -42,6 +46,7 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | Secret/config boundary | `config.py`, bootstrap contract | config/secret/bootstrap tests | Protected main supports Fernet but does not require it by default; the compatibility path is not a production confidentiality claim. PR #193's stricter default is ACTIVE-PR, while enterprise secret-manager choice remains host-owned. | | Diagnostic confidentiality | health/error/logging contracts | traceback/health/redaction tests | Generic `ValidationError` rejected-value confidentiality remains incomplete on protected main; PR #202 is ACTIVE-PR and its evidence does not transfer until merge. | | Checkpoint concurrency/integrity | `checkpoint_store.py` | CAS/concurrency/RLS/rollback tests | PostgreSQL atomicity does not extend to external systems. | +| Recovery evidence confidentiality/integrity | `postgres_recovery_receipt.py`, `postgres_backup_evidence.py`, `postgres_schema_evidence.py` | recovery receipt/artifact/schema evidence regressions | Evidence is bounded and content-free but does not authenticate an operator, prove backup provenance beyond caller-supplied receipt inputs, or prove restore semantics. | | Release artifact integrity | `release_evidence.py` + release contracts | descriptor/dirfd/reproducibility tests | Publication credentials and external registry availability are operational dependencies. | ## Data and persistence traceability @@ -52,6 +57,7 @@ The reference protected-main tree is `d0a4b30be1f46536e352443309f3a35533156767`. | Result checkpoints | `(tenant_scope, checkpoint_consumer_name, endpoint_alias, remote_batch_id)` | ADR 0006, ADR 0007, `docs/result-streaming.md` | Prefix checkpoint is not provider authentication or whole-stream immutability; cross-system exactly-once is not claimed. | | Package JSONL/payload state | package-owned schema identities and virtual payload references | PRD/TRD, schema-integrity and payload docs/tests | Persisted package data is canonical, not a disposable cache. | | Configuration/secrets | `com_config`, `com_secrets` | PRD/TRD and config tests | Shipped compatibility mode can still persist `is_encrypted = FALSE`; runtime/provisioning least-privilege separation, required-encryption default, legacy-row migration, key rotation/recovery, and redacted policy-readiness evidence are not all protected-main behavior. | +| PostgreSQL recovery evidence | bounded receipt metadata plus backup/schema SHA-256 and byte-size evidence | protected-main recovery evidence modules/tests; canonical PRD/TRD/this traceability map once integrated | The evidence does not itself persist a backup, execute backup/restore, prove isolated target parity, or establish PITR/RPO/RTO/HA/DR. Backup storage, encryption, retention, key custody, and external copies remain deployment/operator responsibilities unless a separately integrated contract says otherwise. | | Release evidence | descriptor/artifact identity contracts | ADR 0003, ADR 0004 | Evidence proves the reviewed artifact path, not organizational certification. | ## Active overlay register @@ -66,6 +72,10 @@ The following open pull requests are intentionally represented only as overlays - **#193** — runtime-store/schema-provisioning separation plus an encryption-required `SecretStore` default with explicit local/development compatibility opt-out; both remain active overlays, and this PR does not by itself establish legacy-row migration, key rotation/recovery, or readiness-policy lifecycle completion. - **#194** — atomic local result-effect/checkpoint application; a current-main-compatible implementation is under review but remains an active overlay and is not shipped. - **#202** — compatibility-aware `ValidationError` rejected-value confidentiality hardening. It is Draft because the current `safe_value` guard accepts `str` subclasses via `isinstance`; the branch must first require an exact built-in `str` and pass a hostile-subclass regression. Protected main still discloses arbitrary rejected values by default, and the intended safe-default redaction/bounded `safe_value` contract remains ACTIVE-PR until integrated. +- **#208** — bounded logical PostgreSQL backup executor candidate using `pg_dump`; active source only and not evidence that protected main can create a restorable backup. +- **#209** — bounded direct logical PostgreSQL restore executor candidate using `pg_restore`; active source only and not evidence of isolated restore acceptance or end-to-end recovery readiness. + +Merged recovery PRs #205, #206, and #207 are no longer active overlays; their bounded receipt, backup-artifact, and packaged-schema evidence primitives are present on the referenced protected-main tree. Their historical PR descriptions remain integration evidence, not durable runtime authority. This register is descriptive, not a substitute for refetching GitHub. Before changing a status, verify the PR still exists, its exact contributor head, live protected-main base, ancestry, current reviews/threads, exact-head gates, and resulting protected-main integration. From e2076821bf944ad4c16b44d1d2266455f06e68ff Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 17 Aug 2026 00:03:10 +0900 Subject: [PATCH 24/29] docs(recovery): refresh documentation fitness at protected main --- docs/DOCUMENTATION_FITNESS.md | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index 94e538004..083d1bd5b 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -2,38 +2,42 @@ ## Authority and status model -This inventory evaluates the documentation graph against the exact protected-main tree `d0a4b30be1f46536e352443309f3a35533156767`. It does not promote pull-request content into shipped truth. The canonical status vocabulary is shared with `docs/product/PRD.md` and `docs/product/TRD.md`: **IMPLEMENTED-ON-PROTECTED-MAIN**, **ACTIVE-PR**, **PARTIAL**, **PLANNED**, and **SUPERSEDED**. +This inventory evaluates the documentation graph against the exact protected-main tree `d2f1e32271910a6db98a0757d67194ddadca4566`. It does not promote pull-request content into shipped truth. The canonical status vocabulary is shared with `docs/product/PRD.md` and `docs/product/TRD.md`: **IMPLEMENTED-ON-PROTECTED-MAIN**, **ACTIVE-PR**, **PARTIAL**, **PLANNED**, and **SUPERSEDED**. A document is *fit* only when it states the correct authority boundary, is consistent with the protected-main code/schema/tests it describes, avoids transferring predecessor or active-branch evidence into shipped claims, and gives an operator or reviewer enough information to use, verify, recover, or reject the relevant behavior safely. The tenant lifecycle material in this canonical cohort is a reconstruction of already integrated protected-main behavior, not a new runtime tenant/RLS/migration contract. Companion-document verification against protected main confirms the same contract is already present in the public `README.md`, root `ARCHITECTURE.md`, topic operator guide `docs/remote-batch-lifecycle.md`, accepted ADR 0002, `docs/doctoring/tenant-scoped-lifecycle.md`, and `CHANGELOG.md`. Those authorities already cover trusted tenant identity, `NOSUPERUSER NOBYPASSRLS`, transaction-local/forced RLS, direct-SQL limits, legacy-to-`standalone` migration, and rollback constraints. The canonical PRD/TRD therefore may cite that existing evidence without rewriting those protected-main files or racing PR #184's separate legacy-extension-retirement documentation lane. +Protected main now also contains the bounded recovery-evidence primitives integrated by #205, #206, and #207: content-free PostgreSQL recovery receipts, descriptor-pinned backup-artifact hash/size evidence, and packaged-schema hash/size evidence. These are evidence primitives only. They do not make executable logical backup/restore, isolated restore acceptance, WAL/PITR, key/config custody, or any RPO/RTO/HA/DR objective shipped; #208 and #209 remain active source candidates and Issue #204 remains the end-to-end recovery authority. + ## Current fitness matrix | Documentation surface | Status | Fitness assessment | Required next action | | --- | --- | --- | --- | -| `README.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Useful public entry point; protected-main tenant lifecycle guidance already reflects trusted tenant scope and standalone compatibility. It is not the sole product/architecture authority and must not absorb transient PR/check state. | Keep beginner-readable; update only when a bounded shipped capability actually changes after merge. | -| `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, direct-SQL trust limits, rollback considerations, interoperability, and verification, but does not by itself form a complete system/UML/data model. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file. | -| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states and cites deterministic protected-main tenant/standalone acceptance authority. | Validate exact-head docs gates and integrate only through normal PR governance. | -| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries, tenant pre-effect validation, standalone recorder compatibility, and release/testing invariants are explicit. | Validate exact-head docs gates and integrate only through normal PR governance. | -| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. ADR 0002 already records trusted tenant identity, explicit `standalone` compatibility, RLS/direct-SQL role boundaries, migration, and rollback. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. | -| `docs/adr/README.md` | ACTIVE-PR | This canonical branch now supplies the missing protected-main ADR navigation/status index, explains numbering gaps, separates ADR decision status from implementation status, and forbids implicit supersession. | Validate exact-head documentation gates and integrate only through normal PR governance. | +| `README.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Useful public entry point; protected-main tenant lifecycle guidance already reflects trusted tenant scope and standalone compatibility. It is not the sole product/architecture authority and must not absorb transient PR/check state. The newly integrated recovery-evidence primitives are not yet a complete operator recovery procedure. | Keep beginner-readable; add shipped recovery semantics only when adjacent writer ownership permits and do not present evidence primitives as a restorable-backup guarantee. | +| `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, direct-SQL trust limits, rollback considerations, interoperability, and verification, but does not by itself form a complete system/UML/data model. Recovery evidence has advanced on protected main without establishing end-to-end recovery semantics. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file, and update recovery architecture only through a non-conflicting governed documentation lane. | +| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states and cites deterministic protected-main tenant/standalone acceptance authority. | Reflect the integrated recovery-evidence primitives while keeping executable backup/restore and end-to-end recovery PARTIAL/ACTIVE-PR; validate exact-head docs gates and integrate only through normal PR governance. | +| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries, tenant pre-effect validation, standalone recorder compatibility, and release/testing invariants are explicit. | Add the bounded recovery-evidence technical boundary without promoting #208/#209; validate exact-head docs gates and integrate only through normal PR governance. | +| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. ADR 0002 already records trusted tenant identity, explicit `standalone` compatibility, RLS/direct-SQL role boundaries, migration, and rollback. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. No protected-main ADR is inferred for the newly integrated recovery-evidence primitives merely because code exists. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. Add or amend an ADR only if the recovery execution/rollback architecture becomes a durable decision requiring one. | +| `docs/adr/README.md` | ACTIVE-PR | This canonical branch supplies the missing protected-main ADR navigation/status index, explains numbering gaps, separates ADR decision status from implementation status, and forbids implicit supersession. | Keep the referenced protected-main tree current; do not invent a recovery ADR that is not integrated. | | `docs/result-streaming.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Describes bounded provider result streaming/checkpoint behavior; checkpoint persistence has a separate accepted ADR. | Update only when the integrated result-application contract changes the operator-facing boundary. | | `docs/remote-batch-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Serves as the topic-specific operator authority for durable lifecycle tenancy. It already specifies trusted scope selection, pre-effect validation, four-argument standalone recorder compatibility, tenant-qualified identity/conflict/read behavior, forced RLS/application-role limits, migration, direct-SQL impact, rollback, recovery, and deterministic verification. | Keep synchronized with tenant/RLS and lifecycle migrations; do not duplicate this contract into a competing operator file. | | `docs/doctoring/tenant-scoped-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Supporting evidence independently records pre-effect validation, explicit standalone compatibility, database role/RLS limits, migration and rollback, plus current primary references. | Keep supporting references current; do not promote doctoring into the sole product authority. | | `CHANGELOG.md` tenant lifecycle entry | IMPLEMENTED-ON-PROTECTED-MAIN | The protected-main Unreleased section already records tenant-scoped durable lifecycle, forced default-deny RLS, standalone compatibility, and the atomic tenant migration fix. | Change only when integrated shipped behavior changes; do not log active-PR claims as shipped. | | Topic-specific doctoring/reference docs | IMPLEMENTED-ON-PROTECTED-MAIN | Useful deep evidence exists, but doctoring material is supporting evidence, not the canonical product-status authority. | Keep primary-reference and APA-style citations current where material. | +| PostgreSQL recovery evidence | IMPLEMENTED-ON-PROTECTED-MAIN / end-to-end PARTIAL | `postgres_recovery_receipt.py`, `postgres_backup_evidence.py`, and `postgres_schema_evidence.py` are integrated protected-main primitives with focused tests. They produce bounded content-free metadata/hash/size evidence without executing SQL or claiming cluster parity. | Keep the evidence/guarantee distinction explicit. Do not claim restorable backup, isolated restore, PITR, RPO/RTO/HA/DR, or compliance readiness from these primitives alone. | +| PostgreSQL logical backup/restore execution | ACTIVE-PR | #208 proposes bounded `pg_dump` execution and #209 proposes bounded direct `pg_restore` execution. Neither is protected-main truth. #209 also introduces a direct-SQL/rollback contract that requires permanent operator/architecture/ADR/doctoring/CHANGELOG documentation once writer ownership is coordinated. | Do not race those source branches or adjacent documentation writers. Promote only after exact-head gates, valid findings, required documentation, qualifying approval, and protected-main integration. | | Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation for retiring legacy `http` / `pg_cron`; it is a different contract from the already-shipped tenant lifecycle migration and is not protected-main truth. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. | | OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. | | Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. | | Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay; FR-4/end-to-end result application stays `PARTIAL` on protected main, and neither this row nor the TRD content-fidelity invariant promotes #194 into shipped truth. | Keep FR-4 and `docs/TRACEABILITY.md` as the validation/status authority until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. | | Runtime config/schema provisioning separation | ACTIVE-PR | PR #193 is a current-main least-privilege candidate: runtime constructors move away from DDL/default seeding, validate the search-path-resolved base-table shape and current-role read capability through bounded catalog probes, and retain provisioning/default seeding at the explicit schema boundary. It is not shipped until normal governance integrates it. | Preserve the trusted-search-path/non-ownership caveat and reacquire exact-head checks/review after every push; do not transfer predecessor or earlier-head evidence. | -| Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs. | +| Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map and now distinguishes integrated recovery evidence from active executable recovery candidates. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs. | | Threat model | PLANNED | No canonical protected-main `docs/THREAT_MODEL.md` is present. Security rules are distributed across AGENTS, architecture, ADRs, tests, and issue/PR evidence. | Add a threat model that distinguishes assets, trust boundaries, attacker capabilities, mitigations, residual risk, and non-guarantees without certification claims. | | Data governance | PLANNED | No single canonical data-governance document currently maps data classes, retention/ownership, tenant authority, privacy exposure, backup/restore, and deletion limits. | Add a protected-main-grounded data-governance contract. | | UML/component/sequence views | PLANNED | Root architecture is prose-first; no canonical UML authority is present. | Add textual Mermaid/PlantUML-compatible component and critical sequence diagrams after active source contracts stabilize. | | ERD / schema model | PLANNED | SQL and schema tests are authoritative, but no canonical ERD summarizes package-owned identities, tenancy, checkpoint state, and relationships. | Generate/maintain an ERD from protected-main schema; label migration-only/active-PR objects separately. | -| General standalone operator guide | PARTIAL | Protected main has strong operational guidance across `README.md` and topic authorities such as `docs/remote-batch-lifecycle.md`; there is no single omnibus protected-main `docs/OPERABILITY.md`. This is a navigation/consolidation gap, not absence of tenant lifecycle operator guidance. | Establish one general operator index/authority only after checking adjacent writers; do not race #184's active migration-specific operability work. | +| General standalone operator guide | PARTIAL | Protected main has strong operational guidance across `README.md` and topic authorities such as `docs/remote-batch-lifecycle.md`; there is no single omnibus protected-main `docs/OPERABILITY.md`. This is a navigation/consolidation gap, not absence of tenant lifecycle operator guidance. | Establish one general operator index/authority only after checking adjacent writers; do not race #184 or #209 documentation surfaces. | | Release governance | PARTIAL | Release-evidence ADRs/tests/workflows are strong, but the canonical graph lacks one concise protected-main release/operator authority tying versioning, SBOM, provenance, artifact identity, rollback/recovery, and publication verification together. | Add a stable release contract after checking active release writers; do not copy workflow-run IDs. | | Licensing / third-party notices | PARTIAL | Apache-2.0 source headers and repository licensing exist, but acquisition diligence should explicitly trace package license, dependency/SBOM evidence, and third-party notice process. | Evaluate a concise licensing/compliance evidence index without claiming legal certification. | @@ -48,6 +52,8 @@ The tenant lifecycle material in this canonical cohort is a reconstruction of al - PostgreSQL RLS is defense in depth, not host authentication/authorization, SQL-injection prevention, or correct identity mapping; production application roles for the tenant lifecycle boundary are `NOSUPERUSER NOBYPASSRLS`. - Provider/model content never becomes tenant, credential, endpoint, filesystem, or database authority. - Content-fidelity invariants constrain any result-application path that exists, but they do not prove that end-to-end result application is shipped; FR-4 and `docs/TRACEABILITY.md` remain the capability-status authority while protected main is `PARTIAL`. +- Bounded recovery receipts, artifact hashes, and packaged-schema hashes are evidence primitives, not proof that a backup is restorable or that a stated recovery objective is met. +- A future direct logical restore contract must keep caller-owned source trust, target isolation, allowed libpq environment, transaction rollback behavior, and post-restore acceptance explicit; those active-PR semantics are not shipped until integrated. - Distributed exactly-once processing is never implied by PostgreSQL checkpoint atomicity alone. - Security, privacy, SOC 2, and CSAP material is evidence-readiness documentation only unless an external certification actually exists. - Database object names, migration/rollback behavior, exact owned coverage requirements, Python-version acceptance, packaging/SBOM/provenance expectations, and bounded diagnostics must remain synchronized with repository tests and live governance. From e5901d177e345601a93ca1a86dd3e7cea41ea7b4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 17 Aug 2026 00:05:06 +0900 Subject: [PATCH 25/29] docs(recovery): align product contract with protected main --- docs/product/PRD.md | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/docs/product/PRD.md b/docs/product/PRD.md index 5fac473bb..47ef86feb 100644 --- a/docs/product/PRD.md +++ b/docs/product/PRD.md @@ -33,6 +33,7 @@ The product must let a qualified host: - submit, poll, wait, cancel, and retrieve through validated OpenAI-compatible Batch API clients; - operate in exact `standalone` mode or with a trusted tenant-qualified lifecycle identity; - recover or reconcile provider lifecycle state without introducing a second database-side networking authority; +- derive bounded content-free PostgreSQL recovery evidence for backup artifacts and the packaged schema without treating evidence as proof of restorability; and - prove package quality, security, reproducibility, release-artifact identity, and rollback assumptions through repository evidence. ## Protected-main capability contract @@ -47,6 +48,12 @@ The product must let a qualified host: | Durable resumable result checkpoint/CAS storage | IMPLEMENTED-ON-PROTECTED-MAIN | PostgreSQL supplies tenant-qualified durable checkpoint authority and conflict detection; this is not a distributed exactly-once claim. | | Redacted readiness reporting | IMPLEMENTED-ON-PROTECTED-MAIN | Public readiness evidence does not disclose arbitrary lower-layer diagnostic content. | | Scheduler-independent bounded provider reconciliation primitive | IMPLEMENTED-ON-PROTECTED-MAIN | A host can submit a finite, validated candidate set for polling/retrieval through the existing bounded provider client; candidate discovery, scheduling, and cross-process lease ownership remain outside this primitive. | +| Bounded PostgreSQL recovery receipt | IMPLEMENTED-ON-PROTECTED-MAIN | The package can encode/decode deterministic bounded content-free metadata that identifies package/source/PostgreSQL/schema/backup evidence without carrying credentials, DSNs, SQL, business payloads, or arbitrary diagnostics. | +| Bounded PostgreSQL backup-artifact integrity evidence | IMPLEMENTED-ON-PROTECTED-MAIN | The package can derive SHA-256 and byte-size evidence from one private regular backup artifact under descriptor-pinned, no-follow, finite-work constraints without executing backup or restore. | +| Bounded packaged PostgreSQL schema evidence | IMPLEMENTED-ON-PROTECTED-MAIN | The package can derive SHA-256 and byte-size evidence from the exact distributed `schema.sql` resource under a finite package-owned work budget without executing SQL or asserting live-cluster parity. | +| PostgreSQL logical backup execution | ACTIVE-PR | A `pg_dump` candidate exists in #208 but is not shipped; protected main must not be described as creating a restorable backup from the evidence primitives alone. | +| PostgreSQL logical restore execution | ACTIVE-PR | A direct `pg_restore` candidate exists in #209 but is not shipped; its caller-owned source trust, target-isolation responsibility, libpq allowlist, transactional failure boundary, and archive-integrity contract remain active-PR semantics. | +| End-to-end PostgreSQL recovery readiness | PARTIAL | Integrated evidence primitives do not yet prove an isolated restore with schema/RLS/constraint/extension parity, migration compatibility, external key/config custody, physical/WAL/PITR recovery, or a stated RPO/RTO/HA/DR objective. | | Durable reconciliation candidate discovery | ACTIVE-PR | Discovery must be tenant-qualified, bounded, deterministic, and database-authoritative before it can become product truth. | | Cross-process reconciliation single-flight | ACTIVE-PR | Concurrent workers must not race the same tenant/provider identity; merge eligibility remains governed by live repository policy. | | Existing-volume legacy `http` / `pg_cron` retirement | ACTIVE-PR | Existing deployments need fail-closed, reversible migration evidence before compatibility packages can be removed. | @@ -76,14 +83,18 @@ Standalone source compatibility is also normative. `DurableBatchAPIClient` retai Protected-main acceptance authority for these invariants is deterministic: `tests/test_tenant_durable_client.py` proves construction-time pre-effect tenant validation and the four-argument standalone recorder seam; `tests/test_tenant_lifecycle_persistence.py` proves explicit `standalone` delegation, tenant-qualified conflict targets and reads, malformed-scope rejection before database access, and distinct tenant identities; `pg_llm_batch/schema.sql` supplies the tenant-qualified unique key, forced RLS policy, and tenant-qualified operational status index. `docs/remote-batch-lifecycle.md`, ADR 0002, and `docs/doctoring/tenant-scoped-lifecycle.md` describe the same protected-main contract. -### FR-4: Reconciliation and recovery +### FR-4: Reconciliation and provider-effect recovery Reconciliation shall be finite, deterministic, payload-free in its operational evidence, and use the same validated provider client boundary as normal operations. Candidate discovery, concurrency control, scheduling, and result-application semantics must be explicit capabilities rather than inferred from polling code. Provider-success/database-failure cases must remain observable recovery states rather than being rewritten as if the provider effect never occurred. -### FR-5: Persistence integrity +### FR-5: Persistence integrity and PostgreSQL recovery evidence Package-owned database rows shall use descriptive two-or-more-word `snake_case` object names where applicable, explicit durable identities, parameterized SQL, and migrations that are idempotent or have a documented one-way boundary. Schema copies maintained for package and Docker initialization shall remain synchronized where the repository contract requires it. Malformed durable payload/state shall fail closed before downstream credential or provider effects when correctness depends on that state. +Protected main shall provide bounded, content-free recovery evidence without overstating its guarantee. `PostgresRecoveryReceipt` identifies one evidence set using package version, exact source commit, PostgreSQL major version, schema SHA-256, a reviewed backup-method vocabulary, backup-artifact SHA-256 and size, and bounded start/completion epochs. Backup-artifact inspection shall pin directory/file identity, reject symlink traversal and unsafe file shapes, remain within a caller-visible finite hashing budget, and return only hash/size evidence. Packaged-schema inspection shall stream the exact distributed `schema.sql` under a finite package-owned budget and return only hash/size evidence. Malformed, ambiguous, duplicate-member, hostile-subclass, oversized, mutating, unreadable, or cleanup-failing evidence shall fail closed through fixed package diagnostics. + +These primitives do not execute `pg_dump` or `pg_restore`, persist backup bytes, prove backup provenance or restorability, prove a live-cluster schema matches the package, authorize a tenant/operator, manage encryption keys, manage WAL, establish PITR, or prove RPO/RTO/HA/DR/compliance. Executable logical backup/restore and isolated-restore acceptance remain separate governed capabilities. A future direct restore must make caller-owned source-superuser trust, target isolation, allowed libpq credential/service environment, transaction rollback behavior, and post-restore acceptance explicit before it can become protected-main truth. + ### FR-6: Configuration and secrets PostgreSQL-backed configuration and encrypted-secret support remain available for standalone use. Environment variables are bootstrap transport only where explicitly documented. Embedding hosts may inject credential providers without changing provider protocol semantics. The package shall not invent an external secret-management product dependency. @@ -110,11 +121,11 @@ The package must not apply blanket masking or lossy transformation to authorized ### Reliability -Network, response, retry, wait, candidate-scan, payload, and release-evidence operations must be explicitly bounded. Recovery and rollback must be documented before migrations or release changes are considered complete. Queued, skipped, cancelled, absent, stale, predecessor-head, synthetic-merge-only, or infrastructure-failed evidence is not success for an exact source head. +Network, response, retry, wait, candidate-scan, payload, recovery-evidence, and release-evidence operations must be explicitly bounded. Recovery receipts and hashes are integrity/identity evidence, not restoration success. Recovery and rollback must be documented before migrations or release changes are considered complete. Queued, skipped, cancelled, absent, stale, predecessor-head, synthetic-merge-only, or infrastructure-failed evidence is not success for an exact source head. ### Interoperability -Provider interaction stays OpenAI-Batch-compatible behind a validated Python client seam. Embedding hosts can provide credentials and control-plane context without changing the package's durable/provider semantics. +Provider interaction stays OpenAI-Batch-compatible behind a validated Python client seam. Embedding hosts can provide credentials and control-plane context without changing the package's durable/provider semantics. PostgreSQL backup/restore executors, when integrated, must remain caller-targeted infrastructure seams rather than hidden dependencies on a specific ContextualWisdomLab host. ### Packaging and release @@ -127,10 +138,11 @@ Dependencies must be locked/reproducible according to repository policy. Release - Making provider payload/model output an authority for tenant or endpoint selection. - Providing an unbounded general-purpose HTTP proxy. - Reintroducing provider networking or independent scheduling inside PostgreSQL. +- Claiming a backup is restorable, a live cluster matches packaged schema, or a recovery objective is met from receipt/hash evidence alone. - Claiming distributed exactly-once processing without an integrated transaction/recovery contract spanning every external effect. - Requiring a specific ContextualWisdomLab host service for standalone operation. - Claiming SOC 2, CSAP, or other certification solely from repository controls. ## Product acceptance boundary -A product capability moves to **IMPLEMENTED-ON-PROTECTED-MAIN** only after its unchanged source has satisfied the live ruleset, required exact-head CI/security/coverage/package/provenance/release checks, valid review findings are resolved, and the resulting tree is integrated into the protected default branch. Documentation must then be updated to move the capability out of `ACTIVE-PR`, `PARTIAL`, or `PLANNED`; historical evidence must not be transferred as if it were proof for a different head. \ No newline at end of file +A product capability moves to **IMPLEMENTED-ON-PROTECTED-MAIN** only after its unchanged source has satisfied the live ruleset, required exact-head CI/security/coverage/package/provenance/release checks, valid review findings are resolved, and the resulting tree is integrated into the protected default branch. Documentation must then be updated to move the capability out of `ACTIVE-PR`, `PARTIAL`, or `PLANNED`; historical evidence must not be transferred as if it were proof for a different head. From ad761d336f113b2ac8eb879e9402975ac6af2623 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 17 Aug 2026 00:06:31 +0900 Subject: [PATCH 26/29] docs(recovery): align technical contract with protected main --- docs/product/TRD.md | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/docs/product/TRD.md b/docs/product/TRD.md index 918ec518a..3449661f1 100644 --- a/docs/product/TRD.md +++ b/docs/product/TRD.md @@ -8,7 +8,7 @@ Status vocabulary is shared with the PRD: **IMPLEMENTED-ON-PROTECTED-MAIN**, **A ## System boundary -`pg-llm-batch` is a Python package plus PostgreSQL schema and container assets. It may run standalone or be embedded by another service. The package owns batch preparation, package persistence, validated provider Batch API access, durable lifecycle projection, resumable checkpoint storage, and bounded reconciliation primitives. It does not own host authentication, business authorization, ingress/WAF, infrastructure TLS policy, external secret-manager choice, global OpenTelemetry configuration, or a cross-system distributed transaction. +`pg-llm-batch` is a Python package plus PostgreSQL schema and container assets. It may run standalone or be embedded by another service. The package owns batch preparation, package persistence, validated provider Batch API access, durable lifecycle projection, resumable checkpoint storage, bounded reconciliation primitives, and bounded content-free PostgreSQL recovery-evidence primitives. It does not own host authentication, business authorization, ingress/WAF, infrastructure TLS policy, external secret-manager choice, global OpenTelemetry configuration, backup-storage infrastructure, WAL/archive infrastructure, or a cross-system distributed transaction. ## Component contract @@ -21,6 +21,9 @@ Status vocabulary is shared with the PRD: **IMPLEMENTED-ON-PROTECTED-MAIN**, **A | `db.py` | schema application and parameterized persistence/read helpers, including tenant-qualified lifecycle context, conflict identity, and exact-row lookup | database-side provider networking, authenticating tenant callers | | `checkpoint_store.py` | tenant-qualified PostgreSQL checkpoint/CAS operations and conflict semantics | distributed exactly-once claims | | `reconciliation.py` | finite host-selected polling/retrieval pass using the existing validated client surface | candidate discovery, scheduling, cross-process leasing | +| `postgres_recovery_receipt.py` | encode/decode one deterministic bounded content-free PostgreSQL recovery evidence receipt | proving backup success/restorability, authenticating operators, carrying DSNs/credentials/business content | +| `postgres_backup_evidence.py` | derive SHA-256 and byte-size evidence from one private regular backup artifact through descriptor-pinned no-follow traversal and finite work | executing `pg_dump`/`pg_restore`, persisting backup bytes, proving restore semantics | +| `postgres_schema_evidence.py` | derive SHA-256 and byte-size evidence from the exact distributed `pg_llm_batch/schema.sql` resource under a finite package-owned budget | executing SQL, proving live-cluster parity or migration currency | | `config.py` | PostgreSQL-backed configuration and encrypted secret storage for standalone composition | prescribing an embedding host's external secret manager | | `observability.py` | opt-in bounded traces/metrics around reviewed operations | configuring global SDK/exporter/resource policy | | `health.py` | readiness aggregation and redacted public health response | general-purpose web serving or arbitrary diagnostic reflection | @@ -28,6 +31,8 @@ Status vocabulary is shared with the PRD: **IMPLEMENTED-ON-PROTECTED-MAIN**, **A The protected-main validation authority for the `durable_client.py` compatibility row is `pg_llm_batch/durable_client.py`, `tests/test_tenant_durable_client.py`, and `tests/test_tenant_lifecycle_persistence.py`: the tests prove construction-time tenant rejection before downstream effects, exact four-argument standalone recorder invocation, explicit `standalone` persistence/read delegation, and tenant-qualified lifecycle persistence/read identities. +The protected-main validation authority for recovery evidence is the three recovery modules plus their focused test suites. Historical merged PRs #205, #206, and #207 are integration evidence, not runtime authority. `docs/TRACEABILITY.md` remains the canonical status map for the distinction between integrated evidence primitives and active backup/restore execution candidates. + ## Runtime architecture ### Batch preparation @@ -77,6 +82,18 @@ Protected main supplies a scheduler-independent `reconcile_batch_candidates(...) The host still owns candidate discovery, tenant authorization, scheduling, and cross-process concurrency. Durable candidate discovery and tenant-qualified advisory single-flight are **ACTIVE-PR** surfaces; neither is treated as shipped until protected-main integration. Package-owned autonomous scheduling, crash/restart completion semantics, terminal-work retirement after durable result application, and an end-to-end exactly-once worker remain **PARTIAL** or **PLANNED** capabilities. +### PostgreSQL recovery evidence + +Protected main supplies three deliberately non-executing recovery-evidence primitives. + +1. `PostgresRecoveryReceipt` binds exact built-in primitive metadata for package version, source commit, PostgreSQL major, packaged-schema SHA-256, reviewed backup-method vocabulary (`logical`, `physical`, or `pitr`), backup artifact SHA-256/size, and bounded timestamps. Its JSON representation is deterministic and size-bounded, rejects duplicate/unknown fields and hostile subclasses, and maps ordinary malformed input/decoder failures to fixed content-free diagnostics. +2. `inspect_postgres_backup_artifact(...)` traverses path components through pinned directory descriptors with no-follow semantics, rejects `..`, symlinked parents/final components, non-regular/empty/oversized files and unsafe link counts/permissions as defined by the implementation contract, hashes under an explicit finite maximum-size work budget, bounds each read request by remaining budget, compares descriptor identity/metadata before and after hashing, and treats cleanup failures as bounded evidence without masking an already-selected primary error. +3. `inspect_postgres_schema()` streams the exact distributed `pg_llm_batch/schema.sql` resource through SHA-256 under a finite package-owned work budget and returns only SHA-256 plus byte size. Missing, unreadable, empty, oversized, malformed-chunk, hostile-subclass, or cleanup-failing resources fail closed through fixed content-free diagnostics. + +These primitives do not execute SQL or database mutation. They do not prove the backup command succeeded, prove backup provenance beyond caller-controlled receipt fields, prove restorability, prove a live database matches the packaged schema, provide target isolation, manage keys/secrets, manage physical/WAL/PITR infrastructure, or establish RPO/RTO/HA/DR/compliance. Those are separate acceptance domains. + +Logical `pg_dump` execution in #208 and direct `pg_restore` execution in #209 remain **ACTIVE-PR**. Until integration, no protected-main technical contract may rely on those executors. The direct-restore candidate additionally requires permanent operator/architecture/ADR/doctoring/CHANGELOG coverage for caller-owned source-superuser trust, the non-authorizing service selector, permitted inherited libpq variables, single-transaction rollback behavior, target isolation, and post-restore acceptance before it may be represented as shipped. + ## Persistence requirements ### Naming and schema ownership @@ -93,12 +110,16 @@ Package-owned persisted virtual JSONL is canonical state, not a best-effort cach Durable provider/resource identifiers and tenant identities are validated before they become persistence or authorization inputs. Database/query failures must not be silently reclassified as an authoritative no-row result when correctness depends on distinguishing those cases. +Recovery evidence is identity/integrity metadata, not a second persistence authority. The package does not own backup storage, replica lifecycle, object-store retention, WAL archives, encryption-at-rest infrastructure, or backup deletion merely because it can hash an operator-selected artifact. Operators/deployments must keep those responsibilities explicit. + ## Security and privacy requirements ### Secrets Standalone provider configuration and encrypted secrets are PostgreSQL-backed. Environment variables are limited to explicitly documented bootstrap transport such as the database DSN and optional encryption key. CLI secret entry uses no-echo prompting or bounded standard input rather than plaintext process arguments. Embedding hosts may supply another credential provider through the supported seam. +Recovery evidence must not carry DSNs, passwords, Fernet keys, prompts/results, ciphertext, arbitrary SQL, provider payloads, paths where not required by the callable interface, dynamic exception names, or reflected lower-layer diagnostics. A future backup/restore executor must isolate credentials from process arguments and ambient environment according to its separately reviewed contract. + ### Authorized content fidelity The package does not gain authority to mask, tokenize away, truncate for privacy, or otherwise alter an authorized prompt, request, JSONL record, or provider result merely because the content may contain PII. Silent transformation would change token counts, provider semantics, persisted evidence, replay behavior, and downstream business meaning. Serialization, token accounting, persistence, upload, and retrieval paths implemented on protected main therefore preserve authorized business content unless an explicit reviewed feature contract says otherwise. The same content-fidelity invariant constrains any result-application path that exists, but end-to-end result application remains **PARTIAL** under FR-4 and `docs/TRACEABILITY.md`; PR #194 is an **ACTIVE-PR** transaction-seam candidate, not protected-main proof of a completed result-application capability. @@ -117,6 +138,10 @@ RLS augments a trusted host authorization boundary; it is not itself authenticat Provider URLs, statuses, IDs, headers, JSON, JSONL, retry guidance, and metadata are untrusted external input. Validation occurs before the downstream effect that relies on that value. Model/provider output never grants tenant, endpoint, credential, or filesystem authority. +### Recovery authority boundary + +A backup artifact, receipt, service selector, schema hash, or backup method string is not authorization to read, write, restore, or replace a database. Authentication, authorization, target isolation, backup custody, key custody, destructive-operation approval, and recovery-objective ownership remain external until a separately integrated contract explicitly supplies and verifies them. Recovery evidence must never be used to infer those authorities. + ## Observability requirements OpenTelemetry support is opt-in. Base installations must not require OpenTelemetry packages solely to use normal batch functionality. Operation names, outcomes, and error categories are finite. Telemetry attributes exclude endpoint aliases, provider URLs, resource identifiers, credentials, metadata, prompts, and provider response bodies. @@ -129,10 +154,14 @@ Readiness covers the required PostgreSQL/tokenizer/configuration boundary and re Operational migrations require backup/preflight/acceptance/recovery documentation before release. A recovery instruction must preserve package and operator-owned state; destructive `CASCADE`, hidden history rewrite, or deletion of unknown operator objects is not an acceptable shortcut. +A recovery drill must distinguish artifact identity from restore acceptance. At minimum, acceptance criteria for a future end-to-end logical restore must address exact schema/package identity, required schema/RLS/constraint/extension behavior, migration compatibility, intended target isolation, credential/key availability, and rollback/recovery behavior. Physical/WAL/PITR drills additionally require their own timeline/target and infrastructure acceptance criteria. No repository evidence should claim universal RPO/RTO/HA/DR without an explicit measured deployment objective. + ## Concurrency requirements Batch preparation uses database coordination/transactionality appropriate to its package-owned state. Durable lifecycle writes use explicit ordering/conflict semantics. Any cross-process reconciliation exclusion must be tenant-qualified, non-blocking or finitely bounded, exception-safe, and clear about whether it is transient session state or durable lease state. Session advisory locking must never be promoted into a durable lease or distributed exactly-once claim. +Recovery-evidence file inspection must remain finite and fail closed under concurrent mutation. A successful hash/size result is valid only for the descriptor identity/metadata contract that was revalidated by the implementation; it is not a lock or lease over external backup infrastructure. + ## Testing requirements Every source defect follows realistic RED → narrow fix → GREEN → focused/full validation. Required repository evidence includes: @@ -144,6 +173,7 @@ Every source defect follows realistic RED → narrow fix → GREEN → focused/f - realistic PostgreSQL integration for SQL/RLS/migration/concurrency behavior; - package and container installation/health validation; - migration idempotency and rollback/recovery coverage where applicable; +- recovery-evidence tests for exact primitive types, duplicate/unknown metadata, finite work, descriptor/path boundaries, concurrent mutation, cleanup failure, and content-free diagnostics; - confidentiality regressions that inspect full exception/traceback surfaces when relevant; - security scanning and SAST; - dependency-lock and packaging reproducibility; @@ -155,6 +185,10 @@ Queued, pending, skipped, cancelled, absent, neutral, stale, predecessor-head, s A release may originate only from the exact integrated protected head after all live required quality, security, review, migration, rollback/recovery, operational, packaging, provenance, and release-acceptance gates are terminal-success. Versioning and CHANGELOG updates are followed by publication and artifact verification. Repository control evidence may support SOC 2 / CSAP readiness but must not be represented as certification. +The presence of recovery-evidence primitives does not make a release recovery-ready for a deployment. A release or operator contract that claims restore/PITR/RPO/RTO/HA/DR readiness must cite the exact integrated executor/drill/acceptance evidence for that deployment objective rather than extrapolating from hash/receipt modules. + ## Documentation requirements Canonical product/technical/architecture/ADR/UML/ERD/security/operability/release/data-governance/traceability documents must distinguish shipped protected-main state from active or planned work. Durable documents should describe contracts rather than transient check-run IDs. When an active capability merges, the canonical graph is updated in the same governance model rather than relying on stale PR body claims. + +Direct-SQL or rollback authority introduced by a future backup/restore executor requires coordinated permanent README, operator, architecture, ADR (or explicit accepted amendment/no-new-decision rationale), doctoring/reference, and CHANGELOG coverage. Active PR source may document its own callable boundary, but canonical docs must not represent it as shipped before integration. From 458a1056ac6dbaad7e5ad61e9924ec29848fb64f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 17 Aug 2026 00:07:16 +0900 Subject: [PATCH 27/29] docs(recovery): refresh ADR index authority and recovery boundary --- docs/adr/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/adr/README.md b/docs/adr/README.md index d169ab9a0..d9f677522 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -4,7 +4,7 @@ This index is a navigation and status aid for architecture decision records that exist on protected `main`. The decision record itself remains the normative source for its context, decision, consequences, security boundary, and supersession rules. -The protected-main tree used to reconstruct this index is `d0a4b30be1f46536e352443309f3a35533156767`. A pull request, historical branch, generated merge commit, workflow run, review comment, or proposed file that is not on protected main is not silently promoted into this index. +The protected-main tree used to reconstruct this index is `d2f1e32271910a6db98a0757d67194ddadca4566`. A pull request, historical branch, generated merge commit, workflow run, review comment, or proposed file that is not on protected main is not silently promoted into this index. ## Protected-main decisions @@ -17,6 +17,8 @@ The protected-main tree used to reconstruct this index is `d0a4b30be1f46536e3524 | [0007](0007-durable-result-checkpoint-store.md) | Durable tenant-isolated result checkpoint store | Accepted | Adds optional PostgreSQL persistence, tenant isolation, compare-and-swap concurrency, and a caller-owned transaction seam without claiming distributed exactly-once delivery. Depends on ADR 0006. | | [0015](0015-http-425-too-early-retry.md) | HTTP 425 retry for bounded idempotent GETs | Accepted for the bounded retry slice | Keeps the default GET retry-status set closed, side-effecting POSTs single-attempt, and TLS/certificate/fingerprint failures outside automatic retry. | +Protected main also contains bounded PostgreSQL recovery-evidence primitives in `postgres_recovery_receipt.py`, `postgres_backup_evidence.py`, and `postgres_schema_evidence.py`. Their integration does **not** create an implicit ADR. They are implementation/evidence contracts recorded by the PRD/TRD/traceability map, while executable logical backup/restore and end-to-end recovery remain active/partial work. If a future restore executor introduces a durable architectural decision about direct SQL authority, target isolation, source-superuser trust, libpq credential/environment handling, or rollback/acceptance semantics, that decision must be added or explicitly amended through normal ADR governance rather than inferred from the implementation. + ## Numbering and missing identifiers ADR numbers are stable identifiers, not a promise of contiguous numbering. The absence of `0001`, `0005`, or `0008` through `0014` from the protected-main directory does **not** mean those decisions are missing, rejected, accepted elsewhere, or safe to reconstruct from old branches. Gaps may reflect historical work, superseded proposals, unmerged branches, or reserved identifiers. Only a reviewed protected-main document may establish the status of a missing number. @@ -46,4 +48,4 @@ For a material architecture change: ## Non-guarantees that must remain visible -The current ADR set does not establish authentication from PostgreSQL RLS alone, provider authenticity from result checkpointing, full-stream immutability from a prefix checkpoint, distributed exactly-once processing, retry permission for arbitrary HTTP failures, or organizational security/compliance certification. Those boundaries must not be weakened by summaries, operator docs, marketing material, or future ADR titles. \ No newline at end of file +The current ADR set does not establish authentication from PostgreSQL RLS alone, provider authenticity from result checkpointing, full-stream immutability from a prefix checkpoint, distributed exactly-once processing, retry permission for arbitrary HTTP failures, backup restorability or PITR/RPO/RTO/HA/DR from recovery receipts/hashes alone, or organizational security/compliance certification. Those boundaries must not be weakened by summaries, operator docs, marketing material, or future ADR titles. From 89f4e52df9e36c6e9af6576b995f00ae5ded60c6 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 17 Aug 2026 00:14:34 +0900 Subject: [PATCH 28/29] docs(recovery): mark seek-safe restore successor --- docs/DOCUMENTATION_FITNESS.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/DOCUMENTATION_FITNESS.md b/docs/DOCUMENTATION_FITNESS.md index 083d1bd5b..2ef97bca1 100644 --- a/docs/DOCUMENTATION_FITNESS.md +++ b/docs/DOCUMENTATION_FITNESS.md @@ -8,7 +8,7 @@ A document is *fit* only when it states the correct authority boundary, is consi The tenant lifecycle material in this canonical cohort is a reconstruction of already integrated protected-main behavior, not a new runtime tenant/RLS/migration contract. Companion-document verification against protected main confirms the same contract is already present in the public `README.md`, root `ARCHITECTURE.md`, topic operator guide `docs/remote-batch-lifecycle.md`, accepted ADR 0002, `docs/doctoring/tenant-scoped-lifecycle.md`, and `CHANGELOG.md`. Those authorities already cover trusted tenant identity, `NOSUPERUSER NOBYPASSRLS`, transaction-local/forced RLS, direct-SQL limits, legacy-to-`standalone` migration, and rollback constraints. The canonical PRD/TRD therefore may cite that existing evidence without rewriting those protected-main files or racing PR #184's separate legacy-extension-retirement documentation lane. -Protected main now also contains the bounded recovery-evidence primitives integrated by #205, #206, and #207: content-free PostgreSQL recovery receipts, descriptor-pinned backup-artifact hash/size evidence, and packaged-schema hash/size evidence. These are evidence primitives only. They do not make executable logical backup/restore, isolated restore acceptance, WAL/PITR, key/config custody, or any RPO/RTO/HA/DR objective shipped; #208 and #209 remain active source candidates and Issue #204 remains the end-to-end recovery authority. +Protected main now also contains the bounded recovery-evidence primitives integrated by #205, #206, and #207: content-free PostgreSQL recovery receipts, descriptor-pinned backup-artifact hash/size evidence, and packaged-schema hash/size evidence. These are evidence primitives only. They do not make executable logical backup/restore, isolated restore acceptance, WAL/PITR, key/config custody, or any RPO/RTO/HA/DR objective shipped. #208 remains the active logical-backup candidate. Direct-restore predecessor #209 is not mergeable as a product contract because its EOF-consumption postcondition conflicts with seekable PostgreSQL custom archives; Draft #212 is the active successor and Issue #204 remains the end-to-end recovery authority. ## Current fitness matrix @@ -16,28 +16,28 @@ Protected main now also contains the bounded recovery-evidence primitives integr | --- | --- | --- | --- | | `README.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Useful public entry point; protected-main tenant lifecycle guidance already reflects trusted tenant scope and standalone compatibility. It is not the sole product/architecture authority and must not absorb transient PR/check state. The newly integrated recovery-evidence primitives are not yet a complete operator recovery procedure. | Keep beginner-readable; add shipped recovery semantics only when adjacent writer ownership permits and do not present evidence primitives as a restorable-backup guarantee. | | `ARCHITECTURE.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Covers standalone/embedded deployment, durable tenancy/RLS, migration boundary, direct-SQL trust limits, rollback considerations, interoperability, and verification, but does not by itself form a complete system/UML/data model. Recovery evidence has advanced on protected main without establishing end-to-end recovery semantics. | Keep architectural contracts synchronized; add separate UML/data views rather than overloading this file, and update recovery architecture only through a non-conflicting governed documentation lane. | -| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states and cites deterministic protected-main tenant/standalone acceptance authority. | Reflect the integrated recovery-evidence primitives while keeping executable backup/restore and end-to-end recovery PARTIAL/ACTIVE-PR; validate exact-head docs gates and integrate only through normal PR governance. | -| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries, tenant pre-effect validation, standalone recorder compatibility, and release/testing invariants are explicit. | Add the bounded recovery-evidence technical boundary without promoting #208/#209; validate exact-head docs gates and integrate only through normal PR governance. | -| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. ADR 0002 already records trusted tenant identity, explicit `standalone` compatibility, RLS/direct-SQL role boundaries, migration, and rollback. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. No protected-main ADR is inferred for the newly integrated recovery-evidence primitives merely because code exists. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. Add or amend an ADR only if the recovery execution/rollback architecture becomes a durable decision requiring one. | -| `docs/adr/README.md` | ACTIVE-PR | This canonical branch supplies the missing protected-main ADR navigation/status index, explains numbering gaps, separates ADR decision status from implementation status, and forbids implicit supersession. | Keep the referenced protected-main tree current; do not invent a recovery ADR that is not integrated. | +| `docs/product/PRD.md` | ACTIVE-PR | First current-main-compatible canonical product contract. It explicitly separates shipped, active, partial, planned, and superseded capability states and cites deterministic protected-main tenant/standalone acceptance authority. | Reflect the integrated recovery-evidence primitives while keeping executable backup/restore and end-to-end recovery PARTIAL/ACTIVE-PR; treat #209 as an unsafe predecessor and #212 as the Draft direct-restore successor until protected integration. | +| `docs/product/TRD.md` | ACTIVE-PR | First current-main-compatible technical requirements authority; component boundaries, tenant pre-effect validation, standalone recorder compatibility, and release/testing invariants are explicit. | Keep the bounded recovery-evidence technical boundary without promoting #208/#212; remove predecessor #209 from any implied merge path before canonical integration. | +| ADR set (`0002`, `0003`, `0004`, `0006`, `0007`, `0015`) | IMPLEMENTED-ON-PROTECTED-MAIN | Material tenant, release-evidence, result-checkpoint, and retry decisions exist. ADR 0002 already records trusted tenant identity, explicit `standalone` compatibility, RLS/direct-SQL role boundaries, migration, and rollback. Individual ADR status remains authoritative: `0003` and `0004` are still `Proposed`, while the accepted records keep their own bounded decision status. No protected-main ADR is inferred for the newly integrated recovery-evidence primitives merely because code exists. | Preserve those record-local statuses; do not infer architectural acceptance from related implementation. Draft #212 proposes ADR 0016 for custom-format restore seek semantics, but that record is not protected-main authority until integrated. | +| `docs/adr/README.md` | ACTIVE-PR | This canonical branch supplies the missing protected-main ADR navigation/status index, explains numbering gaps, separates ADR decision status from implementation status, and forbids implicit supersession. | Keep the referenced protected-main tree current; do not list #212's proposed ADR 0016 as a protected-main decision before integration. | | `docs/result-streaming.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Describes bounded provider result streaming/checkpoint behavior; checkpoint persistence has a separate accepted ADR. | Update only when the integrated result-application contract changes the operator-facing boundary. | | `docs/remote-batch-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Serves as the topic-specific operator authority for durable lifecycle tenancy. It already specifies trusted scope selection, pre-effect validation, four-argument standalone recorder compatibility, tenant-qualified identity/conflict/read behavior, forced RLS/application-role limits, migration, direct-SQL impact, rollback, recovery, and deterministic verification. | Keep synchronized with tenant/RLS and lifecycle migrations; do not duplicate this contract into a competing operator file. | | `docs/doctoring/tenant-scoped-lifecycle.md` | IMPLEMENTED-ON-PROTECTED-MAIN | Supporting evidence independently records pre-effect validation, explicit standalone compatibility, database role/RLS limits, migration and rollback, plus current primary references. | Keep supporting references current; do not promote doctoring into the sole product authority. | | `CHANGELOG.md` tenant lifecycle entry | IMPLEMENTED-ON-PROTECTED-MAIN | The protected-main Unreleased section already records tenant-scoped durable lifecycle, forced default-deny RLS, standalone compatibility, and the atomic tenant migration fix. | Change only when integrated shipped behavior changes; do not log active-PR claims as shipped. | | Topic-specific doctoring/reference docs | IMPLEMENTED-ON-PROTECTED-MAIN | Useful deep evidence exists, but doctoring material is supporting evidence, not the canonical product-status authority. | Keep primary-reference and APA-style citations current where material. | | PostgreSQL recovery evidence | IMPLEMENTED-ON-PROTECTED-MAIN / end-to-end PARTIAL | `postgres_recovery_receipt.py`, `postgres_backup_evidence.py`, and `postgres_schema_evidence.py` are integrated protected-main primitives with focused tests. They produce bounded content-free metadata/hash/size evidence without executing SQL or claiming cluster parity. | Keep the evidence/guarantee distinction explicit. Do not claim restorable backup, isolated restore, PITR, RPO/RTO/HA/DR, or compliance readiness from these primitives alone. | -| PostgreSQL logical backup/restore execution | ACTIVE-PR | #208 proposes bounded `pg_dump` execution and #209 proposes bounded direct `pg_restore` execution. Neither is protected-main truth. #209 also introduces a direct-SQL/rollback contract that requires permanent operator/architecture/ADR/doctoring/CHANGELOG documentation once writer ownership is coordinated. | Do not race those source branches or adjacent documentation writers. Promote only after exact-head gates, valid findings, required documentation, qualifying approval, and protected-main integration. | +| PostgreSQL logical backup/restore execution | ACTIVE-PR | #208 proposes bounded `pg_dump` execution. #209's direct-restore predecessor has a verified false-failure defect because a seekable custom-format `pg_restore` need not leave the shared descriptor at EOF after successful transactional restore. Draft #212 is the active successor: it removes that invalid EOF postcondition while preserving metadata-fingerprint verification and owns the permanent README/architecture/ADR/doctoring/CHANGELOG restore-contract documentation. None is protected-main truth. | Freeze #209 rather than merge or patch it in parallel; do not race #208/#212 active writers. Promote only the unchanged successor after exact-head gates, valid findings, required documentation, qualifying approval, and protected-main integration. | | Existing-volume legacy PostgreSQL retirement operability material | ACTIVE-PR | PR #184 contains bounded migration/operator documentation for retiring legacy `http` / `pg_cron`; it is a different contract from the already-shipped tenant lifecycle migration and is not protected-main truth. | Do not duplicate or rewrite #184-owned README/architecture/operability/retirement surfaces here. | | OpenTelemetry installation/operation documentation | ACTIVE-PR | PR #175 owns the packaging-extra lane; protected main must not imply the optional dependency lock is integrated. | Promote only after exact lock/materialization and release gates are proven. | | Durable reconciliation candidate/single-flight documentation | ACTIVE-PR | PRs #190 and #191 own current reconciliation additions; protected main has only the scheduler-independent bounded reconciliation primitive. | Update canonical shipped status only after each exact head merges. | | Atomic durable result application | ACTIVE-PR | PR #194 contains a current-main-compatible test-first caller-owned PostgreSQL transaction seam for applying one validated streamed result/error record together with checkpoint advancement. It remains an active overlay; FR-4/end-to-end result application stays `PARTIAL` on protected main, and neither this row nor the TRD content-fidelity invariant promotes #194 into shipped truth. | Keep FR-4 and `docs/TRACEABILITY.md` as the validation/status authority until #194 integrates normally; preserve the explicit same-transaction and no-distributed-exactly-once boundary. | | Runtime config/schema provisioning separation | ACTIVE-PR | PR #193 is a current-main least-privilege candidate: runtime constructors move away from DDL/default seeding, validate the search-path-resolved base-table shape and current-role read capability through bounded catalog probes, and retain provisioning/default seeding at the explicit schema boundary. It is not shipped until normal governance integrates it. | Preserve the trusted-search-path/non-ownership caveat and reacquire exact-head checks/review after every push; do not transfer predecessor or earlier-head evidence. | -| Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map and now distinguishes integrated recovery evidence from active executable recovery candidates. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs. | +| Canonical traceability | ACTIVE-PR | This branch is establishing the first current-main-compatible requirements-to-evidence map and distinguishes integrated recovery evidence from active executable candidates and unsafe predecessor heads. | Maintain `docs/TRACEABILITY.md` using stable code/test/doc authorities, not run IDs; record #212 only as an active overlay until integration. | | Threat model | PLANNED | No canonical protected-main `docs/THREAT_MODEL.md` is present. Security rules are distributed across AGENTS, architecture, ADRs, tests, and issue/PR evidence. | Add a threat model that distinguishes assets, trust boundaries, attacker capabilities, mitigations, residual risk, and non-guarantees without certification claims. | | Data governance | PLANNED | No single canonical data-governance document currently maps data classes, retention/ownership, tenant authority, privacy exposure, backup/restore, and deletion limits. | Add a protected-main-grounded data-governance contract. | | UML/component/sequence views | PLANNED | Root architecture is prose-first; no canonical UML authority is present. | Add textual Mermaid/PlantUML-compatible component and critical sequence diagrams after active source contracts stabilize. | | ERD / schema model | PLANNED | SQL and schema tests are authoritative, but no canonical ERD summarizes package-owned identities, tenancy, checkpoint state, and relationships. | Generate/maintain an ERD from protected-main schema; label migration-only/active-PR objects separately. | -| General standalone operator guide | PARTIAL | Protected main has strong operational guidance across `README.md` and topic authorities such as `docs/remote-batch-lifecycle.md`; there is no single omnibus protected-main `docs/OPERABILITY.md`. This is a navigation/consolidation gap, not absence of tenant lifecycle operator guidance. | Establish one general operator index/authority only after checking adjacent writers; do not race #184 or #209 documentation surfaces. | +| General standalone operator guide | PARTIAL | Protected main has strong operational guidance across `README.md` and topic authorities such as `docs/remote-batch-lifecycle.md`; there is no single omnibus protected-main `docs/OPERABILITY.md`. This is a navigation/consolidation gap, not absence of tenant lifecycle operator guidance. | Establish one general operator index/authority only after checking adjacent writers; do not race #184 or #212 documentation surfaces. | | Release governance | PARTIAL | Release-evidence ADRs/tests/workflows are strong, but the canonical graph lacks one concise protected-main release/operator authority tying versioning, SBOM, provenance, artifact identity, rollback/recovery, and publication verification together. | Add a stable release contract after checking active release writers; do not copy workflow-run IDs. | | Licensing / third-party notices | PARTIAL | Apache-2.0 source headers and repository licensing exist, but acquisition diligence should explicitly trace package license, dependency/SBOM evidence, and third-party notice process. | Evaluate a concise licensing/compliance evidence index without claiming legal certification. | @@ -53,7 +53,7 @@ Protected main now also contains the bounded recovery-evidence primitives integr - Provider/model content never becomes tenant, credential, endpoint, filesystem, or database authority. - Content-fidelity invariants constrain any result-application path that exists, but they do not prove that end-to-end result application is shipped; FR-4 and `docs/TRACEABILITY.md` remain the capability-status authority while protected main is `PARTIAL`. - Bounded recovery receipts, artifact hashes, and packaged-schema hashes are evidence primitives, not proof that a backup is restorable or that a stated recovery objective is met. -- A future direct logical restore contract must keep caller-owned source trust, target isolation, allowed libpq environment, transaction rollback behavior, and post-restore acceptance explicit; those active-PR semantics are not shipped until integrated. +- A direct logical restore contract must keep caller-owned source trust, target isolation, allowed libpq environment, transaction rollback behavior, and post-restore acceptance explicit. Its archive-completion checks must match PostgreSQL custom-format random-access semantics: final descriptor offset is not proof of complete restore. #209 is therefore an unsafe predecessor, while #212 remains only an ACTIVE-PR successor until integrated. - Distributed exactly-once processing is never implied by PostgreSQL checkpoint atomicity alone. - Security, privacy, SOC 2, and CSAP material is evidence-readiness documentation only unless an external certification actually exists. - Database object names, migration/rollback behavior, exact owned coverage requirements, Python-version acceptance, packaging/SBOM/provenance expectations, and bounded diagnostics must remain synchronized with repository tests and live governance. From eafbfa9b301dcd35fd49b0521abd250eb35e5815 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 17 Aug 2026 00:15:46 +0900 Subject: [PATCH 29/29] docs(recovery): trace seek-safe restore successor --- docs/TRACEABILITY.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/TRACEABILITY.md b/docs/TRACEABILITY.md index 36df331a9..1a67bb9c7 100644 --- a/docs/TRACEABILITY.md +++ b/docs/TRACEABILITY.md @@ -20,7 +20,7 @@ The reference protected-main tree is `d2f1e32271910a6db98a0757d67194ddadca4566`. | FR-5 package persistence integrity | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/db.py`, `pg_llm_batch/schema.sql`, Docker schema mirror | schema-integrity, payload, lifecycle, checkpoint migration tests | Existing-volume legacy-extension retirement is separately ACTIVE-PR #184. | | FR-5 bounded PostgreSQL recovery evidence primitives | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/postgres_recovery_receipt.py`, `pg_llm_batch/postgres_backup_evidence.py`, `pg_llm_batch/postgres_schema_evidence.py` | focused receipt/artifact/schema evidence tests; merged PRs #205, #206, #207 are historical integration evidence | Protected main can derive bounded content-free receipt, backup-artifact hash/size, and packaged-schema hash/size evidence. These primitives do not prove that a backup command ran successfully, that an artifact is restorable, that a live cluster matches packaged schema, or that PITR/RPO/RTO/HA/DR objectives are satisfied. | | FR-5 executable PostgreSQL logical backup | ACTIVE-PR | none on protected main beyond the recovery-evidence primitives | PR #208 exact-head evidence only | The `pg_dump` execution candidate is not shipped. Its service selector is not tenant authorization, and current exact-head review/check evidence must be reacquired before integration. | -| FR-5 executable PostgreSQL logical restore | ACTIVE-PR | none on protected main beyond the recovery-evidence primitives | PR #209 exact-head evidence only | The direct `pg_restore` candidate is not shipped. Its caller-owned source-superuser trust assertion, target-isolation responsibility, libpq allowlist, single-transaction failure boundary, and archive-integrity checks remain active-PR contracts until integration. | +| FR-5 executable PostgreSQL logical restore | ACTIVE-PR | none on protected main beyond the recovery-evidence primitives | PR #212 exact-head evidence only; PR #209 is predecessor defect evidence | Draft #212 is the active direct-`pg_restore` successor. PR #209 must not merge because its EOF-consumption postcondition conflicts with seekable PostgreSQL custom archives and can report failure after transactional restore has already succeeded. The successor's caller-owned source-superuser trust, target-isolation responsibility, libpq allowlist, single-transaction failure boundary, metadata-integrity checks, and permanent documentation remain unshipped until integration. | | FR-5 end-to-end PostgreSQL recovery readiness | PARTIAL | bounded receipt/artifact/schema evidence primitives are integrated | Issue #204 acceptance authority plus protected-main recovery evidence tests | No protected-main end-to-end isolated restore drill yet proves schema/RLS/constraint/extension parity, migration compatibility, external key/config custody, physical/WAL/PITR recovery, or a stated RPO/RTO/HA/DR objective. | | FR-5 legacy `http` / `pg_cron` authority retirement on existing volumes | ACTIVE-PR | protected main does not yet contain the retirement migration contract | PR #184 migration/smoke/operator evidence only | Must remain active until unchanged exact head satisfies migration/security/release/review gates and merges. | | FR-6 PostgreSQL-backed configuration/secrets | IMPLEMENTED-ON-PROTECTED-MAIN | `pg_llm_batch/config.py`, schema | config/secret tests and bootstrap docs | Protected main deliberately retains `SecretStore(require_encryption=False)` as the compatibility default; Fernet support is therefore not an encryption-required production claim. PR #193 proposes an encryption-required default plus explicit local/development opt-out while also separating runtime construction from provisioning. | @@ -73,7 +73,8 @@ The following open pull requests are intentionally represented only as overlays - **#194** — atomic local result-effect/checkpoint application; a current-main-compatible implementation is under review but remains an active overlay and is not shipped. - **#202** — compatibility-aware `ValidationError` rejected-value confidentiality hardening. It is Draft because the current `safe_value` guard accepts `str` subclasses via `isinstance`; the branch must first require an exact built-in `str` and pass a hostile-subclass regression. Protected main still discloses arbitrary rejected values by default, and the intended safe-default redaction/bounded `safe_value` contract remains ACTIVE-PR until integrated. - **#208** — bounded logical PostgreSQL backup executor candidate using `pg_dump`; active source only and not evidence that protected main can create a restorable backup. -- **#209** — bounded direct logical PostgreSQL restore executor candidate using `pg_restore`; active source only and not evidence of isolated restore acceptance or end-to-end recovery readiness. +- **#209** — unsafe direct logical-restore predecessor. Its EOF-consumption postcondition is incompatible with seekable PostgreSQL custom archives and can produce a false post-commit failure. It must not merge in this state and is retained only as defect/history context while the successor is active. +- **#212** — Draft direct logical-restore successor. It replaces the invalid EOF postcondition with metadata-fingerprint verification and owns the accompanying README/architecture/ADR/doctoring/CHANGELOG restore-contract documentation. It remains ACTIVE-PR and is not evidence of isolated restore acceptance or end-to-end recovery readiness. Merged recovery PRs #205, #206, and #207 are no longer active overlays; their bounded receipt, backup-artifact, and packaged-schema evidence primitives are present on the referenced protected-main tree. Their historical PR descriptions remain integration evidence, not durable runtime authority.