From 66220fffbf273c113a9c87eee2eca7620485024d Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 14:08:29 +0900 Subject: [PATCH 01/14] docs: add public product landing page --- docs/index.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 docs/index.md diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 000000000..742d566fa --- /dev/null +++ b/docs/index.md @@ -0,0 +1,44 @@ +# pg-llm-batch + +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/pg-llm-batch) + +`pg-llm-batch` is a standalone and embeddable PostgreSQL-backed engine for token-aware LLM batch preparation, submission, polling, retrieval, and durable lifecycle evidence. + +## Product responsibility + +- Count model tokens authoritatively inside PostgreSQL with `pg_tiktoken`. +- Assemble OpenAI-compatible JSONL batches under explicit token, byte, and record limits. +- Submit, poll, wait for, and retrieve batches through OpenAI-compatible Batch APIs. +- Persist standalone or tenant-scoped lifecycle state with default-deny PostgreSQL row-level security. +- Keep configuration and encrypted secrets in PostgreSQL-backed stores rather than ordinary runtime environment variables. +- Expose bounded health/readiness, recovery, and optional OpenTelemetry operations without leaking prompt/provider content into telemetry. + +## Quick start + +```bash +docker compose up -d --build +export PG_LLM_BATCH_DSN=postgresql://pgllm:pgllm@localhost:5432/pgllm +python -m pg_llm_batch init-db +python -m pg_llm_batch health +``` + +Use the [repository README](../README.md) for gateway configuration, secret input, batch submission, durable lifecycle modes, embedding as a submodule, recovery, observability, and test instructions. + +## Architecture + +PostgreSQL owns authoritative token counting and package-owned lifecycle state. Python owns validated provider-facing I/O and orchestration. Shared-table hosts bind tenant scope from trusted host authorization context and rely on forced RLS; provider metadata never selects tenant identity. + +Key references: + +- [Remote batch lifecycle](remote-batch-lifecycle.md) +- [Tenant-scoped lifecycle](doctoring/tenant-scoped-lifecycle.md) +- [CLI secret input](doctoring/cli-secret-input.md) +- [Count-tokens stdin privacy](doctoring/count-tokens-stdin-privacy.md) +- [OpenTelemetry operations](doctoring/opentelemetry-operations.md) + +## Releases and verification + +Use protected-branch history, GitHub Releases, current checks, and repository test evidence to determine what is shipped. A documentation source commit is not evidence that GitHub Pages is already published; Pages completion requires live repository settings and HTTPS content verification after reconciliation. + +- [Repository](https://github.com/ContextualWisdomLab/pg-llm-batch) +- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/pg-llm-batch) From a8a1246b03f208189af5b29c335c571fe0d602a7 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 14:55:30 +0900 Subject: [PATCH 02/14] docs(readme): expose Psycopg commercial-license blocker --- README.md | 42 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 38 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 935ba5be4..1838bcd38 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,17 @@ Azure OpenAI, or a LiteLLM gateway). Extracted from ContextualWisdomLab's `xtrmLLMBatchPython` batch core and relicensed to **Apache-2.0** (see [`NOTICE`](NOTICE) for provenance). +> **Commercial dependency status:** the repository's own source is Apache-2.0, +> but the current runtime dependency `psycopg[binary]>=3.1` resolves to +> LGPL-3.0-only Psycopg/Psycopg Binary distributions. That GPL-family inbound +> path is outside ContextualWisdomLab's commercial dependency baseline and is +> tracked in [issue #322](https://github.com/ContextualWisdomLab/pg-llm-batch/issues/322). +> The current install/compose/submodule commands below document development and +> verification behavior; they are **not** evidence that this dependency graph is +> approved for commercial incorporation or distribution. Do not suppress the +> license inventory or treat this repository's Apache-2.0 grant as relicensing +> Psycopg. + ## Why it exists - **Token counting is authoritative.** Counts come from `pg_tiktoken` in the @@ -63,13 +74,20 @@ than a second database-side network authority. - PostgreSQL with `pg_tiktoken`. Fresh bundled database initialization does not create `pg_cron` or `http`; their image packages are retained temporarily only for existing-volume cleanup and rollback compatibility. -- Python 3.10+ with `psycopg[binary]` and `aiohttp` (installed via `pip install .`). +- Python 3.10+ with the **current** `psycopg[binary]` and `aiohttp` dependency + graph (installed via `pip install .`). Psycopg is LGPL-3.0-only and therefore + remains an unresolved commercial-policy blocker under issue #322; this line + describes present execution requirements, not an approved inbound dependency. - Tenant-scoped lifecycle deployments require an application database role with `NOSUPERUSER NOBYPASSRLS` and a trusted host authorization boundary. --- -## Standalone use +## Standalone development and verification + +The following path exercises the current implementation. Because it installs the +LGPL-family Psycopg runtime path described above, it is not the approved +commercial distribution path while issue #322 remains open. ### 1. Bring up the stack @@ -219,7 +237,14 @@ exits zero, treat the target as unsafe and do not retry into the same service. See [`docs/doctoring/postgres-logical-restore.md`](docs/doctoring/postgres-logical-restore.md) for the operator steps. -## Embed as a git submodule +## Embedding boundary + +The codebase supports submodule-style embedding mechanically, but the current +runtime graph contains the unapproved LGPL-family Psycopg dependency. The +commands below document existing developer integration only; do not incorporate +this package into a commercial ContextualWisdomLab distribution until issue +#322 removes/replaces that dependency and exact-head package/license evidence is +clean. ```bash git submodule add https://github.com/ContextualWisdomLab/pg-llm-batch.git \ @@ -306,6 +331,9 @@ for signals, ownership boundaries, privacy rules, and APA 7 references. ## Tests +The current test environment also installs Psycopg and therefore verifies the +present implementation, not the eventual commercially compatible replacement. + ```bash pip install -e '.[test]' pytest # unit tests (fakes, no DB needed) @@ -339,4 +367,10 @@ PG_LLM_BATCH_TEST_DSN=postgresql://pgllm:pgllm@localhost:5432/pgllm \ ## License -Apache-2.0. See [`LICENSE`](LICENSE) and [`NOTICE`](NOTICE). +The pg-llm-batch repository's original source is Apache-2.0; see +[`LICENSE`](LICENSE) and [`NOTICE`](NOTICE). That grant does not relicense +third-party dependencies. The current direct runtime path includes +`psycopg[binary]>=3.1` / Psycopg Binary under LGPL-3.0-only, which is not +accepted by ContextualWisdomLab's commercial inbound baseline. Issue #322 owns +its removal/replacement; do not present the current dependency graph as +commercial-policy clean until that work is integrated and reverified. From 227b2de65320e01c30b270b7de0f6115b3d8b056 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 14:55:47 +0900 Subject: [PATCH 03/14] docs(pages): expose commercial dependency boundary --- docs/index.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/index.md b/docs/index.md index 742d566fa..4f19f318c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -13,7 +13,13 @@ - Keep configuration and encrypted secrets in PostgreSQL-backed stores rather than ordinary runtime environment variables. - Expose bounded health/readiness, recovery, and optional OpenTelemetry operations without leaking prompt/provider content into telemetry. -## Quick start +## Commercial dependency status + +The repository's original source is Apache-2.0, but the current runtime manifest directly depends on `psycopg[binary]>=3.1`. Current Psycopg/Psycopg Binary package metadata identifies that path as LGPL-3.0-only, which is outside ContextualWisdomLab's commercial inbound baseline. [Issue #322](https://github.com/ContextualWisdomLab/pg-llm-batch/issues/322) owns replacement while preserving PostgreSQL transaction, RLS, type-adaptation, concurrency, recovery, packaging, and integration behavior. + +The current compose/install commands document development and verification of today's implementation; they are not evidence that the dependency graph is approved for commercial incorporation or distribution. The Apache-2.0 repository grant does not relicense Psycopg. + +## Development quick start ```bash docker compose up -d --build @@ -22,7 +28,7 @@ python -m pg_llm_batch init-db python -m pg_llm_batch health ``` -Use the [repository README](../README.md) for gateway configuration, secret input, batch submission, durable lifecycle modes, embedding as a submodule, recovery, observability, and test instructions. +Use the [repository README](https://github.com/ContextualWisdomLab/pg-llm-batch/blob/main/README.md) for gateway configuration, secret input, batch submission, durable lifecycle modes, the currently blocked embedding path, recovery, observability, and test instructions. ## Architecture @@ -40,5 +46,10 @@ Key references: Use protected-branch history, GitHub Releases, current checks, and repository test evidence to determine what is shipped. A documentation source commit is not evidence that GitHub Pages is already published; Pages completion requires live repository settings and HTTPS content verification after reconciliation. +Commercial-readiness evidence additionally requires issue #322 to be resolved with a dependency graph that no longer contains a disallowed GPL/LGPL/AGPL-family runtime package. Passing product tests do not override that license boundary. + - [Repository](https://github.com/ContextualWisdomLab/pg-llm-batch) +- [README](https://github.com/ContextualWisdomLab/pg-llm-batch/blob/main/README.md) +- [Apache-2.0 source license](https://github.com/ContextualWisdomLab/pg-llm-batch/blob/main/LICENSE) +- [Commercial dependency blocker #322](https://github.com/ContextualWisdomLab/pg-llm-batch/issues/322) - [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/pg-llm-batch) From f9249513fb84870ca6c688a48f30b8ab951225ba Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 16:06:56 +0900 Subject: [PATCH 04/14] docs(security): remove reusable quick-start database password --- docs/index.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index 4f19f318c..27ddd95d8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,13 +21,18 @@ The current compose/install commands document development and verification of to ## Development quick start +Create a fresh development-only database password for this run. Docker Compose consumes the value through its named `postgres_password` secret, and the same generated value is used only as bootstrap transport for the local CLI DSN. + ```bash +export PG_LLM_BATCH_POSTGRES_PASSWORD="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')" docker compose up -d --build -export PG_LLM_BATCH_DSN=postgresql://pgllm:pgllm@localhost:5432/pgllm +export PG_LLM_BATCH_DSN="postgresql://pgllm:${PG_LLM_BATCH_POSTGRES_PASSWORD}@localhost:5432/pgllm" python -m pg_llm_batch init-db python -m pg_llm_batch health ``` +Do not replace the generated value with a shared example password. Production deployments should supply the Compose secret and application bootstrap credential through their reviewed secret-management path. + Use the [repository README](https://github.com/ContextualWisdomLab/pg-llm-batch/blob/main/README.md) for gateway configuration, secret input, batch submission, durable lifecycle modes, the currently blocked embedding path, recovery, observability, and test instructions. ## Architecture From 304a276ed3e75bc7e8a4ef5cf24a6718a77697f5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 16:08:50 +0900 Subject: [PATCH 05/14] docs(reliability): preserve quick-start database password lifecycle --- docs/index.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/index.md b/docs/index.md index 27ddd95d8..6d977c7ea 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,17 +21,24 @@ The current compose/install commands document development and verification of to ## Development quick start -Create a fresh development-only database password for this run. Docker Compose consumes the value through its named `postgres_password` secret, and the same generated value is used only as bootstrap transport for the local CLI DSN. +For a **new disposable Compose project**, generate a development-only database password once, retain it in your normal local secret store, and reuse that same value for later starts of the existing `pgdata` volume. PostgreSQL applies the initialization password only when it first creates the data directory; changing the Compose secret later does not rotate the existing database role password. + +First-time initialization: ```bash export PG_LLM_BATCH_POSTGRES_PASSWORD="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')" +# Save this generated value outside the repository in your local secret manager. docker compose up -d --build export PG_LLM_BATCH_DSN="postgresql://pgllm:${PG_LLM_BATCH_POSTGRES_PASSWORD}@localhost:5432/pgllm" python -m pg_llm_batch init-db python -m pg_llm_batch health ``` -Do not replace the generated value with a shared example password. Production deployments should supply the Compose secret and application bootstrap credential through their reviewed secret-management path. +For subsequent starts that reuse the same `pgdata` volume, restore the **same** development password into `PG_LLM_BATCH_POSTGRES_PASSWORD` before `docker compose up`, then build `PG_LLM_BATCH_DSN` from that value as above. + +If this is a disposable development database and the original password is intentionally unavailable, `docker compose down -v` removes the persisted database volume; the next start is a new initialization and permanently deletes the old local database contents. For a retained database, rotate the PostgreSQL role credential deliberately and update the Compose/application secret together instead of changing only the environment value. + +Do not use a shared example password. Production deployments should supply database credentials through their reviewed secret-management and rotation path. Use the [repository README](https://github.com/ContextualWisdomLab/pg-llm-batch/blob/main/README.md) for gateway configuration, secret input, batch submission, durable lifecycle modes, the currently blocked embedding path, recovery, observability, and test instructions. From 6e6e4aa21bb06c334ca62b22ab635554a975c1c1 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 16:10:22 +0900 Subject: [PATCH 06/14] docs(security): align public password lifecycle examples --- README.md | 46 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 1838bcd38..04e1f8ebb 100644 --- a/README.md +++ b/README.md @@ -25,13 +25,15 @@ relicensed to **Apache-2.0** (see [`NOTICE`](NOTICE) for provenance). - **Token counting is authoritative.** Counts come from `pg_tiktoken` in the database, so the numbers used to pack a batch are exactly what the DB sees — there is no drifting Python-side tokenizer. -- **No secrets in the environment.** All configuration and credentials live in - Postgres KV tables (`com_config`, `com_secrets`). The environment is only a - *bootstrap transport* for the DSN and an optional Fernet key. This replaces - the ~75 `os.getenv` reads in the upstream app. CLI secret values are entered - through a no-echo prompt or bounded standard input, never as process arguments. - Content-bearing `count-tokens` input is likewise accepted only through bounded - UTF-8 standard input, so prompt text is not placed in process arguments. +- **Provider secrets stay out of ordinary process arguments.** Runtime provider + configuration and credentials live in Postgres KV tables (`com_config`, + `com_secrets`). Environment variables are limited to explicit bootstrap + transport such as the DSN and optional Fernet key; the bundled Compose path + mounts the PostgreSQL password as a named secret instead of embedding it in + the component DSN. CLI secret values are entered through a no-echo prompt or + bounded standard input, never as process arguments. Content-bearing + `count-tokens` input is likewise accepted only through bounded UTF-8 standard + input, so prompt text is not placed in process arguments. - **Disk-free assembly.** JSONL payloads are stored as `JSONB` and reconstructed by JOIN, never written to disk. - **Standalone or tenant-scoped lifecycle state.** `DurableBatchAPIClient` @@ -91,17 +93,34 @@ commercial distribution path while issue #322 remains open. ### 1. Bring up the stack +For a **new disposable Compose project**, generate the PostgreSQL development +password once and retain it in your normal local secret store. PostgreSQL applies +that password only when it initializes the `pgdata` volume, so later starts of +the same volume must reuse the same value unless you deliberately rotate the +role credential inside PostgreSQL. + ```bash +export PG_LLM_BATCH_POSTGRES_PASSWORD="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')" +# Save the generated value outside the repository in your local secret manager. docker compose up -d --build # postgres becomes healthy only once pg_tiktoken + com_config are ready; # the component then serves GET /healthz on :8080 curl -fsS localhost:8080/healthz ``` -### 2. Point it at your gateway (config + secret in the DB, not env) +On subsequent starts that reuse the existing `pgdata` volume, restore the same +password to `PG_LLM_BATCH_POSTGRES_PASSWORD` before running `docker compose up`. +Changing only the Compose secret does not rotate the persisted database role. If +a disposable development password is intentionally lost, `docker compose down +-v` removes the persisted database volume so the next start can initialize a new +password; that command permanently deletes the old local database contents. +Retained environments should use a deliberate PostgreSQL credential-rotation +procedure and update the deployment/application secret together. + +### 2. Point it at your gateway (config + secret in the DB, not argv) ```bash -export PG_LLM_BATCH_DSN=postgresql://pgllm:pgllm@localhost:5432/pgllm +export PG_LLM_BATCH_DSN="postgresql://pgllm:${PG_LLM_BATCH_POSTGRES_PASSWORD}@localhost:5432/pgllm" python -m pg_llm_batch init-db # idempotent python -m pg_llm_batch config set gateway base_url https://your-gateway/v1 python -m pg_llm_batch config set-secret gateway_api_key.default # no-echo prompt @@ -152,9 +171,11 @@ expires, including the last observed remote status. Assemble a batch programmatically: ```python +import os + from pg_llm_batch import PostgresBatchOrchestrator -orch = PostgresBatchOrchestrator("postgresql://pgllm:pgllm@localhost:5432/pgllm") +orch = PostgresBatchOrchestrator(os.environ["PG_LLM_BATCH_DSN"]) result = orch.prepare_batches(batch_uuid="") for payload in result["ready"]: print(payload.file_path, payload.request_count, payload.total_tokens) @@ -338,8 +359,11 @@ present implementation, not the eventual commercially compatible replacement. pip install -e '.[test]' pytest # unit tests (fakes, no DB needed) +# Reuse the same development password as the existing pgdata volume, or generate +# one once before the first initialization of a new disposable test volume. +export PG_LLM_BATCH_POSTGRES_PASSWORD="" docker compose up -d --build postgres -PG_LLM_BATCH_TEST_DSN=postgresql://pgllm:pgllm@localhost:5432/pgllm \ +PG_LLM_BATCH_TEST_DSN="postgresql://pgllm:${PG_LLM_BATCH_POSTGRES_PASSWORD}@localhost:5432/pgllm" \ pytest -m integration # against the real pg_tiktoken PostgreSQL container ``` From eb0bbb3656bcdc0d71e4aedf76371dc2907a703f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 16:15:42 +0900 Subject: [PATCH 07/14] docs(security): use libpq passfile for local database auth --- README.md | 69 ++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 51 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 04e1f8ebb..971f0a70b 100644 --- a/README.md +++ b/README.md @@ -28,12 +28,13 @@ relicensed to **Apache-2.0** (see [`NOTICE`](NOTICE) for provenance). - **Provider secrets stay out of ordinary process arguments.** Runtime provider configuration and credentials live in Postgres KV tables (`com_config`, `com_secrets`). Environment variables are limited to explicit bootstrap - transport such as the DSN and optional Fernet key; the bundled Compose path - mounts the PostgreSQL password as a named secret instead of embedding it in - the component DSN. CLI secret values are entered through a no-echo prompt or - bounded standard input, never as process arguments. Content-bearing - `count-tokens` input is likewise accepted only through bounded UTF-8 standard - input, so prompt text is not placed in process arguments. + transport such as the DSN, Compose secret source, passfile path, and optional + Fernet key; the bundled Compose path mounts the PostgreSQL password as a named + secret instead of embedding it in the component DSN. CLI secret values are + entered through a no-echo prompt or bounded standard input, never as process + arguments. Content-bearing `count-tokens` input is likewise accepted only + through bounded UTF-8 standard input, so prompt text is not placed in process + arguments. - **Disk-free assembly.** JSONL payloads are stored as `JSONB` and reconstructed by JOIN, never written to disk. - **Standalone or tenant-scoped lifecycle state.** `DurableBatchAPIClient` @@ -99,28 +100,58 @@ that password only when it initializes the `pgdata` volume, so later starts of the same volume must reuse the same value unless you deliberately rotate the role credential inside PostgreSQL. +For the host-side CLI, create a mode-0600 libpq passfile before starting Compose. +The passfile escapes libpq delimiters, so arbitrary generated or restored +passwords are not interpolated into a connection URI. After Compose has read its +bootstrap secret, remove the plaintext password from the shell environment; the +CLI environment contains only the passfile path and a credential-free DSN. + ```bash export PG_LLM_BATCH_POSTGRES_PASSWORD="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')" # Save the generated value outside the repository in your local secret manager. +export PGPASSFILE="$(mktemp "${TMPDIR:-/tmp}/pg-llm-batch.pgpass.XXXXXX")" +chmod 600 "$PGPASSFILE" +python - <<'PY' +import os +from pathlib import Path + +password = os.environ["PG_LLM_BATCH_POSTGRES_PASSWORD"] +escaped = password.replace("\\", "\\\\").replace(":", "\\:") +Path(os.environ["PGPASSFILE"]).write_text( + f"localhost:5432:pgllm:pgllm:{escaped}\n", + encoding="utf-8", +) +PY + docker compose up -d --build +unset PG_LLM_BATCH_POSTGRES_PASSWORD +export PG_LLM_BATCH_DSN="postgresql://pgllm@localhost:5432/pgllm" # postgres becomes healthy only once pg_tiktoken + com_config are ready; # the component then serves GET /healthz on :8080 curl -fsS localhost:8080/healthz ``` On subsequent starts that reuse the existing `pgdata` volume, restore the same -password to `PG_LLM_BATCH_POSTGRES_PASSWORD` before running `docker compose up`. -Changing only the Compose secret does not rotate the persisted database role. If -a disposable development password is intentionally lost, `docker compose down --v` removes the persisted database volume so the next start can initialize a new -password; that command permanently deletes the old local database contents. -Retained environments should use a deliberate PostgreSQL credential-rotation -procedure and update the deployment/application secret together. +password from your local secret manager, recreate the mode-0600 passfile with the +same Python escaping step, run `docker compose up`, and unset +`PG_LLM_BATCH_POSTGRES_PASSWORD` again. Changing only the Compose secret does not +rotate the persisted database role. If a disposable development password is +intentionally lost, `docker compose down -v` removes the persisted database +volume so the next start can initialize a new password; that command permanently +deletes the old local database contents. Retained environments should use a +deliberate PostgreSQL credential-rotation procedure and update the +deployment/application secret together. + +Remove the temporary passfile when the local CLI session is finished: + +```bash +rm -f "$PGPASSFILE" +unset PGPASSFILE PG_LLM_BATCH_DSN +``` ### 2. Point it at your gateway (config + secret in the DB, not argv) ```bash -export PG_LLM_BATCH_DSN="postgresql://pgllm:${PG_LLM_BATCH_POSTGRES_PASSWORD}@localhost:5432/pgllm" python -m pg_llm_batch init-db # idempotent python -m pg_llm_batch config set gateway base_url https://your-gateway/v1 python -m pg_llm_batch config set-secret gateway_api_key.default # no-echo prompt @@ -359,12 +390,14 @@ present implementation, not the eventual commercially compatible replacement. pip install -e '.[test]' pytest # unit tests (fakes, no DB needed) -# Reuse the same development password as the existing pgdata volume, or generate -# one once before the first initialization of a new disposable test volume. +# Restore the same development password used by the existing pgdata volume, +# recreate a mode-0600 PGPASSFILE as in the quick start, then unset the plaintext +# password after Compose has consumed it. export PG_LLM_BATCH_POSTGRES_PASSWORD="" docker compose up -d --build postgres -PG_LLM_BATCH_TEST_DSN="postgresql://pgllm:${PG_LLM_BATCH_POSTGRES_PASSWORD}@localhost:5432/pgllm" \ - pytest -m integration # against the real pg_tiktoken PostgreSQL container +unset PG_LLM_BATCH_POSTGRES_PASSWORD +PG_LLM_BATCH_TEST_DSN="postgresql://pgllm@localhost:5432/pgllm" \ + pytest -m integration # libpq reads the password from PGPASSFILE ``` ## Docs From a597da5a648706bbe9267f7f5c2cb9869f3f32eb Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 16:16:23 +0900 Subject: [PATCH 08/14] docs(security): keep database password out of DSN --- docs/index.md | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/docs/index.md b/docs/index.md index 6d977c7ea..e3159135a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -23,21 +23,43 @@ The current compose/install commands document development and verification of to For a **new disposable Compose project**, generate a development-only database password once, retain it in your normal local secret store, and reuse that same value for later starts of the existing `pgdata` volume. PostgreSQL applies the initialization password only when it first creates the data directory; changing the Compose secret later does not rotate the existing database role password. -First-time initialization: +Create a mode-0600 libpq passfile for host-side CLI access so the password is not embedded in `PG_LLM_BATCH_DSN`. The passfile writer escapes libpq delimiters before Compose consumes the bootstrap environment secret. ```bash export PG_LLM_BATCH_POSTGRES_PASSWORD="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')" # Save this generated value outside the repository in your local secret manager. +export PGPASSFILE="$(mktemp "${TMPDIR:-/tmp}/pg-llm-batch.pgpass.XXXXXX")" +chmod 600 "$PGPASSFILE" +python - <<'PY' +import os +from pathlib import Path + +password = os.environ["PG_LLM_BATCH_POSTGRES_PASSWORD"] +escaped = password.replace("\\", "\\\\").replace(":", "\\:") +Path(os.environ["PGPASSFILE"]).write_text( + f"localhost:5432:pgllm:pgllm:{escaped}\n", + encoding="utf-8", +) +PY + docker compose up -d --build -export PG_LLM_BATCH_DSN="postgresql://pgllm:${PG_LLM_BATCH_POSTGRES_PASSWORD}@localhost:5432/pgllm" +unset PG_LLM_BATCH_POSTGRES_PASSWORD +export PG_LLM_BATCH_DSN="postgresql://pgllm@localhost:5432/pgllm" python -m pg_llm_batch init-db python -m pg_llm_batch health ``` -For subsequent starts that reuse the same `pgdata` volume, restore the **same** development password into `PG_LLM_BATCH_POSTGRES_PASSWORD` before `docker compose up`, then build `PG_LLM_BATCH_DSN` from that value as above. +For subsequent starts that reuse the same `pgdata` volume, restore the **same** development password from your local secret manager, recreate the mode-0600 passfile with the same escaping step, run `docker compose up`, and unset `PG_LLM_BATCH_POSTGRES_PASSWORD` again. The CLI then uses the credential-free DSN plus `PGPASSFILE`. Changing only the Compose secret does not rotate the existing database role password. If this is a disposable development database and the original password is intentionally unavailable, `docker compose down -v` removes the persisted database volume; the next start is a new initialization and permanently deletes the old local database contents. For a retained database, rotate the PostgreSQL role credential deliberately and update the Compose/application secret together instead of changing only the environment value. +Remove the temporary passfile when the local CLI session ends: + +```bash +rm -f "$PGPASSFILE" +unset PGPASSFILE PG_LLM_BATCH_DSN +``` + Do not use a shared example password. Production deployments should supply database credentials through their reviewed secret-management and rotation path. Use the [repository README](https://github.com/ContextualWisdomLab/pg-llm-batch/blob/main/README.md) for gateway configuration, secret input, batch submission, durable lifecycle modes, the currently blocked embedding path, recovery, observability, and test instructions. From 2deb8b899ea9e620fed217ca7cd5a0e70fde36bd Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 2 Sep 2026 16:17:21 +0900 Subject: [PATCH 09/14] docs(lint): quote inline issue reference --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 971f0a70b..24cb5517f 100644 --- a/README.md +++ b/README.md @@ -295,8 +295,8 @@ The codebase supports submodule-style embedding mechanically, but the current runtime graph contains the unapproved LGPL-family Psycopg dependency. The commands below document existing developer integration only; do not incorporate this package into a commercial ContextualWisdomLab distribution until issue -#322 removes/replaces that dependency and exact-head package/license evidence is -clean. +`#322` removes/replaces that dependency and exact-head package/license evidence +is clean. ```bash git submodule add https://github.com/ContextualWisdomLab/pg-llm-batch.git \ From 2bc5d7c7471e1adc1932d79af608fcd018dc785b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Tue, 8 Sep 2026 21:41:18 +0900 Subject: [PATCH 10/14] fix(stack): preserve parent code in public docs restack --- .github/workflows/ci.yml | 101 +++++++-- docs/product-technical-gap-baseline.md | 12 +- tests/test_candidate_source_wheel_parity.py | 131 ++++++++++++ .../test_candidate_wheel_license_verifier.py | 4 +- tests/test_pg8000_candidate_python_matrix.py | 35 ++++ tests/test_workflow_contracts.py | 9 +- tools/verify_candidate_source_wheel_parity.py | 194 ++++++++++++++++++ 7 files changed, 466 insertions(+), 20 deletions(-) create mode 100644 tests/test_candidate_source_wheel_parity.py create mode 100644 tests/test_pg8000_candidate_python_matrix.py create mode 100644 tools/verify_candidate_source_wheel_parity.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3158790f8..daf93c94c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -114,16 +114,51 @@ jobs: run: bash tests/smoke_postgres_container_logging.sh - name: Run legacy SQL cleanup integration smoke run: bash tests/smoke_legacy_sql_cleanup.sh + - name: Set up Python 3.10 for candidate parity + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.10" + - name: Create Python 3.10 candidate environment + run: python -m venv /tmp/pg8000-candidate-py310 + - name: Set up Python 3.12 for candidate parity + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.12" + - name: Create Python 3.12 candidate environment + run: python -m venv /tmp/pg8000-candidate-py312 - name: Set up Python 3.14 for candidate parity uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: "3.14" + - name: Create Python 3.14 candidate environment + run: python -m venv /tmp/pg8000-candidate-py314 - name: Set up uv for candidate parity uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 with: prune-cache: true - name: Install locked project dependencies - run: uv sync --locked + shell: bash + run: | + uv sync --locked + for environment in \ + /tmp/pg8000-candidate-py310 \ + /tmp/pg8000-candidate-py312 \ + /tmp/pg8000-candidate-py314; do + UV_PROJECT_ENVIRONMENT="$environment" \ + uv sync --locked --no-dev --no-install-project \ + --python "$environment/bin/python" + done + - name: Build exact pg-llm-batch wheel for candidate parity + shell: bash + run: | + mkdir -p /tmp/pg-llm-batch-candidate-wheel + uv build --wheel --no-sources --out-dir /tmp/pg-llm-batch-candidate-wheel + mapfile -t product_wheels < <( + find /tmp/pg-llm-batch-candidate-wheel -maxdepth 1 -type f \ + -name 'pg_llm_batch-*.whl' -print + ) + test "${#product_wheels[@]}" -eq 1 + echo "PG_LLM_BATCH_CANDIDATE_WHEEL=${product_wheels[0]}" >> "$GITHUB_ENV" - name: Download exact pg8000 candidate dependency closure run: >- python -m pip download --no-deps --only-binary=:all: @@ -143,18 +178,54 @@ jobs: db4e40728b728508912cbb3d44f19ce188f218e9eba635821bb4b68564f8fd67 /tmp/pg8000-candidate/asn1crypto-1.5.1-py2.py3-none-any.whl 4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274 /tmp/pg8000-candidate/six-1.17.0-py2.py3-none-any.whl EOF - - name: Verify pg8000 candidate dependency licenses - run: python tools/verify_candidate_wheel_licenses.py /tmp/pg8000-candidate - - name: Install exact candidate closure into the CI environment + - name: Download exact pg8000 candidate source distribution + shell: bash + run: | + mkdir -p /tmp/pg8000-candidate-source + curl --fail --location --proto '=https' --tlsv1.2 \ + --output /tmp/pg8000-candidate-source/pg8000-1.31.5.tar.gz \ + https://files.pythonhosted.org/packages/c8/9a/077ab21e700051e03d8c5232b6bcb9a1a4d4b6242c9a0226df2cfa306414/pg8000-1.31.5.tar.gz + - name: Verify pg8000 candidate source digest run: >- - uv pip install --python .venv/bin/python --no-deps + echo "46ebb03be52b7a77c03c725c79da2ca281d6e8f59577ca66b17c9009618cae78 /tmp/pg8000-candidate-source/pg8000-1.31.5.tar.gz" + | sha256sum --check --strict + - name: Verify pg8000 candidate source-wheel parity + run: >- + python tools/verify_candidate_source_wheel_parity.py + /tmp/pg8000-candidate-source/pg8000-1.31.5.tar.gz /tmp/pg8000-candidate/pg8000-1.31.5-py3-none-any.whl - /tmp/pg8000-candidate/python_dateutil-2.9.0.post0-py2.py3-none-any.whl - /tmp/pg8000-candidate/scramp-1.4.17-py3-none-any.whl - /tmp/pg8000-candidate/asn1crypto-1.5.1-py2.py3-none-any.whl - /tmp/pg8000-candidate/six-1.17.0-py2.py3-none-any.whl - - name: Verify candidate environment dependency consistency - run: uv pip check --python .venv/bin/python + - name: Verify pg8000 candidate dependency licenses + run: python tools/verify_candidate_wheel_licenses.py /tmp/pg8000-candidate + - name: Install exact candidate closure into release Python environments + shell: bash + run: | + for interpreter in \ + /tmp/pg8000-candidate-py310/bin/python \ + /tmp/pg8000-candidate-py312/bin/python \ + /tmp/pg8000-candidate-py314/bin/python; do + uv pip install --python "$interpreter" --no-deps \ + /tmp/pg8000-candidate/pg8000-1.31.5-py3-none-any.whl \ + /tmp/pg8000-candidate/python_dateutil-2.9.0.post0-py2.py3-none-any.whl \ + /tmp/pg8000-candidate/scramp-1.4.17-py3-none-any.whl \ + /tmp/pg8000-candidate/asn1crypto-1.5.1-py2.py3-none-any.whl \ + /tmp/pg8000-candidate/six-1.17.0-py2.py3-none-any.whl + done + - name: Install exact pg-llm-batch wheel into candidate environments + shell: bash + run: | + test -f "$PG_LLM_BATCH_CANDIDATE_WHEEL" + uv pip install --python "/tmp/pg8000-candidate-py310/bin/python" --no-deps \ + "$PG_LLM_BATCH_CANDIDATE_WHEEL" + uv pip install --python "/tmp/pg8000-candidate-py312/bin/python" --no-deps \ + "$PG_LLM_BATCH_CANDIDATE_WHEEL" + uv pip install --python "/tmp/pg8000-candidate-py314/bin/python" --no-deps \ + "$PG_LLM_BATCH_CANDIDATE_WHEEL" + - name: Verify pg8000 candidate Python 3.10 environment + run: uv pip check --python /tmp/pg8000-candidate-py310/bin/python + - name: Verify pg8000 candidate Python 3.12 environment + run: uv pip check --python /tmp/pg8000-candidate-py312/bin/python + - name: Verify pg8000 candidate Python 3.14 environment + run: uv pip check --python /tmp/pg8000-candidate-py314/bin/python - name: Start candidate PostgreSQL runtime shell: bash run: | @@ -202,8 +273,12 @@ jobs: done docker logs "$PG8000_CANDIDATE_CONTAINER" exit 1 - - name: Run real pg8000 candidate PostgreSQL smoke - run: uv run --no-sync python tests/smoke_pg8000_candidate_postgres.py + - name: Run real pg8000 candidate PostgreSQL smoke on Python 3.10 + run: cd "$RUNNER_TEMP" && /tmp/pg8000-candidate-py310/bin/python "$GITHUB_WORKSPACE/tests/smoke_pg8000_candidate_postgres.py" + - name: Run real pg8000 candidate PostgreSQL smoke on Python 3.12 + run: cd "$RUNNER_TEMP" && /tmp/pg8000-candidate-py312/bin/python "$GITHUB_WORKSPACE/tests/smoke_pg8000_candidate_postgres.py" + - name: Run real pg8000 candidate PostgreSQL smoke on Python 3.14 + run: cd "$RUNNER_TEMP" && /tmp/pg8000-candidate-py314/bin/python "$GITHUB_WORKSPACE/tests/smoke_pg8000_candidate_postgres.py" - name: Tear down candidate PostgreSQL runtime if: ${{ always() }} shell: bash diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 60313f98e..391a91316 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -16,17 +16,19 @@ The repository has no immutable GitHub release at the latest refresh. A release PR #233 remains the dependency-root delivery lane and must be judged from its live head and live base, not predecessor evidence. -PR #323 is the active Draft migration lane for issue #322. It establishes a driver-neutral PostgreSQL anti-corruption port, retains Psycopg only as the current baseline adapter, and evaluates pg8000 1.31.5 as candidate evidence without promoting it into the production manifest. The lane exercises parameter binding, tuple-row normalization, row-count semantics, transaction and cleanup precedence, forced-RLS tenant scope, JSONB/UUID/timestamp behavior, exact candidate dependency hashes and license metadata, URI/keyword/explicit-service selection, packaged restore-catalog inspection, thread-affinity rejection at the anti-corruption boundary, and real PostgreSQL candidate execution. Exact branch evidence now also terminates a live candidate backend from a second authenticated session, requires the severed capability to fail and become terminal, and proves recovery only by opening a fresh connection. That is candidate recovery evidence; it is not production-driver promotion. +PR #323 is the active Draft migration lane for issue #322. It establishes a driver-neutral PostgreSQL anti-corruption port, retains Psycopg only as the current baseline adapter, and evaluates pg8000 1.31.5 as candidate evidence without promoting it into the production manifest. The lane exercises parameter binding, tuple-row normalization, row-count semantics, transaction and cleanup precedence, forced-RLS tenant scope, JSONB/UUID/timestamp behavior, exact candidate dependency hashes and license metadata, URI/keyword/explicit-service selection, packaged restore-catalog acceptance, thread-affinity rejection at the anti-corruption boundary, and real PostgreSQL candidate execution. Exact branch evidence now also terminates a live candidate backend from a second authenticated session, requires the severed capability to fail and become terminal, and proves recovery only by opening a fresh connection. That is candidate recovery evidence; it is not production-driver promotion. -The current candidate supply-chain work verifies license metadata for the exact five-wheel pg8000 candidate closure before installation. The verifier reads bounded wheel `METADATA` without importing candidate code, rejects GPL/LGPL/AGPL-family declarations, requires positive reviewed permissive-license evidence for every exact package/version, and rejects an unexpected wheel set. This strengthens candidate admission but does not itself approve a production driver replacement. +Candidate runtime parity now executes against the built pg-llm-batch wheel rather than repository import leakage. CI builds the exact source head with `uv build --wheel --no-sources`, installs that wheel without dependency resolution into each isolated Python 3.10, 3.12, and 3.14 candidate environment, runs `uv pip check`, changes out of the repository working tree, and runs the same real pg8000/PostgreSQL smoke without `PYTHONPATH`. This proves package-installed behavior for the candidate lane while the committed product metadata still intentionally retains the Psycopg baseline. + +Candidate supply-chain admission now verifies both the exact five-wheel pg8000 closure and the published pg8000 1.31.5 source distribution before candidate installation. CI pins the wheel and source-distribution SHA-256 digests, reads bounded wheel `METADATA` without importing candidate code, rejects GPL/LGPL/AGPL-family declarations, requires positive reviewed permissive-license evidence for every exact package/version, and compares every Python source path and byte digest under the pg8000 package between the pinned source distribution and universal wheel. The parity verifier does not extract archives, import candidate code, follow archive links, or execute a source build. This closes the published source-to-wheel executable-payload parity gap for the selected candidate artifacts; it does not itself approve a production driver replacement or provide an upstream build attestation. ## Highest-priority gaps | Gap | Current state | Required next evidence | | --- | --- | --- | | Commercial PostgreSQL runtime dependency | P0 / active | Complete issue #322: preserve shipped DB semantics while removing every disallowed GPL/LGPL/AGPL-family runtime package from the committed dependency graph. | -| Candidate driver contract parity | Active Draft | Real server-terminated-session discard and fresh-session recovery are now proven on the candidate. Close remaining selector/conninfo compatibility, realistic concurrency beyond the deterministic anti-cross-thread guard, timeout/health, remaining schema/recovery surfaces, and package-installed behavior before production promotion. | -| Candidate supply-chain admission | Active / strengthened | Exact wheel hashes and license metadata are gated; complete vulnerability/SBOM/provenance and final runtime-graph evidence before promotion. | +| Candidate driver contract parity | Active Draft | Real server-terminated-session recovery, the Python 3.10/3.12/3.14 matrix, and built-wheel execution are proven. Close remaining selector/conninfo compatibility, realistic concurrency beyond the deterministic anti-cross-thread guard, timeout/health, and remaining schema/recovery surfaces before production promotion. | +| Candidate supply-chain admission | Active / strengthened | Exact wheel/source hashes, source-to-wheel Python payload parity, closure license metadata, and installed-product-wheel execution are gated; complete vulnerability/SBOM/provenance and final production runtime-graph evidence before promotion. | | Dependency-root governance | External owner paths / non-passing | #233 has leaf CI/release/security evidence but still requires authenticated current-head compatibility CodeQL/OpenCode/Noema settlement and a structurally satisfiable independent approval path before normal protected integration. | | Immutable product release | Not yet published | After the production dependency replacement and all gates pass on one integrated protected head, perform version/CHANGELOG/tag/package/SBOM/provenance/reproducibility/rollback publication and verify artifact identity. | | Context Graph / EA projection | Candidate-only until released authority exists | `context-graph-contracts` and `enterprise-architecture-core` currently expose no immutable GitHub release. Do not pin mutable producer heads. Continue pg-owned release-readiness seams and adopt only a verified released contract. | @@ -53,4 +55,4 @@ Prompt, response, batch-result, and user data remain pg/product-domain data and ## Evidence discipline -Queued, pending, skipped-required, cancelled, absent, predecessor-head, model-only, and status-only evidence is non-passing. A current blocker is the next work item at its actual owner: pg-owned causes require a realistic RED, the smallest causal repair, focused/full GREEN, and exact-head refetch; foreign-owned causes require advancement of the existing owner path followed by independent pg work. A report, comment, handoff, or documentation-only change is never completion while executable code/test/release work remains. \ No newline at end of file +Queued, pending, skipped-required, cancelled, absent, predecessor-head, model-only, and status-only evidence is non-passing. A current blocker is the next work item at its actual owner: pg-owned causes require a realistic RED, the smallest causal repair, focused/full GREEN, and exact-head refetch; foreign-owned causes require advancement of the existing owner path followed by independent pg work. A report, comment, handoff, or documentation-only change is never completion while executable code/test/release work remains. diff --git a/tests/test_candidate_source_wheel_parity.py b/tests/test_candidate_source_wheel_parity.py new file mode 100644 index 000000000..c84a41cd4 --- /dev/null +++ b/tests/test_candidate_source_wheel_parity.py @@ -0,0 +1,131 @@ +"""Regression tests for immutable pg8000 source-to-wheel parity evidence. + +The commercial PostgreSQL-driver migration pins the published pg8000 wheel, but +a wheel digest alone does not show that its executable package sources match the +published source distribution. These tests require a bounded, non-executing +archive verifier and require CI to run it before candidate installation. +""" + +from __future__ import annotations + +import importlib.util +from io import BytesIO +from pathlib import Path +import tarfile +import zipfile + +import pytest + + +_REPOSITORY_ROOT = Path(__file__).parents[1] +_TOOL_PATH = _REPOSITORY_ROOT / "tools" / "verify_candidate_source_wheel_parity.py" + + +def _load_verifier(): + """Load the repository-owned parity verifier without making tools a package.""" + assert _TOOL_PATH.is_file(), "candidate source-wheel parity verifier is missing" + spec = importlib.util.spec_from_file_location("candidate_source_wheel_parity", _TOOL_PATH) + if spec is None or spec.loader is None: + raise AssertionError("candidate source-wheel parity verifier could not be loaded") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _write_sdist(path: Path, sources: dict[str, bytes]) -> None: + """Write a minimal pg8000 source distribution with executable package bytes.""" + with tarfile.open(path, "w:gz") as archive: + for relative_path, payload in sources.items(): + member = tarfile.TarInfo(f"pg8000-1.31.5/src/pg8000/{relative_path}") + member.size = len(payload) + archive.addfile(member, BytesIO(payload)) + documentation = b"candidate docs\n" + member = tarfile.TarInfo("pg8000-1.31.5/README.md") + member.size = len(documentation) + archive.addfile(member, BytesIO(documentation)) + + +def _write_wheel(path: Path, sources: dict[str, bytes]) -> None: + """Write a minimal pg8000 wheel carrying the supplied package source bytes.""" + with zipfile.ZipFile(path, "w") as archive: + for relative_path, payload in sources.items(): + archive.writestr(f"pg8000/{relative_path}", payload) + archive.writestr( + "pg8000-1.31.5.dist-info/METADATA", + "Metadata-Version: 2.4\nName: pg8000\nVersion: 1.31.5\n", + ) + + +def test_candidate_source_and_wheel_require_identical_python_payloads(tmp_path: Path) -> None: + """Exact executable Python sources must agree across the two published artifacts.""" + verifier = _load_verifier() + sources = { + "__init__.py": b"__version__ = '1.31.5'\n", + "core.py": b"def marker():\n return 'same'\n", + } + sdist_path = tmp_path / "pg8000-1.31.5.tar.gz" + wheel_path = tmp_path / "pg8000-1.31.5-py3-none-any.whl" + _write_sdist(sdist_path, sources) + _write_wheel(wheel_path, sources) + + verifier.verify_candidate_source_wheel_parity(sdist_path, wheel_path) + + +def test_candidate_source_wheel_parity_rejects_changed_executable_source(tmp_path: Path) -> None: + """A wheel-side source mutation must fail even when both archive names are expected.""" + verifier = _load_verifier() + sdist_path = tmp_path / "pg8000-1.31.5.tar.gz" + wheel_path = tmp_path / "pg8000-1.31.5-py3-none-any.whl" + _write_sdist(sdist_path, {"core.py": b"VALUE = 'source'\n"}) + _write_wheel(wheel_path, {"core.py": b"VALUE = 'wheel'\n"}) + + with pytest.raises( + verifier.CandidateSourceWheelParityError, + match="package payload differs", + ): + verifier.verify_candidate_source_wheel_parity(sdist_path, wheel_path) + + +def test_candidate_source_wheel_parity_rejects_extra_wheel_python_source(tmp_path: Path) -> None: + """The built wheel must not introduce executable Python absent from the sdist.""" + verifier = _load_verifier() + sdist_path = tmp_path / "pg8000-1.31.5.tar.gz" + wheel_path = tmp_path / "pg8000-1.31.5-py3-none-any.whl" + _write_sdist(sdist_path, {"core.py": b"VALUE = 1\n"}) + _write_wheel( + wheel_path, + { + "core.py": b"VALUE = 1\n", + "injected.py": b"VALUE = 'unexpected'\n", + }, + ) + + with pytest.raises( + verifier.CandidateSourceWheelParityError, + match="package payload differs", + ): + verifier.verify_candidate_source_wheel_parity(sdist_path, wheel_path) + + +def test_candidate_source_wheel_parity_runs_before_candidate_install() -> None: + """CI must hash and compare the source artifact before candidate code is installed.""" + workflow = (_REPOSITORY_ROOT / ".github" / "workflows" / "ci.yml").read_text( + encoding="utf-8" + ) + source_download_step = "- name: Download exact pg8000 candidate source distribution" + source_digest_step = "- name: Verify pg8000 candidate source digest" + parity_step = "- name: Verify pg8000 candidate source-wheel parity" + install_step = "- name: Install exact candidate closure into release Python environments" + verifier_command = "python tools/verify_candidate_source_wheel_parity.py" + source_path = "/tmp/pg8000-candidate-source/pg8000-1.31.5.tar.gz" + wheel_path = "/tmp/pg8000-candidate/pg8000-1.31.5-py3-none-any.whl" + + assert source_download_step in workflow + assert source_digest_step in workflow + assert parity_step in workflow + assert verifier_command in workflow + assert source_path in workflow + assert wheel_path in workflow + assert workflow.index(source_download_step) < workflow.index(source_digest_step) + assert workflow.index(source_digest_step) < workflow.index(parity_step) + assert workflow.index(parity_step) < workflow.index(install_step) diff --git a/tests/test_candidate_wheel_license_verifier.py b/tests/test_candidate_wheel_license_verifier.py index 8088a3c40..a9a9f0f36 100644 --- a/tests/test_candidate_wheel_license_verifier.py +++ b/tests/test_candidate_wheel_license_verifier.py @@ -175,7 +175,9 @@ def test_candidate_license_gate_runs_before_candidate_install() -> None: encoding="utf-8" ) verification_step = "- name: Verify pg8000 candidate dependency licenses" - install_step = "- name: Install exact candidate closure into the CI environment" + install_step = ( + "- name: Install exact candidate closure into release Python environments" + ) assert verification_step in workflow assert "python tools/verify_candidate_wheel_licenses.py /tmp/pg8000-candidate" in workflow diff --git a/tests/test_pg8000_candidate_python_matrix.py b/tests/test_pg8000_candidate_python_matrix.py new file mode 100644 index 000000000..23277f922 --- /dev/null +++ b/tests/test_pg8000_candidate_python_matrix.py @@ -0,0 +1,35 @@ +"""Regression for exact-artifact pg8000 parity across shipped Python runtimes.""" + +from __future__ import annotations + +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + + +def test_real_pg8000_postgres_smoke_covers_release_runtime_matrix() -> None: + """Run the installed package plus real candidate/PostgreSQL contract per minor.""" + workflow = (ROOT / ".github/workflows/ci.yml").read_text(encoding="utf-8") + + expected_runtimes = { + "3.10": "/tmp/pg8000-candidate-py310/bin/python", + "3.12": "/tmp/pg8000-candidate-py312/bin/python", + "3.14": "/tmp/pg8000-candidate-py314/bin/python", + } + assert "Build exact pg-llm-batch wheel for candidate parity" in workflow + assert "uv build --wheel --no-sources" in workflow + assert "/tmp/pg-llm-batch-candidate-wheel" in workflow + assert "Install exact pg-llm-batch wheel into candidate environments" in workflow + + for python_version, interpreter in expected_runtimes.items(): + assert f'python-version: "{python_version}"' in workflow + assert f'uv pip install --python "{interpreter}" --no-deps' in workflow + assert f"uv pip check --python {interpreter}" in workflow + assert f'cd "$RUNNER_TEMP" && {interpreter} ' in workflow + assert '"$GITHUB_WORKSPACE/tests/smoke_pg8000_candidate_postgres.py"' in workflow + + assert "PYTHONPATH=." not in workflow + assert workflow.count("tests/smoke_pg8000_candidate_postgres.py") == len( + expected_runtimes + ) diff --git a/tests/test_workflow_contracts.py b/tests/test_workflow_contracts.py index deb21a8d8..4d5304309 100644 --- a/tests/test_workflow_contracts.py +++ b/tests/test_workflow_contracts.py @@ -250,8 +250,15 @@ def test_ci_pg8000_candidate_pins_and_hashes_full_dependency_closure() -> None: assert requirement in workflow assert digest in workflow + expected_interpreters = ( + "/tmp/pg8000-candidate-py310/bin/python", + "/tmp/pg8000-candidate-py312/bin/python", + "/tmp/pg8000-candidate-py314/bin/python", + ) assert "pip download --no-deps --only-binary=:all:" in workflow - assert "uv pip install --python .venv/bin/python --no-deps" in workflow + assert 'uv pip install --python "$interpreter" --no-deps' in workflow + for interpreter in expected_interpreters: + assert interpreter in workflow assert "/tmp/pg8000-candidate/pg8000-1.31.5-py3-none-any.whl" in workflow assert ( "/tmp/pg8000-candidate/python_dateutil-2.9.0.post0-py2.py3-none-any.whl" diff --git a/tools/verify_candidate_source_wheel_parity.py b/tools/verify_candidate_source_wheel_parity.py new file mode 100644 index 000000000..833a9d78c --- /dev/null +++ b/tools/verify_candidate_source_wheel_parity.py @@ -0,0 +1,194 @@ +#!/usr/bin/env python3 +"""Verify executable pg8000 sources agree between pinned sdist and wheel artifacts. + +PyPI publishes pg8000 1.31.5 as both a source distribution and a universal wheel, +but the release was uploaded without Trusted Publishing. The repository already +pins and license-checks the wheel used for candidate execution. This verifier +adds an independent, non-executing artifact-consistency check: every Python +source shipped under the pg8000 package in the exact wheel must have the same +path and bytes in the exact source distribution, and neither artifact may add a +Python module absent from the other. + +The verifier never extracts archives, imports candidate code, follows archive +links, or accepts alternate artifact names. Finite member, file-count, and total +payload limits keep malformed archives from turning provenance inspection into an +unbounded resource operation. +""" + +from __future__ import annotations + +from hashlib import sha256 +from pathlib import Path, PurePosixPath +import sys +import tarfile +import zipfile + + +_EXPECTED_SDIST_NAME = "pg8000-1.31.5.tar.gz" +_EXPECTED_WHEEL_NAME = "pg8000-1.31.5-py3-none-any.whl" +_SDIST_PACKAGE_PREFIX = "pg8000-1.31.5/src/pg8000/" +_WHEEL_PACKAGE_PREFIX = "pg8000/" +_MAX_PACKAGE_FILES = 512 +_MAX_MEMBER_BYTES = 2 * 1024 * 1024 +_MAX_PACKAGE_BYTES = 8 * 1024 * 1024 + + +class CandidateSourceWheelParityError(RuntimeError): + """Reject candidate artifacts that cannot prove source-to-wheel parity.""" + + +def _relative_python_path(member_name: str, *, prefix: str) -> str | None: + """Return one bounded package-relative Python path or ``None`` for other files.""" + if not member_name.startswith(prefix): + return None + relative = member_name[len(prefix) :] + if not relative or relative.endswith("/") or not relative.endswith(".py"): + return None + path = PurePosixPath(relative) + if path.is_absolute() or any(part in {"", ".", ".."} for part in path.parts): + raise CandidateSourceWheelParityError("candidate package member path is invalid") + return path.as_posix() + + +def _record_payload( + payloads: dict[str, str], + *, + relative_path: str, + payload: bytes, + total_bytes: int, +) -> int: + """Record one source digest while enforcing finite and unique package evidence.""" + if len(payload) > _MAX_MEMBER_BYTES: + raise CandidateSourceWheelParityError("candidate package member exceeds size limit") + if relative_path in payloads: + raise CandidateSourceWheelParityError("candidate package member identity is duplicated") + if len(payloads) >= _MAX_PACKAGE_FILES: + raise CandidateSourceWheelParityError("candidate package file count exceeds limit") + total_bytes += len(payload) + if total_bytes > _MAX_PACKAGE_BYTES: + raise CandidateSourceWheelParityError("candidate package payload exceeds size limit") + payloads[relative_path] = sha256(payload).hexdigest() + return total_bytes + + +def _sdist_python_payloads(sdist_path: Path) -> dict[str, str]: + """Read bounded Python-source digests from the exact pg8000 source distribution.""" + if sdist_path.name != _EXPECTED_SDIST_NAME or not sdist_path.is_file(): + raise CandidateSourceWheelParityError("candidate source artifact identity is invalid") + + payloads: dict[str, str] = {} + total_bytes = 0 + try: + with tarfile.open(sdist_path, mode="r:gz") as archive: + for member in archive.getmembers(): + relative = _relative_python_path( + member.name, + prefix=_SDIST_PACKAGE_PREFIX, + ) + if relative is None: + continue + if not member.isfile(): + raise CandidateSourceWheelParityError( + "candidate source package member is not a regular file" + ) + if member.size < 0 or member.size > _MAX_MEMBER_BYTES: + raise CandidateSourceWheelParityError( + "candidate package member exceeds size limit" + ) + stream = archive.extractfile(member) + if stream is None: + raise CandidateSourceWheelParityError( + "candidate source package member could not be inspected" + ) + payload = stream.read(_MAX_MEMBER_BYTES + 1) + if len(payload) != member.size: + raise CandidateSourceWheelParityError( + "candidate source package member size is inconsistent" + ) + total_bytes = _record_payload( + payloads, + relative_path=relative, + payload=payload, + total_bytes=total_bytes, + ) + except CandidateSourceWheelParityError: + raise + except (OSError, tarfile.TarError, EOFError): + raise CandidateSourceWheelParityError( + "candidate source artifact could not be inspected" + ) from None + + if not payloads: + raise CandidateSourceWheelParityError("candidate source package payload is empty") + return payloads + + +def _wheel_python_payloads(wheel_path: Path) -> dict[str, str]: + """Read bounded Python-source digests from the exact pg8000 universal wheel.""" + if wheel_path.name != _EXPECTED_WHEEL_NAME or not wheel_path.is_file(): + raise CandidateSourceWheelParityError("candidate wheel artifact identity is invalid") + + payloads: dict[str, str] = {} + total_bytes = 0 + try: + with zipfile.ZipFile(wheel_path) as archive: + for member in archive.infolist(): + relative = _relative_python_path( + member.filename, + prefix=_WHEEL_PACKAGE_PREFIX, + ) + if relative is None: + continue + if member.is_dir() or member.file_size > _MAX_MEMBER_BYTES: + raise CandidateSourceWheelParityError( + "candidate package member exceeds size limit" + ) + payload = archive.read(member) + if len(payload) != member.file_size: + raise CandidateSourceWheelParityError( + "candidate wheel package member size is inconsistent" + ) + total_bytes = _record_payload( + payloads, + relative_path=relative, + payload=payload, + total_bytes=total_bytes, + ) + except CandidateSourceWheelParityError: + raise + except (OSError, zipfile.BadZipFile, RuntimeError, ValueError): + raise CandidateSourceWheelParityError( + "candidate wheel artifact could not be inspected" + ) from None + + if not payloads: + raise CandidateSourceWheelParityError("candidate wheel package payload is empty") + return payloads + + +def verify_candidate_source_wheel_parity(sdist_path: Path, wheel_path: Path) -> None: + """Require exact Python package path and byte parity across pinned artifacts.""" + if not isinstance(sdist_path, Path) or not isinstance(wheel_path, Path): + raise CandidateSourceWheelParityError("candidate artifact path is invalid") + source_payloads = _sdist_python_payloads(sdist_path) + wheel_payloads = _wheel_python_payloads(wheel_path) + if source_payloads != wheel_payloads: + raise CandidateSourceWheelParityError("candidate source and wheel package payload differs") + + +def main(argv: list[str] | None = None) -> int: + """Run source-to-wheel parity verification for one exact pg8000 candidate pair.""" + arguments = sys.argv[1:] if argv is None else argv + if len(arguments) != 2: + raise SystemExit( + "usage: verify_candidate_source_wheel_parity.py SDIST_PATH WHEEL_PATH" + ) + try: + verify_candidate_source_wheel_parity(Path(arguments[0]), Path(arguments[1])) + except CandidateSourceWheelParityError as exc: + raise SystemExit(str(exc)) from None + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 224ed124b675eaf0ec1f558a458286610387500b Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 9 Sep 2026 03:19:18 +0900 Subject: [PATCH 11/14] docs(commercial): restore parent operator contracts --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 2bde3f9c6..f4a4726e0 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,8 @@ llm_requests ──▶ PostgresBatchOrchestrator.prepare_batches() durable lifecycle + tenant/RLS + reconciliation evidence ``` -Provider-facing polling and retrieval stay outside PostgreSQL. The former bundled `pg_cron` + `pgsql-http` provider retriever is retired; automatic reconciliation is a separate product capability rather than a second database-side network authority. +Provider-facing polling and retrieval stay outside PostgreSQL. The former bundled `pg_cron` + `pgsql-http` provider retriever is +retired; automatic reconciliation is a separate product capability rather than a second database-side network authority. | Piece | Module | | --- | --- | @@ -129,7 +130,7 @@ python -m pg_llm_batch wait --endpoint default --batch-id --poll- python -m pg_llm_batch retrieve --endpoint default --batch-id ``` -`count-tokens` accepts strict UTF-8 only through the explicit stdin source and enforces its bounded input contract before configuration-store or PostgreSQL acquisition. Prompt text is not accepted through an argv option and rejected content is not reflected into parser/runtime diagnostics. +`count-tokens` accepts at most 1 MiB of strict UTF-8 only through the explicit stdin source and enforces that bound before configuration-store or PostgreSQL acquisition. Prompt text is not accepted through an argv option and rejected content is not reflected into parser/runtime diagnostics. Programmatic preparation remains available: @@ -199,7 +200,7 @@ See [`docs/remote-batch-lifecycle.md`](docs/remote-batch-lifecycle.md) for migra The repository contains bounded backup, restore, catalog, replay, and recovery-evidence primitives. Each primitive proves only its documented slice; none by itself establishes end-to-end PITR, RPO/RTO, HA/DR, CSAP, SOC 2, or a deployment certification. -For a caller-owned logical archive, `restore_postgres_logical_backup()` uses an isolated libpq-service execution boundary for `pg_restore`. That subprocess contract is distinct from package-created pg8000 connections and retains its own restricted libpq environment. See [`docs/doctoring/postgres-logical-restore.md`](docs/doctoring/postgres-logical-restore.md). +For a caller-owned logical archive, use `restore_postgres_logical_backup()` only against an isolated libpq service after you can assert `source_superusers_trusted=True`. The service name is not an authorization boundary. Only `PGPASSWORD`, `PGPASSFILE`, and `PGSERVICEFILE` may be inherited. The executor runs `pg_restore --single-transaction --exit-on-error`. Custom-format restore seeks through the archive, so success is not required to leave the descriptor at end-of-file. If metadata changes after `pg_restore` exits zero, treat the target as unsafe and do not retry into the same service. This subprocess contract is distinct from package-created pg8000 connections. See [`docs/doctoring/postgres-logical-restore.md`](docs/doctoring/postgres-logical-restore.md). ## Embedding boundary @@ -234,7 +235,7 @@ The credentials provider is an anti-corruption seam: callers may use the package Files/Batches control-plane JSON uses an independent decoded-byte budget before strict UTF-8 and JSON-object parsing. Provider result/error files are streamed in bounded chunks and checked against `max_download_bytes` before JSONL parsing. Adapters that cannot provide the required bounded stream contract fail closed. -Idempotent provider `GET` operations may retry reviewed transient HTTP/transport failures within bounded attempts and backoff. Upload, batch creation, and cancellation `POST` operations are not retried automatically. TLS handshake/certificate and peer-identity failures are not treated as ordinary transient retries. +Idempotent provider `GET` operations use up to three total attempts by default for transient `408`, `425`, `429`, `502`, `503`, and `504` responses and for retryable aiohttp transport failures. TLS handshake and certificate failures are never retried automatically; they fail after the first attempt because repeating a request cannot repair peer identity or TLS policy. Certificate fingerprint mismatches are never retried automatically for the same peer-identity reason. A bounded RFC `Retry-After` delta or HTTP-date is honored. Upload, batch creation, and cancellation `POST` operations are not retried automatically. ## Observability From 27ba31da6f45055d95d805108f1b0d1be8b0c433 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 16 Sep 2026 13:04:31 +0900 Subject: [PATCH 12/14] docs(recovery): expose bounded target observer contract --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f4a4726e0..961ce529c 100644 --- a/README.md +++ b/README.md @@ -200,6 +200,8 @@ See [`docs/remote-batch-lifecycle.md`](docs/remote-batch-lifecycle.md) for migra The repository contains bounded backup, restore, catalog, replay, and recovery-evidence primitives. Each primitive proves only its documented slice; none by itself establishes end-to-end PITR, RPO/RTO, HA/DR, CSAP, SOC 2, or a deployment certification. +The Draft recovery-target observer in `pg_llm_batch.postgres_recovery_target_configuration` remains module-scoped. `observe_postgres_recovery_target_configuration(...)` performs one fixed catalog-qualified read over the eight reviewed PostgreSQL recovery-target settings plus `pg_is_in_recovery()` on an already-connected isolated recovery target; `postgres_recovery_target_configuration_was_observed(...)` checks the resulting bounded evidence. This proves only that the effective settings visible to that connection match the reviewed target contract while recovery is active. It does not write PostgreSQL configuration, create `recovery.signal`, supply `restore_command`, validate or replay WAL bytes, prove archive completeness or timeline ancestry, prove target attainment or replay completion, pause/resume/promote recovery, prove application readiness, or establish achieved RPO/RTO, HA/DR, CSAP, SOC 2, or certification. The caller owns connection and timeout policy, and the observer's diagnostics/evidence remain content-minimal. + For a caller-owned logical archive, use `restore_postgres_logical_backup()` only against an isolated libpq service after you can assert `source_superusers_trusted=True`. The service name is not an authorization boundary. Only `PGPASSWORD`, `PGPASSFILE`, and `PGSERVICEFILE` may be inherited. The executor runs `pg_restore --single-transaction --exit-on-error`. Custom-format restore seeks through the archive, so success is not required to leave the descriptor at end-of-file. If metadata changes after `pg_restore` exits zero, treat the target as unsafe and do not retry into the same service. This subprocess contract is distinct from package-created pg8000 connections. See [`docs/doctoring/postgres-logical-restore.md`](docs/doctoring/postgres-logical-restore.md). ## Embedding boundary @@ -274,4 +276,4 @@ The pg-llm-batch repository's original source is Apache-2.0; see [`LICENSE`](LIC On this Draft stack, the default runtime manifest pins `pg8000==1.31.5`; Psycopg is optional test/development-only legacy-adapter evidence. Issue #322 remains open because commercial acceptance is not a branch-local dependency declaration: the change must reach protected main through normal governance and the immutable release must re-prove the final package, dependency-license inventory, vulnerability state, SBOM, provenance, reproducibility, and rollback evidence. -Issue #123 independently remains open for package-created remote PostgreSQL transport encryption and authenticated server identity. Do not present a green #323/#321 Draft as completion of either protected release or secure-remote transport policy. +Issue #123 independently remains open for package-created remote PostgreSQL transport encryption and authenticated server identity. Do not present a green #323/#321 Draft as completion of either protected release or secure-remote transport policy. \ No newline at end of file From 36261459b9acaddfeb1bbd45ae027d09ff92cb9f Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Wed, 16 Sep 2026 13:04:47 +0900 Subject: [PATCH 13/14] docs(recovery): align public index with target observer --- docs/index.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/index.md b/docs/index.md index 0709ed6d2..5786c5f22 100644 --- a/docs/index.md +++ b/docs/index.md @@ -53,6 +53,12 @@ Key references: - [Count-tokens stdin privacy](doctoring/count-tokens-stdin-privacy.md) - [OpenTelemetry operations](doctoring/opentelemetry-operations.md) +## Recovery evidence boundary + +The Draft recovery-target observer remains module-scoped under `pg_llm_batch.postgres_recovery_target_configuration`. It reads the eight reviewed PostgreSQL recovery-target settings plus `pg_is_in_recovery()` in one fixed catalog-qualified query on an already-connected isolated recovery target and emits bounded evidence for whether those effective settings match the reviewed target contract while recovery is active. + +That observation is deliberately narrower than recovery completion. It does not mutate PostgreSQL configuration, create `recovery.signal`, supply `restore_command`, validate or replay WAL bytes, prove archive completeness or timeline ancestry, prove target attainment or replay completion, pause/resume/promote recovery, prove application readiness, or establish achieved RPO/RTO, HA/DR, CSAP, SOC 2, or certification. Connection and timeout policy remain caller-owned, and evidence/diagnostics must not expose DSNs, credentials, provider payloads, or dynamic database exception text. + ## Releases and verification Use protected-branch history, GitHub Releases, current checks, and exact-source repository evidence to determine what is shipped. A documentation source commit is not evidence that GitHub Pages is published, and a green Draft is not evidence that a dependency migration is commercially released. @@ -64,4 +70,4 @@ Commercial driver readiness requires #322 to reach protected release authority w - [Apache-2.0 source license](https://github.com/ContextualWisdomLab/pg-llm-batch/blob/main/LICENSE) - [Commercial dependency transition #322](https://github.com/ContextualWisdomLab/pg-llm-batch/issues/322) - [PostgreSQL transport-security gap #123](https://github.com/ContextualWisdomLab/pg-llm-batch/issues/123) -- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/pg-llm-batch) +- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/pg-llm-batch) \ No newline at end of file From d5057439ecf41e237958905c001894b045383d10 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sat, 19 Sep 2026 23:02:37 +0900 Subject: [PATCH 14/14] docs: bound secret and health disclosure claims --- docs/index.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/index.md b/docs/index.md index 5786c5f22..28ceedaee 100644 --- a/docs/index.md +++ b/docs/index.md @@ -44,6 +44,12 @@ PostgreSQL owns authoritative token accounting and package-owned durable lifecyc The current Draft runtime selects exact pg8000 behind `PostgresDriverPort`. Psycopg remains only optional legacy-adapter test evidence. Unsupported libpq connection semantics fail closed at the pg8000 anti-corruption boundary rather than being silently approximated. +### Secret-storage and diagnostic boundaries + +`SecretStore` is not an encryption-at-rest guarantee by default. With a Fernet key it encrypts package-managed secret values; without one it retains the compatibility path that base64-obfuscates values unless the caller explicitly sets `require_encryption=True`. [Issue #121](https://github.com/ContextualWisdomLab/pg-llm-batch/issues/121) remains the owner for mandatory encryption, migration of existing unencrypted rows, key rotation/recovery, and external key custody. + +`/healthz` publishes only fixed required-component names and boolean readiness. Operator-facing health internals are a separate boundary and must not be treated as tenant-visible or public telemetry merely because the HTTP readiness projection is content-minimal. [Issue #203](https://github.com/ContextualWisdomLab/pg-llm-batch/issues/203) remains the authority for CLI diagnostic-disclosure hardening. + Key references: - [Remote batch lifecycle](remote-batch-lifecycle.md)