diff --git a/docs/adr/0023-pg8000-remote-tls-server-identity.md b/docs/adr/0023-pg8000-remote-tls-server-identity.md new file mode 100644 index 000000000..48ee9f9bb --- /dev/null +++ b/docs/adr/0023-pg8000-remote-tls-server-identity.md @@ -0,0 +1,134 @@ +# ADR 0023: Authenticate remote PostgreSQL server identity in the pg8000 production adapter + +- Status: Proposed +- Date: 2026-09-11 +- Owners: PostgreSQL infrastructure boundary / issue #123 +- Decision branch: Draft #342 + +## Problem + +The production-driver migration in Draft #323 selects pg8000 1.31.5 behind `PostgresDriverPort`. The inherited `Pg8000CandidateDriverAdapter.connect()` passes the validated host, port, database, user, optional password, and optional connect timeout to pg8000, but supplies no `ssl_context`. + +pg8000 documents `ssl_context=None` as attempting SSL and then falling back to an ordinary socket when the server refuses SSL. That is incompatible with the package-created remote PostgreSQL connection boundary: a remote PostgreSQL selector must not silently lose transport encryption, and encryption without certificate and hostname verification is not server authentication. + +The same package is used for local development and embedding. Existing local PostgreSQL runtime smokes use explicit loopback targets that are not provisioned with a trusted TLS identity. Host applications may also inject another `PostgresDriverPort` when they intentionally own connection construction. The package therefore needs a narrow default policy with an explicit distinction between host trust-store authority and package connection configuration. + +Python's TLS defaults create a second security consideration. `ssl.create_default_context()` enables TLS key logging when process environment variable `SSLKEYLOGFILE` is set. Default CA loading also follows the host OpenSSL/platform trust configuration; on OpenSSL-backed platforms, the default verify paths expose the process-level CA environment keys conventionally named `SSL_CERT_FILE` and `SSL_CERT_DIR`. A package-created database connection must not silently export TLS session keys, while host-level CA trust remains a deployment/platform concern rather than a DSN or package-secret concern. + +TLS verification failures are a third boundary. pg8000 passes its supplied `SSLContext` into `wrap_socket(..., server_hostname=host)`, so certificate and hostname verification failures can surface as Python `ssl.SSLError` subclasses whose rendered text contains the remote identity or other TLS details. pg8000 1.31.5 separately reports a server that refuses SSL as `dbapi.InterfaceError("Server refuses SSL")` even when an explicit `SSLContext` requires TLS. Passing either raw TLS failure shape through would contradict this repository's content-free production-diagnostic contract even though the connection itself failed securely. + +## Constraints + +- Preserve the provider-neutral `PostgresDriverPort` and its current selector, timeout, transaction, thread-affinity, JSONB, SQLSTATE, and service-file semantics. +- Preserve exact pg8000 1.31.5 distribution and import-origin admission in `pg8000_driver_adapter`. +- Do not add ambient `PG*`, `.env`, arbitrary DSN TLS flags, certificate paths, or per-connection trust-material discovery as a second package configuration authority. +- Admit the Python/OpenSSL/platform default CA store as host trust authority. On platforms where OpenSSL default verify paths honor process-level CA location variables, those variables are part of the host trust-store boundary, not package selector grammar. +- Do not honor `SSLKEYLOGFILE` as package-created PostgreSQL TLS key-export authority. +- Keep the package's explicit injected-driver seam for embedding hosts that own a different connection policy. +- Keep diagnostics content-free: TLS-policy construction and handshake failures must not expose certificate-store paths, hosts, DSNs, usernames, passwords, or platform details through package-authored diagnostics or acceptance output. +- Preserve non-TLS database/authentication and unrelated DB-API interface failures instead of converting every pg8000 connection failure into a TLS-policy error. +- Keep the deliberate local-development exception explicit and mechanically bounded to `localhost`, IPv4 loopback (`127.0.0.0/8`), and IPv6 loopback (`::1`). +- Do not treat a unit-level SSL-context contract as proof of real certificate or PostgreSQL TLS behavior. + +## Alternatives considered + +### Keep pg8000's default `ssl_context=None` + +Rejected. pg8000 may fall back to plaintext if the remote server refuses SSL. This fails the remote fail-closed requirement and provides no package-level server-identity guarantee. + +### Pass `ssl_context=True` + +Rejected. pg8000 documents this as an SSL context with minimum checks. It requires SSL but does not establish the explicit certificate and hostname-verification contract required by issue #123. + +### Require verified TLS for every connection, including loopback + +Deferred. This is the preferred long-term deployment posture, but the current local PostgreSQL development/runtime-smoke path has no reviewed certificate provisioning contract. Making that unrelated infrastructure migration part of this repair would broaden the bounded change and would obscure the remote downgrade defect. Loopback is therefore a temporary explicit exception, not a general private-network exception. + +### Parse caller-supplied TLS flags or trust material through DSN/package environment configuration + +Rejected for this slice. That would expand the connection grammar and create new trust-material precedence, secrecy, validation, and configuration-governance contracts. A future caller-owned trust-policy object may be added through a separately reviewed port if a deployment needs per-connection or per-tenant trust selection. + +This rejection does not redefine the operating system/Python/OpenSSL CA store as package configuration. The default CA loader may consume host-managed trust locations, including OpenSSL default-path environment variables where the platform supports them. That authority is process/deployment scoped and is deliberately covered by the real TLS acceptance harness. + +### Use `ssl.create_default_context()` unchanged + +Rejected after review. Its peer-verification defaults are suitable, but Python documents that it enables TLS key logging when `SSLKEYLOGFILE` is present. Package-created PostgreSQL sessions must not acquire an ambient session-key export sink merely because another process-level debugging setting exists. + +### Construct a strict client context, load host default CAs, and never auto-enable key logging + +Selected. The adapter constructs `SSLContext(PROTOCOL_TLS_CLIENT)`, explicitly enables `VERIFY_X509_PARTIAL_CHAIN | VERIFY_X509_STRICT`, calls `load_default_certs()`, and validates `check_hostname=True`, `verify_mode=CERT_REQUIRED`, and `keylog_filename is None` before raw pg8000 access. This retains the platform trust store and hostname authentication while preventing `SSLKEYLOGFILE` from enabling key export through this package path. Explicit loopback targets keep the existing local-development behavior. + +### Propagate raw TLS handshake exceptions from pg8000 + +Rejected after review. Python certificate-verification diagnostics can embed the server identity and other TLS detail. pg8000 1.31.5 also uses its DB-API `InterfaceError` with exact argument `"Server refuses SSL"` when the server rejects the SSLRequest. The package owns the remote TLS requirement and therefore owns the confidentiality boundary for both forms. Python `ssl.SSLError` and only that exact admitted pg8000 refusal shape are normalized; unrelated pg8000/database/interface exceptions retain their existing semantics. + +## Decision + +`Pg8000DriverAdapter`, the production implementation selected by `retained_postgres_driver()`, wraps only the already-admitted pg8000 DB-API connection factory. + +For a validated non-loopback host it: + +1. creates a fresh `SSLContext(PROTOCOL_TLS_CLIENT)`; +2. explicitly enables `VERIFY_X509_PARTIAL_CHAIN` and `VERIFY_X509_STRICT`; +3. loads the platform/Python/OpenSSL default CA trust through `load_default_certs()`; +4. verifies `check_hostname is True`, `verify_mode == ssl.CERT_REQUIRED`, and `keylog_filename is None`; +5. supplies that exact context as pg8000's `ssl_context` argument; +6. fails before raw driver access with `PostgreSQL TLS policy is unavailable` if the trust context cannot be constructed or any of those invariants is weakened; +7. converts Python `ssl.SSLError` raised by that remote connection attempt into the same fixed content-free TLS-policy error with exception chaining suppressed; and +8. converts only the exact admitted pg8000 DB-API `InterfaceError("Server refuses SSL")` shape into that same fixed error, while re-raising every unrelated interface/database failure unchanged. + +For exact loopback identities (`localhost`, IPv4 loopback, IPv6 loopback), it does not inject `ssl_context`; this preserves the current bounded development exception. Private RFC1918/ULA addresses, Kubernetes/service DNS names, and other non-loopback hosts are remote for this policy and receive verified TLS. The loopback path also does not apply the remote TLS exception normalizer because the package does not own a TLS handshake on that deliberate development path. + +The lower-level candidate adapter remains policy-neutral so its parser/adapter behavior is not duplicated or forked. Host software that deliberately owns a different TLS connection policy continues to inject a `PostgresDriverPort`; it does not mutate this package's DSN grammar. + +The host CA store is an explicit deployment authority. On OpenSSL-backed platforms, Python's default verify paths may honor `SSL_CERT_FILE` and `SSL_CERT_DIR`; the package does not parse those values, copy trust material, or expose them through its selectors. `SSLKEYLOGFILE` is different: it controls export of TLS session keys rather than trust anchors, so package-created PostgreSQL contexts deliberately do not inherit it. + +The existing permanent pg8000 candidate PostgreSQL smoke is also the realistic acceptance owner for this boundary. It uses the disposable repository PostgreSQL container's non-loopback bridge address, provisions an ephemeral CI-only CA and server identity, and executes the production `Pg8000DriverAdapter` rather than a TLS mock. The acceptance harness binds the ephemeral CA through the same host trust-store path used by `load_default_certs()`. The matrix requires: + +- trusted CA plus matching IP subject alternative name to establish TLS, confirmed from `pg_catalog.pg_stat_ssl`; +- an unrelated CA to fail verification; +- a CA-trusted certificate with a mismatching IP subject alternative name to fail peer-identity verification; +- the same remote selector to fail when PostgreSQL TLS is disabled, proving no plaintext downgrade; +- every negative real-PostgreSQL TLS case to surface exactly `Pg8000DriverTlsPolicyError("PostgreSQL TLS policy is unavailable")` with no chained cause rather than an arbitrary pg8000/OpenSSL exception; +- ambient `SSLKEYLOGFILE` not to become a key-log sink for the constructed production context; and +- TLS failure rendering used by the acceptance harness not to contain the ephemeral database password. + +Focused unit contracts additionally inject a secret-bearing `SSLCertVerificationError` and pg8000's pinned `InterfaceError("Server refuses SSL")` at the exact raw-driver seam and require the externally rendered package exception to be only `PostgreSQL TLS policy is unavailable`, with no chained cause. A separate regression requires an unrelated `InterfaceError` to remain native. Together these contracts make diagnostic confidentiality deterministic without weakening the real PostgreSQL matrix or collapsing unrelated driver semantics. + +The test PKI itself must remain RFC 5280-conforming. The ephemeral CA asserts critical `basicConstraints = CA:TRUE` and critical `keyUsage = keyCertSign,cRLSign`; leaf certificates assert critical `CA:FALSE`, TLS server key usage, `extendedKeyUsage = serverAuth`, subject/authority key identifiers, and the tested IP SAN. The harness does not disable `VERIFY_X509_STRICT` to make malformed test certificates pass. + +## Evidence + +The test-first head `eee15706ffe214cd744667740fab75303a9835f8` added only the remote-TLS regression. Hosted CI run `34561345681` reached the non-integration suite and produced the expected RED: three remote-host cases failed because pg8000 kwargs contained no `ssl_context`; 1681 tests passed, 3 failed, and 5 were deselected. The later workflow cancellation caused by descendant commits does not change that already-terminal failing job evidence. + +Minimum production repair `77089494bed2ee4b81cda0d7bc46446f6cc81fc8` added the first production TLS policy without changing the abstract port. Exact candidate `7f6864cd4fb94ef9a0e76955b06babad990c8c00` additionally covered trust-context construction failure and rejected contexts with either hostname verification or `CERT_REQUIRED` disabled. + +The first realistic TLS acceptance head `dc6b1cc66065117fbd6a93acce8dcb0b94afcc56` then produced a useful compatibility RED in CI `34564646091`. Python 3.10 and 3.12 completed the real pg8000/PostgreSQL smoke, while Python 3.14 rejected the harness-generated CA during the matching-identity success case with `CERTIFICATE_VERIFY_FAILED` because the CA certificate did not contain a key-usage extension. This was a test-PKI defect, not a reason to weaken production verification. Python 3.13+ enables `VERIFY_X509_STRICT` in `create_default_context()` by default, and RFC 5280 defines the CA/basic-constraints and certificate-signing key-usage relationship. Descendant `92e5b8ec4246329080f34bc772dbd60102d3df11` repaired only the generated CI certificates with explicit CA/leaf constraints and key usages; it did not disable strict verification. + +Review of exact `c61a91a36e76cd9b9127e1d9eb98aff776bfdf48` found a second policy-authority defect. Python 3.14 documents that `create_default_context()` honors `SSLKEYLOGFILE`, while the branch claimed that no ambient TLS environment authority existed. The existing real smoke also intentionally used the OpenSSL CA environment path, showing that CA trust and key export had been conflated in the ADR. Test-first `506fc36499ac191d6ea328e0bdf20e2df1e65d95` adds the regression that package-created remote TLS must not inherit ambient key logging. Descendants replace `create_default_context()` with an explicit client context, retain strict X.509 and host default CA loading, and reject any constructed context with key logging enabled. + +Fresh review then found the diagnostic half of the same trust boundary incomplete: `secure_connect()` supplied the verified context but returned `raw_connect(**kwargs)` without normalizing `ssl.SSLError`, even though this ADR already required handshake diagnostics to be content-free. Test-first `571ac5524c8f28106386f441ad269ab9eebf1f80` injects an `SSLCertVerificationError` containing a host, credential token, and CA path and requires the fixed package TLS-policy error instead. Its hosted workflows were only queued/in progress when the ordinary causal repair followed, so that generation is not claimed as terminal hosted RED evidence. The source-level RED is direct: the predecessor returned the raw connect call without an exception boundary, so the injected `SSLCertVerificationError` escaped unchanged. Ordinary child `4035b19fc19da4443995f95460889f9c5b8f9163` adds the remote-only `SSLError` normalization while preserving non-TLS driver errors and loopback behavior. + +A subsequent exact-surface review found that this exception boundary was still incomplete for the real no-downgrade path. pg8000 1.31.5 documents `InterfaceError` as the interface exception used when an SSL connection is attempted and the server refuses it, while the real PostgreSQL smoke accepted any exception for its negative TLS cases. Test-first `09a8e3151317b38362df89cb0b4eddcfe75cc9d0` pins the admitted `InterfaceError("Server refuses SSL")` shape and independently proves an unrelated `InterfaceError` must remain native. Its workflow generation did not materialize before the causal source descendant, so it is not claimed as hosted RED. The predecessor source deterministically caught only `ssl.SSLError`, so the injected pg8000 refusal escaped unchanged. Source repair `ec3676b43762509dcf4ef7693ef8f4e3dd1f297f` narrowly recognizes the exact admitted pg8000 exception type/argument pair and maps only that shape to the fixed package TLS-policy error. Real-smoke descendant `f93ac63e119d6dc23c9eed579a770ec5a656456d` then tightens every negative real-PostgreSQL TLS case to require that exact package error with no chained cause instead of treating an arbitrary exception as acceptance success. + +Earlier ADR-bearing validation also exposed repository contracts rather than TLS-policy defects: the ADR heading must use canonical `# ADR NNNN:` form, and every owned production nested callable must carry a docstring to preserve 100% docstring coverage. Those findings were repaired on ordinary descendants without weakening either gate. + +## Consequences and follow-up + +Remote package-created pg8000 connections can no longer rely on pg8000's plaintext fallback once this branch is normally integrated. The trust anchor is the Python/platform/OpenSSL default CA store, hostname verification uses the validated host supplied to pg8000, strict X.509 validation is enabled consistently across supported Python versions, and process-level `SSLKEYLOGFILE` does not enable PostgreSQL TLS session-key export through this adapter. Certificate/hostname failures and the admitted pg8000 server-refuses-SSL negotiation failure from the package-owned remote TLS path no longer expose raw Python/OpenSSL/pg8000 diagnostic detail to callers. Unrelated DB-API interface failures retain their native semantics. + +Enterprise private CAs can participate through the deployment's default trust-store authority where the platform supports it. A future explicit caller-owned trust-policy capability is still appropriate when trust selection must vary per connection, tenant, or application boundary. That design must define authority, public certificate custody, precedence, cache/lifetime behavior, diagnostics, and interaction with service-file parsing rather than silently expanding DSN grammar. + +The branch contains realistic TLS-enabled PostgreSQL acceptance for the core issue #123 transport matrix. Issue closure still requires this exact capability to survive current-head CI, independent review/thread resolution, normal protected-stack integration, post-integration acceptance, and immutable release evidence. The acceptance does not prove certificate rotation/revocation operations or caller-specific trust-policy semantics. + +## References + +Cooper, D., Santesson, S., Farrell, S., Boeyen, S., Housley, R., & Polk, W. (2008). *Internet X.509 public key infrastructure certificate and certificate revocation list (CRL) profile* (RFC 5280). RFC Editor. https://www.rfc-editor.org/rfc/rfc5280 + +pg8000 project. (n.d.). *pg8000 1.31.5 documentation*. PyPI. https://pypi.org/project/pg8000/1.31.5/ + +PostgreSQL Global Development Group. (n.d.). *SSL support*. PostgreSQL 17 documentation. https://www.postgresql.org/docs/17/libpq-ssl.html + +PostgreSQL Global Development Group. (n.d.). *Database connection control functions*. PostgreSQL 17 documentation. https://www.postgresql.org/docs/17/libpq-connect.html + +Python Software Foundation. (2026). *ssl — TLS/SSL wrapper for socket objects*. Python 3.14 documentation. https://docs.python.org/3.14/library/ssl.html diff --git a/pg_llm_batch/pg8000_driver_adapter.py b/pg_llm_batch/pg8000_driver_adapter.py index 99558b5c8..85c09441d 100644 --- a/pg_llm_batch/pg8000_driver_adapter.py +++ b/pg_llm_batch/pg8000_driver_adapter.py @@ -15,15 +15,29 @@ from importlib import import_module from importlib.metadata import Distribution, PackageNotFoundError, distribution from importlib.util import find_spec +from ipaddress import ip_address from pathlib import Path +from ssl import ( + CERT_REQUIRED, + PROTOCOL_TLS_CLIENT, + SSLContext, + SSLError, + VERIFY_X509_PARTIAL_CHAIN, + VERIFY_X509_STRICT, +) from types import ModuleType +from typing import Any -from .pg8000_candidate_driver_port import Pg8000CandidateDriverAdapter +from .pg8000_candidate_driver_port import ( + Pg8000CandidateDriverAdapter, + ServiceResolver, +) from .pg8000_candidate_service_file import Pg8000CandidateServiceFileResolver from .postgres_driver_port import PostgresDriverPort PG8000_ADMITTED_VERSION = "1.31.5" +_PG8000_SSL_REFUSAL = "Server refuses SSL" class Pg8000DriverUnavailableError(RuntimeError): @@ -35,14 +49,109 @@ class Pg8000DriverUnavailableError(RuntimeError): """ +class Pg8000DriverTlsPolicyError(RuntimeError): + """Report that the package cannot construct or complete verified remote TLS. + + The diagnostic is deliberately fixed and content-free. It does not expose + certificate-store paths, selectors, hosts, credentials, or platform details. + """ + + +def _is_explicit_loopback_host(host: str) -> bool: + """Return whether a validated host is an explicit local-development target.""" + if host.casefold() == "localhost": + return True + try: + return ip_address(host).is_loopback + except ValueError: + return False + + +def _new_remote_ssl_context() -> SSLContext: + """Create a strict client context without ambient TLS session-key logging.""" + context = SSLContext(PROTOCOL_TLS_CLIENT) + context.verify_flags |= VERIFY_X509_PARTIAL_CHAIN | VERIFY_X509_STRICT + context.load_default_certs() + return context + + +def _verified_remote_ssl_context() -> SSLContext: + """Construct one host-trust TLS context with peer verification enabled.""" + try: + context = _new_remote_ssl_context() + except (OSError, ValueError): + context = None + if ( + context is None + or not context.check_hostname + or context.verify_mode != CERT_REQUIRED + or context.keylog_filename is not None + ): + # Raise after the handler so sensitive TLS details are not retained in + # the public exception object's implicit ``__context__`` reference. + raise Pg8000DriverTlsPolicyError( + "PostgreSQL TLS policy is unavailable" + ) from None + return context + + +def _is_pg8000_ssl_refusal(error: BaseException, interface_error: object) -> bool: + """Recognize only pg8000 1.31.5's pinned remote SSL-refusal exception shape.""" + if not isinstance(interface_error, type) or not issubclass( + interface_error, BaseException + ): + return False + return type(error) is interface_error and error.args == (_PG8000_SSL_REFUSAL,) + + class Pg8000DriverAdapter(Pg8000CandidateDriverAdapter): - """Expose the fully proved pg8000 port semantics under the production name. + """Expose proved pg8000 semantics with verified TLS for remote TCP targets. - The implementation deliberately inherits the already exercised cursor, - connection, selector, JSONB, SQLSTATE, and thread-affinity behavior instead - of copying that logic into a second concrete-driver authority. + The implementation inherits the already exercised cursor, connection, + selector, JSONB, SQLSTATE, timeout, and thread-affinity behavior rather than + copying those contracts. Production construction adds one policy boundary: + non-loopback TCP targets always receive a host-trust ``SSLContext`` with + certificate and hostname verification and without ambient TLS key logging. + TLS handshake failures from that package-owned context, including pg8000's + exact server-refuses-SSL interface error, are normalized at the same + content-free boundary. Explicit localhost/loopback selectors retain the + documented development exception. Embedding hosts that need a different + connection policy retain the existing injected ``PostgresDriverPort`` seam + instead of mutating package connection grammar. """ + def __init__( + self, + dbapi_module: ModuleType, + *, + service_resolver: ServiceResolver | None = None, + ) -> None: + """Bind the admitted module and inject remote TLS into its connect seam.""" + super().__init__(dbapi_module, service_resolver=service_resolver) + raw_connect = self._connect + interface_error = vars(dbapi_module).get("InterfaceError") + + def secure_connect(**kwargs: Any) -> object: + """Inject verified TLS and redact its remote handshake diagnostics.""" + host = kwargs["host"] + if _is_explicit_loopback_host(host): + return raw_connect(**kwargs) + kwargs["ssl_context"] = _verified_remote_ssl_context() + try: + return raw_connect(**kwargs) + except SSLError: + pass + except Exception as error: + if not _is_pg8000_ssl_refusal(error, interface_error): + raise + # Leave the lower-layer handler before raising so the public error + # cannot retain certificate/peer detail through ``__context__``. + raise Pg8000DriverTlsPolicyError( + "PostgreSQL TLS policy is unavailable" + ) from None + + self._connect = secure_connect + def _resolve_origin_path(value: str | Path) -> Path: """Resolve one origin path without exposing filesystem lookup diagnostics.""" @@ -112,7 +221,8 @@ def load_pg8000_driver(*, service_file: Path | None = None) -> PostgresDriverPor or preloaded ``pg8000.dbapi`` module from a different filesystem location while avoiding a second distribution-metadata lookup. Service-file support is opt-in through one caller-selected path; ambient ``PGSERVICEFILE`` - discovery remains outside the admitted contract. + discovery remains outside the admitted contract. Remote TCP selectors then + receive the verified TLS policy owned by ``Pg8000DriverAdapter``. Args: service_file: Optional explicit ``pg_service.conf`` path. When omitted, diff --git a/tests/smoke_pg8000_candidate_postgres.py b/tests/smoke_pg8000_candidate_postgres.py index b28b69f29..3f70ae39c 100644 --- a/tests/smoke_pg8000_candidate_postgres.py +++ b/tests/smoke_pg8000_candidate_postgres.py @@ -6,23 +6,33 @@ checks cover the candidate URI, keyword, and explicit service connection selectors, portable connection/cursor ACL, thread-affine connection use, transaction, parameter, JSONB, UUID/timestamp, affected-row, narrow PostgreSQL -error classification, restore-catalog inspection, transport recovery, and -transaction-local tenant semantics that must be proven before candidate -promotion. +error classification, restore-catalog inspection, transport recovery, +transaction-local tenant semantics, and the production adapter's authenticated +remote-TLS boundary that must be proven before candidate promotion. """ from __future__ import annotations +from contextlib import contextmanager from datetime import datetime, timezone from importlib import metadata +from ipaddress import ip_address import os from pathlib import Path +import subprocess +from tempfile import TemporaryDirectory +import time +from typing import Iterator import uuid from pg8000 import dbapi from pg_llm_batch.pg8000_candidate_driver_port import Pg8000CandidateDriverAdapter from pg_llm_batch.pg8000_candidate_service_file import Pg8000CandidateServiceFileResolver +from pg_llm_batch.pg8000_driver_adapter import ( + Pg8000DriverAdapter, + Pg8000DriverTlsPolicyError, +) from pg_llm_batch.pg8000_driver_candidate_jsonb import adapt_pg8000_jsonb from pg_llm_batch.postgres_restore_acceptance import inspect_postgres_restore_catalog @@ -30,6 +40,7 @@ _EXPECTED_DATABASE = "pgllm" _EXPECTED_USER = "pgllm" _CREDENTIAL_FREE_DSN = "postgresql://pgllm@127.0.0.1:5432/pgllm" +_TLS_DIRECTORY = "/tmp/pg-llm-batch-tls" def _candidate_driver() -> Pg8000CandidateDriverAdapter: @@ -60,6 +71,398 @@ def _connection() -> object: return driver.connect(private_dsn, connect_timeout_seconds=5) +def _run_command(arguments: list[str], *, timeout_seconds: int = 30) -> str: + """Run one bounded local acceptance command with content-free failure output.""" + try: + completed = subprocess.run( + arguments, + check=True, + capture_output=True, + text=True, + timeout=timeout_seconds, + ) + except (OSError, subprocess.SubprocessError): + raise AssertionError("remote TLS acceptance command failed") from None + return completed.stdout.strip() + + +def _candidate_container() -> str: + """Return the CI-owned PostgreSQL container identity without guessing it.""" + container = os.environ.get("PG8000_CANDIDATE_CONTAINER") + if not container: + raise AssertionError("candidate PostgreSQL container identity is unavailable") + return container + + +def _candidate_container_ip(container: str) -> str: + """Resolve and validate the real non-loopback container address used for TLS.""" + address = _run_command( + [ + "docker", + "inspect", + "--format", + "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}", + container, + ] + ) + try: + parsed = ip_address(address) + except ValueError: + raise AssertionError("candidate PostgreSQL container address is invalid") from None + if parsed.is_loopback or parsed.is_unspecified: + raise AssertionError("candidate PostgreSQL container address is not remote") + return address + + +def _generate_ca(directory: Path, stem: str) -> tuple[Path, Path]: + """Create one ephemeral CI-only certificate authority for TLS acceptance.""" + key_path = directory / f"{stem}.key" + certificate_path = directory / f"{stem}.crt" + _run_command( + [ + "openssl", + "req", + "-x509", + "-newkey", + "rsa:2048", + "-nodes", + "-sha256", + "-days", + "1", + "-subj", + f"/CN={stem}", + "-addext", + "basicConstraints=critical,CA:TRUE", + "-addext", + "keyUsage=critical,keyCertSign,cRLSign", + "-addext", + "subjectKeyIdentifier=hash", + "-keyout", + str(key_path), + "-out", + str(certificate_path), + ] + ) + return key_path, certificate_path + + +def _generate_server_certificate( + directory: Path, + *, + stem: str, + ca_key: Path, + ca_certificate: Path, + identity_ip: str, + serial: int, +) -> tuple[Path, Path]: + """Create one ephemeral server certificate with an explicit IP SAN.""" + key_path = directory / f"{stem}.key" + request_path = directory / f"{stem}.csr" + certificate_path = directory / f"{stem}.crt" + extension_path = directory / f"{stem}.ext" + extension_path.write_text( + "basicConstraints=critical,CA:FALSE\n" + "keyUsage=critical,digitalSignature,keyEncipherment\n" + f"subjectAltName=IP:{identity_ip}\n" + "extendedKeyUsage=serverAuth\n" + "subjectKeyIdentifier=hash\n" + "authorityKeyIdentifier=keyid:always,issuer\n", + encoding="utf-8", + ) + _run_command( + [ + "openssl", + "req", + "-newkey", + "rsa:2048", + "-nodes", + "-sha256", + "-subj", + f"/CN={stem}", + "-keyout", + str(key_path), + "-out", + str(request_path), + ] + ) + _run_command( + [ + "openssl", + "x509", + "-req", + "-sha256", + "-days", + "1", + "-in", + str(request_path), + "-CA", + str(ca_certificate), + "-CAkey", + str(ca_key), + "-set_serial", + str(serial), + "-extfile", + str(extension_path), + "-out", + str(certificate_path), + ] + ) + return key_path, certificate_path + + +def _install_server_certificate( + container: str, + *, + key_path: Path, + certificate_path: Path, +) -> None: + """Install a test-only server identity with PostgreSQL-required key ownership.""" + _run_command( + ["docker", "exec", "--user", "0", container, "mkdir", "-p", _TLS_DIRECTORY] + ) + _run_command( + ["docker", "cp", str(key_path), f"{container}:{_TLS_DIRECTORY}/server.key"] + ) + _run_command( + [ + "docker", + "cp", + str(certificate_path), + f"{container}:{_TLS_DIRECTORY}/server.crt", + ] + ) + _run_command( + [ + "docker", + "exec", + "--user", + "0", + container, + "chown", + "postgres:postgres", + f"{_TLS_DIRECTORY}/server.key", + f"{_TLS_DIRECTORY}/server.crt", + ] + ) + _run_command( + [ + "docker", + "exec", + "--user", + "0", + container, + "chmod", + "600", + f"{_TLS_DIRECTORY}/server.key", + ] + ) + _run_command( + [ + "docker", + "exec", + "--user", + "0", + container, + "chmod", + "644", + f"{_TLS_DIRECTORY}/server.crt", + ] + ) + + +def _alter_system(container: str, setting: str, value: str) -> None: + """Set one bounded PostgreSQL server parameter through the CI superuser.""" + if setting not in {"ssl", "ssl_cert_file", "ssl_key_file"}: + raise AssertionError("remote TLS acceptance setting is not admitted") + literal = value.replace("'", "''") + _run_command( + [ + "docker", + "exec", + container, + "psql", + "-U", + _EXPECTED_USER, + "-d", + _EXPECTED_DATABASE, + "-v", + "ON_ERROR_STOP=1", + "-c", + f"ALTER SYSTEM SET {setting} = '{literal}'", + ] + ) + + +def _restart_candidate_postgres(container: str) -> None: + """Restart the disposable CI PostgreSQL and wait for its real readiness probe.""" + _run_command(["docker", "restart", container], timeout_seconds=60) + deadline = time.monotonic() + 45 + while time.monotonic() < deadline: + completed = subprocess.run( + [ + "docker", + "exec", + container, + "pg_isready", + "-U", + _EXPECTED_USER, + "-d", + _EXPECTED_DATABASE, + ], + check=False, + capture_output=True, + text=True, + timeout=5, + ) + if completed.returncode == 0: + return + time.sleep(0.5) + raise AssertionError("candidate PostgreSQL did not become ready after TLS restart") + + +def _configure_server_tls(container: str, *, enabled: bool) -> None: + """Enable or disable TLS on the disposable PostgreSQL test boundary.""" + if enabled: + _alter_system(container, "ssl_cert_file", f"{_TLS_DIRECTORY}/server.crt") + _alter_system(container, "ssl_key_file", f"{_TLS_DIRECTORY}/server.key") + _alter_system(container, "ssl", "on" if enabled else "off") + _restart_candidate_postgres(container) + + +@contextmanager +def _trusted_ca(certificate_path: Path) -> Iterator[None]: + """Temporarily bind Python's default trust loading to one CI-only CA.""" + previous = os.environ.get("SSL_CERT_FILE") + os.environ["SSL_CERT_FILE"] = str(certificate_path) + try: + yield + finally: + if previous is None: + os.environ.pop("SSL_CERT_FILE", None) + else: + os.environ["SSL_CERT_FILE"] = previous + + +def _remote_tls_driver(address: str, password: str) -> Pg8000DriverAdapter: + """Construct the production adapter with a credential-free service selector.""" + def resolve_service(service_name: str) -> dict[str, str]: + if service_name != "tls-acceptance": + raise AssertionError("unexpected TLS acceptance service selector") + return { + "host": address, + "port": "5432", + "dbname": _EXPECTED_DATABASE, + "user": _EXPECTED_USER, + "password": password, + } + + return Pg8000DriverAdapter(dbapi, service_resolver=resolve_service) + + +def _assert_remote_tls_failure(driver: Pg8000DriverAdapter, password: str) -> None: + """Require the exact content-free package TLS failure on real PostgreSQL.""" + try: + connection = driver.connect( + "service=tls-acceptance", + connect_timeout_seconds=5, + ) + except Pg8000DriverTlsPolicyError as error: + if str(error) != "PostgreSQL TLS policy is unavailable": + raise AssertionError("remote TLS diagnostic contract changed") from None + if error.__cause__ is not None: + raise AssertionError("remote TLS failure retained a chained cause") from None + rendered = f"{error!s}\n{error!r}" + if password in rendered: + raise AssertionError("TLS failure disclosed credential material") from None + return + except Exception: + raise AssertionError( + "remote PostgreSQL TLS failure escaped package policy boundary" + ) from None + connection.close() + raise AssertionError("remote PostgreSQL TLS failure was accepted") + + +def _assert_production_remote_tls_contract() -> None: + """Exercise authenticated TLS, peer identity, and no-downgrade on real PostgreSQL.""" + container = _candidate_container() + address = _candidate_container_ip(container) + password = _candidate_password() + restored_plaintext = False + + with TemporaryDirectory(prefix="pg-llm-batch-tls-") as temporary_directory: + directory = Path(temporary_directory) + ca_key, ca_certificate = _generate_ca(directory, "pg-llm-batch-ci-ca") + _, untrusted_ca_certificate = _generate_ca( + directory, + "pg-llm-batch-ci-untrusted-ca", + ) + matching_key, matching_certificate = _generate_server_certificate( + directory, + stem="matching-server", + ca_key=ca_key, + ca_certificate=ca_certificate, + identity_ip=address, + serial=1001, + ) + mismatch_key, mismatch_certificate = _generate_server_certificate( + directory, + stem="mismatch-server", + ca_key=ca_key, + ca_certificate=ca_certificate, + identity_ip="192.0.2.1", + serial=1002, + ) + + try: + _install_server_certificate( + container, + key_path=matching_key, + certificate_path=matching_certificate, + ) + _configure_server_tls(container, enabled=True) + driver = _remote_tls_driver(address, password) + + with _trusted_ca(ca_certificate): + connection = driver.connect( + "service=tls-acceptance", + connect_timeout_seconds=5, + ) + try: + with connection.cursor() as cursor: + cursor.execute( + "SELECT ssl FROM pg_catalog.pg_stat_ssl " + "WHERE pid = pg_backend_pid()" + ) + if cursor.fetchone() != (True,): + raise AssertionError("remote PostgreSQL connection is not TLS") + finally: + connection.close() + + with _trusted_ca(untrusted_ca_certificate): + _assert_remote_tls_failure(driver, password) + + _install_server_certificate( + container, + key_path=mismatch_key, + certificate_path=mismatch_certificate, + ) + _restart_candidate_postgres(container) + with _trusted_ca(ca_certificate): + _assert_remote_tls_failure(driver, password) + + _configure_server_tls(container, enabled=False) + restored_plaintext = True + with _trusted_ca(ca_certificate): + _assert_remote_tls_failure(driver, password) + finally: + if not restored_plaintext: + try: + _configure_server_tls(container, enabled=False) + except AssertionError: + pass + + def _assert_keyword_and_service_selector_connections() -> None: """Prove exact-artifact connection parity beyond the URI-only happy path. @@ -422,6 +825,7 @@ def main() -> None: _assert_typed_rls_read(evidence_uuid, evidence_time) finally: _cleanup() + _assert_production_remote_tls_contract() if __name__ == "__main__": diff --git a/tests/test_pg8000_remote_tls_identity.py b/tests/test_pg8000_remote_tls_identity.py new file mode 100644 index 000000000..3b9c593d8 --- /dev/null +++ b/tests/test_pg8000_remote_tls_identity.py @@ -0,0 +1,273 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Remote TLS policy regressions for the admitted pg8000 driver.""" + +from __future__ import annotations + +from pathlib import Path +import ssl +from types import ModuleType, SimpleNamespace + +import pytest + +import pg_llm_batch.pg8000_driver_adapter as driver_module +from pg_llm_batch.pg8000_driver_adapter import ( + Pg8000DriverAdapter, + Pg8000DriverTlsPolicyError, +) + + +class _RawConnection: + """Stand in for a raw pg8000 connection without performing network I/O.""" + + +class _Pg8000InterfaceError(Exception): + """Model the admitted pg8000 DB-API interface-error boundary.""" + + +def _dbapi_module(calls: list[dict[str, object]]) -> ModuleType: + """Return the exact DB-API metadata shape plus a recording connect factory.""" + module = ModuleType("pg8000.dbapi") + module.apilevel = "2.0" # type: ignore[attr-defined] + module.paramstyle = "format" # type: ignore[attr-defined] + module.threadsafety = 1 # type: ignore[attr-defined] + module.InterfaceError = _Pg8000InterfaceError # type: ignore[attr-defined] + + def connect(**kwargs: object) -> _RawConnection: + calls.append(dict(kwargs)) + return _RawConnection() + + module.connect = connect # type: ignore[attr-defined] + return module + + +def _connect_kwargs(host: str) -> dict[str, object]: + """Open one fake connection and return the kwargs crossing into pg8000.""" + calls: list[dict[str, object]] = [] + adapter = Pg8000DriverAdapter(_dbapi_module(calls)) + adapter.connect( + f"user=pgllm password=secret host={host} dbname=pgllm", + connect_timeout_seconds=7, + ) + assert len(calls) == 1 + return calls[0] + + +@pytest.mark.parametrize("host", ["db.example.invalid", "10.20.30.40", "192.168.50.10"]) +def test_remote_tcp_requires_authenticated_tls(host: str) -> None: + """Require CA and hostname verification for every non-loopback TCP target.""" + kwargs = _connect_kwargs(host) + + context = kwargs["ssl_context"] + assert isinstance(context, ssl.SSLContext) + assert context.check_hostname is True + assert context.verify_mode == ssl.CERT_REQUIRED + assert kwargs["host"] == host + assert kwargs["timeout"] == 7 + + +@pytest.mark.parametrize( + "host", + ["localhost", "LOCALHOST", "127.0.0.1", "127.42.0.9", "::1"], +) +def test_explicit_loopback_keeps_the_local_development_exception(host: str) -> None: + """Keep plaintext fallback confined to explicit loopback selectors.""" + kwargs = _connect_kwargs(host) + + assert "ssl_context" not in kwargs + assert kwargs["host"] == host + + +def test_remote_tls_context_construction_failure_is_content_free( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Fail before raw driver access when the host trust context cannot be built.""" + calls: list[dict[str, object]] = [] + adapter = Pg8000DriverAdapter(_dbapi_module(calls)) + + def fail_context() -> ssl.SSLContext: + raise OSError("secret certificate store path") + + monkeypatch.setattr(driver_module, "_new_remote_ssl_context", fail_context) + + with pytest.raises( + Pg8000DriverTlsPolicyError, + match="^PostgreSQL TLS policy is unavailable$", + ) as failure: + adapter.connect("user=pgllm host=db.example.invalid dbname=pgllm") + + assert failure.value.__cause__ is None + assert failure.value.__context__ is None + assert calls == [] + + +def test_remote_tls_ssl_context_failure_is_content_free( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Normalize OpenSSL failures while constructing the host trust context.""" + calls: list[dict[str, object]] = [] + adapter = Pg8000DriverAdapter(_dbapi_module(calls)) + + def fail_context() -> ssl.SSLContext: + raise ssl.SSLError("secret default CA loader detail") + + monkeypatch.setattr(driver_module, "_new_remote_ssl_context", fail_context) + + with pytest.raises( + Pg8000DriverTlsPolicyError, + match="^PostgreSQL TLS policy is unavailable$", + ) as failure: + adapter.connect("user=pgllm host=db.example.invalid dbname=pgllm") + + assert str(failure.value) == "PostgreSQL TLS policy is unavailable" + assert failure.value.__cause__ is None + assert failure.value.__context__ is None + assert calls == [] + + +@pytest.mark.parametrize( + "context", + [ + SimpleNamespace( + check_hostname=False, + verify_mode=ssl.CERT_REQUIRED, + keylog_filename=None, + ), + SimpleNamespace( + check_hostname=True, + verify_mode=ssl.CERT_NONE, + keylog_filename=None, + ), + SimpleNamespace( + check_hostname=True, + verify_mode=ssl.CERT_REQUIRED, + keylog_filename="tls.keys", + ), + ], +) +def test_remote_rejects_weakened_tls_context( + monkeypatch: pytest.MonkeyPatch, + context: SimpleNamespace, +) -> None: + """Reject a TLS context if peer identity or key-log isolation is disabled.""" + calls: list[dict[str, object]] = [] + adapter = Pg8000DriverAdapter(_dbapi_module(calls)) + monkeypatch.setattr(driver_module, "_new_remote_ssl_context", lambda: context) + + with pytest.raises( + Pg8000DriverTlsPolicyError, + match="^PostgreSQL TLS policy is unavailable$", + ): + adapter.connect("user=pgllm host=db.example.invalid dbname=pgllm") + + assert calls == [] + + +def test_remote_tls_does_not_honor_ambient_key_logging( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Keep process-level SSLKEYLOGFILE from becoming a package TLS key sink.""" + monkeypatch.setenv("SSLKEYLOGFILE", str(tmp_path / "postgres-tls.keys")) + + kwargs = _connect_kwargs("db.example.invalid") + context = kwargs["ssl_context"] + + assert isinstance(context, ssl.SSLContext) + assert context.keylog_filename is None + assert context.verify_flags & ssl.VERIFY_X509_PARTIAL_CHAIN + assert context.verify_flags & ssl.VERIFY_X509_STRICT + + +def test_remote_tls_handshake_failure_is_content_free() -> None: + """Do not leak remote identity, credentials, or CA paths from TLS failures.""" + calls: list[dict[str, object]] = [] + module = _dbapi_module(calls) + + def fail_connect(**kwargs: object) -> _RawConnection: + calls.append(dict(kwargs)) + raise ssl.SSLCertVerificationError( + 1, + "certificate verify failed for db.example.invalid; " + "password=secret; ca=/private/operator-ca.pem", + ) + + module.connect = fail_connect # type: ignore[attr-defined] + adapter = Pg8000DriverAdapter(module) + + with pytest.raises( + Pg8000DriverTlsPolicyError, + match="^PostgreSQL TLS policy is unavailable$", + ) as failure: + adapter.connect( + "user=pgllm password=secret host=db.example.invalid dbname=pgllm" + ) + + assert str(failure.value) == "PostgreSQL TLS policy is unavailable" + assert failure.value.__cause__ is None + assert failure.value.__context__ is None + assert len(calls) == 1 + + +def test_remote_server_ssl_refusal_is_content_free() -> None: + """Normalize pg8000's pinned server-refuses-SSL InterfaceError shape.""" + calls: list[dict[str, object]] = [] + module = _dbapi_module(calls) + + def fail_connect(**kwargs: object) -> _RawConnection: + calls.append(dict(kwargs)) + raise _Pg8000InterfaceError("Server refuses SSL") + + module.connect = fail_connect # type: ignore[attr-defined] + adapter = Pg8000DriverAdapter(module) + + with pytest.raises( + Pg8000DriverTlsPolicyError, + match="^PostgreSQL TLS policy is unavailable$", + ) as failure: + adapter.connect("user=pgllm host=db.example.invalid dbname=pgllm") + + assert str(failure.value) == "PostgreSQL TLS policy is unavailable" + assert failure.value.__cause__ is None + assert failure.value.__context__ is None + assert len(calls) == 1 + + +def test_unrelated_pg8000_interface_error_remains_native() -> None: + """Do not collapse unrelated DB-API interface failures into TLS policy.""" + calls: list[dict[str, object]] = [] + module = _dbapi_module(calls) + native_failure = _Pg8000InterfaceError("connection setup failed") + + def fail_connect(**kwargs: object) -> _RawConnection: + calls.append(dict(kwargs)) + raise native_failure + + module.connect = fail_connect # type: ignore[attr-defined] + adapter = Pg8000DriverAdapter(module) + + with pytest.raises(_Pg8000InterfaceError) as failure: + adapter.connect("user=pgllm host=db.example.invalid dbname=pgllm") + + assert failure.value is native_failure + assert len(calls) == 1 + + +def test_missing_interface_error_authority_does_not_reclassify_failure() -> None: + """Re-raise failures when the injected DB-API exposes no classifiable authority.""" + calls: list[dict[str, object]] = [] + module = _dbapi_module(calls) + del module.InterfaceError # type: ignore[attr-defined] + native_failure = RuntimeError("connection setup failed") + + def fail_connect(**kwargs: object) -> _RawConnection: + calls.append(dict(kwargs)) + raise native_failure + + module.connect = fail_connect # type: ignore[attr-defined] + adapter = Pg8000DriverAdapter(module) + + with pytest.raises(RuntimeError) as failure: + adapter.connect("user=pgllm host=db.example.invalid dbname=pgllm") + + assert failure.value is native_failure + assert len(calls) == 1