feat: qualify CA-issued TLS and plan SCAP scanning

.github/workflows/ci.yml

@@ -128,6 +128,12 @@ jobs:
           IMAGE: ${{ env.TEST_IMAGE }}
         run: bash tests/smoke.sh
 
+      - name: Rehearse CA-issued TLS
+        env:
+          CONTAINER_RUNTIME: docker
+          IMAGE: ${{ env.TEST_IMAGE }}
+        run: bash tests/tls-rehearsal.sh
+
       - name: Scan image
         uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0
         with:

.gitignore

@@ -5,6 +5,7 @@
 .DS_Store
 .pre-commit-cache/
 .smoke-secrets.*/
+.tls-rehearsal.*/
 /clickhouse-server-ubi9.spdx.json
 /grype.sarif
 /grype-all.json

CHANGELOG.md

@@ -26,6 +26,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - Native AMD64 and ARM64 CI builds, smoke tests, vulnerability evidence, and release-manifest architecture validation.
 - Explicit repository scope and official-image storage compatibility guidance, including the XML-based replacement for the unreleased `CLICKHOUSE_DATA_DIR` interface.
 - Podman-first user procedures, a tested Podman support baseline, and rootless user-namespace permission guidance.
+- Native CA-issued TLS rehearsal covering HTTPS, native TCP, connected/disconnected outbound trust, negative cases, renewal, rollback, and operator evidence procedures.
+- Dedicated loopback-only TLS health client configuration that supports CA-issued certificate chains without weakening server-side outbound verification.
 
 ### Changed
 

Containerfile

@@ -82,6 +82,7 @@ COPY --from=builder /runtime/etc/ /etc/
 COPY --from=builder /runtime/var/ /var/
 COPY --from=builder /runtime/docker-entrypoint-initdb.d/ /docker-entrypoint-initdb.d/
 COPY --chown=101:0 --chmod=0755 container/entrypoint.sh /usr/local/bin/clickhouse-entrypoint
+COPY --chown=101:0 --chmod=0444 container/health-client.xml /usr/local/share/clickhouse-health-client.xml
 COPY --chown=101:0 --chmod=0644 container/config.d/container.xml /etc/clickhouse-server/config.d/container.xml
 
 ENV LANG="C.UTF-8" \

README.md

@@ -109,7 +109,7 @@ Build-time arguments are `CLICKHOUSE_VERSION`, `CLICKHOUSE_CHANNEL`, `UBI_MINIMA
 
 The supported Podman baseline is version 5.3 or newer because 5.3.1 is the oldest engine on which the full smoke suite has been recorded. This is a tested support floor, not a claim that older versions cannot run the image. Docker Engine remains compatible and is used by GitHub Actions for its native architecture jobs and release Buildx workflow. See [Podman compatibility and version support](docs/PODMAN.md) for tested versions, rootless bind mounts, remote clients, and Compose behavior.
 
-The smoke suite verifies startup with a read-only root filesystem and no capabilities, package-manager absence, authenticated local and network queries, first-start initialization, persistent-data restarts, password-file support, the passwordless network restriction, TLS-only native initialization and health, graceful shutdown, and operation under an arbitrary OpenShift-style UID. It requires `openssl` on the test host to create an ephemeral TLS fixture.
+The smoke suite verifies startup with a read-only root filesystem and no capabilities, package-manager absence, authenticated local and network queries, first-start initialization, persistent-data restarts, password-file support, the passwordless network restriction, TLS-only native initialization and health, graceful shutdown, and operation under an arbitrary OpenShift-style UID. The separate CA-issued rehearsal validates chained certificates, HTTPS/native TLS, connected and disconnected outbound trust, negative cases, rotation, and rollback. Both require `openssl` on the test host; the TLS rehearsal also requires `curl`.
 
 ## Release process
 
@@ -139,9 +139,9 @@ ClickHouse commonly benefits from `nofile=262144:262144`. Optional capabilities
 
 Treat the effective ClickHouse data path (default `/var/lib/clickhouse`) as durable state, back it up according to your ClickHouse topology, and pin production deployments to an image digest rather than a mutable tag.
 
-Before a production rollout, follow the [production deployment guide](docs/PRODUCTION.md). Configure inbound encryption and public/private outbound trust with the [TLS guide](docs/TLS.md). The secure ClickHouse ports (`8443`, `9440`, and `9010`) are configuration choices and are not enabled by default.
+Before a production rollout, follow the [production deployment guide](docs/PRODUCTION.md). Configure inbound encryption and public/private outbound trust with the [TLS guide](docs/TLS.md), then execute the [CA-issued TLS rehearsal](docs/TLS-REHEARSAL.md). The secure ClickHouse ports (`8443`, `9440`, and `9010`) are configuration choices and are not enabled by default.
 
-See [SECURITY.md](SECURITY.md) for vulnerability reporting and the support policy. Contributor references include [Podman compatibility](docs/PODMAN.md), [rootless storage and permissions](docs/ROOTLESS.md), [qualification evidence](docs/QUALIFICATION.md), the [vulnerability-management process](docs/VULNERABILITY-MANAGEMENT.md), [official-image comparison](docs/IMAGE-COMPARISON.md), [versioning and release standard](docs/VERSION.md), [first-release roadmap](docs/ROADMAP.md), [CI and security process](docs/CI.md), [Endor Labs posture](docs/ENDOR.md), [badge policy](docs/BADGING.md), and [OpenSSF Scorecard controls](docs/OPENSSF_SCORECARD.md).
+See [SECURITY.md](SECURITY.md) for vulnerability reporting and the support policy. Contributor references include [Podman compatibility](docs/PODMAN.md), [rootless storage and permissions](docs/ROOTLESS.md), [qualification evidence](docs/QUALIFICATION.md), the [vulnerability-management process](docs/VULNERABILITY-MANAGEMENT.md), [official-image comparison](docs/IMAGE-COMPARISON.md), [versioning and release standard](docs/VERSION.md), [first-release roadmap](docs/ROADMAP.md), [CI and security process](docs/CI.md), [SCAP compliance scanning](docs/SCAP.md), [Endor Labs posture](docs/ENDOR.md), [badge policy](docs/BADGING.md), and [OpenSSF Scorecard controls](docs/OPENSSF_SCORECARD.md).
 
 ## License
 

container/entrypoint.sh

@@ -3,6 +3,7 @@ set -Eeuo pipefail
 shopt -s nullglob
 
 readonly CONFIG_FILE="${CLICKHOUSE_CONFIG:-/etc/clickhouse-server/config.xml}"
+readonly HEALTH_CLIENT_CONFIG="/usr/local/share/clickhouse-health-client.xml"
 readonly GENERATED_DIR="/tmp/clickhouse-entrypoint"
 readonly USERS_FILE="${GENERATED_DIR}/users.xml"
 readonly INIT_DIR="/docker-entrypoint-initdb.d"
@@ -145,7 +146,13 @@ client_command() {
     local -n command_ref=$1
     local port
 
-    command_ref=(clickhouse-client --host 127.0.0.1 --user default)
+    command_ref=(
+        env -u CLICKHOUSE_CONFIG
+        clickhouse-client
+        --config-file "${HEALTH_CLIENT_CONFIG}"
+        --host 127.0.0.1
+        --user default
+    )
     if [[ -n "${CLICKHOUSE_PASSWORD}" ]]; then
         command_ref+=(--password "${CLICKHOUSE_PASSWORD}")
     fi
@@ -166,7 +173,7 @@ client_command() {
 
     # This client connects only over container loopback for initialization and
     # health. External clients must validate the server certificate normally.
-    command_ref+=(--port "${port}" --secure --accept-invalid-certificate)
+    command_ref+=(--port "${port}" --secure)
 }
 
 healthcheck() {

container/health-client.xml

@@ -0,0 +1,14 @@
+<?xml version="1.0"?>
+<config>
+    <!-- Used only by the entrypoint's loopback initialization and health query. -->
+    <accept-invalid-certificate>1</accept-invalid-certificate>
+    <openSSL>
+        <client>
+            <loadDefaultCAFile>false</loadDefaultCAFile>
+            <verificationMode>none</verificationMode>
+            <invalidCertificateHandler>
+                <name>AcceptCertificateHandler</name>
+            </invalidCertificateHandler>
+        </client>
+    </openSSL>
+</config>

docs/CI.md

@@ -69,7 +69,7 @@ Record accepted findings in the release pull request with the advisory, affected
 Each native image matrix job runs these controls in order. AMD64 uses `ubuntu-24.04` and `linux/amd64`; ARM64 uses `ubuntu-24.04-arm` and `linux/arm64`. QEMU is not installed and does not count as native-runtime evidence.
 
 1. **Trivy configuration scan** checks the `Containerfile`, Compose configuration, and repository infrastructure configuration for high and critical misconfigurations.
-2. **Build and smoke tests** exercise startup, authentication, initialization, persistence, shutdown, read-only operation, dropped capabilities, and arbitrary UIDs.
+2. **Build and runtime tests** exercise startup, authentication, initialization, persistence, shutdown, read-only operation, dropped capabilities, arbitrary UIDs, chained CA-issued HTTPS/native TLS, public/private outbound trust, disconnected isolation, negative certificate cases, renewal, and rollback.
 3. **Trivy image scan** blocks fixed high and critical operating-system or application vulnerabilities and reports its detected OS and package count for review.
 4. **Complete SPDX inventory** uses Syft to inventory the tested filesystem and RPM database, then `scripts/augment-spdx.py` declares the three pinned ClickHouse TGZ components that have no RPM metadata. The script takes their version and channel from `Containerfile`, records Apache-2.0 licensing and package identifiers, and fails instead of duplicating a component Syft already found.
 5. **Blocking Grype SBOM scan** scans that exact SPDX document and blocks fixed high and critical vulnerabilities.

docs/ROADMAP.md

@@ -10,7 +10,8 @@ This roadmap is the release gate for the first supported image. A checked item m
 2. Validate [TLS.md](TLS.md) with CA-issued certificates for HTTPS and native TCP, including rotation and a fully disconnected rehearsal.
 3. Execute [PRODUCTION.md](PRODUCTION.md) on native `amd64`, native `arm64`, and OpenShift, recording resource, storage, backup/restore, shutdown, and recovery evidence.
 4. Complete the ClickHouse/UBI notice and SBOM review described in [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md).
-5. Publish the first signed GHCR release only after every blocker below is complete. Reconsider Docker Hub and paid security services after that release has real consumer demand.
+5. Establish the tailored container SCAP baseline in [SCAP.md](SCAP.md), retain both architecture reports, and review every applicability decision before enforcing selected rules.
+6. Publish the first signed GHCR release only after every blocker below is complete. Reconsider Docker Hub and paid security services after that release has real consumer demand.
 
 ### Incremental delivery plan
 
@@ -94,7 +95,29 @@ This repository owns image-specific behavior, basic usage, and minimal platform
 
 - [ ] If no real OpenShift environment can be obtained, run a restricted Kubernetes proxy test and explicitly mark OpenShift as unvalidated and unsupported in the first release. A Kind/K3s test does not satisfy or replace the OpenShift checklist above.
 
-#### 5. Final candidate and signed release
+#### 5. Tailored SCAP image-compliance baseline
+
+**Profile discovery and tailoring**
+
+- [ ] Pin a UBI 9 OpenSCAP scanner image by digest and pin the OpenSCAP and ComplianceAsCode content versions. Record the RHEL 9 data-stream SHA-256 and reject an unexpected stream.
+- [ ] Run the upstream RHEL 9 Standard profile in report-only discovery mode against the exported image filesystem. Inventory every pass, failure, error, not-applicable, and not-checked result without claiming host or deployment compliance.
+- [ ] Create a reviewed XCCDF tailoring profile containing only rules that are applicable to and controlled by this image. Commit a rule-rationale matrix and document every host/platform exclusion.
+- [ ] Exclude kernel, boot-loader, partition, mount-layout, systemd, audit, host-networking, sysctl, SELinux-mode, and FIPS-mode controls unless the image later gains direct ownership of one. Do not use automatic remediation.
+
+**Safe CI integration**
+
+- [ ] Export the stopped, already-tested image's merged filesystem into an ephemeral directory and mount that directory read-only into the scanner. Never execute target-image content to prepare the scan.
+- [ ] Run `oscap-chroot` in a digest-pinned scanner with no Docker/Podman socket, no host namespace, no workflow secrets, and no evaluation-time network. Prove the minimum chroot-related capability; do not use `--privileged`, Podman-in-Podman, or broad host mounts.
+- [ ] Generate architecture-specific ARF XML, XCCDF XML, and HTML reports containing the image digest, architecture, scanner/content versions, data-stream hash, and tailoring hash. Retain them with the other image-security evidence.
+- [ ] Run report-only on native AMD64 and ARM64 for at least three scheduled or `main` executions. Evaluation errors fail immediately; selected-rule findings become blocking only after the baseline is stable and reviewed.
+- [ ] Cross-check one exact image digest with `oscap-podman` on a disposable RHEL 9 host. Reconcile platform/applicability differences before enforcement; do not grant routine hosted CI root or engine access merely to match that command.
+
+**Exit evidence**
+
+- [ ] Retain the discovery report, final tailoring, rule-rationale/exclusion review, three stable two-architecture runs, capability inspection, and `oscap-chroot` versus `oscap-podman` comparison.
+- [ ] State precisely that the result covers selected image-filesystem controls and is not CIS/STIG certification of the host, OpenShift cluster, or production deployment.
+
+#### 6. Final candidate and signed release
 
 - [ ] Refresh and review the UBI Minimal and Micro manifest-list digests together. Confirm both architectures resolve, rebuild from scratch, and retain the old/new digest and vulnerability comparison.
 - [ ] Confirm the selected ClickHouse release/channel and archive checksums, run both native CI jobs, and complete the current unfixed-finding triage with owner, rationale, compensating controls, and review expiry.
@@ -124,6 +147,10 @@ This repository owns image-specific behavior, basic usage, and minimal platform
 - [ ] Review workflow permissions, immutable action pins, secret-scanning alerts, and CodeQL/Scorecard findings.
 - [ ] Test private vulnerability reporting and confirm that `SECURITY.md` names a monitored response path.
 - [ ] Complete license, redistribution, trademark, and upstream-notice review for ClickHouse and Red Hat UBI content.
+- [ ] Complete an image threat model covering build inputs, CI trust, registry/release publication, runtime identity, storage, ingress/egress TLS, secrets, and the boundary with `clickhouse-production-stack`.
+- [ ] Define and exercise a UBI rebuild cadence and response SLA for exploitable critical/high findings, including unfixed findings that later receive a vendor fix.
+- [ ] Compare final SBOM and filesystem/package inventories against the reviewed baseline and investigate unexpected additions, removals, setuid/setgid files, or world-writable paths.
+- [ ] Exercise password/key/certificate rotation and failure paths while confirming logs and retained CI evidence do not expose secret material.
 
 ### Release mechanics and documentation