ci: harden release and document security process

.github/ISSUE_TEMPLATE/bug.yml

@@ -0,0 +1,66 @@
+name: Bug report
+description: Report reproducible image, runtime, or documentation behavior
+title: "bug: "
+labels:
+  - bug
+body:
+  - type: markdown
+    attributes:
+      value: >-
+        Do not report vulnerabilities here. Use the private security-reporting link
+        shown before opening an issue.
+  - type: input
+    id: image
+    attributes:
+      label: Image tag and digest
+      description: Include both when available; do not report only `latest`.
+      placeholder: ghcr.io/datopsis/clickhouse-server-ubi9:v...@sha256:...
+    validations:
+      required: true
+  - type: input
+    id: runtime
+    attributes:
+      label: Runtime and version
+      placeholder: Docker 28.x, Podman 5.x, or OpenShift 4.x
+    validations:
+      required: true
+  - type: dropdown
+    id: architecture
+    attributes:
+      label: Architecture
+      options:
+        - linux/amd64
+        - linux/arm64
+        - Other or unknown
+    validations:
+      required: true
+  - type: textarea
+    id: deployment
+    attributes:
+      label: Minimal deployment configuration
+      description: Remove passwords, tokens, internal hostnames, and other secrets.
+      render: yaml
+    validations:
+      required: true
+  - type: textarea
+    id: behavior
+    attributes:
+      label: Reproduction and observed behavior
+      description: List exact steps, the result, and the result you expected.
+    validations:
+      required: true
+  - type: textarea
+    id: logs
+    attributes:
+      label: Relevant logs
+      description: Redact all secrets and personal or internal information.
+      render: text
+  - type: checkboxes
+    id: checks
+    attributes:
+      label: Checks
+      options:
+        - label: I searched existing issues and tested a currently supported image.
+          required: true
+        - label: I removed secrets and sensitive data from this report.
+          required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,8 @@
+blank_issues_enabled: true
+contact_links:
+  - name: Report a vulnerability privately
+    url: https://github.com/datopsis/clickhouse-server-ubi9/security/advisories/new
+    about: Never disclose a suspected vulnerability in a public issue.
+  - name: ClickHouse upstream support
+    url: https://github.com/ClickHouse/ClickHouse/issues
+    about: Report behavior that reproduces in an official ClickHouse distribution upstream.

.github/pull_request_template.md

@@ -0,0 +1,16 @@
+## Summary
+
+Describe the user-facing or operational outcome and why the change is needed.
+
+## Validation
+
+- [ ] I ran the relevant local checks from `docs/CI.md`.
+- [ ] I reviewed CI logs, warnings, annotations, skipped steps, and retained security evidence rather than relying only on green status checks.
+- [ ] I added or updated tests for behavior changes.
+- [ ] I updated user and operator documentation where needed.
+- [ ] I added a `CHANGELOG.md` entry when the change is notable to users or security reviewers.
+- [ ] I did not weaken an image, workflow, or release control without documenting the threat, rationale, compensating control, owner, and expiry.
+
+## Security and release impact
+
+State whether this changes image contents or runtime behavior. If it does, identify the required packaging-revision action under `docs/VERSION.md`. List accepted scanner findings or write `None`.

.github/workflows/ci.yml

@@ -12,9 +12,14 @@ on:
 permissions:
   contents: read
 
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   lint:
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     steps:
       - name: Check out repository
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -35,6 +40,9 @@ jobs:
       - name: Run repository checks
         run: pre-commit run --all-files --show-diff-on-failure
 
+      - name: Test release tag validation
+        run: bash tests/release-tag.sh
+
       - name: Audit GitHub Actions security
         uses: zizmorcore/zizmor-action@70fb788f84895a7701f5643d103d587e460b5c99 # v0.6.3
         with:
@@ -44,6 +52,7 @@ jobs:
 
   image:
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     permissions:
       contents: read
       security-events: write

.github/workflows/codeql.yml

@@ -15,6 +15,10 @@ on:
 
 permissions: read-all
 
+concurrency:
+  group: codeql-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   analyze-actions:
     name: Analyze GitHub Actions

.github/workflows/release.yml

@@ -7,12 +7,17 @@ on:
 
 permissions: read-all
 
+concurrency:
+  group: release-${{ github.ref }}
+  cancel-in-progress: false
+
 env:
   IMAGE: ghcr.io/datopsis/clickhouse-server-ubi9
 
 jobs:
   release:
     runs-on: ubuntu-latest
+    timeout-minutes: 90
     permissions:
       contents: write
       id-token: write
@@ -22,8 +27,12 @@ jobs:
       - name: Check out repository
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
+          fetch-depth: 0
           persist-credentials: false
 
+      - name: Validate release tag and changelog
+        run: bash scripts/validate-release-tag.sh "${GITHUB_REF_NAME}" --require-annotated
+
       - name: Set up QEMU
         uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4.2.0
 

.github/workflows/scorecard.yml

@@ -11,6 +11,10 @@ on:
 
 permissions: read-all
 
+concurrency:
+  group: scorecard-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   analysis:
     name: Scorecard analysis

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 All notable changes to this project will be documented in this file.
 
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and releases use container-version tags documented in `README.md`.
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). GitHub Releases correspond only to published container images; repository-only changes remain under `Unreleased` until the next image release. See [docs/VERSION.md](docs/VERSION.md).
 
 ## [Unreleased]
 
@@ -15,3 +15,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 - OpenSSF Scorecard publishing, CodeQL analysis for GitHub Actions, hash-locked CI tooling, CODEOWNERS, and signed GitHub release evidence.
 - Syft SPDX SBOM generation and Grype vulnerability gates for CI and release images, with SARIF and release artifacts retained according to their useful lifecycle.
 - First-release roadmap, repository badge policy, and contributor-facing CI and artifact documentation.
+- A documented container and repository versioning and release standard.
+- CI security-layer, enforcement, result-review, and Endor Labs guidance.
+- Structured bug reporting and pull-request review checklists.

README.md

@@ -117,7 +117,7 @@ The smoke suite verifies startup with a read-only root filesystem and no capabil
 1. Update and locally test the versions and digests in `Containerfile`.
 2. Merge the change to `main` after CI passes.
 3. Complete the release gates in [docs/ROADMAP.md](docs/ROADMAP.md).
-4. Create a tag such as `v26.8.2.7-ubi9.8-1`.
+4. Choose the next release version according to [docs/VERSION.md](docs/VERSION.md), validate it with `bash scripts/validate-release-tag.sh <tag>`, and create its tag, such as `v26.8.2.7-ubi9.8-1`.
 5. Push the tag. GitHub Actions builds both architectures, scans the image with Trivy and Grype, publishes it to GHCR, attaches SBOM and provenance attestations, signs the resulting digest, and creates a GitHub release containing the SPDX SBOM, Sigstore bundle, and provenance evidence.
 
 Verify a release with GitHub as the keyless identity provider:
@@ -135,7 +135,7 @@ ClickHouse commonly benefits from `nofile=262144:262144`. Optional capabilities
 
 Treat `/var/lib/clickhouse` as durable state, back it up according to your ClickHouse topology, and pin production deployments to an image digest rather than a mutable tag.
 
-See [SECURITY.md](SECURITY.md) for vulnerability reporting and the support policy. Contributor references include the [first-release roadmap](docs/ROADMAP.md), [CI and artifact design](docs/CI.md), [badge policy](docs/BADGING.md), and [OpenSSF Scorecard controls](docs/OPENSSF_SCORECARD.md).
+See [SECURITY.md](SECURITY.md) for vulnerability reporting and the support policy. Contributor references include the [versioning and release standard](docs/VERSION.md), [first-release roadmap](docs/ROADMAP.md), [CI and security process](docs/CI.md), [Endor Labs posture](docs/ENDOR.md), [badge policy](docs/BADGING.md), and [OpenSSF Scorecard controls](docs/OPENSSF_SCORECARD.md).
 
 ## License
 

docs/CI.md

@@ -1,6 +1,6 @@
 # Continuous integration and release automation
 
-This repository treats the built image as the primary deliverable. CI therefore checks repository quality, builds and exercises the container, inventories its contents, and applies two independent vulnerability scanners before release.
+This repository treats the built image as the primary deliverable. CI therefore checks repository quality, builds and exercises the container, inventories its contents, and applies two independent vulnerability scanners before release. Required checks are deliberately few; each one groups related tools so branch protection stays understandable and runner use remains modest.
 
 ## Workflow map
 
@@ -9,19 +9,70 @@ This repository treats the built image as the primary deliverable. CI therefore
 | `CI` | Pull requests, pushes to `main`, weekly schedule, manual dispatch | Lint, workflow audit, configuration scan, image build, smoke tests, Trivy scan, Syft SBOM, and Grype scan. |
 | `CodeQL` | Workflow changes, weekly schedule, manual dispatch | Static analysis of GitHub Actions with the security-extended query suite. |
 | `OpenSSF Scorecard` | Pushes to `main`, ruleset changes, weekly schedule, manual dispatch | Supply-chain posture analysis, SARIF upload, and public Scorecard publication. |
-| `Release image` | Tags matching `v*` | Multi-architecture publish, digest scans, evidence generation, keyless signing, and GitHub release creation. |
+| `Release image` | Tags matching `v*` | Tag/input/changelog validation, multi-architecture publish, digest scans, evidence generation, keyless signing, and GitHub release creation. |
 
 Workflow-level permissions default to read-only. Write scopes are applied only to jobs that publish code-scanning results, packages, attestations, signatures, or releases. Third-party actions are pinned to full commit SHAs and tracked by Dependabot.
 
+## Analysis layers and why each exists
+
+| Layer | Controls | What it can establish | Important limit |
+| --- | --- | --- | --- |
+| Repository hygiene | pre-commit built-ins and release-tag tests | Parseable YAML/JSON, normalized text, no obvious private keys, merge markers, unsafe symlinks, or oversized additions; release identifiers match image inputs and changelog state. | Pattern checks do not prove that no secret exists. GitHub secret scanning and push protection provide the native enforcement layer. |
+| Source-specific lint | ShellCheck and Hadolint | Common shell defects and unsafe or wasteful container-build patterns. | These are static rules, not runtime evidence. |
+| Workflow security | Actionlint, Zizmor, and CodeQL `actions` with `security-extended` | Workflow syntax, dangerous expressions, excessive permissions, untrusted checkout/data flows, artifact risks, and immutable references. | CodeQL runs on workflow changes and a schedule; Zizmor runs in every `lint` job. |
+| Build configuration | Trivy configuration scan | High/critical Containerfile and infrastructure misconfigurations. | A clean configuration scan says nothing about packages in the built image. |
+| Runtime behavior | Buildx plus `tests/smoke.sh` | The exact test image starts and stops correctly under production-oriented restrictions and supports documented initialization/authentication behavior. | Hosted-runner tests do not replace native-architecture and OpenShift release qualification. |
+| Image vulnerabilities | Trivy image scan | No fixed high/critical findings according to Trivy's current databases and vendor severity selection. | `ignore-unfixed` intentionally leaves unfixed risk for human release review. |
+| Independent inventory and scan | Syft plus Grype | SPDX inventory of the tested image and a second vulnerability matcher/database; fixed high/critical findings block. | Overlap is intentional, but scanner agreement is not proof of absence. |
+| Supply-chain posture | OpenSSF Scorecard | Repository and build-pipeline practice signals published independently. | Historical and popularity signals improve only through genuine project operation. |
+| Release integrity | BuildKit attestations, Cosign, GHCR, and GitHub Releases | Digest-bound multi-architecture artifact, SBOM/provenance evidence, keyless signature, and durable release assets. | The tag workflow publishes before post-build scans; a failed candidate must be quarantined or removed. |
+
+No additional general-purpose scanner is currently justified. Dependency Review has little useful input without a supported package manifest; another image CVE scanner would duplicate Trivy and Grype; another secret action would duplicate GitHub secret scanning and the private-key hook; and a generic SAST action would add little beyond ShellCheck, CodeQL Actions, and Zizmor for this shell/container repository. Reconsider when the repository gains a new language, manifest, deployment format, or credible fuzz target. See [ENDOR.md](ENDOR.md) for the conditional Endor Labs adoption decision.
+
+## Pull request and merge process
+
+1. A pull request starts the two required checks: `lint` and `image`. Workflow-file changes also start CodeQL.
+2. Reviewers inspect the diff, check annotations, scanner summaries, smoke-test version, and the retained SBOM/SARIF. They confirm skipped steps are expected for the event and review warnings or ignored findings.
+3. The active `main` ruleset requires a pull request, resolved review threads, and the latest `lint` and `image` results before merge; it also blocks deletion and force pushes. Required approving reviews remain deliberately disabled until the post-first-release review described in [ROADMAP.md](ROADMAP.md).
+4. A merge starts `CI` on the exact `main` commit. Workflow changes start CodeQL, and every main push refreshes Scorecard. These post-merge runs are reviewed because merge-commit context, secrets, permissions, and SARIF publication differ from pull requests.
+5. Weekly schedules refresh time-sensitive vulnerability and workflow analysis even when source has not changed. Manual dispatch supports investigation; it is not a substitute for the pull-request checks.
+6. Only a validated annotated container-version tag starts a release. The workflow verifies that the tag's ClickHouse and UBI versions match `Containerfile` and that the changelog has a dated matching section before publishing.
+
+Concurrency cancels superseded CI, CodeQL, and Scorecard work for the same ref. Releases are never automatically cancelled. Job timeouts bound stuck or compromised work without hiding a failed control.
+
+## Enforced GitHub settings
+
+Repository configuration is part of the security boundary, even though it is not stored in Git:
+
+- Actions are enabled, the default `GITHUB_TOKEN` permission is read-only, and workflows cannot approve pull requests.
+- GitHub requires third-party Actions to be referenced by a full commit SHA. Workflow files also keep the release tag in a comment for review and Dependabot updates.
+- The `Protect main` ruleset requires pull requests, resolved review threads, and successful, up-to-date `lint` and `image` checks; it prevents branch deletion and non-fast-forward updates. The required approval count is intentionally zero for now.
+- Secret scanning, push protection, Dependabot security updates, and private vulnerability reporting are enabled.
+- Workflow permissions are narrowed per job; only code-scanning publication, OIDC signing, package publication, and release creation receive write scopes.
+
+Audit these settings before each release and after organization policy changes. A file review cannot detect a disabled ruleset or broadened repository-level token policy.
+
+## Reviewing a result rather than a color
+
+For every required run, verify the event and head SHA first. Then review the following evidence:
+
+- `lint`: every hook and the release-tag test ran, Zizmor audited every workflow, and there are no warnings or annotations hidden behind a successful wrapper.
+- `image`: the configuration scan count, ClickHouse version printed by the smoke suite, Trivy target/OS/package count and result count, SBOM package count, and Grype found-versus-ignored counts.
+- `CodeQL` and Scorecard: analysis covered the intended files, SARIF processing completed, and the Security tab has no new open alert. A successful upload is not the same as zero findings.
+- skipped steps: PR SARIF publication is intentionally skipped to avoid permission failures from untrusted forks; it runs on `main`. A skipped build, smoke test, or scanner is not acceptable.
+- warnings: Trivy may use another vendor's severity when Red Hat data is absent. Grype's `only-fixed` option can ignore real but currently unfixable findings. Review both against Red Hat and ClickHouse advisories before a release.
+
+Record accepted findings in the release pull request with the advisory, affected package, architecture, fix availability, rationale, owner, compensating control, and expiry. Do not rerun until a transient failure turns green or treat an empty SARIF file as proof that the scanner evaluated every risk category.
+
 ## Image security pipeline
 
 The image job runs these controls in order:
 
 1. **Trivy configuration scan** checks the `Containerfile`, Compose configuration, and repository infrastructure configuration for high and critical misconfigurations.
 2. **Build and smoke tests** exercise startup, authentication, initialization, persistence, shutdown, read-only operation, dropped capabilities, and arbitrary UIDs.
-3. **Trivy image scan** blocks fixed high and critical operating-system or application vulnerabilities.
+3. **Trivy image scan** blocks fixed high and critical operating-system or application vulnerabilities and reports its detected OS and package count for review.
 4. **Syft inventory** generates `clickhouse-server-ubi9.spdx.json` in SPDX JSON format from the tested image.
-5. **Grype SBOM scan** scans that exact SPDX document and blocks fixed high and critical vulnerabilities.
+5. **Grype SBOM scan** scans that exact SPDX document and blocks fixed high and critical vulnerabilities. Its log may also report ignored unfixed matches, which remain part of release review.
 6. **Artifact and SARIF publication** retains the inventory and result for investigation and publishes non-PR Grype results to GitHub code scanning.
 
 Trivy and Grype deliberately overlap. They use different databases and matching logic, so a clean result from one does not replace the other. Both gates ignore vulnerabilities without an upstream fix; unfixed findings still require periodic review before release. Scanner disagreements should be investigated against the vendor advisory and documented if accepted.