Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,28 @@ jobs:
# (synthetic RFC 3161 tokens in committed demo/golden packets). Any
# finding not fingerprinted there still fails this gate.

threat-model-baseline:
name: threat-model baseline (frozen tags must match their pins; drift is report-only)
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Harden Runner (audit mode)
uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1
with:
egress-policy: audit
- name: Checkout (full history and tags — the check reads the document at each baseline tag)
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0
fetch-tags: true
persist-credentials: false
# Fails if docs/threat-model.md at a baseline tag no longer hashes to the value pinned in
# docs/audits/threat-model-baseline.md. Drift of the current file from the latest
# baseline is printed as a warning annotation and never fails the job (issue #366).
# Standard library only; no environment to build.
- name: Check the pinned baselines
run: python3 scripts/check_threat_model_baseline.py

verifier-portability:
name: verifier subset compiles on older Pythons
runs-on: ubuntu-latest
Expand Down
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,15 @@ follow [Semantic Versioning](https://semver.org/). The **packet format** and the

### Added

- **A CI check for the pinned threat-model baseline** (#366).
`scripts/check_threat_model_baseline.py` reads each baseline in
`docs/audits/threat-model-baseline.md` and fails if `docs/threat-model.md` at that
baseline's tag no longer hashes to the pinned value, or if the tag cannot be read.
It also reports, without failing, when the current document differs from the latest
baseline, naming the commits and lines changed since the freeze. It runs as the
`threat-model baseline` job in `ci.yml` (and `make threat-model-baseline`), with tests
in `tests/test_threat_model_baseline_check.py`. The pinned hash and the baseline
document are untouched; cutting B2 still needs a maintainer re-review.
- **Google Analytics 4 on the documentation website, and only there** (owner decision,
2026-09-17: GA4 on every public site, with the privacy copy updated to match). The
app, the CLI and the relay keep the no-telemetry rule unchanged. `site/analytics.js`
Expand Down
5 changes: 5 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,11 @@ i18n-usage: ## Report app i18n keys with no rendering path (reports; never gates
doc-links: ## Validate local Markdown links and capability-ledger evidence paths
uv run python scripts/check_doc_links.py

threat-model-baseline: ## Fail if a frozen threat-model baseline tag no longer matches its pinned hash; report (never fail) drift since the freeze (issue #366)
# Not in `verify`: it needs the baseline tags, which a shallow or tagless clone does
# not have. CI runs it in its own job with `fetch-tags`. Drift is printed, not gated.
python3 scripts/check_threat_model_baseline.py

markers: ## No bare TODO/FIXME/HACK (must reference an issue, e.g. TODO(#142)); no un-issued noqa/type:ignore
@bad=$$(grep -rnE '(TODO|FIXME|HACK)' --include='*.py' --include='*.js' src tests app scripts fuzz 2>/dev/null \
| grep -vE '\(#[0-9]+\)' || true); \
Expand Down
193 changes: 193 additions & 0 deletions scripts/check_threat_model_baseline.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
# SPDX-License-Identifier: AGPL-3.0-or-later
# Copyright 2026 Chelsea Kelly-Reif
"""Check the frozen threat-model baselines, and report drift from the latest one.

``docs/audits/threat-model-baseline.md`` pins a SHA-256 for each baseline (B1, ...) and names
the annotated git tag that marks its freeze commit. Two things follow from "immutable once
tagged", and this script treats them differently:

1. **Fails** when the frozen document, read from a baseline's tag, no longer hashes to the value
pinned in the audit file. That is the immutability the baseline claims: the tag was moved or
the pin was edited. It also fails when a pinned tag cannot be found, because a check that
cannot read the tag has verified nothing (a shallow checkout must fetch tags).
2. **Reports, and does not fail,** when the current ``docs/threat-model.md`` differs from the
latest baseline, naming the commits and lines changed since the freeze. Drift is expected
between baselines; the point is that it stays visible until a new baseline is cut and
re-reviewed by a person. This script never writes the pin or the threat model.

Standard library only, so it runs before project dependencies are installed.
"""

from __future__ import annotations

import argparse
import hashlib
import os
import re
import shutil
import subprocess
import sys
from dataclasses import dataclass
from pathlib import Path

BASELINE_FILE = Path("docs/audits/threat-model-baseline.md")
_DEFAULT_DOCUMENT = "docs/threat-model.md"

_HEADING = re.compile(r"^## Baseline (B\d+)\s*$", re.MULTILINE)
_NEXT_HEADING = re.compile(r"^## ", re.MULTILINE)
_HASH_ROW = re.compile(r"^\|\s*Document SHA-256\s*\|\s*`([0-9a-f]{64})`\s*\|", re.MULTILINE)
_TAG_ROW = re.compile(r"^\|\s*Git tag\s*\|\s*`([^`\s]+)`", re.MULTILINE)
_DOC_ROW = re.compile(r"^\|\s*Frozen document\s*\|\s*\[`([^`\s]+)`\]", re.MULTILINE)


@dataclass(frozen=True)
class Baseline:
"""One pinned baseline as written in the audit file."""

baseline_id: str
document: str
sha256: str
tag: str


class BaselineError(Exception):
"""The audit file does not carry a readable pin."""


def parse_baselines(text: str) -> list[Baseline]:
"""Return the baselines in file order. Raises if a section is missing its hash or tag."""
headings = list(_HEADING.finditer(text))
if not headings:
raise BaselineError("no '## Baseline B<n>' section found")
baselines: list[Baseline] = []
for heading in headings:
start = heading.end()
following = _NEXT_HEADING.search(text, start)
section = text[start : following.start() if following else len(text)]
digest = _HASH_ROW.search(section)
tag = _TAG_ROW.search(section)
if digest is None or tag is None:
raise BaselineError(
f"baseline {heading.group(1)} has no readable 'Document SHA-256' and 'Git tag' rows"
)
document = _DOC_ROW.search(section)
baselines.append(
Baseline(
baseline_id=heading.group(1),
document=document.group(1) if document else _DEFAULT_DOCUMENT,
sha256=digest.group(1),
tag=tag.group(1),
)
)
return baselines


def _git(root: Path, *args: str) -> subprocess.CompletedProcess[bytes]:
git = shutil.which("git")
if git is None:
raise BaselineError("git is not installed; the baseline tags cannot be read")
return subprocess.run( # noqa: S603 - resolved git path and constant arguments
[git, "-C", str(root), *args], capture_output=True, check=False
)


def _file_at_tag(root: Path, tag: str, document: str) -> bytes | None:
result = _git(root, "show", f"refs/tags/{tag}:{document}")
return result.stdout if result.returncode == 0 else None


def verify_immutable(root: Path, baselines: list[Baseline]) -> list[str]:
"""Return one problem per baseline whose tagged document no longer matches its pin."""
problems: list[str] = []
for baseline in baselines:
frozen = _file_at_tag(root, baseline.tag, baseline.document)
if frozen is None:
problems.append(
f"{baseline.baseline_id}: cannot read {baseline.document} at tag "
f"{baseline.tag} (is the tag missing? a CI checkout needs "
"fetch-depth: 0 and fetch-tags: true)"
)
continue
actual = hashlib.sha256(frozen).hexdigest()
if actual != baseline.sha256:
problems.append(
f"{baseline.baseline_id}: {baseline.document} at tag {baseline.tag} hashes to "
f"{actual}, but the audit file pins {baseline.sha256}. A baseline is immutable "
"once tagged; do not move the tag or edit the pin. Cut a new baseline instead."
)
return problems


def drift_report(root: Path, baseline: Baseline) -> str | None:
"""Describe how the working-tree document differs from ``baseline``, or ``None`` if it does not.

Reports commits (touching the document, since the tag) and added/removed lines (tag against
the working tree, so uncommitted edits count too). Report-only: callers must not fail on it.
"""
path = root / baseline.document
if not path.is_file():
return f"{baseline.document} is missing; the latest baseline is {baseline.baseline_id}"
if hashlib.sha256(path.read_bytes()).hexdigest() == baseline.sha256:
return None

commits = "an unknown number of"
counted = _git(
root, "rev-list", "--count", f"refs/tags/{baseline.tag}..HEAD", "--", baseline.document
)
if counted.returncode == 0:
commits = counted.stdout.decode().strip()

lines = "an unknown number of lines changed"
numstat = _git(root, "diff", "--numstat", f"refs/tags/{baseline.tag}", "--", baseline.document)
if numstat.returncode == 0 and numstat.stdout.strip():
added, removed, *_ = numstat.stdout.decode().split()
lines = f"{added} lines added, {removed} removed"

return (
f"{baseline.document} differs from baseline {baseline.baseline_id} "
f"(tag {baseline.tag}): {commits} commit(s) since the freeze, {lines}. "
"Report only: a maintainer re-review cuts the next baseline."
)


def check(root: Path) -> tuple[list[str], str | None]:
"""Return ``(problems, drift)``. Only ``problems`` should fail a build."""
audit = root / BASELINE_FILE
try:
baselines = parse_baselines(audit.read_text(encoding="utf-8"))
except (OSError, BaselineError) as exc:
return [f"{BASELINE_FILE}: {exc}"], None
try:
problems = verify_immutable(root, baselines)
return problems, drift_report(root, baselines[-1])
except BaselineError as exc:
return [str(exc)], None


def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--root",
type=Path,
default=Path(__file__).resolve().parent.parent,
help="repository root (defaults to the parent of scripts/)",
)
args = parser.parse_args()
problems, drift = check(args.root)

# GitHub renders ::warning:: as an annotation; on a terminal it reads as a plain line.
on_actions = os.environ.get("GITHUB_ACTIONS") == "true"
if drift is not None:
prefix = "::warning title=Threat model drift::" if on_actions else "DRIFT (report only): "
print(f"{prefix}{drift}")
if problems:
print("threat-model baseline gate failed:")
for problem in problems:
print(f"- {problem}")
return 1
print("threat-model baselines: every pinned tag matches its pinned hash")
return 0


if __name__ == "__main__":
sys.exit(main())
Loading
Loading