diff --git a/.github/workflows/hourly-pr-gap-loop.yml b/.github/workflows/hourly-pr-gap-loop.yml new file mode 100644 index 000000000..500cc7b58 --- /dev/null +++ b/.github/workflows/hourly-pr-gap-loop.yml @@ -0,0 +1,45 @@ +name: Hourly Gap Baseline Freshness Audit + +# Read-only Orgmetra heartbeat for buyer-facing gap truth. +# +# The central ContextualWisdomLab/.github scheduler already owns review dispatch, +# branch updates and protected PR integration on its established cadence. This +# workflow deliberately does not invoke or duplicate that writer. It only audits +# docs/product-technical-gap-baseline.md against fresh GitHub state so a quiet +# repository cannot leave buyer-facing baseline copy silently stale. + +on: + schedule: + - cron: "37 * * * *" + workflow_dispatch: + +permissions: + contents: read + pull-requests: read + issues: read + +concurrency: + group: hourly-gap-baseline-freshness + cancel-in-progress: false + +jobs: + gap-baseline-freshness: + name: Gap baseline freshness audit + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout default branch truth + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.event.repository.default_branch }} + persist-credentials: false + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.13" + - name: Audit gap baseline against live repository state + env: + GH_TOKEN: ${{ github.token }} + run: | + python3 scripts/ops/gap_baseline_freshness.py \ + --baseline docs/product-technical-gap-baseline.md >> "$GITHUB_STEP_SUMMARY" diff --git a/docs/doctoring/hourly-pr-gap-loop-references.md b/docs/doctoring/hourly-pr-gap-loop-references.md new file mode 100644 index 000000000..8bd9f3176 --- /dev/null +++ b/docs/doctoring/hourly-pr-gap-loop-references.md @@ -0,0 +1,20 @@ +# Hourly gap-baseline audit primary references + +Verified against GitHub's current official documentation on **2026-08-25**. These sources define the execution and least-privilege assumptions used by the active Orgmetra read-only freshness audit. They do not replace the central `.github` repository's published scheduler contract or Orgmetra's effective ruleset. + +## APA 7 references + +GitHub. (n.d.). *Workflow syntax for GitHub Actions*. GitHub Docs. Retrieved August 25, 2026, from https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax + +GitHub. (n.d.). *Reusing workflow configurations*. GitHub Docs. Retrieved August 25, 2026, from https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations + +## Decision relevance + +- GitHub permits workflow- and job-level `permissions` to constrain `GITHUB_TOKEN`. The Orgmetra heartbeat needs only repository contents, pull-request metadata and issue metadata reads, so its token remains read-only. +- GitHub defines reusable workflow calls as writer-capable job boundaries when the caller grants write permissions. Orgmetra deliberately does **not** call the central review/merge scheduler from this hourly workflow because the central `.github` automation already owns that mutation lane on its established cadence. +- The scheduled audit runs from Orgmetra's default branch after integration and only reports whether the buyer-facing gap baseline is current. It does not dispatch reviews, update branches, approve, enable auto-merge, merge, forward secrets, or obtain an OIDC mutation token. +- The baseline's `Inventory date` is explicitly an **Asia/Seoul calendar date**, not a midnight-UTC timestamp. A `develop` commit later on the same Korea calendar day therefore does not by itself make a date-only snapshot stale; a commit on a later Korea calendar day does. +- The audit compares the recorded open pull-request and non-PR issue counts with the complete live queues. A queue change is reported as a refresh candidate even when `develop` has not advanced, because active-PR and issue truth can change without a protected-branch commit. +- A baseline inventory date later than the current Asia/Seoul calendar date is internally impossible evidence and fails closed before any live GitHub read. A future-dated snapshot must never be reported as current merely because no `develop` commit is later than that future date. +- Live GitHub state is authoritative evidence for this audit. If the PR/issue/commit reads cannot be established or their payload shape is invalid, the audit fails closed with a non-zero result rather than reporting a successful/current audit from missing evidence. +- The central scheduler remains the single writer for review dispatch and protected PR integration. This active PR only adds a read-only truth-audit surface. diff --git a/scripts/ops/gap_baseline_freshness.py b/scripts/ops/gap_baseline_freshness.py new file mode 100644 index 000000000..49a002296 --- /dev/null +++ b/scripts/ops/gap_baseline_freshness.py @@ -0,0 +1,256 @@ +#!/usr/bin/env python3 +"""Audit the product-technical gap baseline against live repository truth. + +This script is the operational regression required by the baseline document's +own execution-loop contract: every loop must refetch live GitHub state and +reject stale buyer copy before acting. It never hard-codes volatile payloads; +every comparison is computed at runtime from the recorded inventory date, the +live default branch, and the complete live open pull-request/issue queues. + +Exit codes: + 0 audit completed, or the baseline has not integrated yet; findings are reported + 2 contract violation or live-state evidence could not be established + +The output is plain Markdown so callers can append it directly to +``$GITHUB_STEP_SUMMARY``. +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +from datetime import datetime +from pathlib import Path +from zoneinfo import ZoneInfo + +INVENTORY_DATE_PATTERN = re.compile( + r"^Inventory date:\s*(\d{4}-\d{2}-\d{2})", re.MULTILINE +) +SNAPSHOT_QUEUE_PATTERN = re.compile( + r"^At this snapshot,\s+(?P\d+)\s+pull requests\s+" + r"and\s+(?P\d+|zero|one|two|three|four|five|six|seven|" + r"eight|nine|ten)\s+non-PR issues?(?:\s+\([^)]*\))?\s+are open\b", + re.MULTILINE, +) +REPOSITORY_PATTERN = re.compile(r"^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$") +BASELINE_TIMEZONE = ZoneInfo("Asia/Seoul") +COUNT_WORDS = { + "zero": 0, + "one": 1, + "two": 2, + "three": 3, + "four": 4, + "five": 5, + "six": 6, + "seven": 7, + "eight": 8, + "nine": 9, + "ten": 10, +} + + +def _run_gh_json( + arguments: list[str], *, paginate: bool = False +) -> list[dict[str, object]]: + """Return a list payload from ``gh api``, flattening all pages when asked.""" + command = ["gh", *arguments] + if paginate: + command.extend(["--paginate", "--slurp"]) + completed = subprocess.run( + command, + check=False, + capture_output=True, + text=True, + ) + if completed.returncode != 0: + raise RuntimeError(f"gh {' '.join(arguments)} failed: {completed.stderr.strip()}") + + payload = json.loads(completed.stdout) + if paginate: + if not isinstance(payload, list): + raise RuntimeError("unexpected paginated GitHub payload shape") + flattened: list[dict[str, object]] = [] + for page in payload: + if not isinstance(page, list) or not all(isinstance(item, dict) for item in page): + raise RuntimeError("unexpected paginated GitHub page shape") + flattened.extend(page) + return flattened + + if not isinstance(payload, list) or not all(isinstance(item, dict) for item in payload): + raise RuntimeError("unexpected GitHub list payload shape") + return payload + + +def _live_repository() -> str: + """Return the repository this workflow actually executes in.""" + repository = os.environ.get("GITHUB_REPOSITORY", "ContextualWisdomLab/Orgmetra") + if REPOSITORY_PATTERN.fullmatch(repository) is None: + raise RuntimeError("invalid GITHUB_REPOSITORY shape") + return repository + + +def _snapshot_queue_counts(baseline_text: str) -> tuple[int, int]: + """Read the recorded open PR and non-PR issue counts from the baseline.""" + match = SNAPSHOT_QUEUE_PATTERN.search(baseline_text) + if match is None: + raise RuntimeError("missing snapshot open queue counts") + issue_count = match.group("open_issues") + parsed_issue_count = ( + int(issue_count) if issue_count.isdigit() else COUNT_WORDS[issue_count] + ) + return int(match.group("open_pull_requests")), parsed_issue_count + + +def _live_state() -> dict[str, object]: + """Fetch complete live open PR/issue queues and newest develop integration.""" + repository = _live_repository() + open_pull_requests = _run_gh_json( + ["api", f"repos/{repository}/pulls?state=open&per_page=100"], + paginate=True, + ) + open_issues = _run_gh_json( + ["api", f"repos/{repository}/issues?state=open&per_page=100"], + paginate=True, + ) + # Issues and PRs share the issues endpoint; keep only genuine issues. + genuine_issues = [item for item in open_issues if "pull_request" not in item] + commits = _run_gh_json( + ["api", f"repos/{repository}/commits?sha=develop&per_page=1"] + ) + if not commits: + raise RuntimeError("develop commit payload is empty") + newest_commit_date = None + if commits: + commit = commits[0].get("commit") + if not isinstance(commit, dict): + raise RuntimeError("unexpected commit payload shape") + committer = commit.get("committer") + if not isinstance(committer, dict) or not isinstance(committer.get("date"), str): + raise RuntimeError("unexpected commit committer payload shape") + newest_commit_date = committer["date"] + return { + "open_pull_requests": len(open_pull_requests), + "open_issues": len(genuine_issues), + "newest_develop_commit_date": newest_commit_date, + } + + +def main() -> int: + """Print a Markdown freshness report for the gap baseline snapshot.""" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--baseline", default="docs/product-technical-gap-baseline.md") + arguments = parser.parse_args() + + baseline_path = Path(arguments.baseline) + try: + baseline_text = baseline_path.read_text(encoding="utf-8") + except FileNotFoundError: + print( + "gap-baseline freshness: baseline not present on this integrated " + "branch yet; audit remains non-mutating and will activate when the " + "baseline owner integrates" + ) + return 0 + except OSError as error: + print(f"gap-baseline freshness: FAIL unreadable baseline: {error}") + return 2 + + match = INVENTORY_DATE_PATTERN.search(baseline_text) + if match is None: + print("gap-baseline freshness: FAIL missing 'Inventory date:' header") + return 2 + + inventory_date = datetime.strptime(match.group(1), "%Y-%m-%d").date() + now_date = datetime.now(tz=BASELINE_TIMEZONE).date() + if inventory_date > now_date: + print( + "gap-baseline freshness: FAIL future inventory date " + f"{inventory_date.isoformat()} exceeds current Asia/Seoul date " + f"{now_date.isoformat()}" + ) + return 2 + age_days = (now_date - inventory_date).days + try: + snapshot_open_pull_requests, snapshot_open_issues = _snapshot_queue_counts( + baseline_text + ) + except (RuntimeError, ValueError) as error: + print(f"gap-baseline freshness: FAIL malformed baseline: {error}") + return 2 + + lines = [ + "## Gap baseline freshness", + "", + f"- Baseline file: `{baseline_path.as_posix()}`", + f"- Inventory date: {inventory_date.isoformat()} " + f"(age {float(age_days):.1f} days)", + f"- Snapshot open pull requests: {snapshot_open_pull_requests}", + f"- Snapshot open issues: {snapshot_open_issues}", + ] + + try: + state = _live_state() + except (RuntimeError, ValueError, KeyError) as error: + lines.append(f"- FAIL live-state fetch: {error}") + print("\n".join(lines)) + return 2 + + lines.extend( + [ + f"- Live open pull requests: {state['open_pull_requests']}", + f"- Live open issues: {state['open_issues']}", + "- Newest develop integration: " + f"{state['newest_develop_commit_date']}", + ] + ) + + newest_integration = state.get("newest_develop_commit_date") + integrations_after_snapshot = False + try: + if not isinstance(newest_integration, str): + raise ValueError("newest develop integration timestamp is missing") + integration_time = datetime.fromisoformat( + newest_integration.replace("Z", "+00:00") + ) + integrations_after_snapshot = ( + integration_time.astimezone(BASELINE_TIMEZONE).date() > inventory_date + ) + except (TypeError, ValueError) as error: + lines.append(f"- FAIL live-state fetch: invalid develop timestamp: {error}") + print("\n".join(lines)) + return 2 + + queue_changed = ( + state["open_pull_requests"] != snapshot_open_pull_requests + or state["open_issues"] != snapshot_open_issues + ) + + if integrations_after_snapshot or queue_changed: + reasons = [] + if integrations_after_snapshot: + reasons.append("develop integrated commits after the recorded inventory date") + if queue_changed: + reasons.append("the live open PR/issue queue differs from the snapshot") + lines.append( + "- Result: **refresh candidate** — " + + " and ".join(reasons) + + ". Per the execution loop, refresh the baseline only when " + "buyer/product-visible truth changed; otherwise record this audit " + "as observed." + ) + else: + lines.append( + "- Result: current — no develop integration is newer than the " + "recorded inventory snapshot." + ) + + print("\n".join(lines)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_hourly_pr_gap_loop.py b/tests/test_hourly_pr_gap_loop.py new file mode 100644 index 000000000..c57c97b0f --- /dev/null +++ b/tests/test_hourly_pr_gap_loop.py @@ -0,0 +1,280 @@ +"""Regression tests for the scheduled Orgmetra gap-baseline audit.""" + +from __future__ import annotations + +import importlib.util +import json +from pathlib import Path +from types import ModuleType + +ROOT = Path(__file__).resolve().parents[1] +WORKFLOW = ROOT / ".github" / "workflows" / "hourly-pr-gap-loop.yml" +SCRIPT = ROOT / "scripts" / "ops" / "gap_baseline_freshness.py" +CENTRAL_SCHEDULER = "pr-review-merge-scheduler.yml" +BASELINE = ( + "Inventory date: 2026-08-25 (Asia/Seoul).\n" + "At this snapshot, 4 pull requests and one non-PR issue are open.\n" +) + + +def _load_script() -> ModuleType: + spec = importlib.util.spec_from_file_location("gap_baseline_freshness", SCRIPT) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def test_hourly_audit_does_not_create_a_second_pr_writer() -> None: + """The Orgmetra heartbeat must stay read-only beside the central writer.""" + text = WORKFLOW.read_text(encoding="utf-8") + + assert CENTRAL_SCHEDULER not in text + assert "contents: write" not in text + assert "pull-requests: write" not in text + assert "actions: write" not in text + assert "id-token: write" not in text + assert "secrets: inherit" not in text + assert "permissions:\n contents: read\n pull-requests: read\n issues: read" in text + assert "ref: ${{ github.event.repository.default_branch }}" in text + + +def test_live_queue_counts_request_all_pages(monkeypatch) -> None: + """Open PR/issue counts must not silently truncate at the first 100 rows.""" + module = _load_script() + calls: list[list[str]] = [] + + class Completed: + returncode = 0 + stderr = "" + + def __init__(self, stdout: str) -> None: + self.stdout = stdout + + def fake_run(arguments: list[str], **_kwargs: object) -> Completed: + calls.append(arguments) + joined = " ".join(arguments) + if "/pulls?" in joined: + return Completed(json.dumps([[{"number": 1}], [{"number": 2}]])) + if "/issues?" in joined: + return Completed( + json.dumps([[{"number": 3}], [{"number": 4, "pull_request": {}}]]) + ) + if "/commits?" in joined: + return Completed( + json.dumps( + [ + { + "commit": { + "committer": {"date": "2026-08-25T00:00:00Z"} + } + } + ] + ) + ) + raise AssertionError(f"unexpected gh invocation: {arguments}") + + monkeypatch.setattr(module.subprocess, "run", fake_run) + state = module._live_state() + + assert state["open_pull_requests"] == 2 + assert state["open_issues"] == 1 + queue_calls = [ + call + for call in calls + if "/pulls?" in " ".join(call) or "/issues?" in " ".join(call) + ] + assert len(queue_calls) == 2 + for call in queue_calls: + assert "--paginate" in call + assert "--slurp" in call + + +def test_missing_baseline_is_nonfatal_until_dependency_integrates( + monkeypatch, tmp_path, capsys +) -> None: + """A pre-#100 scheduled run must report absence without staying permanently red.""" + module = _load_script() + missing = tmp_path / "product-technical-gap-baseline.md" + monkeypatch.setattr( + module.sys, + "argv", + ["gap-baseline-freshness", "--baseline", str(missing)], + ) + + assert module.main() == 0 + assert "baseline not present" in capsys.readouterr().out.lower() + + +def test_future_inventory_date_is_nonpassing_without_live_read( + monkeypatch, tmp_path, capsys +) -> None: + """A future-dated buyer snapshot cannot be accepted as current repository truth.""" + module = _load_script() + baseline = tmp_path / "product-technical-gap-baseline.md" + baseline.write_text( + BASELINE.replace("2026-08-25", "2999-12-31"), encoding="utf-8" + ) + live_calls = 0 + + def live_state() -> dict[str, object]: + nonlocal live_calls + live_calls += 1 + return { + "open_pull_requests": 4, + "open_issues": 1, + "newest_develop_commit_date": "2026-08-25T12:00:00Z", + } + + monkeypatch.setattr(module, "_live_state", live_state) + monkeypatch.setattr( + module.sys, + "argv", + ["gap-baseline-freshness", "--baseline", str(baseline)], + ) + + assert module.main() == 2 + assert live_calls == 0 + assert "future inventory date" in capsys.readouterr().out.lower() + + +def test_same_inventory_day_integration_is_not_newer( + monkeypatch, tmp_path, capsys +) -> None: + """A same-Korea-calendar-day commit must not make a date-only snapshot stale.""" + module = _load_script() + baseline = tmp_path / "product-technical-gap-baseline.md" + baseline.write_text(BASELINE, encoding="utf-8") + monkeypatch.setattr( + module, + "_live_state", + lambda: { + "open_pull_requests": 4, + "open_issues": 1, + "newest_develop_commit_date": "2026-08-25T12:00:00Z", + }, + ) + monkeypatch.setattr( + module.sys, + "argv", + ["gap-baseline-freshness", "--baseline", str(baseline)], + ) + + assert module.main() == 0 + assert "result: current" in capsys.readouterr().out.lower() + + +def test_next_korea_calendar_day_integration_requires_refresh( + monkeypatch, tmp_path, capsys +) -> None: + """UTC timestamps crossing midnight in Korea must stale the prior local date.""" + module = _load_script() + baseline = tmp_path / "product-technical-gap-baseline.md" + baseline.write_text(BASELINE, encoding="utf-8") + monkeypatch.setattr( + module, + "_live_state", + lambda: { + "open_pull_requests": 4, + "open_issues": 1, + "newest_develop_commit_date": "2026-08-25T16:00:00Z", + }, + ) + monkeypatch.setattr( + module.sys, + "argv", + ["gap-baseline-freshness", "--baseline", str(baseline)], + ) + + assert module.main() == 0 + assert "refresh candidate" in capsys.readouterr().out.lower() + + +def test_live_state_failure_is_nonpassing(monkeypatch, tmp_path, capsys) -> None: + """An unavailable live control plane must fail closed instead of silently passing.""" + module = _load_script() + baseline = tmp_path / "product-technical-gap-baseline.md" + baseline.write_text(BASELINE, encoding="utf-8") + + def fail_live_state() -> dict[str, object]: + raise RuntimeError("GitHub API unavailable") + + monkeypatch.setattr(module, "_live_state", fail_live_state) + monkeypatch.setattr( + module.sys, + "argv", + ["gap-baseline-freshness", "--baseline", str(baseline)], + ) + + assert module.main() == 2 + output = capsys.readouterr().out.lower() + assert "fail live-state fetch" in output + assert "github api unavailable" in output + + +def test_queue_change_is_reported_as_refresh_candidate( + monkeypatch, tmp_path, capsys +) -> None: + """A changed live queue makes the point-in-time baseline a refresh candidate.""" + module = _load_script() + baseline = tmp_path / "product-technical-gap-baseline.md" + baseline.write_text(BASELINE, encoding="utf-8") + monkeypatch.setattr( + module, + "_live_state", + lambda: { + "open_pull_requests": 5, + "open_issues": 1, + "newest_develop_commit_date": "2026-08-25T12:00:00Z", + }, + ) + monkeypatch.setattr( + module.sys, + "argv", + ["gap-baseline-freshness", "--baseline", str(baseline)], + ) + + assert module.main() == 0 + assert "refresh candidate" in capsys.readouterr().out.lower() + + +def test_empty_develop_commit_payload_fails_closed(monkeypatch) -> None: + """An empty develop response cannot establish a current integration point.""" + module = _load_script() + + class Completed: + returncode = 0 + stderr = "" + stdout = "[]" + + monkeypatch.setattr(module.subprocess, "run", lambda *_args, **_kwargs: Completed()) + try: + module._live_state() + except RuntimeError as error: + assert "empty" in str(error) + else: + raise AssertionError("empty develop payload was accepted") + + +def test_invalid_develop_timestamp_fails_closed(monkeypatch, tmp_path, capsys) -> None: + """Malformed integration timestamps cannot be reported as a current audit.""" + module = _load_script() + baseline = tmp_path / "product-technical-gap-baseline.md" + baseline.write_text(BASELINE, encoding="utf-8") + monkeypatch.setattr( + module, + "_live_state", + lambda: { + "open_pull_requests": 4, + "open_issues": 1, + "newest_develop_commit_date": "not-a-timestamp", + }, + ) + monkeypatch.setattr( + module.sys, + "argv", + ["gap-baseline-freshness", "--baseline", str(baseline)], + ) + + assert module.main() == 2 + assert "invalid develop timestamp" in capsys.readouterr().out.lower()