Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/scripts/integration-governance.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ const MAX_COMMITS = 1000;
const MAX_AUTHOR_BATCH = 50;
// This literal is inside the enrolled .github closure. A caller-supplied
// preparedBy.sourceSHA must never choose a new executable verifier tree.
const TRUSTED_SCRIPTS_TREE = '9affee853918156f2e5e8463562008c8c46b8afa';
const TRUSTED_SCRIPTS_TREE = '1e72143fdd7f0a7231d11c973e30301064899b27';
const SOURCE = 'https://github.com/kubernetes-sigs/kubebuilder-release-tools/tree/012269a88fa4c034a0acf1ba84c26b195c0dbab4';
const sha = value => crypto.createHash('sha256').update(value).digest('hex');
const git = (args, cwd = process.cwd()) =>
Expand Down
63 changes: 63 additions & 0 deletions .github/scripts/monthly-outcomes-pr.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
#!/usr/bin/env bash
# Only sanitized renderer outputs enter this branch; never stage an input tree.
set -euo pipefail
unset GITHUB_TOKEN
: "${MONTH:?missing validated month}"
: "${GITHUB_REPOSITORY:?missing repository}"
[[ "$MONTH" =~ ^[0-9]{4}-[0-9]{2}$ ]] || exit 1
branch="automation/monthly-outcomes-$MONTH"
pr="$(gh pr list --repo "$GITHUB_REPOSITORY" --head "$branch" --state all --limit 2 --json number,state)"
if [ "$(printf '%s' "$pr" | jq length)" -gt 1 ] || printf '%s' "$pr" | jq -e 'any(.[]; .state != "OPEN")' >/dev/null; then
echo "::error::A closed/merged monthly PR already exists; resolve it explicitly rather than reposting."
exit 1
fi
remote_branch=false
if git ls-remote --exit-code --heads origin "$branch" >/dev/null; then
git fetch origin "$branch"
git checkout -b "$branch" FETCH_HEAD
remote_branch=true
else
git checkout -b "$branch"
fi
dest="src/docs/benchmarks"
ensure_pr() {
if [ "$remote_branch" = false ] || [ "$(printf '%s' "$pr" | jq length)" -ne 0 ]; then
return
fi
if git diff --quiet origin/v5...HEAD -- "$dest/outcomes-$MONTH.*" "$dest/assets/outcomes-$MONTH-*" ; then
echo "No monthly report diff against v5; no PR to create."
return
fi
# A successful push can outlive a failed PR creation. Retry without changing
# the approved report or its original provenance.
gh pr create --repo "$GITHUB_REPOSITORY" --base v5 --head "$branch" \
--title "📖 docs: monthly outcomes $MONTH" \
--body "Owner-supplied, sanitized monthly aggregates. Review the cohort, observation time, cost denominator and provenance before merging. No raw evidence, stories or deployment identities are published. Collection completeness and publication approval are owner attestations, not independently verified by this workflow. CI and normal review gates must pass; this workflow does not merge."
}
# Keep the original renderer revision when the approved input is unchanged,
# even if unrelated v5 commits landed while this month's PR was open.
provenance="outcomes-$MONTH.provenance.json"
if [ -f "$dest/$provenance" ] &&
[ "$(jq -er .input_sha256 "$dest/$provenance")" = "$(jq -er .input_sha256 ".monthly-outcomes/$provenance")" ]; then
ensure_pr
echo "Approved input unchanged; preserving the monthly report and provenance."
exit 0
fi
# Replace only this month's outputs, including obsolete cards.
git ls-files -z "$dest/outcomes-$MONTH.*" "$dest/assets/outcomes-$MONTH-*" | xargs -0 -r git rm -q --
cp ".monthly-outcomes/outcomes-$MONTH.json" "$dest/"
cp ".monthly-outcomes/outcomes-$MONTH.md" "$dest/"
cp ".monthly-outcomes/outcomes-$MONTH.provenance.json" "$dest/"
cp .monthly-outcomes/assets/"outcomes-$MONTH-"* "$dest/assets/"
git add -- "$dest/outcomes-$MONTH.json" "$dest/outcomes-$MONTH.md" "$dest/outcomes-$MONTH.provenance.json" "$dest/assets/outcomes-$MONTH-"*
if git diff --cached --quiet; then
ensure_pr
echo "No staged monthly report diff; no commit or push."
exit 0
fi
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git commit -s -m "📖 docs: publish outcomes for $MONTH"
git push origin "$branch"
remote_branch=true
ensure_pr
75 changes: 75 additions & 0 deletions .github/workflows/monthly-outcomes.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
name: Monthly outcomes

on:
schedule:
# Prior month, after every merge has the existing 14-day maturation period.
- cron: '17 6 15 * *'
workflow_dispatch:
inputs:
month:
description: 'Completed UTC month (YYYY-MM); empty selects prior month'
type: string
default: ''
pull_request:
paths:
- '.github/workflows/monthly-outcomes.yml'
- '.github/scripts/monthly-outcomes-pr.sh'
- 'src/scripts/monthly-outcomes.py'
- 'src/scripts/test-monthly-outcomes.py'

permissions:
contents: read

concurrency:
group: monthly-outcomes-${{ github.event_name == 'pull_request' && github.event.pull_request.number || 'publication' }}
cancel-in-progress: false

jobs:
selftest:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: src/go.mod
cache: false
- run: python3 src/scripts/test-monthly-outcomes.py

publish:
if: github.event_name != 'pull_request' && github.repository == 'hivecommons/hive'
needs: selftest
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Require supplied input and existing publication credential
env:
SUPPLIED: ${{ secrets.MONTHLY_OUTCOMES_INPUT }}
PUSH_TOKEN: ${{ secrets.TOPUP_PUSH_TOKEN }}
run: |
if [ -z "$SUPPLIED" ] || [ -z "$PUSH_TOKEN" ]; then
echo "::error::Monthly outcomes is not operational: an owner must supply MONTHLY_OUTCOMES_INPUT and the existing TOPUP_PUSH_TOKEN. No deployment collection is configured."
exit 1
fi
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: v5
fetch-depth: 0
token: ${{ secrets.TOPUP_PUSH_TOKEN }}
persist-credentials: true
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
go-version-file: src/go.mod
cache: false
- name: Validate, sanitize and render supplied aggregate
id: generate
env:
MONTHLY_OUTCOMES_INPUT: ${{ secrets.MONTHLY_OUTCOMES_INPUT }}
MONTH: ${{ inputs.month }}
run: python3 src/scripts/monthly-outcomes.py render --month "$MONTH" --out "$GITHUB_WORKSPACE/.monthly-outcomes"
- name: Open or update the month's reviewed PR
if: steps.generate.outputs.changed == 'true'
env:
GH_TOKEN: ${{ secrets.TOPUP_PUSH_TOKEN }}
MONTH: ${{ steps.generate.outputs.month }}
run: bash .github/scripts/monthly-outcomes-pr.sh
1 change: 1 addition & 0 deletions changelog.d/added-monthly-outcomes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Add a fail-closed monthly outcomes publication wrapper and reviewed-PR cadence for owner-supplied, sanitized evidence aggregates; deployment source access and monthly input remain explicit operator prerequisites.
17 changes: 13 additions & 4 deletions src/docs/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,25 @@ Each report's "What the README shows and leaves out" section records, per card,

## Regenerate

For the fail-closed monthly schedule, owner-supplied input contract and reviewed
PR process, see [monthly automation](monthly-automation.md). Hosted runners
do not have the spoke's evidence; scheduling alone does not collect it.

```sh
# on a spoke, or anywhere with a copy of its evidence tree
src/scripts/outcome-benchmark.sh --root /data/evidence --month 2026-10 \
--spend cost-2026-10.json # saved GET /api/cost, for cost per merged PR
--stories stories-2026-10.json # curated caught → fixed → merged examples
--red-team red-team-2026-10.json # prompt-injection evaluation counts
--hub hub.json # saved GET /api/hub/evidence (admin session), optional
--now 2026-10-27T00:00:00Z \
--spend docs/benchmarks/cost-2026-10.json \
--stories docs/benchmarks/stories-2026-10.json \
--red-team docs/benchmarks/red-team-2026-10.json \
--source "hive-oke spoke snapshot 2026-10-27"
```

The example preserves the historical observation time, not a completed-month
snapshot. `--hub` is optional and requires a separately saved, authorized
`GET /api/hub/evidence` response. Saved input paths above are relative to `src/`,
where the wrapper runs; `--root` must point to the actual private evidence copy.

The script wraps `hivectl evidence benchmark`; see [hivectl.md](../hivectl.md#evidence--benchmark) for every flag. Redraw the committed report and legacy cards without the evidence tree using `src/scripts/outcome-benchmark.sh --rerender`. The wrapper rejects `--readme` with instructions for the separate README renderer. The underlying CLI still supports marked legacy documents, but the root README deliberately has no legacy outcomes markers. To take a read-only copy of a spoke's evidence: `kubectl exec deploy/hive -c hive -- tar czf - -C /data evidence | tar xzf -`.

## Metrics
Expand Down
129 changes: 129 additions & 0 deletions src/docs/benchmarks/monthly-automation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Monthly outcomes publication

## Status and source access

This is a **supplied-input publication cadence, not a deployed data collector**.
The existing `outcome-benchmark.sh` runs `hivectl evidence benchmark` offline. It
reads schema-v1 files under the spoke's `/data/evidence`, optionally a saved
`GET /api/cost`, curated stories, red-team counts and saved hub evidence. It
does not call a dashboard or discover/authenticate to a spoke. Hub evidence
requires an admin session; the ordinary cost response also requires authorized
dashboard access. A hosted runner has neither that filesystem nor an assumed
dashboard session. No deployment credential, new service or private artifact
download is introduced here.

An evidence owner must produce and approve a complete monthly aggregate in the
source environment. Until that owner supplies the input below and the existing
publication credential, the scheduled/manual workflow **fails visibly and
publishes nothing**. Its presence does not establish operational collection.
If source collection is later automated, its timer belongs with the deployment
that owns the evidence, not on a hosted runner with expanded credentials.

## Cadence and cohort

`.github/workflows/monthly-outcomes.yml` runs at **06:17 UTC on the 15th** and
supports manual dispatch with an optional `month: YYYY-MM`. Empty selects the
prior completed UTC month, including December at January rollover. GitHub may
delay scheduled runs. The merge cohort must be exactly `[month start, next
month start)`, and its snapshot must observe at least 14 days after month end,
matching the existing maturation rule. A manual run before that point fails.
Past months can be dispatched explicitly; current/future months cannot.

The existing fold, rates, small-n suppression, Wilson intervals, lower medians,
coverage caveats and trend history are retained. The wrapper validates counts
and rate intervals; it does not invent confidence intervals for medians or
estimated spend. Cost keeps its **own** denominator and window, even if those
differ from the monthly merge cohort. The committed October snapshot remains
historical: approximately $6124.75 / 2869 hive-attributed merges from
2026-08-08 through 2026-10-27, an upper-bound/order-of-magnitude estimate, not
today's universal cost. It is not a completed-month automation input: it was
observed before October ended and contains 10 unreadable bundles.

## Owner-supplied input

1. In the evidence-owning environment, take an immutable, complete snapshot
using existing authorized access. Check collection completion independently:
zero unreadable files alone cannot prove a snapshot is complete. Retain the
raw snapshot privately and calculate its SHA-256. Do not copy it into this
public repository or upload it as an Actions artifact.
2. Record the exact generator Git revision. Use the existing generator against
that snapshot with explicit `--month`, `--now` (snapshot observation time)
and a private output directory. Optional cost input must be its matching
saved response, not an extrapolation from October's figures. Check any
`count_error`/partial or stale-count indicators in that saved response before
approving it: the computed Report cannot reconstruct those source errors.
Omit `Spend` entirely when its source is incomplete or unverifiable. Omit optional
hub, story and evaluation inputs; publication strips them regardless.
3. Wrap the computed `outcomes-YYYY-MM.json` in this envelope, in a private local
file. `report` is the complete existing `outcomebench.Report` object, not raw
bundles. The two attestations are the evidence owner's responsibility.

```json
{
"schema": 1,
"month": "YYYY-MM",
"collection_complete": true,
"publication_approved": true,
"snapshot_sha256": "<64 lowercase hex characters>",
"generator_revision": "<40 lowercase hex characters>",
"report": {}
}
```

4. Sanitize and validate locally, then inspect the prepared file:

```sh
python3 src/scripts/monthly-outcomes.py prepare --month YYYY-MM \
--input PRIVATE-ENVELOPE.json --out PREPARED-AGGREGATE.json
```

The wrapper projects onto known aggregate fields. It replaces repository
identities with `repository-001` labels, replaces source prose with the
snapshot digest, allows only `agent/bot/human/unknown` author categories,
drops unknown report fields, and omits stories, issue/review/log text,
URLs, hub context and evaluation metadata. Dates and fixed cohort/category
names are the only surviving input strings. It never logs input or renderer
stderr. Aggregate counts can still be sensitive: explicit owner approval
and human PR review are required, not just string scrubbing.
5. A repository secret administrator sets `MONTHLY_OUTCOMES_INPUT` to that
**prepared** JSON (maximum 48000 UTF-8 bytes), using the existing GitHub
secret management process. Do not put JSON in dispatch fields, issue bodies
or logs. Replace the secret with a newly approved snapshot each month. A
missing/stale month, premature observation, unreadable/empty cohort,
invalid metric or unapproved/incomplete attestation fails closed.
The workflow also requires the **existing** `TOPUP_PUSH_TOKEN` publication
credential. Do not expand its scope or mint new privileges for this task;
if unavailable, publication remains blocked. It uses no write grant on
`GITHUB_TOKEN` and does not fall back to that token (which may suppress PR CI).

For manual offline rendering, run `monthly-outcomes.py render` with the same
input/month and an **empty** `--out` directory. It calls the existing Go
`--from-json` renderer, emitting its usual report/JSON/SVG/PNG assets plus
`outcomes-YYYY-MM.provenance.json`. Provenance records the approved snapshot
digest, input generator revision, actual renderer revision, observation time
and sanitized input digest. Completeness is an attestation; hashes enable
reproduction by authorized owners, not independent public verification.

## Review, reruns and failure handling

Only this month's rendered outputs and provenance are staged. Raw/supplied
input is never staged or uploaded. Root README graphics and prior reports are
not refreshed. Identical already-merged input skips rendering; identical input
on the open monthly branch makes no commit/push/repost. If the branch push
succeeds but PR creation fails, the run fails visibly; an unchanged rerun
retries the missing PR without another commit or provenance update. A branch
with no monthly report difference against `v5` does not open a PR. A serialized
`automation/monthly-outcomes-YYYY-MM` branch has one PR against `v5`, updated
with ordinary signed-off commits (no force push). Obsolete cards for that
month are removed. A closed/declined or already-merged PR is not automatically
reopened/reposted; a human must resolve any proposed correction.

The workflow never merges. Normal CI and authorized review gates apply,
including privacy, provenance and representativeness review. Input does not
prove that mediated and unmediated PRs are causally comparable. Required
source data/access and credential provisioning remain operator gates.
CI's focused fixtures exercise UTC selection, malformed/partial/stale input,
privacy projection, historical cost preservation, deterministic real rendering
and no-op/error handling, including local-Git/Mock-GitHub one-PR reruns,
failed-creation recovery and closed-PR refusal; no fixture makes a live
deployment/API request.
Loading
Loading