Skip to content

✨ evidence: automate reviewed monthly outcomes publication - #16250

Merged
clubanderson merged 3 commits into
v5from
automation/monthly-outcomes-20261102
Nov 2, 2026
Merged

clubanderson merged 3 commits into
v5from
automation/monthly-outcomes-20261102

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Scope

Automates publication of EXISTING outcome evidence, not a new metrics framework or a hosted deployment collector. Leaves weekly competitor reports, workflow compiler work, root README graphics and the historical October report unchanged.

Source/access assessment

Existing outcome-benchmark.sh reads deployment-local schema-v1 /data/evidence plus optional saved API files; a hosted runner has no assumed source access. New workflow runs 06:17 UTC on the 15th for the prior completed UTC month, with manual dispatch. It fails visibly unless an evidence owner supplies MONTHLY_OUTCOMES_INPUT (approved sanitized aggregate envelope) and the existing TOPUP_PUSH_TOKEN. No credentials were provisioned or expanded and no live collection was run. End-to-end source collection remains blocked on the evidence owner.

Safety and reproducibility

  • Exact UTC month bounds, full 14-day maturation, no unreadable/empty cohorts, validated counts/Wilson intervals and explicit collection/publication attestations.
  • Repository names anonymized; source prose, raw evidence, stories, hub context and evaluation metadata excluded. Only monthly renderer outputs/provenance are staged.
  • Original cost denominator/window and upper-bound estimate methodology preserved; no fabricated cost intervals.
  • SHA-pinned actions, read-only GITHUB_TOKEN, existing publication credential only; no automatic merge.
  • One branch/PR per month; unchanged approved input preserves original provenance despite base changes; closed PRs are not reposted.

Validation

Independent bounded precommit review completed; fixed full-maturation validation and unchanged-input open-PR provenance churn. Added focused CI fixtures including real existing Go rendering and local-Git/Mock-GitHub PR idempotency. Per AGENTS.md no repository suites, builds, lints or guards were run locally. CI is the executable verdict. Trusted scripts tree literal re-pinned.

Remaining operational gate

An authorized evidence owner must produce a complete observed snapshot, approve aggregates/privacy/cost source, provision the per-month sanitized input through existing secret management, and confirm existing publication credential availability. This PR does NOT claim an operational scheduled source pipeline.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 2, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 2, 2026
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 2, 2026
@clubanderson

Copy link
Copy Markdown
Member Author

Implementation checkpoint: exact head 19b2ce8 has green Monthly outcomes selftest (including real offline rendering, sanitization, historical cost preservation and local-Git/Mock-GitHub idempotency), build-and-test, build-go, lint, changelog, DCO and native queue policy. No repository tests/build/lints were executed locally. Remaining checks were overlayfs-exec-guard, patch-coverage and Tide at the last bounded read; no failures. This remains a supplied-input publication mechanism, NOT an operational source collector: an authorized evidence owner must supply an approved complete monthly aggregate and confirm the existing publication credential. Source data/credentials were not accessed or provisioned. Prow says /lgtm then root approver hanthor /approve; author clubanderson cannot provide an independent native approval. No merge or policy bypass attempted. Overlap notice with #16241 is only the mandatory trusted-scripts tree literal; whichever merges second must refresh that pin against actual v5 without changing the other PR’s work.

@hivecommons-hive

Copy link
Copy Markdown
Contributor
19b2ce8

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head 19b2ce8d0dfc) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/monthly-outcomes.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 2, 2026
@clubanderson

Copy link
Copy Markdown
Member Author

Bounded follow-through completed for unchanged head 19b2ce8: every current v5 branch-protection-required check has successful exact-head evidence, and Monthly outcomes selftest, overlayfs-exec-guard, coverage, test and patch-coverage are all terminal SUCCESS. Superseded duplicate DCO-attribution runs were canceled, with the newer required check successful. No source changes were needed.

Remaining real gate is Tide: its exact-head status says “Not mergeable. Needs approved, lgtm labels.” No PR reviews are recorded. Clarification to my earlier wording: hanthor is the notifier-selected/requested approver, NOT the sole eligible approver. Current v5 OWNERS lists clubanderson, hanthor, Danathar, nicholasjackson and kellyaa as approvers/reviewers: https://github.com/hivecommons/hive/blob/v5/OWNERS . Author is clubanderson; I will not self-approve or bypass the missing Prow authorization. An eligible other OWNER (hanthor, Danathar, nicholasjackson or kellyaa) can provide the necessary authorized review/approval under Prow. Branch protection currently has required_pull_request_reviews=null; I am not claiming a universal native second-human approval requirement or that two separate reviewers are required.

PR/branch remain preserved, not merged. Publication also retains its separate approved supplied-input and existing-credential prerequisites: no operational scheduled private-evidence feed is claimed.

@clubanderson

Copy link
Copy Markdown
Member Author

Final moving-base update: while completing the bounded check, v5 advanced through #16241 (ceb80dd), and GitHub now reports mergeable_state=dirty. A read-only git merge-tree against fetched v5 confirms the sole conflict is .github/scripts/integration-governance.cjs, the mandatory TRUSTED_SCRIPTS_TREE literal; no worktree files were changed. Thus there are TWO remaining gates: reconcile that literal against the combined src/scripts tree (then obtain fresh exact-head CI), and obtain the missing eligible Prow approved/lgtm authorization. Current head 19b2ce8 remains clean/preserved with all required checks terminal-success. I am not chasing moving v5 or presenting those results as a verdict on a future conflict-resolution head. The evidence-owner supplied-input/credential prerequisite remains unchanged.

Merge v5 8664848 without changing landed scripts. Independently reviewed complete scripts-tree union: all 67 existing entries unchanged; only the two monthly automation files and the byte-identical Twitch script from #16241 are added. Pin exactly 96c95ac. No runtime enrollment, privilege, weekly report source or canonical workflow changes.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

Completed the identified branch-maintenance repair in our isolated feature worktree. New DCO-signed merge head: 5f2e144; normal push, no force/amend. Merged v5 8664848.

Before updating the literal, independently checked the COMPLETE old/new src/scripts inventory: 67 common-base entries are byte/mode-identical in both parents and the combined tree; the only additions are monthly-outcomes.py, test-monthly-outcomes.py and the unchanged twitch-channel-cta.sh from landed #16241 (blob 653c90868bee2e2f94fd124f4cc084a48a6402a3). Both original parent pins matched their actual trees. A separate bounded read-only repair review independently verified that exact union, all three additions, modes/blob IDs and trust policy (integration-orchestration.md lines 46–51). Only then pinned the reviewed combined tree 96c95ace27dc711bc1367c6a36e5cd4c9b7153ef. No runtime/profile enrollment, execution authority or privileges changed. Relative to merged v5 the PR remains its original eight-file monthly scope. No weekly source or canonical CI source/template changed; no regeneration was needed.

Fresh exact-head CI is now required; previous-head success is not being reused. Independent repair review is NOT Prow merge approval. Missing eligible approved/lgtm authorization remains separate from the operational prerequisite: owner-approved supplied monthly input and existing publication credentials. No live approved source generation or operational private evidence feed is claimed.

@hivecommons-hive

hivecommons-hive Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the review swarm requested changes

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@clubanderson

Copy link
Copy Markdown
Member Author

Repair follow-through is terminal and green on unchanged head 5f2e144. After the normal shared quota reset, bounded checks at three-minute intervals confirmed all 11 branch-protection-required contexts SUCCESS, plus Monthly outcomes selftest, overlayfs-exec-guard and patch-coverage SUCCESS (14/14 required/focused contexts, zero pending or failed). No previous-head result was reused. The shared-pin conflict is reconciled, the overlap label has cleared, and the isolated worktree is clean.

Only merge-authority gate remains: current Tide explicitly says “Not mergeable. Needs approved, lgtm labels.” There are no PR reviews. Eligible authority comes from OWNERS, not a requested-reviewer suggestion: https://github.com/hivecommons/hive/blob/866484828d73d241ffd7f03a84b1173ede658749/OWNERS . hanthor, Danathar, nicholasjackson or kellyaa can provide authorized non-author review/approval under Prow; clubanderson authored this PR, so I have not self-approved, synthesized labels, bypassed Tide or merged. No universal two-distinct-reviewer rule is asserted. PR and signed branch are preserved for that genuine authorization.

Separate operational prerequisite remains unchanged: an evidence owner must supply approved complete monthly aggregates through MONTHLY_OUTCOMES_INPUT and confirm the existing publication credential. CI exercised offline deterministic rendering and mock publication, not live private source collection; no operational scheduled private-evidence feed is claimed. No further source/canonical/weekly changes were made.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review at head 5f2e144. Verified the src/scripts re-pin (git rev-parse 5f2e144:src/scripts = 96c95ace…, matches .github/scripts/integration-governance.cjs:12), ran the 7 non-Go fixtures in src/scripts/test-monthly-outcomes.py locally (pass; the real-renderer fixture passed in CI selftest). One in-scope gap:

monthly-outcomes-pr.sh can orphan a pushed branch and then never open the PR (.github/scripts/monthly-outcomes-pr.sh:23-27, :36-39, :45-47).
git push (line 43) and gh pr create (line 45) are not atomic. If gh pr create fails after the push (rate limit, transient API error), the run exits 1, but every rerun with the same approved input now takes the early-exit at lines 23-27 — gh pr list returns [] (line 9), the remote branch exists (line 14), the provenance digests match — and exits 0 printing "preserving the existing monthly PR" although no PR exists. Reproduced locally with a mock gh whose create fails once: run 1 exit 1, run 2 exit 0, remote branch present, zero PRs. The same happens via the "No monthly report diff" exit at lines 36-39. Net effect: the month is silently never published until the owner changes the input, contradicting the PR body's fail-visibly / one-branch-one-PR claim. Fix: create the PR on both early exits when none exists:

@@ .github/scripts/monthly-outcomes-pr.sh
 branch="automation/monthly-outcomes-$MONTH"
 pr="$(gh pr list --repo "$GITHUB_REPOSITORY" --head "$branch" --state all --limit 2 --json number,state)"
+ensure_pr() {
+  [ "$(printf '%s' "$pr" | jq length)" -eq 0 ] || return 0
+  gh pr create --repo "$GITHUB_REPOSITORY" --base v5 --head "$branch" \
+    --title "📖 docs: monthly outcomes $MONTH" \
+    --body "Owner-supplied, sanitized monthly aggregates. Review the cohort, observation time, cost denominator and provenance before merging. No raw evidence, stories or deployment identities are published. Collection completeness and publication approval are owner attestations, not independently verified by this workflow. CI and normal review gates must pass; this workflow does not merge."
+}
 ...
   echo "Approved input unchanged; preserving the existing monthly PR without a repost."
+  ensure_pr
   exit 0
 ...
   echo "No monthly report diff; no commit, push or stale PR repost."
+  ensure_pr
   exit 0
 ...
 git push origin "$branch"
-if [ "$(printf '%s' "$pr" | jq length)" -eq 0 ]; then
-  gh pr create ... (existing block)
-fi
+ensure_pr

Worth adding a 4th attempt to test_publication_one_pr_rerun_no_diff_and_closed_pr where the mock gh create fails once and the rerun with MOCK_PRS='[]' must call create again.

— hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Merge v5 7f5097c, preserving its landed sources. Retry orphan monthly PR creation on unchanged/no-staged-diff runs without report or provenance commits. Refuse closed/merged PR reposts and branches with no monthly source diff. Add stateful GitHub/local-Git CI regressions for repeated creation failure, recovery and no-op publications.

Requested precommit code-review agent found no significant issues. Verified parent scripts pins and combined inventory: only monthly-outcomes.py and test-monthly-outcomes.py differ from v5 scripts tree 9affee8; pin actual combined tree 1e72143. No local tests/builds/lints/guards; CI is the verdict.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

Owner-directed completion: the owner explicitly requested “finish these”, “we dont need someone elses review” and “get it going”. This is a PR-specific merge decision, not a reviewer approval or fabricated LGTM. Repaired the real push/create atomicity bug: unchanged reruns retry missing PR creation and surface failures without provenance churn; added stateful mock-GitHub/real-local-Git CI regressions. Requested precommit code-review agent found no significant issues. CI-only validation remains mandatory. Once every protected exact-head check, native-queue-policy and focused Monthly outcomes selftest is green and no unresolved hold/security issue remains, use owner-authorized head-matched admin squash only if Prow approval remains. No global rules or security/DCO safeguards are changed. Operational publication still requires approved MONTHLY_OUTCOMES_INPUT and the existing publication credential; no secrets provisioned or private-source collection run.

@clubanderson

Copy link
Copy Markdown
Member Author

Completion receipt for exact repaired head 7dcbb72 (not previous-head CI): all 12 protected entries SUCCESS: both Docker gate entries; DCO; build-and-test; test; coverage; golangci-lint; gosec; govulncheck; changelog-fragment-guard; verify PR contents; Sign-off survives the squash. Waited for the initially absent build/test/coverage aggregates, rather than treating an early 9-entry check watch as completion.

Rechecked GitHub: same head, mergeable=true, no hold labels, no failed/timed-out/cancelled/action-required check runs. The previous correctness finding at 5f2e144 is repaired and covered above. Only pending legacy Prow status is Tide: “Needs approved, lgtm labels.” Per the recorded explicit owner direction, proceeding with PR-specific admin squash matched to this exact head, with a real DCO signoff. No synthetic review/approval/LGTM or repository-wide policy changes.

Operational publication is STILL separately gated on owner-approved MONTHLY_OUTCOMES_INPUT and the existing publication credential. This is code/CI completion, not a claim of live private-source collection or an operational source pipeline.

@clubanderson
clubanderson merged commit a62db5c into v5 Nov 2, 2026
85 of 86 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the automation/monthly-outcomes-20261102 branch November 2, 2026 20:34
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@clubanderson

Copy link
Copy Markdown
Member Author

Verified MERGED at 2026-11-02T20:34:55Z, squash commit a62db5c, from CI-verified head 7dcbb72. Fetched/pulled v5 after merge and verified the squash is an ancestor. Squash commit retains the real Andy Anderson DCO signoff.

Cleanup was strictly limited to the worktree and local repair branch created by this completion: removed /Users/andan02/hive-repair-16250-owner and repair/monthly-outcomes-16250-owner. Did not edit/remove the pre-existing monthly worktree, did not manually delete its pre-existing automation branch, and did not change PR16251 or PR16277. Exact-head CI receipts and the owner-authorized merge rationale are recorded above. The approved-input/existing-publication-credential operational gate remains unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant