Skip to content

✨ Auto-decide reviewer scope questions on hive-authored green PRs - #16598

Merged
hivecommons-hive[bot] merged 4 commits into
v5from
feat/reviewer-question-autodecide
Nov 4, 2026
Merged

hivecommons-hive[bot] merged 4 commits into
v5from
feat/reviewer-question-autodecide

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #16597

Why

docs#618 (hive-authored, CI 4/4, Review 5/5, no findings) parked at 085 "Needs human decision" because one perspective ended with a question — "@clubanderson should decide whether to apply the hive-docs-refresh.yml opt-in … and whether this PR should merge ahead of that workflow edit" — and the word workflow tripped the self-authored risk class. Same shape as hive#16456. Owner: "this is auto-merge material, not needs-human".

What

  • pkg/review/decision_question_class.go — pure ClassifyDecisionQuestion: policy sign-off (workflow/action edits from a fork, OWNERS / CODEOWNERS / SECURITY / rulesets / secrets / key material, auto_merge / human_merge_paths / self_authored_decision_policy config edits, owner-only guard wording: sentinel, GitHub/App permissions, unauthorized, tokens/secrets/credentials, branch protection) vs scope question (hive-authored, CI green, no unresolved finding) vs ineligible. Table test with 17 cases including both real asks verbatim.
  • pkg/dashboard/review_question_autodecide.go — paced sweep from UpdateStatusIfFresh: picks 085 PR rows a reviewer's question parked (not on hold, CI not red / running), reads fork + files from GitHub, classifies, and for a scope question runs the existing runDelegatedDecision pipeline with the hive as requester (pending let-hive-decide ledger row, same job registry as the button). Approve → the pipeline's /lgtm /approve + a follow-up: <question> issue (writeback-gated); direction → fixer; owner-only / recommend-only → park stays and the row already shows "Recommendation recorded" / "needs your sign-off" (🐛 Activity Monitor: say whose sign-off a parked review needs and lead with Approve #16578). Once per head via /data/review-question-autodecide.json; one INFO line reviewer question auto-decided with class + reason; review_questions_autodecided_1h on /api/status.
  • Config governor.review.auto_decide_questions (default true), documented in docs/operator-reference/sweeps-and-operations.md with the two classes; golden updated.

Tests

  • pkg/review: TestClassifyDecisionQuestion (17 cases), TestClassifyDecisionQuestionRealAsksMatchRiskClassByWording.
  • pkg/dashboard: candidate selection, unresolved-finding read, once-per-head marker + 1h counter, follow-up title.

A review perspective that ends with a question rather than a finding parked
hive-authored, green, finding-free PRs at 085 on wording alone (docs#618,
hive#16456). Classify the ask: policy sign-off stays a person's; a scope /
follow-up question runs the existing Let-hive-decide pipeline once per head
(approve -> merge queue + follow-up issue; direction -> fixer; owner-only ->
park with the recommendation). Config governor.review.auto_decide_questions
(default on); review_questions_autodecided_1h on /api/status.

Fixes #16597

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 4, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16615 ✨ feat(ci): approve gated workflows for trusted PR heads same files src/pkg/config/config.go
src/pkg/config/testdata/agent_runtime_golden.json
src/pkg/dashboard/server.go
#16607 ✨ feat(github): enforce pin-only exception in merge-request relay same files src/pkg/config/config.go
#16616 📖 docs: centralize Code activity compatibility guidance same files src/docs/api-reference/dashboard-misc.md

@hivecommons-hive

hivecommons-hive Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the review swarm requested changes

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from reviewer (folded here because this head already carries its hive review):

Read review/decision_question_class.go in full, dashboard/review_question_autodecide.go:215-396, and the paths the classifier depends on (review/review.go:866,985, review/security_finding.go:76, stage_tracker_build.go:381-382). The classifier itself is clear and the policy-sign-off-wins ordering is right. Two things a person should weigh before this lands, since it is merge-policy code:

1. Fail-open when the verdict aggregate is unavailable (medium). DecisionQuestionInput has no risk-marker field; the reviewer's <!-- hive-decision-risk: high --> escape hatch reaches this sweep only indirectly — IsSecuritySensitiveFinding (security_finding.go:76) turns it into agg.DecisionFinding, which reviewQuestionUnresolvedFindings (review_question_autodecide.go:254-256) reads. That chain depends on the lookup succeeding: autoDecideReviewQuestions leaves lookup nil when review.LoadArtifact("") errors (:313-316), and autoDecideReviewQuestion only sets UnresolvedFindings inside if lookup != nil { if agg, ok := ...; ok { (:335-341). With no artifact, or no aggregate for that head, the input is {Question, HiveAuthored (tracker read), CIGreen: true} and a risk-high question whose wording misses decisionOwnerOnlyGuards is classified scope-question and delegated. The conservative shape is to return false (keep for a person) when the aggregate cannot be read, the way the PR-read failure at :343-346 already does.

2. CIGreen is hard-coded true (:334). It is backed by the candidate filter excluding ciRed || ciChecking (:224), which in turn derives from pr.CIStatus/FailingChecks/CIChecksRunning (stage_tracker_build.go:381-382). That holds today; it just means the classifier's CIGreen rule is never exercised by this caller, so a future caller or a tracker row with CIStatus == "" (neither failure nor pending) passes as green. Worth either plumbing the real value or a comment saying the filter is the gate.

Open question, not a finding: decisionWorkflowPaths is only consulted when in.FromFork (decision_question_class.go:122). A same-repo PR adopted in place (AdoptedBy > 0 counts as hive-authored, :238) that edits .github/workflows/** would be a scope question. That may be intended — delegatedecision/policy.go:744 already approves "workflow hygiene" — but the reviewer policy lists workflow edits as a risk class regardless of origin, so I would like that to be a deliberate choice.

CI currently shows test (rest 3/4) failed on this head; I have not read the log and make no claim about the cause.

Evidence bundle `#165985db2829` sha256:ab1f88ad4d1015df4d4117834496e7afa5b5171da8f6e1476f27b44fef7c554a — download from the dashboard or `hivectl review evidence #16598`.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read review_question_autodecide.go, decision_question_class.go, the candidate filter's inputs in stage_tracker_build.go, and EnrichCIStatus in pkg/github/client.go at this head. The classifier and once-per-head ledger are sound; the CI-green precondition is not actually enforced, which matters because an approve verdict admits the PR to the merge queue and files a follow-up issue.

Medium — "CI green" is asserted, not checked (src/pkg/dashboard/review_question_autodecide.go:333, correctness)

  • autoDecideReviewQuestion hardcodes CIGreen: true and relies on reviewQuestionCandidates having excluded red/running CI via it.ciRed || it.ciChecking (line 225).
  • it.ciChecking is pending && pr.CIChecksRunning (stage_tracker_build.go:382). But CIChecksRunning is assigned in exactly one place, client.go:2737, inside the hasFail branch — i.e. only when CIStatus is already failure, which makes ciRed true and ciChecking false. In the plain pending case (client.go:2742, checks still running, none failed) and the no-check-runs case (client.go:2727) CIChecksRunning stays false, so ciChecking is false and the row passes the filter.
  • Net effect: a hive-authored PR whose checks are still running, or has no check runs yet, is classified with CIGreen: true, consumes one of the 3 paid decisions per sweep, is marked once-per-head (so it is not revisited when CI actually settles on that head), and on approve gets a follow-up issue filed. The comment on reviewQuestionCandidates ("CI neither red nor still running"), the CIGreen field doc, and the config doc ("own green, finding-free PR") all promise otherwise.
  • Suggested fix: derive CIGreen from the PR's CIStatus == "success" (or CIRealCheckRuns > 0 && allDone) at the candidate stage instead of inferring it from ciChecking, and add a test row with CIStatus: pending, CIChecksRunning: false that must be excluded. (The ciChecking dead-flag is pre-existing; the new feature is the first thing that depends on its negative.)

Low — marker written before the job is known to start (review_question_autodecide.go:354-363)

reviewQuestionAuto.mark runs before delegatedJobs.start; when start returns !ok (a delegate already running for the key) the function returns without deciding, but the head is now seen and is never retried. Moving mark after a successful start keeps the once-per-head guarantee without burning the head on a race.

For awareness, not a defect: .github/workflows/** is only a sign-off path when FromFork is true (decision_question_class.go:115). A hive-branch PR that edits a workflow with a scope question auto-decides. That is deliberate per the comments; stating it so the owner confirms it is intended.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

CI triage for test (rest 3/4) / test / coverage at a35a309 — PR-local, not a baseline issue (v5 tip is green on test and build-and-test).

Failure: pkg/review TestClassifyDecisionQuestion/auto_merge_config_change_in_a_patch_is_a_sign-off

decision_question_class_test.go:121: reason = "PR touches policy path deploy/hive.yaml.dashboard", want it to mention "changes auto_merge"

Cause: the fixture path deploy/hive.yaml.dashboard matches **/hive.yaml.dashboard in decisionPolicyPaths, so ClassifyDecisionQuestion returns from the policy-path rule before decisionPatchPolicyKey is ever consulted. The case therefore never exercises the patch-key rule it is named for.

Fix (test only): use a YAML path that is not in decisionPolicyPaths so the patch rule is the one that settles the class, e.g. in src/pkg/review/decision_question_class_test.go:

			in: green("Confirm the lane should merge its own PRs.", DecisionQuestionFile{Path: "deploy/spoke-overrides.yaml",

(Any .yaml/.yml/.dashboard path outside the pinned policy list works — the .dashboard branch in decisionPatchPolicyKey is only reachable for a non-hive.yaml.dashboard basename, so if that branch is meant to be live you may also want a second case like deploy/overrides.dashboard.) The test aggregate and coverage jobs are downstream of this one failure.

Not pushing to this branch since it is human-authored.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@hivecommons-hive
hivecommons-hive Bot merged commit b7cc0d0 into v5 Nov 4, 2026
79 of 81 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 5db2829.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ Reviewer scope questions on hive-authored green PRs should auto-decide, not park at 085

1 participant