Skip to content

✨ review: add review.shadow to record would_have_held for the review gate - #12158

Closed
mrbobbytables wants to merge 5 commits into
hivecommons:v5from
mrbobbytables:review-shadow-12104-fork
Closed

mrbobbytables wants to merge 5 commits into
hivecommons:v5from
mrbobbytables:review-shadow-12104-fork

Conversation

@mrbobbytables

@mrbobbytables mrbobbytables commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Closes #12104

Adds review.shadow (default false). When on, the review-approval gate is evaluated as require_approval would (including the artifact-unavailable case) but the PR stays merge-eligible; shadow.review_gate.would_have_held + reason/confidence is recorded on the evidence bundle and an audit log line is emitted. require_approval: true still blocks. Nothing here can block a merge.

Behavior note (intent.shadow): writeIntentShadow now uses the shared writeShadowEvidence helper. When evidence.signing_key_path points at a key that exists but fails to load, the shadow observation is skipped with a Warn instead of being written unsigned (previously it fell back to key = nil), so a misconfigured key can't downgrade a signed bundle to "signed": false. An absent key path/missing key file still records unsigned as before. Covered by changelog.d/changed-12104-intent-shadow-signing-key.md.

Includes dashboard PUT + Settings toggle, docs, changelog fragment, and a table test (shadow on / off / require_approval on). Not run locally per AGENTS.md; CI is the verdict.

— hive: backend=copilot model=claude-sonnet-5.5


🐝 Hive Agent: contributor | SHA: ccd1e5722

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 15, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

This PR conflicts with v5 itself. Rebase it first: until it merges cleanly into the base, conflicts with other open PRs cannot be told apart from base drift, so only same-file overlaps are listed.

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#12332 ✨ feature: Add cadence templates for quickly setting cadence for all agents (#12300) same files src/pkg/dashboard/static/index.html
#12343 ✨ selfreport: carry the affected repositories and owners in self-reported issues same files src/pkg/config/config.go
src/pkg/dashboard/static/index.html
#12346 ✨ nous: clarify passive baseline snapshots in Strategy Lab same files src/pkg/dashboard/static/index.html
#12328 ✨ feat(dashboard): close settings dialog when clicking Save & close with no changes same files src/pkg/dashboard/static/index.html

@hivecommons-hive

Copy link
Copy Markdown
Contributor

CI triage (comment-only — head lives in a fork):


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1

@hivecommons-hive

Copy link
Copy Markdown
Contributor

CI triage for head of this PR (run 38787988867 / 38787988855):

PR-local — needs a push from the fork:

  • test (rest 1/5) — settings_parity_test.go:33: hive.yaml settings are not referenced by the Settings UI and are not allowlisted: review.shadow. Either surface review.shadow in the Settings dialog (src/pkg/dashboard/... settings sections, alongside the other review.* keys) or add it to the parity allowlist in settings_parity_test.go.
  • test (rest 2/5) — TestStyleRatchet (style_ratchet_test.go:88): counts fell below baseline; lower styleRatchetBaselines["hub static pages"] to inlineStyles: 945, rawColors: 448, rawFontSizes: 426 so the ratchet keeps biting (likely from merging v5 into the branch).

Not yours — infra / shared, will clear on rerun:

Head is in a fork, so no push from the hive; merging v5 into the branch and pushing the two test fixes above should turn it green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Check report for head d6e14a0 (fork PR — the hive cannot push here, so here is the diagnosis and the exact fix). v5 is green on the same shards, so all three are PR-local or branch-staleness.

  1. test (rest 4/5) / test / coverage — TestRecordReviewShadowSigningKeyUnusable fails (review_shadow_test.go:175: signed=false shadow=&{ReviewGate:…}).
    evidence.LoadSigningKey returns nil, nil when the key file does not exist (pkg/evidence/sign.go:101, os.ErrNotExist branch), so writeShadowEvidence in src/cmd/hive/shadow_gate.go never takes the signing key unusable branch — it calls MergeReviewEvidenceShadow(path, obs, nil), which re-seals the previously signed bundle unsigned and folds the shadow in. Fix in writeShadowEvidence:

    key, err := evidence.LoadSigningKey(cfg.Evidence.SigningKeyPath())
    if err == nil && key == nil && cfg.Evidence.SigningKeyPath() != "" {
        err = fmt.Errorf("evidence: signing key %q not found", cfg.Evidence.SigningKeyPath())
    }
    if err != nil {
        logger.Warn(kind+" shadow: signing key unusable, evidence not recorded", ...)
        return
    }

    Defensively, MergeReviewEvidenceShadow (pkg/github/review_evidence_shadow.go) should also refuse to downgrade: if b.Signed && key == nil { return errors.New("refusing to rewrite signed bundle without key") } before evidence.Seal.

  2. test (rest 2/5) — TestStyleRatchet: operator static/index.html has 1492 style= attributes, above the baseline of 1491. The one new inline style is this PR's <div class="config-toggle" style="margin-top:10px"> in src/pkg/dashboard/static/index.html. Drop the style= (use an existing spacing class / token, e.g. the same class the neighbouring config-toggle rows use) — the ratchet only goes down.

  3. Also in TestStyleRatchet: contributor landing has 86 style= attributes, below the baseline of 94 / 197 raw font-size … below 201. That is branch staleness — this branch is 36 commits behind v5, where those baselines were already lowered. A rebase onto origin/v5 clears it; no edit needed.

After rebase + (1) + (2), the remaining red should clear; tide merges on green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

mrbobbytables and others added 4 commits October 16, 2026 07:46
…gate

Hive-Run: hivecommons#12104
Hive-Plan: RFC hivecommons#12062 / hivecommons#12068 shadow-mode gates
Hive-Spec: review-shadow#would-have-held

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
(cherry picked from commit a187caa)
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…adow write path

Adds a review.shadow reference in the review config handler so the
Settings parity test sees the new toggle, and a round-trip test for
recordReviewShadow through MergeReviewEvidenceShadow.

Hive-Fix-Attempt: 1/6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…logging

Review follow-ups for hivecommons#12158 (review.shadow, hivecommons#12104):

- Branch the review-verdicts.json load warning on require_approval so a
  shadow-only run no longer claims merge eligibility will fail closed.
- Extract writeShadowEvidence, shared by intent.shadow and review.shadow,
  so the two recorders cannot drift again; recordReviewShadow now gets
  the logger==nil fallback and logs confidence like the intent gate.
- When a signing key is configured but unusable, skip the shadow write
  and Warn instead of rewriting a signed bundle as unsigned.
- Record both shadow observations from one eligible block with a shared
  evaluated_at.
- Move the review shadow code to review_shadow.go beside its test, and
  add the dashboard PUT/GET + Settings toggle parity coverage for
  review.shadow.

Hive-Fix-Attempt: 2/6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…w signing-key change

Add review.shadow to the /api/config/review GET response and PUT request
schemas in dashboard/openapi.json so the spec matches what
handleReviewConfigGet/Put emit and accept. Add a changed- changelog fragment
for writeIntentShadow now skipping (Warn) instead of writing unsigned when a
configured signing key fails to load.

Hive-Fix-Attempt: 3/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…op the inline style

LoadSigningKey treats a missing key file as 'unsigned', so the
signing-key-unusable test never reached the guard; write a malformed key
instead. Use the existing .mt-3 utility on the new review.shadow toggle so
the style ratchet stays at baseline.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member

Taken over in #12337 (rebased onto v5 with your commits preserved as author; fixed the two red checks — the unusable-key test fixture and the style-ratchet inline style — plus the docs nits from the swarm). Once #12337 merges I'll close this one as superseded. Thank you!

hivecommons-hive Bot pushed a commit that referenced this pull request Oct 16, 2026
…gate (#12337)

* ✨ review: add review.shadow to record would_have_held for the review gate

Hive-Run: #12104
Hive-Plan: RFC #12062 / #12068 shadow-mode gates
Hive-Spec: review-shadow#would-have-held

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
(cherry picked from commit a187caa)
Signed-off-by: Andy Anderson <clubanderson@users.noreply.github.com>

* 🧪 review: reference review.shadow for settings parity and test the shadow write path

Adds a review.shadow reference in the review config handler so the
Settings parity test sees the new toggle, and a round-trip test for
recordReviewShadow through MergeReviewEvidenceShadow.

Hive-Fix-Attempt: 1/6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
Signed-off-by: Andy Anderson <clubanderson@users.noreply.github.com>

* 🧹 review: share the shadow evidence writer and tighten review.shadow logging

Review follow-ups for #12158 (review.shadow, #12104):

- Branch the review-verdicts.json load warning on require_approval so a
  shadow-only run no longer claims merge eligibility will fail closed.
- Extract writeShadowEvidence, shared by intent.shadow and review.shadow,
  so the two recorders cannot drift again; recordReviewShadow now gets
  the logger==nil fallback and logs confidence like the intent gate.
- When a signing key is configured but unusable, skip the shadow write
  and Warn instead of rewriting a signed bundle as unsigned.
- Record both shadow observations from one eligible block with a shared
  evaluated_at.
- Move the review shadow code to review_shadow.go beside its test, and
  add the dashboard PUT/GET + Settings toggle parity coverage for
  review.shadow.

Hive-Fix-Attempt: 2/6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
Signed-off-by: Andy Anderson <clubanderson@users.noreply.github.com>

* 📝 review: document review.shadow in openapi.json and the intent.shadow signing-key change

Add review.shadow to the /api/config/review GET response and PUT request
schemas in dashboard/openapi.json so the spec matches what
handleReviewConfigGet/Put emit and accept. Add a changed- changelog fragment
for writeIntentShadow now skipping (Warn) instead of writing unsigned when a
configured signing key fails to load.

Hive-Fix-Attempt: 3/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
Signed-off-by: Andy Anderson <clubanderson@users.noreply.github.com>

* 🧪 review: make the unusable-signing-key test use a malformed key and drop the toggle's inline style

- TestRecordReviewShadowSigningKeyUnusable pointed SigningKeyFile at a
  missing file, which LoadSigningKey treats as "no key" (records unsigned
  by design). Write a malformed key so the test exercises the guard that
  skips the write with a Warn.
- The new Settings toggle added an inline style="margin-top:10px", pushing
  operator index.html over the style-ratchet baseline; .config-toggle
  already spaces stacked toggles via margin-bottom, so drop it.
- Document the signing-key caveat in review-swarm.md / intent-verification.md
  and list review.shadow in hive.yaml.example (swarm docs-currency notes).

Signed-off-by: Andy Anderson <clubanderson@users.noreply.github.com>

---------

Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Signed-off-by: Andy Anderson <clubanderson@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: scanner <scanner@hive.kubestellar.io>
Co-authored-by: Andy Anderson <clubanderson@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (intent-alignment) — diff satisfies #12104; one scope note, non-blocking.

  • Disclosed scope creep: intent.shadow behavior change (src/cmd/hive/shadow_gate.go:25-28, src/cmd/hive/intent_shadow.go:50). #12104 only asks for review.shadow; folding writeIntentShadow into the shared writeShadowEvidence helper changes the existing intent path from "write unsigned on key-load error" (old key = nil fallback) to "skip with Warn". This is a sensible fix and is disclosed in the PR body and changelog.d/changed-12104-intent-shadow-signing-key.md, but it is a behavior change to a different feature riding on this issue — flagging so a maintainer consciously accepts it rather than discovering it later.

Verified against the issue: classifyMergeEligibility review gate still keyed only on requireReviewApproval (merge_eligibility.go:462), so shadow cannot alter a bucket; shadowReview is forced off when require_approval is on (merge_eligibility.go:843); artifact-unavailable case mirrored (review_shadow.go:21-22); dashboard PUT/GET + toggle and table tests present.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency (read at head 77d5118)

  1. Changelog names a config key that does not exist. changelog.d/changed-12104-intent-shadow-signing-key.md:1 says the new skip-on-bad-key behaviour triggers when evidence.signing_key_path points at an unusable key. The hive.yaml key is evidence.signing_key_file (src/pkg/config/evidence.go:17; src/docs/operator-reference.md:164; src/docs/review-evidence.md:104). SigningKeyPath() is only the Go accessor (evidence.go:26). An operator grepping their config for signing_key_path finds nothing. The PR body repeats the same name.

  2. Operator docs now contradict shadow-write behaviour. src/docs/operator-reference.md:164 states that when the signing key is "empty, absent or unusable" bundles are written with "signed": false and a warning logged. After this PR, src/cmd/hive/shadow_gate.go:26-30 skips the write entirely for an unusable key — for both review.shadow and (changed behaviour) intent.shadow. Neither that row nor the intent.shadow section (src/docs/intent-verification.md:76, "Recording needs evidence enabled and an existing bundle for the head") nor the new review.shadow paragraph (src/docs/review-swarm.md:224) mentions that a loadable-but-invalid key suppresses shadow recording. The fact lives only in the changelog fragment; one sentence in the operator-reference row (or the two shadow sections) would close the gap.

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/maintainability review (read at head 77d5118)

Shadow gate duplicates the enforce predicate instead of sharing it — src/cmd/hive/review_shadow.go:19-25 re-implements the require_approval decision (!g.reviewLoaded → "review approval required, but review-verdicts.json is unavailable"; !HasAggregateApproval(...) → "awaiting review approval") as a second copy of the branch in classifyMergeEligibility at src/cmd/hive/merge_eligibility.go:462-468, including both reason strings verbatim. The contract in #12104 is that shadow evaluates the gate exactly as require_approval would; with two hand-synced copies, any later change to the enforce branch (an extra condition, a reworded reason) silently makes would_have_held diverge from what the real gate does — which is the one thing this signal must not do. The sibling intent.shadow avoids this by reusing the same predicate the enforce path uses (intentShadowGate → v.BlocksMerge(true), src/cmd/hive/intent_shadow.go:23-24). Suggest extracting a small reviewGateHold(g mergeGates, fullRepo string, pr github.PullRequest) (held bool, reason string) in merge_eligibility.go and calling it from both classifyMergeEligibility and reviewShadowGate; TestReviewShadowGate would then cover parity for free.

Everything else here (shared writeShadowEvidence extraction, dashboard PUT pointer-field pattern, Settings toggle markup, changelog fragment naming, docs placement) follows the existing idioms.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson

Copy link
Copy Markdown
Member

Landed on v5 via #12337 (your commits preserved as author, plus the two CI fixes — malformed-key test fixture and the style-ratchet inline style). Closing as superseded; #12104 is closed by that merge. Thank you @mrbobbytables!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/scanner Created or modified by the scanner agent dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] ✨ review: review.shadow — record would_have_held for review-gated merge

2 participants