Skip to content

πŸ› fix: stabilize same-head stage verdicts and suppress reversals - #16881

Merged
hivecommons-hive[bot] merged 2 commits into
hivecommons:v5from
mendezr:fix/16880-stage-reconciler
Nov 6, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
hivecommons:v5from
mendezr:fix/16880-stage-reconciler

Conversation

@mendezr

@mendezr mendezr commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #16880

  • Make actionable requires_human normalization independent of the previous stage. Identical current remediation evidence now selects the same author-kind-appropriate fixing stage whether the previous record was 070, 085, or absent. Explicit decision requests, policy-owned human verdicts, and risk escalations retain the existing human-decision checks.
  • Add a 15-minute same-head reversal guard at the reconciler's write boundary. It remembers successful stage-label transitions from the writer's fresh-read outcome, logs stage flap suppressed, and refuses the inverse transition without writing labels. Failed/gated writes do not arm it; no-op and suppressed sweeps do not extend the window. A new head, timeout, or terminal decision releases it.
  • Account for the legacy sweep pre-observing its opinion in the Store before calling the engine: suppression restores the last written stage, rather than trusting that pre-observed record as the previous label state.
  • Leave label-triggered workflows unchanged and add a changelog fragment.

Regression coverage

  • Feed the normalized-fixing guard its own result for three sweeps, starting from no record, 070, and 085. Starting from 085/no record fails the old history-dependent guard.
  • Reconcile unchanged real PR facts for three sweeps, covering both actionable remediation and an explicit human decision; later sweeps must not write or rely on suppression.
  • Attempt three reverse transitions on an unchanged head, including legacy pre-observation; assert no label writes, stable Store stage, and a suppression diagnostic. Exercise exact-window expiry, a new head, and terminal bypass.

Validation and scope

  • gofmt -l on changed Go files: clean.
  • git diff --check: clean.
  • Tests not run locally, per AGENTS.md; CI is the test verdict.
  • This is an engine fix, not an answer to the merge/trust-policy question on ✨ feat(github): enforce pin-only exception in merge-request relayΒ #16607. No policy gate is bypassed and no labels or reviews on that PR were changed. Deployment verification of its live sweep/workflow rate remains for after this fix ships. The anti-flap memory is process-local and requires a known PR head.

β€” hive: backend=pi model=openai/gpt-6.1-sol


🐝 Hive Agent: contributor | SHA: 41f3c4fce

Canonicalize actionable reviewer verdicts independently of the prior stage. Bound repeated label writes with a 15-minute same-head reversal guard, using the actual writer outcome rather than pre-observed legacy store state.

Hive-Run: hivecommons#16880
Hive-Plan: stage-reconciler-idempotence-and-anti-flap
Hive-Spec: hivecommons#16880#recommendation-1-2
Signed-off-by: mendezr <mendezr@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 6, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 6, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@mendezr this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • test (rest 1/4)

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay session received this as a fix-pr task and launched a parallel fix session; your current work continues.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against v5 at merge-base 8b5a160, plus stagelabels/store.go (ObserveProgress), stagelabels/enforce.go (Outcome, applyPlan) and the legacy caller in github/automerge/automerge_stage_labels.go. The change looks correct to me; one low-severity side effect worth knowing about.

Looks correct

  • derive_pr_classify.go:592-604 β€” dropping the prior-070/same-head precondition makes PreserveNormalizedFixingStage a pure function of current evidence, and FixingStage(obs) (line 328) picks 070/071/072 by lane/author kind instead of hard-coding 070. Explicit decisions stay human-owned because prstage.NormalizeReviewVerdict won't return changes_requested for them.
  • reconciler.go:170-176 β€” arming the guard only from out.Removed on a successful, non-gated write is the right source: a hint-no-op write leaves Removed empty (enforce.go applyPlan), so idle sweeps never arm or extend it. suppressStageFlap (writer.go:31-52) releases on new head, 15-min expiry and Terminal, and the test covers all three.

Low: Store EnteredAt/History churn under the legacy pre-observe path
automerge_stage_labels.go:1180 calls store.Observe(key, verdict.Stage, …) before ReconcileBatch. When the engine then suppresses the reversal, reconciler.go:157 calls ObserveProgress(key, last.to, …), so each suppressed sweep flips the Store stage twice. Because ObserveProgress treats any prev.Stage != stage as a stage entry (store.go:722-728, 760), every sweep resets EnteredAt, ReasonChangedAt, EntryCommented/StallCommented, and appends two hops to History. Consequence: for the 15-minute window the dwell/stall clock (store.go:2573-2580 reads EnteredAt) restarts every sweep and the record's history shows an Aβ†’Bβ†’A pair per sweep even though labels never moved. The regression and entry comments are head-keyed (automerge_stage_labels.go:1460-1466, automerge_stage_human.go:646-660) so I don't see a repost risk β€” this is bookkeeping noise, not a wrong label. TestReconcilerSuppressesSameHeadStageReversal asserts rec.Stage but not EnteredAt, so it wouldn't catch it. A cheap fix would be to skip the restore when prev.Stage == last.to already, or to have the legacy sweep stop pre-observing; either is fine as a follow-up.

Open note (by design, not a defect): the guard also delays a legitimate same-head reversal within 15 minutes β€” e.g. a maintainer answering a decision right after the PR entered 085 waits up to one window before 085β†’070 is written. Seems like an acceptable trade for the flap it stops, just worth stating in the changelog or #16880.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Keep unchanged-head flap suppression scoped to unchanged CI, review and adoption evidence. Do not block adoption release, escalation recovery or a check rerun turning green. Ignore prior stage and display outputs so legacy pre-observation still cannot defeat the guard.

Signed-off-by: mendezr <mendezr@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the tree at 41f3c4f, not just the diff. The change looks correct to me; one low-severity trade-off worth knowing about, no blockers.

Normalization fix β€” src/pkg/stageengine/derive_pr_classify.go:592-604: RequiresHumanStage (line 578) already normalizes before deciding 085, so the only way a raw requires_human that normalizes to changes_requested reached 085 was the lane reason path, and the old guard only undid it when a prior 070 on the same head existed β€” i.e. the stage depended on which sweep wrote last. Dropping the history gate and returning FixingStage(obs) makes both the engine path (derive_pr.go:239-252, obs.AuthorKind set at line 238) and the legacy pass (automerge_stage_labels.go:915-934, AuthorKind set at 915) produce the same 07x stage from the same evidence. PreserveStageOnStaleDecisionAnswer keeps its hadPrev gate, so the "no prior" case cannot wrongly park at 085.

Flap guard β€” reconciler.go:153-160 runs before Store.ObserveProgress, and the suppressed path restores last.to. I checked the legacy consumer that worried me: postHumanEntry (automerge_stage_human.go:632) returns early when rec.Stage != route.Stage, so a suppressed 070β†’085 reversal does not post a needs-decision entry comment for a label that was never written. Arming only from stagelabels.Current(out.Removed) (reconciler.go:170-175) correctly ignores first-labelling and gated writes.

Trade-off to be aware of (low) β€” writer.go:44-46 compares stages only, not reasons. If the hive wrote 070β†’085 and within 15 min a trusted human answers the decision on the same head, the derived 070 (reason=decision-answered) is the exact inverse and is suppressed until the window expires; the PR body lists only new head / timeout / terminal as releases, so this is by design, but it delays fixer dispatch by up to 15 min in that one human-driven case. Not a defect; a one-line follow-up could exempt StageReasonDecisionAnswered if that latency matters.

Built ./pkg/stageengine ./pkg/github/automerge and type-checked the new tests locally (the suite itself runs in CI). Changelog fragment is named per changelog.d/README.md.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness delta re-review (read 41f3c4f...0384cb3: writer.go, stage_flap_test.go, plus reconciler.go, derive_pr_classify.go, facts.go, and the legacy caller automerge_stage_labels.go at head 0384cb3).

Previous findings

  1. PreserveNormalizedFixingStage change is unreachable; normalization claimed as the fix β€” still present. derive_pr_classify.go:592-605 is unchanged since 41f3c4f; ClassifyStage and RequiresHumanStage (:578) still normalize before the guard, so :601 cannot see raw=requires_human, normalized=changes_requested. changelog.d/fixed-stage-reconciler-flap.md still leads with "Make review-verdict normalization independent of the previous PR stage". Remediation from the prior review stands (identify the alternating input and test it through Reconcile, or drop the normalization claim from summary/changelog).
  2. Suppressed sweeps re-observe the Store twice per sweep in the legacy path β€” still present (reconciler.go:153-159 unchanged, automerge_stage_labels.go:1180 still pre-observes). Low/advisory; already in backlog.

Delta (writer.go:28-52, 66, 88)
The evidence snapshot is mechanically fine: every field in PRFacts/PRPassFacts/LaneFact is JSON-encodable (stagelabels.CheckSummary, DCOState, VerdictState, Role, MergeCommitState are structs/ints; authorkind.Names is three strings), the copy decouples from caller-owned facts, and the nine release cases in stage_flap_test.go:85-150 exercise both the excluded fields (Prior/Stalled/Lane.Reason/SoakElapsed stay suppressed, :112-115) and the included ones.

One narrowing to be aware of (low): the snapshot keeps Pass.FixerExcluded, which the legacy sweep sets from live dispatcher state each sweep (automerge_stage_labels.go:936-938 β†’ fixDispatchStatusFor, automerge_stage_human.go:763-770). That flag is also the input PreserveStageOnStaleDecisionAnswer keys on (derive_pr_classify.go:614-618), which the prior review named as a candidate driver of the 070↔085 flap. If that is the real driver, a toggling FixerExcluded now releases the guard on exactly the sweeps it was meant to suppress (writer.go:66), and the PR still has no test showing which input alternates on #16607. Not asserting it is the driver β€” I could not verify it from the tree β€” but it is why finding 1 still matters: the guard's coverage now depends on the evidence set, and the oscillating input has not been identified.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Style/maintainability review (read at head 0384cb3, which includes the second commit; the dispatched head was 41f3c4f). No blockers from this perspective β€” the derivation fix and the write-boundary guard match #16880's recommendations 1–2 and stage_flap_test.go covers the three-sweep fixed point. Two advisory points:

  1. stageWriteEvidence silently degrades if PRFacts ever stops being JSON-encodable β€” src/pkg/stageengine/writer.go:48 discards the json.Marshal error behind a comment-only invariant. On an error both snapshots become "", suppressStageFlap (writer.go:64) sees them as equal, and the evidence-release added in 0384cb3 quietly stops working while the guard stays armed. Nothing fails loudly. Suggested fix: treat unknown evidence as "do not arm":

    -	b, _ := json.Marshal(pr) // PRFacts contains only JSON-serializable values.
    -	return string(b)
    +	b, err := json.Marshal(pr)
    +	if err != nil {
    +		return "" // unknown evidence: rememberStageWrite refuses to arm
    +	}
    +	return string(b)
     }
    ...
     func (r *reconciler) rememberStageWrite(f Facts, from, to stagelabels.Stage, reason string) {
    -	if f.PR == nil || f.PR.HeadSHA == "" || from == stagelabels.None || to == stagelabels.None {
    +	evidence := stageWriteEvidence(f)
    +	if f.PR == nil || f.PR.HeadSHA == "" || evidence == "" || from == stagelabels.None || to == stagelabels.None {
     		r.writes.Delete(f.Ref.Key())
     		return
     	}
    -	r.writes.Store(f.Ref.Key(), stageWriteTransition{from: from, to: to, head: f.PR.HeadSHA, reason: reason, evidence: stageWriteEvidence(f), at: r.opts.Now()})
    +	r.writes.Store(f.Ref.Key(), stageWriteTransition{from: from, to: to, head: f.PR.HeadSHA, reason: reason, evidence: evidence, at: r.opts.Now()})
  2. PreserveNormalizedFixingStage no longer preserves anything β€” derive_pr_classify.go:592 now ignores prev/hadPrev (_ params) and always normalizes from current evidence, but the name, the stage_goldens_test.go:91 comment ("the part of the previous Record the post-steps read (PreserveNormalizedFixingStage…)") and PRPassFacts.Prior's doc (facts.go:256, "the normalized-fixing guard reads Stage, Reason and ActivityHead") all still describe the old history-dependent behaviour. Not blocking, but the three call sites (derive_pr.go:250, automerge_stage_labels.go:932, stage_goldens_test.go:265) will mislead the next reader. Either rename to something like NormalizeRequiresHumanToFixing(obs, verdict) or at least update the two stale comments.

Minor: the changelog fragment says reversals are suppressed "for 15 minutes", but writer.go:64 also releases on any evidence change; "same-head, same-evidence stage reversals" would describe the shipped behaviour.

Review findings from ci-maintainer (folded here because this head already carries its hive review):

Intent alignment vs #16880 (read at head 0384cb3): the diff delivers both engine-side remedies the issue asked for β€” a history-independent PreserveNormalizedFixingStage (src/pkg/stageengine/derive_pr_classify.go:585-604, now also used when pass.Prior == nil via derive_pr.go:245-256) and a same-head reversal guard at the reconciler write boundary (reconciler.go:153-160, writer.go:58-88). Workflows untouched, as the issue required. No in-scope blocker. Two advisories:

  1. Store dwell still churns under suppression (legacy sweep). ApplyStageLabels pre-observes its (reversed) opinion at src/pkg/github/automerge/automerge_stage_labels.go:1180 before ReconcileBatch (:1264); the suppression path then re-observes last.to (reconciler.go:157). Each sweep in the window is therefore two Store transitions (085β†’070β†’085), and Store.ObserveProgress resets EnteredAt/ReasonChangedAt and books a regression on every stage change (src/pkg/stagelabels/store.go:722-735). Labels are stable (the issue's done-when), but stall/dwell clocks on the held stage reset every sweep. TestReconcilerSuppressesSameHeadStageReversal asserts rec.Stage only, so this is not covered. Consider asserting EnteredAt stability and skipping the re-observe when the Store already holds last.to, or moving the legacy pre-observe behind the engine.

  2. Residual flap rate vs Native queue policy. The guard releases exactly 15 min after the last successful write (writer.go:66), then the reversal is written and re-arms. If any derivation still oscillates, labels flip once per ~15 min β€” within the 22.5 min evidence wait, so that workflow would still mostly cancel. The window matches the issue's own suggestion, so not a defect; just noting the bound.

Limit of this review: I could not confirm that #16607's own flap goes through the normalized-fixing path (its verdict is an explicit decision request, which NormalizeReviewVerdict leaves as requires_human β€” prstage/types.go:241). If its 070 came from elsewhere, only advisory 2's bounded rate applies there. Tests not run locally.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the diff against the v5 tree at 0384cb3 plus the callers. This looks correct to me; no defects found.

Root cause matches the fix. src/pkg/stageengine/derive_pr_classify.go:592 — the old PreserveNormalizedFixingStage only fired when the prior record was 070 on the same head, so identical requires_human→changes_requested evidence produced 070 or 085 depending on what the last sweep wrote. The new body ignores history and derives the stage from FixingStage(obs) only. At the legacy call site (src/pkg/github/automerge/automerge_stage_labels.go:932) obs.AuthorKind is already populated at line 915, so FixingStageFor picks the kind-appropriate 07x stage rather than falling through to NeedsAuthor. The one-line test update at automerge_stage_labels_test.go:336 (adding Lane: stageLaneSelf) is exactly the adjustment that change requires.

Reversal guard is narrow and correctly placed. src/pkg/stageengine/reconciler.go:153 runs suppressStageFlap under the per-Ref lock, before ObserveProgress, and arms only from the writer's fresh-read out.Removed (reconciler.go:170-174), not from the Store the legacy sweep pre-observes. writer.go:59-82 suppresses only an exact inverse (d.Stage == last.from) on the same head with identical evidence, bypasses on Terminal, and the 15-minute window is anchored to the successful write (last.at is never refreshed on a suppressed sweep). Excluding Lanes[i].Reason, Pass.Prior and SoakElapsed from the evidence hash is intentional and exercised by the test mutating all three (stage_flap_test.go:112-116).

Downstream readers of Outcome.Suppressed. The only non-engine reader is automerge_stage_labels.go:1297, which skips the merge-commit note on a suppressed write β€” on a flap-suppressed sweep the stage did not change, so skipping the note is the right behaviour, not a regression. NoteLabelsSuppressed is not reached on the suppressed path.

Changelog fragment changelog.d/fixed-stage-reconciler-flap.md is present and named per the guard. CI is green on this head.

stage 095-pre-merge-soak β†’ 060-reviewed: untrusted-author-role

Evidence bundle `#168810384cb3` sha256:19ba997b283842e67213fc3f6f2923fc5f732bc6d010c6120ddfb6dd712ee11b β€” download from the dashboard or `hivectl review evidence #16881`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Ready for a human merge: verdict approved @0384cb3, 61/61 checks green, DCO yes, soak 10m0s elapsed. Blocked only by role/untrusted (no trusted-author role).

The hive never approves on your behalf: /lgtm /approve (or merge) if you accept it.

@hivecommons-hive
hivecommons-hive Bot merged commit ad46955 into hivecommons:v5 Nov 6, 2026
68 of 69 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Merged by hive contributor-lane auto-merge: CI green for β‰₯10m, reviewer verdict approve, @mendezr holds the trusted tier in this hive's contributor registry.

@mendezr
mendezr deleted the fix/16880-stage-reconciler branch November 6, 2026 10:44
hivecommons-hive Bot pushed a commit that referenced this pull request Nov 7, 2026
…85 PR to 080 (#17012)

projectbluefin/utah-packages#917 (fork PR, approve verdict, fork-guarded red
required Canary, head 7e52182c0 unchanged) went 085 -> 080 -> 085 about 15
times on 2026-11-07. Each 080 came from a contributor-lane pass refused the
destination-merge lock ("destination merge already in progress", another
sweep was evaluating a PR into the same base). The lane records that as
base-refresh:merge-guard with the error, so the observation is Err, and
ClassifyStage's errored-lane branch ran before the hand-off check that
#16819 added: approve verdict + no CI reason = a weak 080. The 080 seated
the red PR at a merge train's front until the next evaluated pass.

The refusal is now no stage opinion whether or not the lane reports an
error. The reversal guard (#16881) and the dampener (#16980) on v6 tip do
not stop this shape: the replay test fails on tip at the first refusal.

Also corrects two stale derive_pr_classify.go citations in the stage-engine
docs.


Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

1 participant