Skip to content

✨ writeback: content-aware duplicate and burst suppression at the gate - #15450

Merged
clubanderson merged 2 commits into
v5from
feat/writeback-content-dedupe
Oct 29, 2026
Merged

clubanderson merged 2 commits into
v5from
feat/writeback-content-dedupe

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #15448

Why

pkg/writeback.Gate is the one choke point for every unsolicited GitHub write, but it only decided on policy (repo mode, ACMM level, opt-in label, backfill). Duplicates were fixed one emitter at a time. Today at 17:20Z the hive posted this identical comment on kubestellar/kubestellar#3925 and #3928, 6 s apart:

🐝 This PR was auto-promoted for contributor-lane merging …

Earlier: selfreport re-post loop on #15021, repeated stall comments, repeated supersede-failed comments. The emitter in today's burst is being fixed separately; this PR makes the gate stop the whole class.

Design

The content layer runs after the policy gate allows a write, and only for comment kinds whose body the caller passed as preview (writeback.ContentDedupeKinds: sentinel, needs_decision_comment, supersession_notice, duplicate_fix_notice, reporter_trust_hold, contributor_adopt, claim_comment, gate_comment, welcome_comment). Allowed() without a preview — labels, closes, hold-gate upkeep (re-checked every tick with a constant preview), parent_issue_edit, ci_rerun, issue_file — stays policy-only.

Check Key Reason Default
Same item (repo, item, kind, sha256₁₆(body)) — body with <!-- hive:… --> marker args (marker name kept), 7–40-hex SHAs, RFC 3339 timestamps, durations (30m0s, 3h) and whitespace normalised duplicate — "identical <kind> already written to #N at <time>" same_item_ttl: 24h
Burst across items (repo, kind, hash) — additionally strips #N / owner/repo#N, @login, digits burst — "same <kind> already sent to #A within 60s; held to avoid a notification burst"; one WARN per burst with repo/kind/count burst_items: 2, burst_window: 60s
Hourly cap (repo, kind) allowed writes in a sliding hour rate-capped hourly_cap: 60 (-1 disables)
  • Holds are suppressions (callers already handle not-allowed); nothing is queued or replayed.
  • Suppressions go through suppressLocked → Recent() / /api/writeback/suppressed / the Write-back containment drill-down, whose legend now explains the three content reasons.
  • /api/status → writeback.dedupe carries {enabled, dedupe_same_item, dedupe_burst, rate_capped, hourly_cap, burst_items, burst_window_s, same_item_ttl_s} (counters are last-hour).
  • The table is persisted in writeback.json beside the repo records ({"repos":…, "dedupe":…}; legacy bare-map files still load), pruned by TTL and bounded to 5000 same-item entries.
  • Config: writeback.dedupe.{enabled, same_item_ttl, burst_items, burst_window, hourly_cap} with validation; documented in operator-reference.md § "Duplicate-write suppression", hive.yaml.example, and the /api/status row of docs/api-reference/health-status-events.md.

Hourly-cap exempt kinds (writeback.RateCapExemptKinds, tested)

Merge path: verdict_label, needs_decision_comment, contributor_adopt, hold_gate_notice, ci_rerun, issue_close, supersession_close. Plus every label kind (stage_label, covered_by_pr_label, verdict_label) and issue_file (no existing item).

Call-site changes

  • pr_supersession_sweep.go: the pre-check (before file fetches, admitting several later actions) becomes policy-only Allowed(); a content-aware WritebackAllowed(…, body) now runs on the real rendered notice right before ensureSupersessionComment, a refusal taking the same path as "notice already stands". This was the one site passing a synthetic/empty preview.
  • automerge_adoption_close_first.go: the reopen path treats duplicate/burst/rate-capped like a pause (info, retry next poll) via writeback.IsContentReason, not as "policy changed".

Example — today's burst, through the new gate

17:20:00 contributor_adopt kubestellar/kubestellar#3925  allowed (burst key recorded)
17:20:06 contributor_adopt kubestellar/kubestellar#3928  burst: same contributor_adopt already sent to #3925 within 1m0s; held to avoid a notification burst
WARN writeback: notification burst held repo=kubestellar/kubestellar kind=contributor_adopt count=2 first=3925 window=1m0s

Tests (pkg/writeback/dedupe_test.go, fake clock via Deps.Now, no sleep)

Same body twice → second suppressed; marker args / SHA / timestamp / duration / whitespace variants still dedupe; different body passes; TTL expiry; burst of 3 → 1 allowed, 2 held, exactly one WARN, window expiry, burst_items: 3 threshold; hourly cap with per-repo budgets and sliding window; exempt kinds never capped; kind-set membership; no-preview and non-comment kinds untouched; enabled: false; persistence across New() reload and legacy file load; table bound; config validation and defaults; /api/status counters age out after 1h while the drill-down keeps 24h.

pkg/writeback coverage 96.9% → 97.3%; TestDirectWriteRatchet baseline unchanged (no new direct writes). static/index.html regenerated.

The write-back gate decided only on policy (repo mode, ACMM level, opt-in
label, backfill), so every emitter had to carry its own idempotence and a
re-post loop or a fan-out burst was fixed one emitter at a time. The gate
now runs a content layer after policy allows a comment kind with a body:

- same-item dedupe: normalised body hash per (repo, item, kind), TTL 24h
  -> reason "duplicate"
- cross-item burst: body additionally stripped of #N, @mentions and
  digits, per (repo, kind); >= 2 distinct items within 60s hold the rest
  -> reason "burst", one WARN per burst
- hourly cap per (repo, kind), default 60, merge-path kinds exempt
  (writeback.RateCapExemptKinds) -> reason "rate-capped"

The table is persisted in writeback.json beside the repo records (legacy
bare-map files still load), pruned by TTL and bounded to 5000 entries.
Suppressions land in Recent()/the drill-down; /api/status writeback.dedupe
carries settings plus last-hour counters. Config under writeback.dedupe
(enabled, same_item_ttl, burst_items, burst_window, hourly_cap).

The supersession sweep's pre-check becomes policy-only and the real
rendered notice is gated before ensureSupersessionComment; the close-first
reopen path treats content reasons as retry-later rather than a policy
change.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 29, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Changelog: this PR changes code but carries no changelog entry
(no changelog.d/ fragment and no CHANGELOG.md edit).

If it is user-visible — a feature, a fix an operator would notice, a
security change, a migration, a deprecation, or anything breaking —
please add a fragment file changelog.d/<category>-<pr-or-slug>.md
containing exactly your entry (see changelog.d/README.md; the
category is one of added/changed/deprecated/fixed/security). Please
do not append to CHANGELOG.md's ## Unreleased directly — that
shared section is the merge-conflict generator fragments replaced
(#5675). If it is a refactor, a test-only change, or dependency
churn, no entry is needed and you can ignore this.

This is a reminder, not a gate; it never blocks a merge.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15400 🐛 fix: keep merge queue position visible during sweeps same files src/pkg/dashboard/static/index.html
#15425 🐛 fix: identify tracker claims from the local ledger same files src/pkg/dashboard/static/index.html
#15405 🐛 fix: require verified signature for email-only hive commit identity match same files src/docs/operator-reference.md
#15444 🐛 tracker: one-line PR sub-row header same files src/pkg/dashboard/static/index.html
#15449 🐛 tracker: merge queue ordinals, one column, readable train icon, no conflicting riders same files src/docs/operator-reference.md
src/pkg/dashboard/static/index.html

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The layering and persistence look right (legacy bare-map load at gate.go:240-255, prune/bound, exempt sets), and the body's description matches the diff. Two things I verified in the tree that I think need a change before this lands — both come from the same root.

1. The body is recorded as "written" when the gate is asked, not when the post succeeds (high)

dedupe.go:267-270 writes the same-item / burst / hourly entries and saveLocked()s inside contentCheckLocked, i.e. inside AllowedPreview (gate.go:373-376). Every caller then goes on to make the GitHub call, which can fail:

  • sentinel_sweep.go:440 asks with the real SentinelComment(...), then CreateComment at sentinel_sweep.go:505; on error it returns (:509) with nothing posted. Next tick the gate answers duplicate for 24h (dedupe.go:228), the sweep sets event.Suppressed = true and returns at :441-443 — before the ensure label call further down. So one transient 5xx/secondary-rate-limit on the comment leaves a sentinel-alert finding with no comment and no label for same_item_ttl. That is a trust-surface alert going dark for a day because of a network blip.
  • pr_supersession_sweep.go:298 asks with body, then ensureSupersessionComment at :302; on error it returns (:304). Next sweep: duplicate → releaseStaleSupersessionPark (:299), and the notice is never posted until the TTL lapses.
  • pr_reporter_trust.go:375 and pr_duplicate_fix_sweep.go:311 have the same shape.

Nothing in dedupe_test.go exercises "allowed, then the write failed" — the tests go straight from AllowedPreview to the next AllowedPreview. The body says "Holds are suppressions (callers already handle not-allowed)", which is true for policy reasons but not for a hold the gate created by itself after a failed write. A Gate.Recorded(repo, item, kind, body) (or Commit) that callers invoke after the write returns nil, with AllowedPreview only reading the tables, would fix all four sites; alternatively the record could be limited to the burst/hourly tables and the same-item entry written post-hoc.

2. Kinds that re-check every pass with a constant body now emit a duplicate suppression on every pass (medium)

dedupe.go:72-75 says kinds that "re-run their gate check every tick with a constant preview and keep their own idempotence" are deliberately excluded. But supersession_notice is in ContentDedupeKinds (dedupe.go:79) and does exactly that: the sweep renders the same body each pass and relies on ensureSupersessionComment returning changed=false (pr_supersession_sweep.go:306-310). After this PR, pass 2+ instead hits duplicate, and suppressLocked (gate.go:546-548) appends a row to recent unconditionally — so every open superseded contributor PR adds one "Suppressed write-backs (last 24h)" entry per sweep, and /api/status → writeback.dedupe.dedupe_same_item counts them as suppressions even though nothing was lost. sentinel is the same once a hub restart empties its in-memory heads map: the persisted dedupe table now answers duplicate, so the first pass reports the existing alert as Suppressed/duplicate in /api/sentinel and skips the cleared-head re-label logic at sentinel_sweep.go:450-470 that the marker comment was designed to drive.

Either dropping supersession_notice/sentinel from ContentDedupeKinds (they already have marker-level idempotence), or having callers treat IsContentReason as "already stands, not suppressed" (only automerge_adoption_close_first.go:273 does today), would keep the operator panel honest.

Smaller note, not blocking: json:"dedupe,omitzero" on Config.Dedupe (config.go:143) needs Go ≥ 1.24 to have any effect; on older toolchains it is silently ignored, which is harmless here but worth knowing when reading /api/config output.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson
clubanderson merged commit 98d5724 into v5 Oct 29, 2026
75 of 76 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/writeback-content-dedupe branch October 29, 2026 18:16
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Writeback gate should suppress duplicate and burst comments dynamically

1 participant