Repository navigation
✨ writeback: content-aware duplicate and burst suppression at the gate - #15450
Conversation
The write-back gate decided only on policy (repo mode, ACMM level, opt-in label, backfill), so every emitter had to carry its own idempotence and a re-post loop or a fan-out burst was fixed one emitter at a time. The gate now runs a content layer after policy allows a comment kind with a body: - same-item dedupe: normalised body hash per (repo, item, kind), TTL 24h -> reason "duplicate" - cross-item burst: body additionally stripped of #N, @mentions and digits, per (repo, kind); >= 2 distinct items within 60s hold the rest -> reason "burst", one WARN per burst - hourly cap per (repo, kind), default 60, merge-path kinds exempt (writeback.RateCapExemptKinds) -> reason "rate-capped" The table is persisted in writeback.json beside the repo records (legacy bare-map files still load), pruned by TTL and bounded to 5000 entries. Suppressions land in Recent()/the drill-down; /api/status writeback.dedupe carries settings plus last-hour counters. Config under writeback.dedupe (enabled, same_item_ttl, burst_items, burst_window, hourly_cap). The supersession sweep's pre-check becomes policy-only and the real rendered notice is gated before ensureSupersessionComment; the close-first reopen path treats content reasons as retry-later rather than a policy change. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Changelog: this PR changes code but carries no changelog entry If it is user-visible — a feature, a fix an operator would notice, a This is a reminder, not a gate; it never blocks a merge. |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
The layering and persistence look right (legacy bare-map load at gate.go:240-255, prune/bound, exempt sets), and the body's description matches the diff. Two things I verified in the tree that I think need a change before this lands — both come from the same root.
1. The body is recorded as "written" when the gate is asked, not when the post succeeds (high)
dedupe.go:267-270 writes the same-item / burst / hourly entries and saveLocked()s inside contentCheckLocked, i.e. inside AllowedPreview (gate.go:373-376). Every caller then goes on to make the GitHub call, which can fail:
sentinel_sweep.go:440asks with the realSentinelComment(...), thenCreateCommentatsentinel_sweep.go:505; on error itreturns (:509) with nothing posted. Next tick the gate answersduplicatefor 24h (dedupe.go:228), the sweep setsevent.Suppressed = trueand returns at:441-443— before theensure labelcall further down. So one transient 5xx/secondary-rate-limit on the comment leaves asentinel-alertfinding with no comment and no label forsame_item_ttl. That is a trust-surface alert going dark for a day because of a network blip.pr_supersession_sweep.go:298asks withbody, thenensureSupersessionCommentat:302; on error it returns (:304). Next sweep:duplicate→releaseStaleSupersessionPark(:299), and the notice is never posted until the TTL lapses.pr_reporter_trust.go:375andpr_duplicate_fix_sweep.go:311have the same shape.
Nothing in dedupe_test.go exercises "allowed, then the write failed" — the tests go straight from AllowedPreview to the next AllowedPreview. The body says "Holds are suppressions (callers already handle not-allowed)", which is true for policy reasons but not for a hold the gate created by itself after a failed write. A Gate.Recorded(repo, item, kind, body) (or Commit) that callers invoke after the write returns nil, with AllowedPreview only reading the tables, would fix all four sites; alternatively the record could be limited to the burst/hourly tables and the same-item entry written post-hoc.
2. Kinds that re-check every pass with a constant body now emit a duplicate suppression on every pass (medium)
dedupe.go:72-75 says kinds that "re-run their gate check every tick with a constant preview and keep their own idempotence" are deliberately excluded. But supersession_notice is in ContentDedupeKinds (dedupe.go:79) and does exactly that: the sweep renders the same body each pass and relies on ensureSupersessionComment returning changed=false (pr_supersession_sweep.go:306-310). After this PR, pass 2+ instead hits duplicate, and suppressLocked (gate.go:546-548) appends a row to recent unconditionally — so every open superseded contributor PR adds one "Suppressed write-backs (last 24h)" entry per sweep, and /api/status → writeback.dedupe.dedupe_same_item counts them as suppressions even though nothing was lost. sentinel is the same once a hub restart empties its in-memory heads map: the persisted dedupe table now answers duplicate, so the first pass reports the existing alert as Suppressed/duplicate in /api/sentinel and skips the cleared-head re-label logic at sentinel_sweep.go:450-470 that the marker comment was designed to drive.
Either dropping supersession_notice/sentinel from ContentDedupeKinds (they already have marker-level idempotence), or having callers treat IsContentReason as "already stands, not suppressed" (only automerge_adoption_close_first.go:273 does today), would keep the operator panel honest.
Smaller note, not blocking: json:"dedupe,omitzero" on Config.Dedupe (config.go:143) needs Go ≥ 1.24 to have any effect; on older toolchains it is silently ignored, which is harmless here but worth knowing when reading /api/config output.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #15448
Why
pkg/writeback.Gateis the one choke point for every unsolicited GitHub write, but it only decided on policy (repo mode, ACMM level, opt-in label, backfill). Duplicates were fixed one emitter at a time. Today at 17:20Z the hive posted this identical comment on kubestellar/kubestellar#3925 and #3928, 6 s apart:Earlier: selfreport re-post loop on #15021, repeated stall comments, repeated supersede-failed comments. The emitter in today's burst is being fixed separately; this PR makes the gate stop the whole class.
Design
The content layer runs after the policy gate allows a write, and only for comment kinds whose body the caller passed as
preview(writeback.ContentDedupeKinds:sentinel,needs_decision_comment,supersession_notice,duplicate_fix_notice,reporter_trust_hold,contributor_adopt,claim_comment,gate_comment,welcome_comment).Allowed()without a preview — labels, closes, hold-gate upkeep (re-checked every tick with a constant preview),parent_issue_edit,ci_rerun,issue_file— stays policy-only.(repo, item, kind, sha256₁₆(body))— body with<!-- hive:… -->marker args (marker name kept), 7–40-hex SHAs, RFC 3339 timestamps, durations (30m0s,3h) and whitespace normalisedduplicate— "identical<kind>already written to #N at<time>"same_item_ttl: 24h(repo, kind, hash)— additionally strips#N/owner/repo#N,@login, digitsburst— "same<kind>already sent to #A within 60s; held to avoid a notification burst"; one WARN per burst with repo/kind/countburst_items: 2,burst_window: 60s(repo, kind)allowed writes in a sliding hourrate-cappedhourly_cap: 60(-1disables)suppressLocked→Recent()//api/writeback/suppressed/ the Write-back containment drill-down, whose legend now explains the three content reasons./api/status→writeback.dedupecarries{enabled, dedupe_same_item, dedupe_burst, rate_capped, hourly_cap, burst_items, burst_window_s, same_item_ttl_s}(counters are last-hour).writeback.jsonbeside the repo records ({"repos":…, "dedupe":…}; legacy bare-map files still load), pruned by TTL and bounded to 5000 same-item entries.writeback.dedupe.{enabled, same_item_ttl, burst_items, burst_window, hourly_cap}with validation; documented inoperator-reference.md§ "Duplicate-write suppression",hive.yaml.example, and the/api/statusrow ofdocs/api-reference/health-status-events.md.Hourly-cap exempt kinds (
writeback.RateCapExemptKinds, tested)Merge path:
verdict_label,needs_decision_comment,contributor_adopt,hold_gate_notice,ci_rerun,issue_close,supersession_close. Plus every label kind (stage_label,covered_by_pr_label,verdict_label) andissue_file(no existing item).Call-site changes
pr_supersession_sweep.go: the pre-check (before file fetches, admitting several later actions) becomes policy-onlyAllowed(); a content-awareWritebackAllowed(…, body)now runs on the real rendered notice right beforeensureSupersessionComment, a refusal taking the same path as "notice already stands". This was the one site passing a synthetic/empty preview.automerge_adoption_close_first.go: the reopen path treatsduplicate/burst/rate-cappedlike a pause (info, retry next poll) viawriteback.IsContentReason, not as "policy changed".Example — today's burst, through the new gate
Tests (
pkg/writeback/dedupe_test.go, fake clock viaDeps.Now, no sleep)Same body twice → second suppressed; marker args / SHA / timestamp / duration / whitespace variants still dedupe; different body passes; TTL expiry; burst of 3 → 1 allowed, 2 held, exactly one WARN, window expiry,
burst_items: 3threshold; hourly cap with per-repo budgets and sliding window; exempt kinds never capped; kind-set membership; no-preview and non-comment kinds untouched;enabled: false; persistence acrossNew()reload and legacy file load; table bound; config validation and defaults;/api/statuscounters age out after 1h while the drill-down keeps 24h.pkg/writebackcoverage 96.9% → 97.3%;TestDirectWriteRatchetbaseline unchanged (no new direct writes).static/index.htmlregenerated.