From f42e793ef9b62a18a969f1e9062d08cb6ec9b02c Mon Sep 17 00:00:00 2001 From: Andrew Anderson Date: Thu, 29 Oct 2026 13:50:51 -0400 Subject: [PATCH 1/2] =?UTF-8?q?=E2=9C=A8=20writeback:=20content-aware=20du?= =?UTF-8?q?plicate=20and=20burst=20suppression=20at=20the=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The write-back gate decided only on policy (repo mode, ACMM level, opt-in label, backfill), so every emitter had to carry its own idempotence and a re-post loop or a fan-out burst was fixed one emitter at a time. The gate now runs a content layer after policy allows a comment kind with a body: - same-item dedupe: normalised body hash per (repo, item, kind), TTL 24h -> reason "duplicate" - cross-item burst: body additionally stripped of #N, @mentions and digits, per (repo, kind); >= 2 distinct items within 60s hold the rest -> reason "burst", one WARN per burst - hourly cap per (repo, kind), default 60, merge-path kinds exempt (writeback.RateCapExemptKinds) -> reason "rate-capped" The table is persisted in writeback.json beside the repo records (legacy bare-map files still load), pruned by TTL and bounded to 5000 entries. Suppressions land in Recent()/the drill-down; /api/status writeback.dedupe carries settings plus last-hour counters. Config under writeback.dedupe (enabled, same_item_ttl, burst_items, burst_window, hourly_cap). The supersession sweep's pre-check becomes policy-only and the real rendered notice is gated before ensureSupersessionComment; the close-first reopen path treats content reasons as retry-later rather than a policy change. Signed-off-by: Andrew Anderson --- .../added-writeback-content-dedupe.md | 1 + .../api-reference/health-status-events.md | 2 +- src/docs/operator-reference.md | 35 ++ src/hive.yaml.example | 6 + .../static-src/js/config-sentinel.js | 1 + .../dashboard/static-src/js/config-state.js | 2 +- src/pkg/dashboard/static/index.html | 3 +- .../automerge_adoption_close_first.go | 2 +- src/pkg/github/pr_supersession_sweep.go | 12 +- src/pkg/writeback/config.go | 71 ++++ src/pkg/writeback/dedupe.go | 368 +++++++++++++++++ src/pkg/writeback/dedupe_test.go | 391 ++++++++++++++++++ src/pkg/writeback/gate.go | 38 +- 13 files changed, 925 insertions(+), 7 deletions(-) create mode 100644 src/changelog.d/added-writeback-content-dedupe.md create mode 100644 src/pkg/writeback/dedupe.go create mode 100644 src/pkg/writeback/dedupe_test.go diff --git a/src/changelog.d/added-writeback-content-dedupe.md b/src/changelog.d/added-writeback-content-dedupe.md new file mode 100644 index 0000000000..a41f441053 --- /dev/null +++ b/src/changelog.d/added-writeback-content-dedupe.md @@ -0,0 +1 @@ +- The write-back gate is content-aware: an identical comment on the same item within 24h, the same notice fanning out to several items within 60s, or more than 60 comments of one kind per repo per hour is suppressed (`writeback.dedupe`), persisted across restarts, counted on `/api/status` and shown in the Write-back containment drill-down as `duplicate`, `burst` or `rate-capped`. diff --git a/src/docs/api-reference/health-status-events.md b/src/docs/api-reference/health-status-events.md index 8b8043df8c..00d2854b6b 100644 --- a/src/docs/api-reference/health-status-events.md +++ b/src/docs/api-reference/health-status-events.md @@ -12,7 +12,7 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `GET` | `/api/health/deep` | Public | Deep health probe | `pkg/dashboard/server.go:1602` | | `GET` | `/api/livez` | Public | Kubernetes liveness probe | `pkg/dashboard/server.go:1603` | | `GET` | `/metrics` | Registered only when `HIVE_METRICS_ENABLED`; requires `Authorization: ****** (403 if the token is unset) | Prometheus metrics | `pkg/dashboard/server.go:1609` | -| `GET` | `/api/status` | Dashboard auth/session | Dashboard aggregate status; `labelPresets[]` (`{key,label,description,labels[],states[]}`) lists the built-in label-filter presets for the Projects cards, led by `needs-human` whose `labels[]` are the effective hard-suppress labels plus `needs-signal`, the hold labels, `stage/071-needs-author` and the reporter-trust awaiting label (#12172); accepts `?fields=a,b` for top-level selection and `?omit=repos` to drop heavy sections, and honors `Accept-Encoding: gzip`; `agents[]` includes busy-pane visibility fields (`kicksUndeliverable`, `busySince`, `lastTranscriptActivity`, `condition`, `conditionMessage`) when a Working pane cannot accept kicks; a working agent also carries `agents[].run` — the live per-run line (#14576): `{since, elapsedS, costUSD, tokens, priced, tests, filesChanged}`, the estimated spend/tokens since the kick was delivered (`lastKickAt`, falling back to `busySince`) (delta of the token collector's per-agent cumulative figure; `priced:false` when the agent has none yet), the last visible test-runner result (`pass`/`fail`/empty) and the last `N files changed` git summary (`-1` when none) parsed from the captured pane; top-level `fleetRun` `{running, hourUSD, hourPriced}` is the governor strip's `running N · $X this hour` figure; `sla` (fleet) and `repos[].sla` (per repo) carry the Delivery SLA block derived from the lifecycle timeline with no extra GitHub calls: `windows.{7d,30d}.{open_to_triage,triage_to_start,start_to_pr,pr_to_merge,open_to_merge}` as `{p50_h,p95_h,n}` (hours, nearest-rank), `history[]` of up to 30 daily buckets `{day,start_to_pr_p50,pr_to_merge_p50,open_to_merge_p50,n_started,n_pr,n_merged}` persisted under `/data/sla-history.json`, `stale_in_progress[]` (`{number,repo,started_at,age_h,agent}` — issues carrying `hive/in-progress` longer than the agent claim TTL with no linked PR), `stale_count`, `merged_7d`, `claim_ttl_s` and, on the fleet block, the effective `targets` (`start_to_pr_h`, `pr_to_merge_h`, `open_to_merge_h`); `staleSince` and `repos[].countsStale` / `countsStaleSince` mark totals served from last-good figures — the previous enumeration (#13382) or the ForgeState cache's last-good counts (#13376); `ci_failure_signals` `{count, last, kicks, unmatched}` is the webhook CI-failure signal store — distinct red (repo, head, check) signals retained, the most recent one `{repo, number, head_sha, check, conclusion, details_url, agent, at}`, webhook-triggered kicks/nudges since boot, and failure deliveries whose head matched no open PR; `hotspots` `{count, actionable, filed}` is the merge-hotspot advisor's one-line summary (files at/above the thresholds, those with a split recommendation, those with a modularize issue filed), absent when the advisor is not wired — details on `GET /api/hotspots` | `pkg/dashboard/server.go:1614` | +| `GET` | `/api/status` | Dashboard auth/session | Dashboard aggregate status; `labelPresets[]` (`{key,label,description,labels[],states[]}`) lists the built-in label-filter presets for the Projects cards, led by `needs-human` whose `labels[]` are the effective hard-suppress labels plus `needs-signal`, the hold labels, `stage/071-needs-author` and the reporter-trust awaiting label (#12172); accepts `?fields=a,b` for top-level selection and `?omit=repos` to drop heavy sections, and honors `Accept-Encoding: gzip`; `agents[]` includes busy-pane visibility fields (`kicksUndeliverable`, `busySince`, `lastTranscriptActivity`, `condition`, `conditionMessage`) when a Working pane cannot accept kicks; a working agent also carries `agents[].run` — the live per-run line (#14576): `{since, elapsedS, costUSD, tokens, priced, tests, filesChanged}`, the estimated spend/tokens since the kick was delivered (`lastKickAt`, falling back to `busySince`) (delta of the token collector's per-agent cumulative figure; `priced:false` when the agent has none yet), the last visible test-runner result (`pass`/`fail`/empty) and the last `N files changed` git summary (`-1` when none) parsed from the captured pane; top-level `fleetRun` `{running, hourUSD, hourPriced}` is the governor strip's `running N · $X this hour` figure; `sla` (fleet) and `repos[].sla` (per repo) carry the Delivery SLA block derived from the lifecycle timeline with no extra GitHub calls: `windows.{7d,30d}.{open_to_triage,triage_to_start,start_to_pr,pr_to_merge,open_to_merge}` as `{p50_h,p95_h,n}` (hours, nearest-rank), `history[]` of up to 30 daily buckets `{day,start_to_pr_p50,pr_to_merge_p50,open_to_merge_p50,n_started,n_pr,n_merged}` persisted under `/data/sla-history.json`, `stale_in_progress[]` (`{number,repo,started_at,age_h,agent}` — issues carrying `hive/in-progress` longer than the agent claim TTL with no linked PR), `stale_count`, `merged_7d`, `claim_ttl_s` and, on the fleet block, the effective `targets` (`start_to_pr_h`, `pr_to_merge_h`, `open_to_merge_h`); `staleSince` and `repos[].countsStale` / `countsStaleSince` mark totals served from last-good figures — the previous enumeration (#13382) or the ForgeState cache's last-good counts (#13376); `ci_failure_signals` `{count, last, kicks, unmatched}` is the webhook CI-failure signal store — distinct red (repo, head, check) signals retained, the most recent one `{repo, number, head_sha, check, conclusion, details_url, agent, at}`, webhook-triggered kicks/nudges since boot, and failure deliveries whose head matched no open PR; `hotspots` `{count, actionable, filed}` is the merge-hotspot advisor's one-line summary (files at/above the thresholds, those with a split recommendation, those with a modularize issue filed), absent when the advisor is not wired — details on `GET /api/hotspots`; `writeback.dedupe` `{enabled, dedupe_same_item, dedupe_burst, rate_capped, hourly_cap, burst_items, burst_window_s, same_item_ttl_s}` is the write-back gate's content layer — the effective `writeback.dedupe.*` settings and the last-hour counts of writes held as `duplicate`, `burst` and `rate-capped` (each also listed under `GET /api/writeback/suppressed`) | `pkg/dashboard/server.go:1614` | | `GET` | `/api/status/summary` | Dashboard auth/session | Compact agent/governor/budget status summary for pollers; honors `Accept-Encoding: gzip` | `pkg/dashboard/server.go:1615` | | `GET` | `/api/overview/issues.csv` | Dashboard auth/session | Cached Overview issue-band export as browser-compatible CSV; filters `band`, `repo`/`repos`, `stale`, and `held` are ANDed. | `pkg/dashboard/server.go:1616` | | `GET` | `/api/overview/issues.json` | Dashboard auth/session | Cached Overview issue-band export as typed JSON with `bands[]` metadata and row fields matching the CSV columns. | `pkg/dashboard/server.go:1617` | diff --git a/src/docs/operator-reference.md b/src/docs/operator-reference.md index d4c7758a47..ea736ad788 100644 --- a/src/docs/operator-reference.md +++ b/src/docs/operator-reference.md @@ -1054,6 +1054,12 @@ writeback: torch-spyre/torch-spyre: { mode: issues-only } # off | issues-only | labels-only | full kinds: sentinel: { min_level: 5, require_label: true, unlabeled_min_level: 6 } + dedupe: # content-aware suppression, see below + enabled: true + same_item_ttl: 24h + burst_items: 2 + burst_window: 60s + hourly_cap: 60 ``` Everything is editable under **⚙ → Features → Autonomy & ACMM → Write-back @@ -1072,6 +1078,35 @@ one line — `writeback: skipped N pre-existing items on (backfill off)` — and the dashboard shows "hive can now write to ``; N existing PRs were left untouched — enable backfill to process them". +### Duplicate-write suppression + +Policy decides whether a kind may write to an item; the content layer under +`writeback.dedupe` decides whether *this* write is worth making. It applies +only to comment kinds whose body the caller handed the gate as the preview +(`sentinel`, `needs_decision_comment`, `supersession_notice`, +`duplicate_fix_notice`, `reporter_trust_hold`, `contributor_adopt`, +`claim_comment`, `gate_comment`, `welcome_comment`); `Allowed()` without a +preview — labels, closes, hold-gate upkeep, task-list edits — stays +policy-only. Three checks run after the policy gate allows a write: + +| Check | Key | Suppression reason | Default | +|---|---|---|---| +| Same item | `(repo, item, kind, hash)` of the body with `` marker arguments, SHAs, RFC 3339 timestamps, durations and whitespace normalised away | `duplicate` — "identical `` already written to #N at `