Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/added-16264-notification-shadow-mode.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Added `github.notifications.shadow` (default off): would-notify instrumentation that classifies every notification-capable GitHub write by producer and class, counts repeats it would suppress, exports the counters on `/metrics`, and documents the write-site inventory — measurement only, no write changes (#16264).
3 changes: 3 additions & 0 deletions src/cmd/hive/main_sweeps.go
Original file line number Diff line number Diff line change
Expand Up @@ -1983,6 +1983,9 @@ func installReviewRelaySettings(client *github.Client, cfg *config.Config, logge
func() int { return cfg.Review.MaxReviewsPerHead },
)
client.SetNotificationsQuiet(func() bool { return cfg.GitHub.Notifications.Quiet() })
// github.notifications.shadow (hivecommons/hive#16264): would-notify
// instrumentation only, read live; nothing is suppressed.
client.SetNotificationsShadow(func() bool { return cfg.GitHub.Notifications.Shadow })
// review.require_labels (PR approval allow-list): the relay's half of the
// gate, read live so a Features-tab edit applies to the next request.
client.SetReviewRequireLabels(func() []string { return cfg.Review.RequireLabels })
Expand Down
186 changes: 186 additions & 0 deletions src/docs/notifications.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,3 +253,189 @@ notifications:
```

The bot needs the `Create Public Threads` and `Send Messages in Threads` permissions, plus `Read Message History` for reconciliation.

## GitHub writes: inventory and shadow mode (`github.notifications.shadow`)

Every new issue/PR comment, review submission, review request and @-mention
the hive makes on GitHub emails the people involved and everyone watching the
repository. [`github.notifications.mode`](operator-reference/sweeps-and-operations.md#notification-mode-githubnotificationsmode)
already folds the hive's own bookkeeping into one edited status comment;
shadow mode measures what is left so a later change can be judged against a
baseline ([#16075](https://github.com/hivecommons/hive/issues/16075)).

```yaml
github:
notifications:
shadow: true # default false
```

Shadow mode is **instrumentation only**: nothing is suppressed, delayed or
rerouted, and the GitHub writes are byte-for-byte the same with the flag on
or off. With the flag on, each notification-capable write the Go client makes
is attributed to its producer (the inventoried function below), classified,
and given a *would-notify* decision:

- **Record** — `{repo, item, producer, class, condition key, would_notify,
reason}` plus the logins the write mentions or requests. Comment bodies,
other recipients and inbox contents are never kept. The last 512 records
stay in memory (`Client.NotificationShadowRecords`).
- **Condition key** (provisional) — `(repo, item, class, subject)`, where the
subject is what the writer supplies (the status-comment section key for
`PostNoteOrFold`) or the producer name. A later change defines the
authoritative, producer-independent key.
- **Decision** — a write whose condition key was already written within 24
hours is `would_notify=false`, reason `repeat_condition`, **only** for the
`status` and `attention` classes. `protocol_command`, `required_review` and
`security_event` writes are always `would_notify=true` (a repeat gets reason
`exempt_class_repeat`). A failed write is `delivery_failed`.

### Counters

The hourly `github notifications` log line gains `shadow_would_suppress`,
`shadow_reviews`, `shadow_review_requests`, `shadow_mentions`,
`shadow_delivery_failures` and `shadow_active_items` while shadow mode is on
(the line is unchanged while it is off). The opt-in Prometheus endpoint
(`HIVE_METRICS_ENABLED` + `HIVE_METRICS_TOKEN`) exports:

| Series | Labels | Meaning |
|---|---|---|
| `hive_github_comments_total` | `op` = `created` / `edited` / `suppressed` | The always-on comment tally. |
| `hive_github_notification_comments_created_total` | `producer`, `class` | New comments. |
| `hive_github_notification_reviews_submitted_total` | `producer`, `class` | Review submissions. |
| `hive_github_notification_review_requests_total` | `producer`, `class` | Review requests. |
| `hive_github_notification_mentions_total` | `producer`, `class` | @-mentions in posted comments and reviews. |
| `hive_github_notification_suppressed_repeats_total` | `producer`, `class` | Notes the hive actually did not post (folded or already present). |
| `hive_github_notification_would_suppress_repeats_total` | `producer`, `class` | Writes shadow mode *would* suppress. |
| `hive_github_notification_delivery_failures_total` | `producer`, `class` | Failed writes. |
| `hive_github_notification_active_items` | — | Items with a notification-capable write in the last 24 hours. |

All counts are API writes, a proxy for notifications — never "emails
delivered". Counters are cumulative since process start.

### Comparing baseline windows

Compare **per active item**, over **matched windows**: the same length (a
whole week, so weekday traffic is matched), similar open-item load, and no
change to `github.notifications.mode` or the hive version inside either
window. For each window compute

```
writes = increase(hive_github_notification_comments_created_total[7d])
+ increase(hive_github_notification_reviews_submitted_total[7d])
+ increase(hive_github_notification_review_requests_total[7d])
per_item = writes / avg_over_time(hive_github_notification_active_items[7d])
repeat_rate = increase(hive_github_notification_would_suppress_repeats_total[7d])
/ avg_over_time(hive_github_notification_active_items[7d])
```

broken down by `class` and `producer`. The baseline is a shadow-on window
before any suppression ships; the candidate is the same metrics after it. The
80% reduction in repeat/non-actionable writes per active item named in #16075
is an **acceptance target**, not a measured or claimed result: shadow mode
only produces the baseline that target will be checked against.

### Write-site inventory

Every function under `src/pkg` and `src/cmd` that calls a notification-capable
write primitive (`CreateIssueComment`, `CreateIssueCommentBudgeted`,
`PostNoteOrFold`, `PostClaimNote`, `PullRequests.CreateReview`,
`RequestReviewers`) or records a suppressed note is listed here and in
`notificationWriteSites` (`src/pkg/github/notification_write_sites.go`).
`TestNotificationWriteSitesInventory` fails when a new call site is added
without a class. Review submissions and review requests are always counted as
`required_review`, whichever producer made them.

- `status` — bookkeeping readable on demand; repeats may be would-suppress.
- `attention` — asks a person to look or act; repeats may be would-suppress.
- `required_review` — a review or review request a merge depends on; never would-suppress.
- `protocol_command` — machine-consumed (claim markers, `/lgtm`, `/approve`, `/ok-to-test`); never would-suppress.
- `security_event` — trust or integrity signal; never would-suppress.

| Producer | Class | Notes |
|---|---|---|
| `dashboard.(*ContributeWSHub).markAlreadyDoneIssue` | `status` | |
| `dashboard.(*ContributeWSHub).releaseGateParkLabel` | `status` | |
| `dashboard.(*ContributeWSHub).releaseToolingBlockedLabel` | `status` | |
| `dashboard.(*Server).armRebaseThenMerge` | `status` | |
| `dashboard.(*Server).executeDelegatedDecision` | `status` | |
| `dashboard.(*Server).handlePullRequestMergeDecision` | `status` | |
| `dashboard.(*Server).handleRepoItemAdoptNow` | `status` | |
| `dashboard.(*Server).handleRepoItemBlockerClosePartial` | `status` | |
| `dashboard.(*Server).handleRepoItemDecision` | `status` | |
| `dashboard.(*Server).handleReviewQueueSupersededClose` | `status` | |
| `dashboard.(*Server).mirrorPlanWith` | `status` | |
| `github.(*Client).CloseIssueForConfiguredLinePR` | `status` | |
| `github.(*Client).CreateIssueCommentBudgeted` | `status` | passthrough wrapper; attributed to its caller |
| `github.(*Client).PostFleetReportCleared` | `status` | |
| `github.(*Client).PostFleetReportRecovery` | `status` | |
| `github.(*Client).PostFleetReportRollup` | `status` | |
| `github.(*Client).PostNoteOrFold` | `status` | passthrough wrapper; attributed to its caller |
| `github.(*Client).SweepLegacyFleetReportIssues` | `status` | |
| `github.(*Client).UpsertIssueMarkerComment` | `status` | passthrough wrapper; attributed to its caller |
| `github.(*Client).closeDecomposedParent` | `status` | |
| `github.(*Client).closeIssueForMergedPRClaimWithIssue` | `status` | |
| `github.(*Client).closeSupersededPRDecision` | `status` | |
| `github.(*Client).fileReviewBacklog` | `status` | |
| `github.(*Client).joinFleetReportIssue` | `status` | |
| `github.(*Client).markInProgressStarted` | `status` | |
| `github.(*Client).postUnblockedNote` | `status` | |
| `github.(*Client).priorArtGateClose` | `status` | |
| `github.(*Client).reconcileBlockedTargets` | `status` | |
| `github.(*Client).recordReviewEvidenceMerge` | `status` | |
| `github.SyncIssueStageLabelsWith` | `status` | |
| `github.ensureIssueStagePRLink` | `status` | |
| `github.githubIssueBacklogSink.AppendToItem` | `status` | |
| `github.githubIssueBacklogSink.CloseItem` | `status` | |
| `github.githubIssueBacklogSink.LiftBlock` | `status` | |
| `github.handleMergedPRIssueStage` | `status` | |
| `github.runRecheckSoak` | `status` | |
| `main.postReviewAcknowledgements` | `status` | |
| `main.rfcNotifySource.Comment` | `status` | |
| `main.unparkEscalatedPR` | `status` | |
| `dashboard.(*ContributeWSHub).parkWrongBase` | `attention` | |
| `dashboard.(*Server).handleRepoItemReroute` | `attention` | |
| `dashboard.(*Server).handleReviewPipelineSendToHuman` | `attention` | |
| `dashboard.(*Server).reroutePickupLapsed` | `attention` | |
| `dashboard.(*Server).settleRebaseMergeLedger` | `attention` | |
| `github.(*Client).NudgeTrackingIssueClosingRefs` | `attention` | |
| `github.(*Client).appendFoldedFinding` | `attention` | |
| `github.(*Client).bindProwLgtm` | `attention` | |
| `github.(*Client).commentOnHumanFiledTwin` | `attention` | |
| `github.(*Client).createIssue` | `attention` | |
| `github.(*Client).handleOneIssueRequest` | `attention` | |
| `github.(*Client).parkClankerRequestedPR` | `attention` | |
| `github.(*Client).redirectToExistingPR` | `attention` | |
| `github.(*Client).resolveDuplicateFixGroup` | `attention` | |
| `main.(*repoForgeRouter).CreateIssueComment` | `attention` | passthrough wrapper; attributed to its caller |
| `main.claimEscalationLookup` | `attention` | |
| `main.liveTriageCommenter.CreateIssueComment` | `attention` | passthrough wrapper; attributed to its caller |
| `main.maintainHeldPRMergeability` | `attention` | |
| `main.rfcCoverageForge.CreateComment` | `attention` | |
| `main.runEscalationSweep` | `attention` | |
| `prfollowup.postStaleHeadPushNotes` | `attention` | |
| `scheduler.(*Scheduler).judgePriorArt` | `attention` | |
| `scheduler.(*Scheduler).parkUnclearIssue` | `attention` | |
| `scheduler.(*Scheduler).postTriageClarifyComment` | `attention` | |
| `github.(*Client).QueuePRAutoMergeWithNoteResultAtHead` | `required_review` | |
| `github.(*Client).RequestReviewers` | `required_review` | passthrough wrapper; attributed to its caller |
| `github.(*Client).handleOneReviewRequest` | `required_review` | |
| `dashboard.postClaimComment` | `protocol_command` | |
| `github.(*Client).PostClaimNote` | `protocol_command` | |
| `github.(*Client).reportOperatorPullRequestApproval` | `protocol_command` | |
| `main.githubClaimHooks` | `protocol_command` | |
| `github.(*Client).handleOnePRRequest` | `security_event` | |
| `github.(*Client).markReporterTrustAwaiting` | `security_event` | |
| `github.(*Client).reconcileReporterTrustSignal` | `security_event` | |
| `main.enforceHoldGuard` | `security_event` | |

Hive-maintained GitHub Actions workflows that comment (not instrumented by the
Go client; listed for completeness):

| Workflow | Class | What it posts |
|---|---|---|
| `changelog-fragment-guard.yml`, `changelog-reminder.yml`, `docs-index-reminder.yml` | `status` | PR reminders about changelog fragments and the docs index. |
| `issue-closed-reopen-hint.yml`, `issue-reopen-command.yml` | `status` | Reopen hint on closed issues; reply to a reopen command. |
| `backend-smoke.yml`, `ci-runner-canary.yml`, `coverage-hourly.yml`, `dco-post-merge.yml`, `flue-smoke.yml`, `omp-smoke.yml`, `wavefront-smoke.yml`, `v2-ci.yml`, `v2-tests.yml` | `attention` | "Still failing" comments on an open failure-tracking issue. |
| `promote-stable.yml` | `attention` | Emergency stable-promotion exception note. |
| `weekly-competitor-report.md` (`.lock.yml`) | `status` | Weekly report comment on the tracking issue. |
| `hive-ok-to-test.yml` | `protocol_command` | The `/ok-to-test` command that releases held workflow runs. |
8 changes: 8 additions & 0 deletions src/docs/operator-reference/sweeps-and-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,14 @@ already present), so the effect of the mode can be charted from the logs.
`normal` restores the previous behaviour: every note is its own comment,
every perspective's verdict its own review, every new head its own mention.

`github.notifications.shadow: true` (default off) adds would-notify
instrumentation without changing any write: every new comment, review,
review request and @-mention is classified by producer and class, repeats of
the same condition are counted as would-suppress, and the counters are
exported on `/metrics`. See [GitHub writes: inventory and shadow
mode](../notifications.md#github-writes-inventory-and-shadow-mode-githubnotificationsshadow)
for the write-site inventory and how to compare baseline windows.

## Suspicious-activity alerts (`sentinel`)

The sentinel evaluates open PRs in the watched repos — human, bot, or agent —
Expand Down
2 changes: 1 addition & 1 deletion src/docs/scale-envelope.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ ones to:
2. **Truncation must preserve the prioritization signal, not cut an arbitrary slice.**
This holds today and is easy to assume backwards:
- Issues are sorted by actionable rank tier, then **oldest first** within each tier before any cap is applied
(`RankActionableIssues`, `src/pkg/github/client.go:5058-5068`, descending `AgeMinutes` within each tier).
(`RankActionableIssues`, `src/pkg/github/client.go:5068-5078`, descending `AgeMinutes` within each tier).
- PRs are sorted by **review class then oldest-first within class**
(`SortPullRequestsForReview`, `src/pkg/github/review_priority.go:334`;
fixes → refactors/docs → tests).
Expand Down
6 changes: 6 additions & 0 deletions src/pkg/config/github_notifications.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ const (
type GitHubNotificationsConfig struct {
// Mode is "quiet" (default when empty) or "normal".
Mode string `yaml:"mode,omitempty" json:"mode,omitempty"`
// Shadow turns on would-notify instrumentation: every
// notification-capable GitHub write (new comment, review, review
// request, @-mention) is classified and counted, and a would-suppress
// decision is recorded for repeats of the same condition. Measurement
// only — nothing is suppressed or rerouted. Default off.
Shadow bool `yaml:"shadow,omitempty" json:"shadow,omitempty"`
}

// Quiet reports whether the hive should fold its PR notes into one edited
Expand Down
6 changes: 6 additions & 0 deletions src/pkg/config/github_notifications_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,9 @@ func TestGitHubNotificationsConfig(t *testing.T) {
t.Error("unknown mode must be invalid")
}
}

func TestGitHubNotificationsShadowDefaultsOff(t *testing.T) {
if (GitHubNotificationsConfig{}).Shadow {
t.Fatal("github.notifications.shadow must default to off")
}
}
70 changes: 70 additions & 0 deletions src/pkg/dashboard/metrics_github_notifications.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
package dashboard

import (
"fmt"
"strings"

ghpkg "github.com/hivecommons/hive/pkg/github"
)

// githubNotificationSnapshot is what /metrics exports about the hive's
// notification-capable GitHub writes: the always-on comment tally and, when
// github.notifications.shadow is on, the shadow counters.
type githubNotificationSnapshot struct {
Created, Edited, Suppressed int64
Shadow bool
Counts []ghpkg.NotificationShadowCount
ActiveItems int
}

func githubNotificationSnapshotFrom(c *ghpkg.Client) githubNotificationSnapshot {
var snap githubNotificationSnapshot
if c == nil {
return snap
}
snap.Created, snap.Edited, snap.Suppressed = c.NotificationCounts()
snap.Counts = c.NotificationShadowCounts()
snap.Shadow = c.NotificationShadowEnabled() || len(snap.Counts) > 0
snap.ActiveItems = c.NotificationShadowActiveItems()
return snap
}

var githubNotificationShadowMetricHelp = []struct{ metric, help string }{
{ghpkg.NotificationMetricCommentsCreated, "New issue/PR comments the hive created (API writes, not emails delivered), by producer and class."},
{ghpkg.NotificationMetricReviewsSubmitted, "PR reviews the hive submitted, by producer and class."},
{ghpkg.NotificationMetricReviewRequests, "Review requests the hive made, by producer and class."},
{ghpkg.NotificationMetricMentions, "@-mentions carried by comments and reviews the hive posted, by producer and class."},
{ghpkg.NotificationMetricSuppressedRepeats, "Notes the hive chose not to post as a new comment (folded or already present), by producer and class."},
{ghpkg.NotificationMetricWouldSuppressRepeats, "Writes shadow mode would have suppressed as a repeat of the same condition, by producer and class. Nothing is suppressed."},
{ghpkg.NotificationMetricDeliveryFailures, "Notification-capable writes that failed, by producer and class."},
}

// writeGitHubNotificationMetrics emits hive_github_comments_total and, when
// shadow mode is or was on, the hive_github_notification_* series.
func writeGitHubNotificationMetrics(b *strings.Builder, hiveID string, writeHeader func(name, help, typ string), snap githubNotificationSnapshot) {
writeHeader("hive_github_comments_total",
"Hive comments since process start by operation: created (notifies), edited (does not), suppressed (folded or already present).", "counter")
for _, op := range []struct {
name string
v int64
}{{"created", snap.Created}, {"edited", snap.Edited}, {"suppressed", snap.Suppressed}} {
fmt.Fprintf(b, "hive_github_comments_total{hive_id=%s,op=%s} %d\n", promQuote(hiveID), promQuote(op.name), op.v)
}
if !snap.Shadow {
return
}
for _, m := range githubNotificationShadowMetricHelp {
name := "hive_github_notification_" + m.metric + "_total"
writeHeader(name, m.help, "counter")
for _, c := range snap.Counts {
if c.Metric != m.metric {
continue
}
fmt.Fprintf(b, "%s{hive_id=%s,producer=%s,class=%s} %d\n",
name, promQuote(hiveID), promQuote(c.Producer), promQuote(string(c.Class)), c.Value)
}
}
writeHeader("hive_github_notification_active_items",
"Issues and PRs that received a notification-capable hive write in the last 24 hours; the per-active-item denominator.", "gauge")
fmt.Fprintf(b, "hive_github_notification_active_items{hive_id=%s} %d\n", promQuote(hiveID), snap.ActiveItems)
}
Loading
Loading