Skip to content

feat(alerts): add re-notify interval as option, stop notifying on every evaluation - #2754

Open
niladrix719 wants to merge 14 commits into
ClickHouse:mainfrom
niladrix719:alert-renotify-interval#2464
Open

niladrix719 wants to merge 14 commits into
ClickHouse:mainfrom
niladrix719:alert-renotify-interval#2464

Conversation

@niladrix719

Copy link
Copy Markdown
Contributor

Summary

Adds a renotifyIntervalMinutes alert setting and changes the default so alerts stop re-notifying on every evaluation while firing, unset notifies once on the OK→ALERT transition, 0 restores the old every-evaluation behavior, and N re-notifies every N minutes.

fixes #2464

Screenshots or video

Screenshot 2026-07-30 at 12 14 08 PM

How to test on Vercel preview

The "Re-notify every" field lives in the saved-search alert modal (DBSearchPage.tsx) and the dashboard tile alert editor (ChartEditorControls.tsx), both gated behind !IS_LOCAL_MODE. The Vercel preview build runs in LOCAL_MODE

Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

@niladrix719 is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@changeset-bot

changeset-bot Bot commented Jul 30, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 467cb39

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@greptile-apps

greptile-apps Bot commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; no new actionable issues remain.

Summary

Adds renotifyIntervalMinutes across alert settings, APIs, and editors.

  • Alerts repeat notifications only when their interval allows it.
  • Alert editors let people choose how often firing alerts repeat.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Evaluate missed windows] --> B{OK before a later ALERT?}
  B -->|Yes| C[Treat as a fresh incident]
  B -->|No| D[Keep earlier notification time]
  C --> E[Send firing notification]
  D --> F{First send or repeat due?}
  F -->|Yes| E
  F -->|No| G[Skip repeat]
  E --> H{Final state is OK?}
  H -->|Yes| I[Send recovery notification]
  H -->|No| J[Save firing state]
Loading

Reviews (8) · Last reviewed commit: "fix(alerts): catch-up run that resolves ..." · Reviewed by Greptile

niladrix719 and others added 3 commits July 30, 2026 12:22
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
@niladrix719 niladrix719 changed the title feat(alerts): add re-notify interval, stop notifying on every evaluation feat(alerts): add re-notify interval as option, stop notifying on every evaluation Jul 30, 2026
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
…-interval

# Conflicts:
#	packages/api/src/models/alertHistory.ts
#	packages/api/src/tasks/checkAlerts/index.ts
#	packages/api/src/utils/zod.ts
…dals

EditAlertModal and EditInlineAlertModal (added on main after this branch was
cut) rebuild the PUT payload from a fixed field list, so saving an alert there
dropped renotifyIntervalMinutes back to transition-only. Carry it like
numConsecutiveWindows, and show it in AlertPropertiesSummary.
@bcanvural

Copy link
Copy Markdown
Contributor

Hi @niladrix719, we'd like to see this land. We're evaluating HyperDX alerts to replace our current alerting, and #2464 is the main blocker: a 1-minute alert posts to Slack every minute for the whole incident. Your defaults (notify on OK→ALERT, 0 for the old behavior, N as a reminder interval) are what we need, and match Alertmanager's repeat_interval model.

The branch is ~450 commits behind main and now conflicts, so I brought it up to date to check it still holds: https://github.com/bcanvural/hyperdx/tree/claude/alert-renotify-interval. It's your branch plus a merge of main, so yours fast-forwards to it.

  • The conflicts in alertHistory.ts, utils/zod.ts and checkAlerts/index.ts were additive. On main the time-series path sends one ALERT per group per run from its final loop, and every send (single-value, time-series, resolve) still goes through shouldNotifyWhileFiring / hasUnresolvedNotification.
  • EditAlertModal and EditInlineAlertModal (new on main) rebuild the PUT payload from a fixed field list, so saving an alert there reset renotifyIntervalMinutes to transition-only. They now carry it like numConsecutiveWindows, and AlertPropertiesSummary shows it.
  • Added a group-by test: each group notifies once, on its own transition.
  • checkAlerts.int.test.ts + external-api/__tests__/alerts.int.test.ts: 272/272 pass; tsc clean for api and app.

Not covered: the MCP save_alert tool takes neither renotifyIntervalMinutes nor numConsecutiveWindows, so an update through MCP clears both.

Feel free to pull the branch into this PR. If you've moved on, I'm happy to open a new PR from it that credits you. @jordan-simonovski, could this get a review? It's been waiting-on-engineering since August.

@niladrix719

Copy link
Copy Markdown
Contributor Author

Thanks @bcanvural, this helped a lot. I pulled your branch into the PR, so your commits are in with your authorship, and I merged the latest main on top. I also added renotifyIntervalMinutes to the MCP save_alert tool, and fixed two small bugs where a failed first notification wasn't retried, with a test for them. The alert integration tests pass. I'm leaving numConsecutiveWindows in MCP for a follow-up unless the maintainers want it here

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 2 minor

2 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/app/src/components/AlertScheduleFields.tsx:1308 — No test covers the blank-vs-0 distinction in the new form field or the MCP save_alert path → In this field a blank input (stored as null, notify once) and 0 (notify on every evaluation) mean very different things. The onChange maps '' to undefined and keeps 0, but no component test checks that 0 survives a save round-trip or that clearing the field sends null. save_alert's new renotifyIntervalMinutes pass-through (mcp/tools/alerts/saveAlert.ts:116) also has no test. Add an AlertScheduleFields/EditAlertModal test for 0, N and cleared values, and an MCP test that sets the field and then omits it, checking that it resets.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings could not be posted inline.

  • P1 Failed sends advance notification clock packages/api/src/tasks/checkAlerts/index.ts:1108 ▶

    When fireChannelEvent throws, trySendNotification records the error but still returns true, so both firing paths persist lastNotifiedAt as though delivery succeeded. This prevents any retry during the current firing episode when the interval is unset and delays retries until the configured interval expires when it is positive.

          } catch (e) {
            logger.error(
              { alertId: alert.id, group, error: serializeError(e) },
              'Failed to fire channel event',
            );
            executionErrors.push(makeWebhookAlertError(e));
            return false;
          }
          return true;
    

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

This review focuses on the renotifyIntervalMinutes feature: the notification-gating engine in checkAlerts/index.ts, the new lastNotifiedAt history field, per-target retry gating, catch-up new-incident detection, and the field's propagation through the REST/external/MCP contracts and the React alert editors. Prior review rounds (15 comments) already identified and fixed the major reliability bugs — catch-up suppression, failed-first-send retry, and per-target retry — and those fixes were verified to hold in the current code.

✅ No critical (P0/P1) issues found. The core logic is carefully written and the previously-flagged P1s are genuinely resolved.

🟡 P2 -- recommended

  • packages/api/src/tasks/checkAlerts/index.ts:1442 -- When every notification target is unresolvable (e.g. the alert's only webhook was deleted), timings is empty and timings.every(t => t.ok) returns true, so the incident is marked notified (lastNotifiedAt set) and, under the new notify-once default, is not re-attempted even if a target becomes deliverable again within the same incident.
    • Fix: Distinguish "nothing was delivered because all targets were unresolvable" from "delivered to all targets" and emit a visible metric/log for the former so a silently-swallowed incident is observable, while preserving the deliberate no-infinite-retry behavior for permanently-dead targets.
🔵 P3 nitpicks (3)
  • packages/api/src/tasks/checkAlerts/index.ts:1803 -- In a catch-up run that spans an old unresolved incident going OK then a fresh breach, isNewIncidentInRun nulls groupPrevious so the new breach notifies, but no resolve notification is emitted for the intervening OK window, so an operator jumps from old-incident-firing straight to new-incident-firing with no "resolved" in between.
    • Fix: Consider emitting the resolve notification for the intervening OK before treating the later breach as a new incident, or document that catch-up collapses the two transitions.
  • packages/api/src/tasks/checkAlerts/index.ts:1551 -- The lastNotifiedAt carry-forward plus shouldNotifyWhileFiring plus set-on-success bookkeeping is duplicated between the single-value path and the time-series state-transition phase, so a future change to the renotify rule must be made in two places or the paths will diverge.
    • Fix: Extract the "decide, send, record lastNotifiedAt" sequence into one shared helper used by both paths.
  • packages/app/src/components/AlertScheduleFields.tsx:152 -- The new "Re-notify every" NumberInput and its unset/0/N semantics have no frontend test, while the backend behavior gained substantial integration coverage.
    • Fix: Add a component test asserting blank maps to transition-only, 0 to every-evaluation, and N to the interval value on submit.

Reviewers (11): correctness, reliability, adversarial, testing, api-contract, maintainability, kieran-typescript, previous-comments, project-standards, agent-native, learnings-researcher.

Testing gaps:

  • No React component test for the new renotifyIntervalMinutes field (unset / 0 / N paths).
  • MCP save_alert lacks a roundtrip test confirming renotifyIntervalMinutes persists and that omitting it resets to transition-only (the documented replace-semantics).

Notes: makeAlertUpdate persists the field correctly via $set: makeAlert(...) (0 is preserved because 0 ?? null === 0); the field is validated consistently (int, min 0, nullish) across zod.ts, common-utils/types.ts, the external v2 schema, OpenAPI, and the MCP schema. The author has acknowledged that numConsecutiveWindows remains absent from the MCP tool as a separate follow-up; renotifyIntervalMinutes itself does reach the MCP save_alert surface.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

executionErrors.push(makeNotificationAlertError(failure));
}
// Only failed dispatches retry; unresolvable targets would retry forever.
return timings.every(t => t.ok);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — One failing channel makes every healthy channel get the alert again on every evaluation

timings.every(t => t.ok) treats the whole send as failed if any one target rejects. lastNotifiedAt is then never written, and the next evaluation re-sends to all channels. Example: an alert has two webhooks and one returns 404/500 for good (a revoked Slack URL). The healthy webhook then gets a duplicate every interval while the alert fires, which is the spam this PR is meant to stop. The existing test only covers a deleted (unresolvable) webhook. Fix: set lastNotifiedAt once at least one target succeeds (timings.some(t => t.ok)), or track delivery per target so retries go only to the failed ones. Add a test with one rejecting and one accepting channel.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

for (const [groupKey, history] of histories.entries()) {
const previousKey = computeHistoryMapKey(alert.id, groupKey);
let groupPrevious = previousMap.get(previousKey);
const isNewIncidentInRun =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — A catch-up run with resolve → new breach → resolve drops the new incident's ALERT

Setup: the previous history is ALERT and was already notified, and the run's buckets go OK, ALERT, OK. lastOkBucket (2) is later than lastAlertBucket (1), so isNewIncidentInRun is false and groupPrevious stays the old notified ALERT. shouldNotifyWhileFiring then skips the send. Only the resolve goes out, so the second incident is never announced; before this PR both the alert and the resolve were sent. Fix: detect an OK bucket before any ALERT bucket in the run (for example, record the first OK index as well), not only compare the latest indices.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

// nullish() (not optional()): persisted alerts store this as null, which
// optional() would reject.
numConsecutiveWindows: z.number().int().min(1).nullish(),
renotifyIntervalMinutes: z.number().int().min(0).nullish(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — renotifyIntervalMinutes validation is written out again instead of derived from the shared schema

z.number().int().min(0).nullish() now appears in common-utils AlertBaseObjectSchema and AlertsPageItemSchema (packages/common-utils/src/types.ts), and again in EditAlertModal.tsx, DBSearchPageAlertModal.tsx and packages/api/src/utils/zod.ts. Per the DRY convention (REQUIRED), reuse AlertBaseObjectSchema.shape.renotifyIntervalMinutes, or .pick() it, in the app form schemas so the bounds have one source of truth.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

executionErrors.push(makeNotificationAlertError(failure));
}
// Only failed dispatches retry; unresolvable targets would retry forever.
return timings.every(t => t.ok);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — If one channel keeps failing, every healthy channel is re-notified on every evaluation

timings.every(t => t.ok) treats the whole send as failed when any one target rejects, so lastNotifiedAt is never set. Take an alert with a working Slack channel and a generic webhook that keeps returning 5xx: dispatch() rejects (template.ts:990-1010), the next evaluation sees lastNotifiedAt == null, and the message goes to all targets again. Slack then gets a duplicate on every tick for as long as the incident lasts, which is the noise this PR is meant to remove. Fix: either track lastNotifiedAt per target and retry only the targets that failed, or count the send as notified when at least one target delivered. Add a two-channel test where one channel always rejects.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

const previousGroups = Array.from(previousMap.keys(), key =>
extractGroupKeyFromMapKey(key, alert.id),
);
const markAbsentGroupsOk = (bucketIndex: number, present: Set<string>) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — markAbsentGroupsOk re-scans every group seen in the last 7 days on every bucket

previousGroups holds every group in previousMap, which uses a 7-day lookback (getPreviousAlertHistories). For each bucket the code also builds a new spread array [...previousGroups, ...histories.keys()] and may list the same group twice. For a high-cardinality group-by doing a long catch-up, that is O(buckets × groups) of extra work. Only groups that alert in this run can make isNewIncidentInRun true, so do the check once after the loop: for each group in lastAlertBucket, find the first bucket before its last ALERT where it was absent or OK, using a per-bucket record of the groups that were present.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Alerts re-notify on every evaluation while firing

3 participants