Repository navigation
feat(alerts): add re-notify interval as option, stop notifying on every evaluation - #2754
niladrix719 wants to merge 14 commits into
Conversation
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
|
@niladrix719 is attempting to deploy a commit to the HyperDX Team on Vercel. A member of the Team first needs to authorize it. |
🦋 Changeset detectedLatest commit: 467cb39 The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
[High impact] The PR appears safe to merge; no new actionable issues remain. SummaryAdds
|
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
…-interval # Conflicts: # packages/api/src/models/alertHistory.ts # packages/api/src/tasks/checkAlerts/index.ts # packages/api/src/utils/zod.ts
…dals EditAlertModal and EditInlineAlertModal (added on main after this branch was cut) rebuild the PUT payload from a fixed field list, so saving an alert there dropped renotifyIntervalMinutes back to transition-only. Carry it like numConsecutiveWindows, and show it in AlertPropertiesSummary.
|
Hi @niladrix719, we'd like to see this land. We're evaluating HyperDX alerts to replace our current alerting, and #2464 is the main blocker: a 1-minute alert posts to Slack every minute for the whole incident. Your defaults (notify on OK→ALERT, The branch is ~450 commits behind
Not covered: the MCP Feel free to pull the branch into this PR. If you've moved on, I'm happy to open a new PR from it that credits you. @jordan-simonovski, could this get a review? It's been |
|
Thanks @bcanvural, this helped a lot. I pulled your branch into the PR, so your commits are in with your authorship, and I merged the latest main on top. I also added |
PR Review
3 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 2 minor 2 posted as inline comment(s) on the changed lines. 1 listed below. Findings outside the changed lines1 minor
Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
Comments Outside DiffThese findings could not be posted inline.
|
Deep ReviewThis review focuses on the ✅ No critical (P0/P1) issues found. The core logic is carefully written and the previously-flagged P1s are genuinely resolved. 🟡 P2 -- recommended
🔵 P3 nitpicks (3)
Reviewers (11): correctness, reliability, adversarial, testing, api-contract, maintainability, kieran-typescript, previous-comments, project-standards, agent-native, learnings-researcher. Testing gaps:
Notes: No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
| executionErrors.push(makeNotificationAlertError(failure)); | ||
| } | ||
| // Only failed dispatches retry; unresolvable targets would retry forever. | ||
| return timings.every(t => t.ok); |
There was a problem hiding this comment.
🟠 major — One failing channel makes every healthy channel get the alert again on every evaluation
timings.every(t => t.ok) treats the whole send as failed if any one target rejects. lastNotifiedAt is then never written, and the next evaluation re-sends to all channels. Example: an alert has two webhooks and one returns 404/500 for good (a revoked Slack URL). The healthy webhook then gets a duplicate every interval while the alert fires, which is the spam this PR is meant to stop. The existing test only covers a deleted (unresolvable) webhook. Fix: set lastNotifiedAt once at least one target succeeds (timings.some(t => t.ok)), or track delivery per target so retries go only to the failed ones. Add a test with one rejecting and one accepting channel.
Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond
| for (const [groupKey, history] of histories.entries()) { | ||
| const previousKey = computeHistoryMapKey(alert.id, groupKey); | ||
| let groupPrevious = previousMap.get(previousKey); | ||
| const isNewIncidentInRun = |
There was a problem hiding this comment.
🔵 minor — A catch-up run with resolve → new breach → resolve drops the new incident's ALERT
Setup: the previous history is ALERT and was already notified, and the run's buckets go OK, ALERT, OK. lastOkBucket (2) is later than lastAlertBucket (1), so isNewIncidentInRun is false and groupPrevious stays the old notified ALERT. shouldNotifyWhileFiring then skips the send. Only the resolve goes out, so the second incident is never announced; before this PR both the alert and the resolve were sent. Fix: detect an OK bucket before any ALERT bucket in the run (for example, record the first OK index as well), not only compare the latest indices.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| // nullish() (not optional()): persisted alerts store this as null, which | ||
| // optional() would reject. | ||
| numConsecutiveWindows: z.number().int().min(1).nullish(), | ||
| renotifyIntervalMinutes: z.number().int().min(0).nullish(), |
There was a problem hiding this comment.
🔵 minor — renotifyIntervalMinutes validation is written out again instead of derived from the shared schema
z.number().int().min(0).nullish() now appears in common-utils AlertBaseObjectSchema and AlertsPageItemSchema (packages/common-utils/src/types.ts), and again in EditAlertModal.tsx, DBSearchPageAlertModal.tsx and packages/api/src/utils/zod.ts. Per the DRY convention (REQUIRED), reuse AlertBaseObjectSchema.shape.renotifyIntervalMinutes, or .pick() it, in the app form schemas so the bounds have one source of truth.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| executionErrors.push(makeNotificationAlertError(failure)); | ||
| } | ||
| // Only failed dispatches retry; unresolvable targets would retry forever. | ||
| return timings.every(t => t.ok); |
There was a problem hiding this comment.
🟠 major — If one channel keeps failing, every healthy channel is re-notified on every evaluation
timings.every(t => t.ok) treats the whole send as failed when any one target rejects, so lastNotifiedAt is never set. Take an alert with a working Slack channel and a generic webhook that keeps returning 5xx: dispatch() rejects (template.ts:990-1010), the next evaluation sees lastNotifiedAt == null, and the message goes to all targets again. Slack then gets a duplicate on every tick for as long as the incident lasts, which is the noise this PR is meant to remove. Fix: either track lastNotifiedAt per target and retry only the targets that failed, or count the send as notified when at least one target delivered. Add a two-channel test where one channel always rejects.
Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond
| const previousGroups = Array.from(previousMap.keys(), key => | ||
| extractGroupKeyFromMapKey(key, alert.id), | ||
| ); | ||
| const markAbsentGroupsOk = (bucketIndex: number, present: Set<string>) => { |
There was a problem hiding this comment.
🔵 minor — markAbsentGroupsOk re-scans every group seen in the last 7 days on every bucket
previousGroups holds every group in previousMap, which uses a 7-day lookback (getPreviousAlertHistories). For each bucket the code also builds a new spread array [...previousGroups, ...histories.keys()] and may list the same group twice. For a high-cardinality group-by doing a long catch-up, that is O(buckets × groups) of extra work. Only groups that alert in this run can make isNewIncidentInRun true, so do the check once after the loop: for each group in lastAlertBucket, find the first bucket before its last ALERT where it was absent or OK, using a per-bucket record of the groups that were present.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Summary
Adds a renotifyIntervalMinutes alert setting and changes the default so alerts stop re-notifying on every evaluation while firing, unset notifies once on the OK→ALERT transition, 0 restores the old every-evaluation behavior, and N re-notifies every N minutes.
fixes #2464
Screenshots or video
How to test on Vercel preview
The "Re-notify every" field lives in the saved-search alert modal (
DBSearchPage.tsx) and the dashboard tile alert editor (ChartEditorControls.tsx), both gated behind!IS_LOCAL_MODE. The Vercel preview build runs in LOCAL_MODE