Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/webhook-group-by-vars.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
'@hyperdx/api': minor
'@hyperdx/app': minor
'@hyperdx/common-utils': minor
---

feat: expose group-by values to webhook body templates as `{{var.<column>}}`, and keep them on resolve notifications
1 change: 1 addition & 0 deletions docs/alert-webhook-template-variables.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ a HyperDX alert without parsing the human-readable message body.
| `{{thresholdMax}}` | number | Upper bound of a `between`/`outside` condition. Always unset for every other comparator, even if the alert was once a range, so guard it (see below). |
| `{{value}}` | number | The value that triggered or resolved the alert. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Bracket syntax in the docs doesn't cover map-attribute group-bys

parseAlertData (index.ts:940) uses ClickHouse's result column name as the key. For the common OTel group-by ResourceAttributes['service.name'], the key is that whole expression, not service.name. It also contains a ], so the documented {{var.[service.name]}} renders empty and a plain [...] segment can't reach the key. Fix: say that the key is the literal group-by expression unless it is aliased, and document {{lookup var "ResourceAttributes['service.name']"}} for keys like this.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

| `{{groupKey}}` | string | The breaching group, for a grouped alert. |
| `{{var}}` | object | The breaching group's values by group-by column or SQL alias. Use one value at a time, e.g. `{{var.customer_id}}`, or `{{var.[service.name]}}` for a name with dots. Empty when not grouped, and on the resolve of a group that has no rows left. |
| `{{sourceQuery}}` | string | The condition behind the alert — search expression or SQL (see below). |
| `{{teamId}}` | string | Team the alert belongs to. |
| `{{note}}` | string | The alert's freeform note — commonly a runbook link. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1383,6 +1383,7 @@ describe('webhooks router', () => {
thresholdMax: 10,
value: 7,
groupKey: 'test-group',
groupValues: { group: 'test-group' },
sourceQuery: 'SeverityText: "error"',
note: 'Test webhook — no runbook',
teamId: expect.any(String),
Expand Down
1 change: 1 addition & 0 deletions packages/api/src/routers/api/webhooks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -491,6 +491,7 @@ router.post(
thresholdMax: 10,
value: 7,
groupKey: 'test-group',
groupValues: { group: 'test-group' },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Send test fills only var.group, so a body using a real group-by column can't be checked

The PR says the test lets you check a body before an alert fires. But the test payload is hardcoded to { group: 'test-group' }, so {{var.customer_id}} or {{var.ServiceName}} (the PR's own repro) still renders as an empty string. Fix: walk the body's Handlebars AST for var.<name> paths, the way findUndeliverableConstruct in packages/api/src/utils/webhookBodyTemplate.ts already walks it, and fill each one with a sample value.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

sourceQuery: 'SeverityText: "error"',
teamId: teamId.toString(),
note: 'Test webhook — no runbook',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7223,6 +7223,68 @@ describe('checkAlerts', () => {
expect(resolutionCall).toBeDefined();
});

it('Group-by alerts keep the group values on the resolve notification', async () => {
const {
team,
webhook,
connection,
source,
savedSearch,
teamWebhooksById,
clickhouseClient,
} = await setupSavedSearchAlertTest();

const details = await createAlertDetails(
team,
source,
{
source: AlertSource.SAVED_SEARCH,
channel: { type: 'webhook', webhookId: webhook._id.toString() },
interval: '5m',
thresholdType: AlertThresholdType.ABOVE,
threshold: 2,
savedSearchId: savedSearch.id,
groupBy: 'ServiceName',
name: 'Errors for {{attributes.ServiceName}}',
},
{ taskType: AlertTaskType.SAVED_SEARCH, savedSearch },
);

const error = (Timestamp: Date) => ({
ServiceName: 'service-b',
Timestamp,
SeverityText: 'error',
Body: 'Error from service-b',
});
const firstWindow = new Date('2023-11-16T22:05:00.000Z');
// One error in the next window: still present in the data, but below threshold.
await bulkInsertLogs([
error(firstWindow),
error(firstWindow),
error(new Date('2023-11-16T22:10:00.000Z')),
]);

for (const at of [
'2023-11-16T22:12:00.000Z',
'2023-11-16T22:16:00.000Z',
]) {
await processAlertAtTime(
new Date(at),
details,
clickhouseClient,
connection.id,
alertProvider,
teamWebhooksById,
);
}

const texts = jest
.mocked(slack.postMessageToWebhook)
.mock.calls.map(call => call[1].text);
expect(texts).toHaveLength(2);
expect(texts[1]).toBe('✅ Errors for service-b');
});

it('TILE alert (metrics) - slack webhook', async () => {
const { team, webhook, connection, teamWebhooksById, clickhouseClient } =
await setupSavedSearchAlertTest();
Expand Down
5 changes: 5 additions & 0 deletions packages/api/src/tasks/checkAlerts/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -562,6 +562,7 @@ const fireChannelEvent = async ({
teamId,
teamWebhooksById,
sampleLines,
groupValues: attributes,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — No test checks that real alert group values reach a webhook body as {{var.*}}

If you deleted groupValues: attributes from fireChannelEvent, or groupValues: groupValues ?? {} from template.ts:758, every test would still pass. generic.test.ts calls buildWebhookTemplateVariables directly, webhooks.int.test.ts only covers the hardcoded test-send payload, and the new checkAlerts test only checks the Slack title, which comes from view.attributes. Add a checkAlerts int test with a generic webhook whose body is {{var.ServiceName}}, and assert the posted body on both the firing and the resolve notification.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — No test covers the new groupValues path from a real alert to the webhook body

The new unit test in generic.test.ts calls buildWebhookTemplateVariables directly. The new checkAlerts int test only checks the Slack title, which comes from view.attributes, not groupValues. So deleting groupValues: attributes here, or groupValues: groupValues ?? {} in template.ts:758, would still pass every test. Add an int test where a grouped alert uses a Generic webhook with body {{var.ServiceName}}, and assert the rendered body on both fire and resolve.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

});
return { failures, timings, dispatchDurationMs };
};
Expand Down Expand Up @@ -1286,6 +1287,8 @@ export const processAlert = async (
string,
{ value: number; attributes: Record<string, string>; startTime: Date }
>();
// ponytail: a group that vanished from the data resolves without its values.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Stray ponytail: prefix in a code comment

ponytail: doesn't match any comment-tag convention in this file and looks like a leftover personal marker. Drop the prefix and keep the explanation of why a vanished group resolves without its values.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

const groupAttributes = new Map<string, Record<string, string>>();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — groupAttributes repeats attributes already stored in bucketEvaluations and latestAlertContext

groupKey is built deterministically from extraFields, so the new groupAttributes map always holds the same attributes as bucketEvaluations[groupKey].attributes and latestAlertContext[groupKey].attributes. That leaves three copies. Use groupAttributes.get(groupKey) in the fire path too and remove attributes from the other two maps. Also replace the ponytail: prefix on the comment with a plain explanation.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond


// Helper to get or create history for a group
const getOrCreateHistory = (groupKey: string): IAlertHistory => {
Expand Down Expand Up @@ -1471,6 +1474,7 @@ export const processAlert = async (
group: groupKey,
totalCount: lastValue?.count || 0,
startTime: lastValue?.startTime || nowInMinsRoundDown,
attributes: groupAttributes.get(groupKey),
});
}
};
Expand Down Expand Up @@ -1635,6 +1639,7 @@ export const processAlert = async (
? extraFields.map(([k, v]) => `${k}:${v}`).join(', ')
: '';
const attributes = hasGroupBy ? Object.fromEntries(extraFields) : {};
groupAttributes.set(groupKey, attributes);

const exceeds = doesExceedThreshold(alert, value);

Expand Down
4 changes: 4 additions & 0 deletions packages/api/src/tasks/checkAlerts/template.ts
Original file line number Diff line number Diff line change
Expand Up @@ -617,6 +617,7 @@ export const renderAlertTemplate = async ({
teamWebhooksById,
dispatcher = inlineNotificationDispatcher,
sampleLines,
groupValues,
}: {
alertProvider: AlertProvider;
clickhouseClient: ClickhouseClient;
Expand All @@ -634,6 +635,8 @@ export const renderAlertTemplate = async ({
* per group; without one they are fetched here.
*/
sampleLines?: () => Promise<string>;
// Flat, unlike view.attributes, so a dotted column name stays one key.
groupValues?: Record<string, string>;
}): Promise<RenderedAlert> => {
// Internal mutable view with __hdx_query_results__ populated on the
// saved-search path. Untrusted values must flow through the view so
Expand Down Expand Up @@ -752,6 +755,7 @@ export const renderAlertTemplate = async ({
: undefined,
value,
groupKey: group ?? '',
groupValues: groupValues ?? {},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — No test checks that real alert notifications pass groupValues through to the webhook message

The new int test only checks the Slack title, which comes from the separate attributes path. generic.test.ts calls buildWebhookTemplateVariables directly. So if groupValues: attributes in fireChannelEvent or groupValues ?? {} here were dropped, every test would still pass. Fix: add a checkAlerts int test with a Generic webhook whose body uses {{var.ServiceName}}, and assert the rendered request body on both fire and resolve.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

sourceQuery: truncateString(
describeSourceQuery(alert, savedSearch, dashboard),
MAX_SOURCE_QUERY_LENGTH,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,21 @@ describe('buildWebhookTemplateVariables', () => {
expect(vars.sourceQuery).toBe('Body: \\"error\\"');
});

it('exposes group-by values as JSON-escaped var.<column> entries', () => {
const vars = buildWebhookTemplateVariables({
...message,
groupValues: { customer_id: 'acme "eu"' },
});
const rendered = createHandlebarsWithHelpers().compile(
'{"summary": "{{var.customer_id}} has an elevated error rate"}',
{ noEscape: true },
)(vars);
expect(JSON.parse(rendered).summary).toBe(
'acme "eu" has an elevated error rate',
);
expect(buildWebhookTemplateVariables(message).var).toEqual({});
});

it('renders enriched fields as empty strings (never "undefined") when absent', () => {
const vars = buildWebhookTemplateVariables(message);
expect(vars.alertId).toBe('');
Expand Down
7 changes: 6 additions & 1 deletion packages/api/src/tasks/checkAlerts/transports/generic.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import {
WebhookTemplateVariable,
} from '@hyperdx/common-utils/dist/types';
import Handlebars from 'handlebars';
import _ from 'lodash';
import { performance } from 'perf_hooks';
import { serializeError } from 'serialize-error';

Expand Down Expand Up @@ -96,14 +97,18 @@ export const buildWebhookTemplateVariables = (message: Message) =>
alertType: escapeJsonString(message.alertType ?? ''),
comparator: escapeJsonString(message.comparator ?? ''),
groupKey: escapeJsonString(message.groupKey ?? ''),
var: _.mapValues(message.groupValues ?? {}, v => escapeJsonString(v)),
note: escapeJsonString(message.note ?? ''),
sourceQuery: escapeJsonString(message.sourceQuery ?? ''),
status: escapeJsonString(message.status ?? ''),
teamId: escapeJsonString(message.teamId ?? ''),
threshold: message.threshold,
thresholdMax: message.thresholdMax,
value: message.value,
}) satisfies Record<WebhookTemplateVariable, string | number | undefined>;
}) satisfies Record<

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Widened satisfies lets any template variable be an object, which weakens the type check

Adding Record<string, string> to the shared value union means title, body and the rest could become objects without a type error. Fix: give var its own type, e.g. satisfies Record<Exclude<WebhookTemplateVariable, 'var'>, string | number | undefined> & { var: Record<string, string> }.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Widening the satisfies value type lets any variable be an object

Adding Record<string, string> to the shared value union means a mistake like title: someObject now type-checks and renders [object Object]. Keep the old union for the scalar variables and type only var as the object, e.g. satisfies Record<Exclude<WebhookTemplateVariable, 'var'>, string | number | undefined> & { var: Record<string, string> }.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Widened satisfies type lets every template variable be an object

Adding Record<string, string> to the shared value union means a scalar variable such as title or groupKey could become an object and still compile. That silently loosens the guarantee the comment above claims. Keep the scalar union for the other keys and type var alone as Record<string, string>, e.g. Record<Exclude<WebhookTemplateVariable, 'var'>, string | number | undefined> & { var: Record<string, string> }.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

WebhookTemplateVariable,
string | number | undefined | Record<string, string>
>;

/**
* Creates a Handlebars instance with common helpers registered.
Expand Down
2 changes: 2 additions & 0 deletions packages/api/src/tasks/checkAlerts/transports/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ export interface Message {
thresholdMax?: number;
value?: number; // the value that triggered/resolved the alert
groupKey?: string;
// The group's values keyed by group-by column, exposed as `{{var.<column>}}`.
groupValues?: Record<string, string>;
sourceQuery?: string; // the search expr / SQL that defines the alert
teamId?: string;
note?: string; // freeform alert note (markdown); commonly holds a runbook link
Expand Down
1 change: 1 addition & 0 deletions packages/app/src/components/TeamSettings/WebhookForm.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ const describeTemplateVariable = (
thresholdMax: 'Upper bound of a between/outside range (number)',
value: 'Value that triggered or resolved the alert (number)',
groupKey: 'The breaching group, if grouped',
var: 'Group-by values by column, e.g. {{var.customer_id}}',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Form lists {{var}} as a variable, but using it as shown renders [object Object]

getWebhookTemplateVariables displays each entry as {{name}}, so the list shows {{var}} next to scalar variables. That object is useless on its own: a body containing {{var}} renders [object Object]. Fix: display this entry as {{var.<column>}}, for example by special-casing object-valued names in getWebhookTemplateVariables, so the label shows the form users actually need.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Variable list shows {{var}} as a token to use, but on its own it renders [object Object]

getWebhookTemplateVariables wraps every name as {{name}}, so the form lists {{var}}. Pasting that into a body puts [object Object] in the payload. Every other entry in the list works as a complete token. Fix: render this entry as {{var.<column>}}, either with a display-name override for var or by building the name from the descriptions map.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The variable list shows {{var}}, and using it as shown sends [object Object]

getWebhookTemplateVariables displays every entry as {{name}}. Every other entry in the list works if copied as-is, but {{var}} is an object, so Handlebars renders it as the string [object Object] in the request body. Show this entry as {{var.<column>}}, or skip the {{name}} wrapping for var, so the copyable token actually works.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The variable list shows {{var}}, which renders [object Object] if pasted as written

getWebhookTemplateVariables displays every entry as {{${name}}}. Every other row can be pasted as-is, but {{var}} in a body renders the object as [object Object]. Display this entry as {{var.<column>}}, either with a per-variable display-name override or by special-casing object variables in getWebhookTemplateVariables.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

sourceQuery: 'Search expression or SQL behind the alert',
startTime: 'Window start, Unix ms (number)',
endTime: 'Window end, Unix ms (number)',
Expand Down
1 change: 1 addition & 0 deletions packages/common-utils/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -648,6 +648,7 @@ export const WEBHOOK_TEMPLATE_VARIABLES = [
'thresholdMax',
'value',
'groupKey',
'var',
'sourceQuery',
'teamId',
'note',
Expand Down
Loading