Skip to content

Commit ed73826

Browse files
committed
fix(api): address review of provisioned tile alerts
- Validate each declared alert like the alerts API, plus what the alert task needs to evaluate it: alertable display type, no PromQL, and a source, connection and webhooks in the team; apply the threshold and schedule rules. - Validate before the dashboard upsert. A tile whose alert is invalid keeps its last synced version, so an old alert never evaluates a new query. - Never take over an app-created alert: skip the file's alert on that tile. - A unique partial index keeps one provisioned alert per tile. - A failed dashboard or alert sync, or files sharing a dashboard name, skip orphan cleanup; duplicate names sync only the first file. - Read webhooks only when a file names one; a failing team no longer stops later teams. - Move alert and webhook-name logic out of index.ts; add business context and a skipped-alert counter; type the test helpers.
1 parent f1fcbc1 commit ed73826

8 files changed

Lines changed: 735 additions & 367 deletions

File tree

‎.changeset/provision-tile-alerts.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,16 @@ feat: provision tile alerts from dashboard files
66

77
The dashboard file provisioner stored a tile's `config.alert` inside the tile
88
but never created the alert, so a provisioned alert passed validation and never
9-
fired. It now creates or updates each declared tile alert on every sync,
10-
validated the same way the dashboards API validates them, and flags it
11-
`provisioned: true`.
9+
fired. It now creates or updates each declared tile alert on every sync, flagged
10+
`provisioned: true`, after the checks the alerts API applies plus the ones the
11+
alert task needs to evaluate it (display type, no PromQL, and a source,
12+
connection and webhooks in the team).
1213

1314
Channels can name their webhook with `webhookName`, resolved per team on each
1415
sync, instead of a `webhookId` that only exists in one install.
1516

1617
A provisioned alert is deleted once its tile, or its file entry, no longer
1718
declares it, and when its dashboard's file is removed or the dashboard is
18-
renamed. An alert that fails validation is skipped with a warning and keeps its
19-
last valid version. Alerts created in the app on a provisioned dashboard are not
20-
touched unless the file declares an alert on the same tile.
19+
renamed. An alert that fails validation is skipped with a warning, and its tile
20+
keeps its last synced version. Alerts created in the app are never changed or
21+
deleted.

‎packages/api/docs/auto_provision/AUTO_PROVISION.md‎

Lines changed: 45 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -166,22 +166,22 @@ to manage these values.
166166
## Dashboard Provisioning
167167
168168
HyperDX supports file-based dashboard provisioning, similar to Grafana's
169-
provisioning system. A scheduled task reads `.json` files from a directory
170-
and upserts dashboards into MongoDB, matched by name for idempotency.
171-
The task runs on the same schedule as other HyperDX tasks (every minute
172-
when using the built-in scheduler, or on your own schedule when running
173-
tasks externally).
169+
provisioning system. A scheduled task reads `.json` files from a directory and
170+
upserts dashboards into MongoDB, matched by name for idempotency. The task runs
171+
on the same schedule as other HyperDX tasks (every minute when using the
172+
built-in scheduler, or on your own schedule when running tasks externally).
174173

175174
### Environment Variables
176175

177-
| Variable | Required | Default | Description |
178-
| ----------------------------------- | -------- | ------- | --------------------------------------------------------------- |
179-
| `DASHBOARD_PROVISIONER_DIR` | Yes | | Directory to watch for `.json` dashboard files |
180-
| `DASHBOARD_PROVISIONER_TEAM_ID` | No\* | | Scope provisioning to a specific team ID |
181-
| `DASHBOARD_PROVISIONER_ALL_TEAMS` | No\* | `false` | Set to `true` to provision dashboards to all teams |
176+
| Variable | Required | Default | Description |
177+
| --------------------------------- | -------- | ------- | -------------------------------------------------- |
178+
| `DASHBOARD_PROVISIONER_DIR` | Yes | | Directory to watch for `.json` dashboard files |
179+
| `DASHBOARD_PROVISIONER_TEAM_ID` | No\* | | Scope provisioning to a specific team ID |
180+
| `DASHBOARD_PROVISIONER_ALL_TEAMS` | No\* | `false` | Set to `true` to provision dashboards to all teams |
182181

183-
\*One of `DASHBOARD_PROVISIONER_TEAM_ID` or `DASHBOARD_PROVISIONER_ALL_TEAMS=true`
184-
is required when `DASHBOARD_PROVISIONER_DIR` is set.
182+
\*One of `DASHBOARD_PROVISIONER_TEAM_ID` or
183+
`DASHBOARD_PROVISIONER_ALL_TEAMS=true` is required when
184+
`DASHBOARD_PROVISIONER_DIR` is set.
185185

186186
### Dashboard JSON Format
187187

@@ -234,9 +234,18 @@ API accepts. The provisioner creates or updates that tile alert on every sync:
234234
"h": 4,
235235
"config": {
236236
"name": "Error rate",
237-
"source": "Logs",
237+
"source": "<source ObjectId>",
238238
"displayType": "line",
239-
"select": [{ "aggFn": "count", "where": "SeverityText:error" }],
239+
"select": [
240+
{
241+
"aggFn": "count",
242+
"aggCondition": "SeverityText:error",
243+
"aggConditionLanguage": "lucene",
244+
"valueExpression": ""
245+
}
246+
],
247+
"where": "",
248+
"whereLanguage": "lucene",
240249
"alert": {
241250
"interval": "5m",
242251
"threshold": 100,
@@ -247,23 +256,32 @@ API accepts. The provisioner creates or updates that tile alert on every sync:
247256
}
248257
```
249258

250-
- A channel names its webhook with `webhookName`, which is looked up among the
251-
team's webhooks on every sync, so the same file works across installs and,
252-
with `DASHBOARD_PROVISIONER_ALL_TEAMS=true`, across teams that each have a
253-
webhook of that name. `webhookId` still works too. A name that matches no
254-
webhook, or webhooks of more than one service, fails validation
259+
- Each alert gets the checks the alerts API applies, plus the ones the alert
260+
task needs to evaluate it: a Line, Stacked Bar or Number display type, no
261+
PromQL, a valid raw SQL alert query, and a source, connection and webhooks
262+
that exist in the team. `source` is the source's ObjectId, so with
263+
`DASHBOARD_PROVISIONER_ALL_TEAMS=true` an alert is only created in the team
264+
that owns its source
265+
- A channel can name its webhook with `webhookName`, looked up among the team's
266+
webhooks on every sync, instead of a `webhookId` that only exists in one
267+
install. A name that matches no webhook, or more than one, fails validation
255268
- Provisioned alerts are flagged with `provisioned: true`. Removing an alert
256-
from its tile, or the tile from the file, deletes it on the next sync
269+
from its tile, or the tile from the file, deletes it on the next sync. Keep
270+
tile ids stable: changing one replaces the alert and loses its history
257271
- Removing a dashboard's file, or renaming the dashboard, deletes the
258272
provisioned alerts of the dashboard it leaves behind. The dashboard itself is
259273
kept, as above. This cleanup only runs when every file in the directory was
260-
read and validated, and never when the directory has no valid files, so a
261-
broken file or an empty mount does not remove alerts
262-
- An alert that fails validation (for example an unknown webhook, or a raw SQL
263-
tile whose display type or query does not support alerts) is skipped with a
264-
warning and keeps its last valid version
265-
- Alerts created in the app on a provisioned dashboard are left alone, unless
266-
the file declares an alert on the same tile, which then replaces it
274+
read, validated and synced, and never when the directory has no valid files,
275+
so a broken file, a failed sync or an empty mount does not remove alerts
276+
- An alert that fails validation is skipped with a warning. Its tile keeps the
277+
version from the last sync, query and alert together, so an old alert never
278+
evaluates a new query; a new tile is added without its alert
279+
- Alerts created in the app are never changed or deleted. A file that declares
280+
an alert on a tile that already has one from the app is skipped with a warning
281+
- Like the dashboard, a provisioned alert edited or deleted in the app is
282+
restored from its file on the next sync
283+
- Files that share a dashboard name are skipped with a warning, except the first
284+
in file name order
267285

268286
## Note on Security
269287

‎packages/api/src/controllers/alerts.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -429,6 +429,7 @@ export const createOrUpdateDashboardAlerts = async (
429429
tileId,
430430
source: AlertSource.TILE,
431431
team: teamId,
432+
...(provisioned && { provisioned: true }),
432433
};
433434
const alertInput = {
434435
...alert,

‎packages/api/src/models/alert.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -261,7 +261,7 @@ const AlertSchema = new Schema<IAlert>(
261261
},
262262
provisioned: {
263263
type: Boolean,
264-
required: false,
264+
default: false,
265265
},
266266
silenced: {
267267
required: false,
@@ -313,4 +313,14 @@ AlertSchema.index({ team: 1, _id: 1 });
313313
// tie-break, and walk that order with a keyset cursor.
314314
AlertSchema.index({ team: 1, displayName: 1, _id: 1 });
315315

316+
// One alert per provisioned tile, so concurrent provisioner runs upsert the
317+
// same document instead of inserting a duplicate that no sync would remove.
318+
AlertSchema.index(
319+
{ team: 1, dashboard: 1, tileId: 1 },
320+
{
321+
unique: true,
322+
partialFilterExpression: { provisioned: true, source: AlertSource.TILE },
323+
},
324+
);
325+
316326
export default mongoose.model<IAlert>('Alert', AlertSchema);

0 commit comments

Comments
 (0)