` cannot opt back out. If a flow genuinely needs a heading or
+full-contrast body, **extend `useConfirm`** (an optional prop, applied to all
+call sites) rather than forking a one-off modal.
+
+**In component tests**, mock it — `ConfirmProvider` pulls in `next/router`,
+which is not available in jsdom:
+
+```tsx
+jest.mock('@/useConfirm', () => ({ useConfirm: jest.fn() }));
+```
+
+Assert on the arguments (and render the message `ReactNode` if you need to check
+the copy). Exercise the real dialog in E2E instead.
+
### EmptyState Component (REQUIRED)
**Use `EmptyState` (`@/components/EmptyState`) for all empty/no-data states.** Do not create ad-hoc inline empty states.
@@ -127,6 +285,117 @@ This pattern cannot be enforced by ESLint and requires manual code review.
**Title copy**: Treat `title` as a short headline (like `Title` in the UI). Do **not** end it with a period. Use `description` for full sentences, which should use normal punctuation including a trailing period when appropriate. Match listing pages (e.g. dashboards and saved searches use parallel phrasing such as “No matching … yet” / “No … yet” without dots).
+### Object names: use `InlineNameInput` (REQUIRED)
+
+**Name or rename a dashboard, saved search, tile, or any other user-created
+object with `InlineNameInput` (`@/components/InlineNameInput/InlineNameInput`).**
+Do not build a pencil-button edit mode, a bordered name `TextInput` in a
+header, or a "Name" field inside a save modal. See
+[naming_objects.md](naming_objects.md) for which variant to use.
+
+### Code snippets (REQUIRED)
+
+**Do not render raw ``, ad-hoc `Paper` + monospace text, or unstyled ``.** Use the existing Mantine/product components so snippets match Terraform export, onboarding, and Storybook guidelines.
+
+| Kind | Component | When |
+|------|-----------|------|
+| **Inline code** | Mantine `` | Short tokens in prose (`Session`, a column name, a flag). |
+| **Fenced / multi-line / copyable** | `CopySnippet` (`@/components/ClickStackOnboarding/CopySnippet`) | Install commands, HCL, JSX examples, any block the user might copy. Wraps `` plus a Copy button. |
+
+```tsx
+// ✅ inline — Mantine Code
+Create a source with Session type.
+
+// ✅ fenced / copyable — CopySnippet (Code + Copy)
+
+
+// Label is optional when the surrounding heading already names the snippet
+
+
+// ❌ BAD — raw pre / Paper chrome
+{snippet}
+{snippet}
+```
+
+**SQL query previews** (rendered ClickHouse SQL with highlighting) still use `SQLPreview` / `ChartSQLPreview` — those are editors, not snippet chrome.
+
+Storybook: `Components/Code` (live `InlineCode`, `FencedBlocks`, and `Usage`). Guidelines markdown maps fenced blocks → `CopySnippet` and inline `` `code` `` → `` automatically. Follow the same split in app UI.
+
+### Chart cards (ChartCard)
+
+**Use `ChartCard` (`@/components/charts/ChartCard`) to wrap a chart in a card.**
+It is a bordered surface with the same header treatment as a custom dashboard
+tile (a full-bleed divider under the title). It replaces the older `ChartBox`;
+don't hand-roll a bordered ``/`
` around a chart.
+
+`ChartCard` renders the card **chrome only**. The header divider is drawn only
+when a descendant renders a `ChartContainer` with a `title` (or `toolbarItems`) —
+`ChartCard` supplies the `ChartContainerCardHeaderProvider` that switches that
+header into card mode — so put a chart that renders a `ChartContainer` inside it
+(`DBTimeChart`, `DBTableChart`, `DBHeatmapChart`, `DBListBarChart`, …). Content
+with its own heading (e.g. a bespoke table card) should still route that heading
+through a titled `ChartContainer` rather than a bare `Text`, so it gets the same
+card header — divider and top padding included — instead of sitting flush against
+the top border. The tile-level controls (fullscreen, line/bar display switcher,
+kebab menu) belong to dashboard tiles and are intentionally **not** part of
+`ChartCard`.
+
+| Prop | Type | Description |
+|------|------|-------------|
+| `children` | `ReactNode` | The chart, usually a `DB*Chart` (or a titled `ChartContainer`) |
+| `style` | `CSSProperties` | Sizing/overflow override — pass a fixed `height`, or `flex: 1; height: 100%` to fill a flex row (`paddingInline` is pinned to keep the divider aligned) |
+| `data-testid` | `string` | Test hook |
+
+```tsx
+// ✅ GOOD — shared card chrome, consistent with dashboard tiles
+
+
+
+
+// ❌ BAD — hand-rolled card that drifts from the dashboard look
+
+
+
+```
+
+**Give it a height.** `ChartCard` is `width: 100%` and fills its parent, so the
+parent (or a `style={{ height }}`) must define the height. For equal-width
+side-by-side charts (e.g. the RED row) use
+`style={{ flex: 1, minWidth: 0, minHeight: 0, height: '100%' }}` inside a
+`Flex`. See the `Charts/ChartCard` Storybook stories for the variants.
+
+## UI text: use sentence case
+
+All user-facing text uses **sentence case** — capitalize only the **first word**
+and any proper nouns/acronyms. Do **not** use Title Case (capitalizing every
+significant word).
+
+This applies to every string a user reads: field labels, buttons, tab/menu
+items, headings, section titles, modal titles, placeholders, tooltips, table
+column headers, empty states, and toast/notification copy.
+
+| Title Case (avoid) | Sentence case (use) |
+|--------------------|---------------------|
+| `Data Source` | `Data source` |
+| `Chart Name` | `Chart name` |
+| `Add Series` | `Add series` |
+| `Count of Events` | `Count of events` |
+| `Save Changes` | `Save changes` |
+| `Delete Dashboard` | `Delete dashboard` |
+
+**Keep the original casing of proper nouns, product names, and acronyms**
+anywhere in the string — sentence case only changes the words around them:
+HyperDX, ClickHouse, ClickStack, OpenTelemetry/OTel, Lucene, SQL, PromQL,
+MongoDB, Kubernetes, JSON, CSV, URL, ID, API, MCP, CPU, P95. For example:
+`Search your events w/ Lucene`, `Edit SQL`, `Copy as cURL`, `View in ClickHouse`.
+
+Only sentence-case **static UI chrome**. Never rewrite dynamic/user data (column
+names, tag values, log/trace content, source or dashboard names a user typed) —
+render those verbatim.
+
## Semantic design tokens (prefer over raw Mantine colors)
The UI is built with **Mantine components**, but **colors and surfaces** should follow the **semantic CSS custom properties** in our themes (`--color-*`, etc.), not ad-hoc Mantine palette values. Those tokens are defined in `packages/app/src/theme/themes/**/_tokens.scss`, align with a **Click UI**–style system, and keep HyperDX and ClickStack visually consistent. They are the path toward a shared design system even while Mantine remains the component layer.
@@ -137,6 +406,32 @@ The UI is built with **Mantine components**, but **colors and surfaces** should
Mantine theme overrides in `packages/app/src/theme/**` may map Mantine’s scale to our palette; that does not replace using **`var(--color-...)`** in new styling where you need explicit color control.
+## Click UI (incremental adoption)
+
+`@clickhouse/click-ui` is a dependency of `packages/app`. `ThemeWrapper` mounts
+`ClickUIProvider` inside `MantineProvider` with `persistTheme={false}`, so it
+follows the HyperDX color mode and never writes its own theme to localStorage.
+Storybook and `renderWithMantine` in tests get the provider for free.
+
+- **Do**: Prefer a click-ui component over a new Mantine one when click-ui has
+ an equivalent for a simple primitive (`Link`, `Text`, `Title`, `Container`,
+ `Badge`, `Separator`, `Spacer`, `Icon`). Import from `@clickhouse/click-ui`.
+- **Do not**: Mix both libraries inside one small component, and do not
+ replace working Mantine code outside the change you were asked to make.
+- **`Container` is not a `Box`**: click-ui has no plain-wrapper equivalent of
+ Mantine `Box`. `Container` is a flex layout, closer to `Group` or `Stack`. It
+ defaults to `display="flex"`, `orientation="horizontal"`, and
+ `alignItems="center"`, so a direct swap puts the children in one row. It also
+ has no `m`/`p`/`w`-style props, only `gap` and `padding` tokens. To replace a
+ `Box` that only carries attributes such as `id` or `data-testid`, use a plain
+ HTML element. To replace a `Box` that uses Mantine style props, keep the
+ `Box`, or set `Container` props to match the layout and check it visually.
+- **Styling**: click-ui styles live in the `clickui` CSS cascade layer, so any
+ unlayered app or Mantine rule wins over them. Override `--click-*` variables
+ outside the layer if a token must change.
+- **First use**: `packages/app/src/components/ContactSupportText.tsx` and the
+ `Click UI/Link` story in `packages/app/src/stories/ClickUI.stories.tsx`.
+
**Chart and visualization colors are a separate, more specific contract** — they have their own categorical, semantic, and heatmap palettes wired through `packages/app/src/utils.ts` helpers. Don't hard-code series colors or reuse `--color-text-*` for charts. See [`data_viz_colors.md`](./data_viz_colors.md) before touching anything that renders data.
## Refactoring
diff --git a/agent_docs/data_viz_colors.md b/agent_docs/data_viz_colors.md
index 72438f54c1..53fec06188 100644
--- a/agent_docs/data_viz_colors.md
+++ b/agent_docs/data_viz_colors.md
@@ -1,114 +1,224 @@
# Data Visualization Colors
-> Single source of truth for chart and visualization colors in HyperDX.
-> Read this before adding, changing, or hard-coding a color in any chart,
-> sparkline, heatmap, legend, status pill, or other data display.
+> Single source of truth for chart and visualization colors in HyperDX. Read
+> this before adding, changing, or hard-coding a color in any chart, sparkline,
+> heatmap, legend, status pill, or other data display.
## TL;DR
-There are **three** color systems for data viz, with three different
-consumption patterns:
+There are **three** color systems for data viz, with three different consumption
+patterns:
-| System | Use for | Source of truth | How to consume |
-| ------------------------------ | ---------------------------------------- | ---------------------------------------- | --------------------------------------------- |
-| **Categorical 1–10** | Multi-series line/bar/area/pie charts | CSS vars `--color-chart-1`..`-10` | `getColorProps(index, label)` in `utils.ts` |
-| **Semantic (success/warn/err)**| Status indicators, log levels, deltas | CSS vars `--color-chart-{success,...}` | `getChartColorSuccess/Warning/Error()` |
-| **Heatmap continuous** | `DBHeatmapChart` density gradients | `darkPalette`/`lightPalette` arrays | Imported directly from `DBHeatmapChart.tsx` |
+| System | Use for | Source of truth | How to consume |
+| ------------------------------------ | ------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
+| **Categorical (10 hues)** | Multi-series line/bar/area/pie charts | `CATEGORICAL_HEX_BY_TOKEN` in `utils.ts` (CSS vars mirror it) | `getColorProps(index, label)` (positional) / `getColorFromCSSToken('chart-{hue}')` (by name) |
+| **Semantic (success/warn/err/info)** | Status indicators, log levels, deltas | CSS vars `--color-chart-{success,warning,error,info}` | `getChartColor{Success,Warning,Error,Info}()` |
+| **Heatmap continuous** | `DBHeatmapChart` density gradients | `darkPalette`/`lightPalette` arrays | Imported directly from `DBHeatmapChart.tsx` |
**Hard rules**:
- **Never** pass a hex color to a chart series. Always go through one of the
helpers above so theme switching works.
-- **Never** map log levels to raw Mantine colors (`red.5`, `yellow.6`).
- Use `logLevelColor()` / `getColorProps()` — they pick the theme-correct
- semantic chart color.
+- **Never** map log levels to raw Mantine colors (`red.5`, `yellow.6`). Use
+ `logLevelColor()` / `getColorProps()` — they pick the theme-correct semantic
+ chart color.
- The categorical palette and the heatmap palette are **different things**.
- Don't reuse `--color-chart-N` for heatmap density; don't reuse the heatmap
+ Don't reuse `--color-chart-{hue}` for heatmap density; don't reuse the heatmap
arrays for series colors.
## Where the colors live
-### Categorical series palette (`--color-chart-1` through `--color-chart-10`)
-
-| Theme | File | Index 1 (primary) |
-| ----------- | ----------------------------------------------------------------------- | ----------------- |
-| HyperDX | `packages/app/src/theme/themes/hyperdx/_tokens.scss` (lines ~95–117) | `#00c28a` brand green |
-| ClickStack | `packages/app/src/theme/themes/clickstack/_tokens.scss` (lines ~175–201)| `#437eef` Observable blue |
-
-The same ten slots in both themes use the same hue families (blue, orange,
-red, cyan, green, pink, purple, light blue, brown, gray) — only **slot 1**
-differs:
-
-- **HyperDX** leads with brand green, then Observable colors.
-- **ClickStack** leads with Observable blue (Click UI accent yellow doesn't
- pass contrast on a typical chart background, so we don't use it as a series
- color — see "Per-theme considerations" below).
-
-The vars are defined identically inside the dark and light selectors. That
-duplication is intentional and called out in `_tokens.scss`: CSS specificity
-requires it because the parent selectors `[data-mantine-color-scheme='dark']`
-and `[data-mantine-color-scheme='light']` would otherwise drop the vars on
-scheme switch.
-
-### Semantic chart colors
+### Categorical palette (`--color-chart-{hue}`)
+
+The categorical palette is **unified across themes** — based on Observable 10
+([d3 schemeObservable10](https://observablehq.com/@d3/color-schemes)), with
+`chart-blue` swapped to `#437eef` so it matches the brand link color
+(`--click-global-color-text-link-default` for ClickStack, same hue for HyperDX
+series). All other hues are straight from Observable 10. Identical on HyperDX
+and ClickStack:
+
+| Token | Hex |
+| ------------------ | --------- |
+| `chart-blue` | `#437eef` |
+| `chart-orange` | `#efb118` |
+| `chart-red` | `#ff725c` |
+| `chart-cyan` | `#6cc5b0` |
+| `chart-green` | `#3ca951` |
+| `chart-pink` | `#ff8ab7` |
+| `chart-purple` | `#a463f2` |
+| `chart-light-blue` | `#97bbf5` |
+| `chart-brown` | `#9c6b4e` |
+| `chart-gray` | `#9498a0` |
+
+**Source of truth for categorical hues lives in JS.** The matching
+`--color-chart-{hue}` CSS vars in both
+`packages/app/src/theme/themes/hyperdx/_tokens.scss` and
+`packages/app/src/theme/themes/clickstack/_tokens.scss` are a stylesheet-author
+affordance only — `getColorFromCSSVariable` and `getColorFromCSSToken` skip
+`getComputedStyle` for categorical tokens and return values straight from
+`CATEGORICAL_HEX_BY_TOKEN` in `utils.ts`. The palette is the same on every theme
+today, so a DOM round-trip per series buys nothing.
+
+The CSS vars exist for:
+
+- SCSS modules and inline `style={{ background: 'var(--color-chart-blue)' }}`
+ consumers (no JS import needed).
+- Devtools inspection while debugging chart styling.
+- Forward-compat: if a future brand wants to override hues, switch
+ `getColorFromCSSVariable`/`getColorFromCSSToken` back to reading the var and
+ add per-brand entries to `CATEGORICAL_HEX_BY_TOKEN`.
+
+The categorical hues and semantic chart colors live in a single shared partial,
+`packages/app/src/theme/themes/_chart-categorical-tokens.scss` (`chart-categorical-tokens`
+and `chart-semantic-tokens` mixins). Both brand themes `@use` it and `@include`
+both mixins inside their per-theme `chart-tokens` mixin. Each theme's
+`chart-tokens` mixin is then `@include`'d inside both `[data-mantine-color-scheme]`
+selectors. Sass inlines the bodies at each call site, so the emitted CSS has the
+same per-scheme specificity as a hand-duplicated block would — but the source
+lives in **one place** for both layers. **If you change a categorical hex in the
+shared partial, change it in `CATEGORICAL_HEX_BY_TOKEN` in `utils.ts` too. If
+you change a semantic hex, update `SEMANTIC_CHART_PALETTE` in `utils.ts` too —
+the SCSS and JS sources are intentionally mirrored.**
+
+Brand identity for charts is carried by non-chart UI chrome (Mantine accent,
+sidebar gradient, Click UI globals), not by per-brand chart semantic colors —
+`success` and `info` reuse categorical `chart-green` and `chart-blue` on both
+brands.
+
+### Semantic chart colors (`--color-chart-{success|warning|error|info}`)
```text
---color-chart-success # success / OK / ingested / info-level fills
+--color-chart-success # success / OK / ingested fills
--color-chart-warning # warnings, throttling, slowdowns
--color-chart-error # failures, errors, alerts firing
+--color-chart-info # info-level logs, neutral "primary" series
--color-chart-success-highlight # hover/selected variants (lighter shades)
--color-chart-warning-highlight
--color-chart-error-highlight
```
-Defined in both `_tokens.scss` files. **HyperDX success uses brand green
-(`#00c28a`)**; **ClickStack success uses Observable green (`#3ca951`)** so it
-doesn't collide with the yellow brand accent. Warning and error are the same
-across themes (orange `#efb118`, red `#ff725c`).
+Defined in both `_tokens.scss` files. **Unified on both brands**: `success`
+reuses categorical `chart-green` (`#3ca951`); `info` reuses categorical
+`chart-blue` (`#437eef`). Warning and error use the same Observable hexes across
+themes.
-### JavaScript fallback (`packages/app/src/utils.ts`)
+Unlike the categorical hues, **the semantic CSS vars are read at runtime** via
+`getComputedStyle` (see `getSemanticChartColor` in `utils.ts`). That keeps
+inline `var(--color-chart-warning)` consumers like `LogLevel.tsx` reacting to
+theme switches without a React re-render, and it lets JS callers like
+`getChartColorSuccess()` return the correct hex for the active theme.
-The CSS vars are the source of truth at runtime, but two palette objects in
-`utils.ts` are the SSR fallback **and** the storybook reference:
+### Single source of truth in TS (`packages/app/src/utils.ts`)
```text
-CHART_PALETTE # HyperDX (green-first), lines ~356-374
-CLICKSTACK_CHART_PALETTE # ClickStack (blue-first), lines ~376-393
-COLORS # Exported, ordered, HyperDX-default array, lines ~398-409
+CATEGORICAL_HEX_BY_TOKEN # { 'chart-blue': '#437eef', ... } — file-private,
+ # authoritative for categorical hues
+SEMANTIC_CHART_PALETTE # { hyperdx: {...}, clickstack: {...} } — file-private,
+ # SSR/fallback for the semantic CSS vars
+COLORS # ordered hex array, derived from CATEGORICAL_PALETTE_TOKENS
```
-`COLORS[0]` corresponds to `--color-chart-1`, `COLORS[1]` to `--color-chart-2`,
-and so on. **Keep them in sync.** If you change a hex in one place, change it
-in all three (HyperDX SCSS, ClickStack SCSS, and `CHART_PALETTE` /
-`CLICKSTACK_CHART_PALETTE` / `COLORS`).
+`COLORS[i]` equals `CATEGORICAL_HEX_BY_TOKEN[CATEGORICAL_PALETTE_TOKENS[i]]`.
+`COLORS` is what `getColorFromCSSVariable(i)` returns — on both server and
+client, since categorical hues no longer round-trip through the CSS var.
+
+The hue-named `CHART_PALETTE_TOKENS`, `CATEGORICAL_PALETTE_TOKENS`, and
+`SEMANTIC_PALETTE_TOKENS` constants live in `packages/common-utils/src/types.ts`
+so the Zod schema can reference them (shared with the API).
### Reader functions (`packages/app/src/utils.ts`)
These are the only functions React code should call:
-| Function | Returns |
-| -------------------------------- | ---------------------------------------------------- |
-| `getColorProps(index, level)` | Categorical color, with log-level override applied |
-| `semanticKeyedColor(key, index)` | Same, but driven by `key` (e.g. series name) |
-| `getChartColorSuccess()` | `var(--color-chart-success)` resolved to a hex string|
-| `getChartColorWarning()` | `var(--color-chart-warning)` resolved |
-| `getChartColorError()` | `var(--color-chart-error)` resolved |
-| `getChartColor*Highlight()` | Hover/selected variants |
-| `logLevelColor(key)` | Maps `'error' \| 'warn' \| 'info'` → semantic color |
-| `getLogLevelColorOrder()` | Stable ordering for log-level series |
+| Function | Returns |
+| -------------------------------- | ----------------------------------------------------------- |
+| `getColorProps(index, level)` | Categorical color by index, with log-level override applied |
+| `semanticKeyedColor(key, index)` | Same, but driven by `key` (e.g. series name) |
+| `getColorFromCSSToken(token)` | Resolves any `ChartPaletteToken` (categorical or semantic) |
+| `getChartColorSuccess()` | `var(--color-chart-success)` resolved to a hex string |
+| `getChartColorWarning()` | `var(--color-chart-warning)` resolved |
+| `getChartColorError()` | `var(--color-chart-error)` resolved |
+| `getChartColorInfo()` | `var(--color-chart-info)` resolved (brand-primary for info) |
+| `getChartColor*Highlight()` | Hover/selected variants |
+| `logLevelColor(key)` | Maps `'error' \| 'warn' \| 'info'` → semantic color |
+| `getLogLevelColorOrder()` | Stable ordering for log-level series |
Internals worth knowing:
-- `getColorFromCSSVariable(index)` reads `--color-chart-{index+1}` from
- `documentElement` via `getComputedStyle`. On SSR or if the var is missing,
- it falls back to `COLORS[index % COLORS.length]`.
-- `getSemanticChartColor(varName, hyperdxFallback, clickstackFallback)` does
- the same for the semantic vars and uses `detectActiveTheme()` (checks for
- the `theme-clickstack` class on ``) to pick the correct fallback when
- CSS isn't available.
-- During SSR, semantic readers return the **HyperDX** default — the
- hydration mismatch window is tiny because charts render after data fetch
- on the client.
+- `getColorFromCSSVariable(index)` returns `COLORS[index % 10]` directly — no
+ DOM read. Categorical hues are unified across themes, so the JS palette is
+ authoritative.
+- `getColorFromCSSToken(token)` is split:
+ - Categorical tokens (`chart-blue`, etc.) come straight from
+ `CATEGORICAL_HEX_BY_TOKEN` — same shortcut, no `getComputedStyle`.
+ - Semantic tokens (`chart-success`, `-warning`, `-error`) read
+ `--color-{token}` from `documentElement` via `getComputedStyle` and fall
+ back to the active theme's `SEMANTIC_CHART_PALETTE` entry when running
+ server-side or when the DOM read fails.
+- `getSemanticChartColor(varName, key)` is the shared helper that backs the
+ `getChartColor{Success,Warning,Error,Info}()` readers. It uses
+ `detectActiveTheme()` (checks for the `theme-clickstack` class on ``) to
+ pick the correct per-brand fallback.
+- During SSR, semantic readers return the **HyperDX** default — the hydration
+ mismatch window is tiny because charts render after data fetch on the client.
+
+### Legacy `chart-1` … `chart-10` tokens
+
+The number-tile color picker (#2265) initially shipped with numeric tokens
+(`chart-1` … `chart-10`). Renamed here to hue-named tokens so stored configs and
+the upcoming external API surface are self-documenting.
+
+Existing stored configs keep working. The mapping:
+
+```text
+chart-1 -> chart-green (was HyperDX brand green at slot 1)
+chart-2 -> chart-blue (was HyperDX slot 2)
+chart-3 -> chart-orange
+...
+chart-10 -> chart-gray
+```
+
+It preserves the HyperDX slot ordering from #2265, so HyperDX users see no
+visual change.
+
+> **⚠️ ClickStack legacy color caveat.** Pre-rename ClickStack used a different
+> slot ordering than HyperDX (`--color-chart-1` was brand blue `#437eef`, not
+> brand green; `--color-chart-2` was orange, not blue; etc.). Because the
+> migration map preserves HyperDX slot ordering, any ClickStack dashboard saved
+> via #2265 will visually shift: stored `chart-1` flips from brand blue to
+> Observable green, `chart-2` flips from orange to blue, and so on. We chose
+> this trade-off deliberately over branching the legacy map by active theme:
+> `LEGACY_CHART_PALETTE_TOKEN_MAP` lives in `common-utils` (shared with the
+> API), and migration is one-shot persisted on next save — theme-branching would
+> couple common-utils to browser DOM state and still produce wrong results for
+> users whose active theme changed since the original pick. Affected users can
+> manually re-pick the desired hue via the (now hue-labeled) color picker. Use
+> `chart-info` semantic if you need the brand-primary appearance.
+
+**`ChartPaletteTokenSchema` itself stays strict** (a plain `z.enum`). Wrapping
+it in `z.preprocess` would force the schema's `z.input` type to `unknown`, which
+poisons `validateRequest`'s `req.body` inference in the API package all the way
+up to `Dashboard.tiles[i].config.color`. Strict input/output equality is more
+important than a one-line runtime migration buried in the schema.
+
+Migration happens at two complementary layers instead:
+
+1. **Fetch-time _and_ write-time** — `normalizeDashboardTileColors` in
+ `packages/app/src/dashboard.ts` walks every tile and rewrites any legacy
+ `config.color` to its hue-named equivalent via `resolveChartPaletteToken`. It
+ runs on every read (`useDashboards` / `fetchLocalDashboards`) and on every
+ write (`useUpdateDashboard` / `useCreateDashboard`). The write-time pass is
+ what lets JSON imports (`DBDashboardImportPage`), presets, and
+ MCP-constructed payloads pass the strict server-side validator, and it
+ converges the DB-side data on next save instead of leaving legacy tokens in
+ storage forever.
+2. **Render-time** — `DBNumberChart` and `ColorSwatchInput` also call
+ `resolveChartPaletteToken` as belt-and-suspenders for tiles constructed in
+ memory between fetch and save (e.g. `ChartEditor` form state, unit-test
+ fixtures, hand-rolled `Tile` literals).
+
+`LEGACY_CHART_PALETTE_TOKEN_MAP` and `resolveChartPaletteToken` live in
+`packages/common-utils/src/types.ts` next to the enum.
### Heatmap palette (component-local)
@@ -119,9 +229,9 @@ Internals worth knowing:
- Selected at the call site by `useMantineColorScheme()` and
`colorScheme === 'light' ? lightPalette : darkPalette`.
-These are **scheme-aware, not brand-aware**: HyperDX dark and ClickStack
-dark share the same heatmap gradient, same for light. Red is intentionally
-omitted from the high end so it can be reserved for error overlays.
+These are **scheme-aware, not brand-aware**: HyperDX dark and ClickStack dark
+share the same heatmap gradient, same for light. Red is intentionally omitted
+from the high end so it can be reserved for error overlays.
`DBHeatmapChart.tsx` re-exports `darkPalette` and `lightPalette`.
`DBSearchHeatmapChart.tsx` imports them via that re-export — **do not**
@@ -133,55 +243,31 @@ A few component-local accents that are **not** part of the categorical or
semantic palettes:
- `ALL_SPANS_COLOR = 'var(--mantine-color-blue-6)'` in
- `packages/app/src/components/deltaChartUtils.ts` — the "all spans"
- reference bar in `DBDeltaChart`. Keep using this var; don't replace it
- with `--color-chart-1` (it's a comparison reference, not a series).
-- Trace waterfall span tints in `DBTraceWaterfallChart.tsx` — derived from
- span attributes, not from this palette.
+ `packages/app/src/components/deltaChartUtils.ts` — the "all spans" reference
+ bar in `DBDeltaChart`. Keep using this var; don't replace it with a
+ categorical token (it's a comparison reference, not a series).
+- Trace waterfall span tints in `DBTraceWaterfallChart.tsx` — derived from span
+ attributes, not from this palette.
## Storybook reference
-The visual reference for the categorical and semantic palettes is the
-storybook story at:
-
-```1:22:packages/app/src/theme/ChartColors.stories.tsx
-import React from 'react';
-
-import {
- COLORS,
- getChartColorError,
- getChartColorSuccess,
- getChartColorWarning,
-} from '@/utils';
-
-// Labels for chart colors - brand green first, then Observable palette
-const COLOR_LABELS = [
- 'Green (Brand)',
- 'Blue',
- 'Orange',
- 'Red',
- 'Cyan',
- 'Pink',
- 'Purple',
- 'Light Blue',
- 'Brown',
- 'Gray',
-];
-```
+The visual reference for the categorical and semantic palettes is the storybook
+story at `packages/app/src/theme/ChartColors.stories.tsx`. It renders
+`AllChartColors`, `BarChartPreview`, `LineChartPreview`,
+`SemanticColorsPreview`, and `AccessibilityCheck`. Run storybook in the `app`
+package to inspect both schemes side by side.
-It renders `AllChartColors`, `BarChartPreview`, `LineChartPreview`,
-`SemanticColorsPreview`, and `AccessibilityCheck`. Run storybook in the
-`app` package to inspect both schemes side by side.
+The number-tile color picker (`ColorSwatchInput.stories.tsx`) renders the same
+tokens through the user-facing picker UI.
## How to consume (recipes)
### Multi-series time-series chart
`ChartUtils.tsx → setLineColors` already wires the categorical palette per
-series via `getColorProps(index, level)` in
-`packages/app/src/ChartUtils.tsx`. **You should not have to think about
-chart colors when adding a new chart that goes through `seriesToTimeSeries`
-/ `setLineColors`** — they handle it.
+series via `getColorProps(index, level)` in `packages/app/src/ChartUtils.tsx`.
+**You should not have to think about chart colors when adding a new chart that
+goes through `seriesToTimeSeries` / `setLineColors`** — they handle it.
If you're rendering a custom chart outside that pipeline:
@@ -195,9 +281,26 @@ const series = data.map((s, i) => ({
}));
```
-`level` (the second arg) is used to override with semantic colors when the
-label looks like a log level (`'error'`, `'warn'`, `'info'`, etc.). Pass
-`s.label` if it might encode a log level, otherwise pass an empty string.
+`level` (the second arg) is used to override with semantic colors when the label
+looks like a log level (`'error'`, `'warn'`, `'info'`, etc.). Pass `s.label` if
+it might encode a log level, otherwise pass an empty string.
+
+### Identity color ("this thing is always blue")
+
+For UI surfaces that should always render a specific hue regardless of
+multi-series ordering:
+
+```tsx
+import { getColorFromCSSToken } from '@/utils';
+
+Always blue;
+```
+
+Or directly in CSS:
+
+```tsx
+
+```
### Status pill / delta indicator
@@ -209,8 +312,8 @@ import { getChartColorError, getChartColorSuccess } from '@/utils';
```
These functions return resolved hex strings, so they can be used in inline
-styles or passed to libraries that don't understand CSS vars (e.g.
-`uPlot`'s canvas fills).
+styles or passed to libraries that don't understand CSS vars (e.g. `uPlot`'s
+canvas fills).
If you're styling a DOM element with regular CSS, prefer the var directly:
@@ -218,9 +321,15 @@ If you're styling a DOM element with regular CSS, prefer the var directly:
```
-The var route reacts instantly to theme switches without re-running React.
-Use the function form only when you need a string at compute time
-(canvas/WebGL, library config objects, etc.).
+The var route reacts instantly to theme switches without re-running React. Use
+the function form only when you need a string at compute time (canvas/WebGL,
+library config objects, etc.).
+
+### User-customizable chart color (number tile etc.)
+
+`ColorSwatchInput` stores the choice as a `ChartPaletteToken`. Resolve to a hex
+at paint time via `getColorFromCSSToken(token)`. Never store hex strings in
+chart configs — tokens reflow correctly across themes and color modes.
### Heatmap
@@ -232,15 +341,15 @@ const { colorScheme } = useMantineColorScheme();
const palette = colorScheme === 'light' ? lightPalette : darkPalette;
```
-That's the only correct way to consume the heatmap gradient. Never
-reconstruct it.
+That's the only correct way to consume the heatmap gradient. Never reconstruct
+it.
### Pie / donut where slice order matters
-The categorical palette is **ordered for distinguishability** — adjacent
-slots are designed to contrast. If you're drawing a pie chart where the
-largest slice should always be the most prominent color, sort your data
-first and let the index map to the palette naturally:
+The categorical palette is **ordered for distinguishability** — adjacent slots
+are designed to contrast. If you're drawing a pie chart where the largest slice
+should always be the most prominent color, sort your data first and let the
+index map to the palette naturally:
```tsx
const sorted = data.toSorted((a, b) => b.value - a.value);
@@ -250,125 +359,143 @@ const slices = sorted.map((d, i) => ({
}));
```
-`ChartUtils.tsx → buildPieChartData` already does this — the comment "Sort
-in descending order so the largest slice is always first and gets the
-first color in the palette" is at line ~444.
+`ChartUtils.tsx → buildPieChartData` already does this.
## Per-theme considerations
-### HyperDX (green-first)
+The categorical palette is identical on both themes — Observable 10. Semantic
+`success` and `info` also reuse categorical hues on both brands.
+
+### HyperDX
-- Slot 1 is brand green (`#00c28a`) so single-series charts feel
- on-brand without any extra config.
-- Semantic success **also** uses brand green, so success indicators and
- primary series share a hue. This is intentional but worth knowing:
- if a chart juxtaposes a "success" pill with a green series, that's
- expected, not a bug.
+- `--color-chart-success` uses categorical `chart-green` (`#3ca951`).
+- `--color-chart-info` uses categorical `chart-blue` (`#437eef`), so info-level
+ logs and `getChartColorInfo()` render the same blue as multi-series slot 0.
+- Multi-series charts start at brand blue (slot 0, `#437eef`) and proceed
+ through the canonical palette — brand identity is preserved via the Mantine
+ green accent and sidebar gradient, not via chart semantic colors.
-### ClickStack (blue-first)
+### ClickStack
-- The brand accent is **yellow** (`--palette-brand-300: #faff69`). It is
- **not** in the chart palette and should not be added — yellow on a
- light background fails contrast, and yellow as a series color reads as
- "warning" in most contexts.
-- Slot 1 falls back to Observable blue (`#437eef`).
-- Semantic success is **Observable green** (`#3ca951`), distinct from
- the yellow brand accent. Don't try to "brand" success with yellow.
+- The brand accent is **yellow** (`--palette-brand-300: #faff69`). It is **not**
+ in the chart palette and should not be added — yellow on a light background
+ fails contrast, and yellow as a series color reads as "warning" in most
+ contexts.
+- `--color-chart-success` uses categorical `chart-green` (`#3ca951`).
+- `--color-chart-info` uses categorical `chart-blue` (`#437eef`, same as
+ `--click-global-color-text-link-default`), so info-level logs and
+ `getChartColorInfo()` render the brand blue.
### Both themes
-- The dark/light scheme split is handled by CSS (the same vars get
- redefined inside each scheme selector). React code does not need to
- branch on scheme except for the heatmap palette.
-- The chart vars in `_tokens.scss` are intentionally duplicated across
- the dark and light blocks. If you change one, change the other.
+- The dark/light scheme split is handled by CSS (the same vars get redefined
+ inside each scheme selector). React code does not need to branch on scheme
+ except for the heatmap palette.
+- The chart vars in `_tokens.scss` are intentionally duplicated across the dark
+ and light blocks. If you change one, change the other.
## Adding new entries
-### A new categorical slot (slot 11+)
+### A new categorical hue (11th token)
-Don't, unless you have a real need. Ten distinguishable hues is the upper
-bound for readable categorical legends — beyond that, viewers can't tell
-slices apart, and color-blind viewers definitely can't. Solutions in
-descending order of preference:
+Don't, unless you have a real need. Ten distinguishable hues is the upper bound
+for readable categorical legends — beyond that, viewers can't tell slices apart,
+and color-blind viewers definitely can't. Solutions in descending order of
+preference:
1. **Group small categories** into "Other" before charting.
2. **Reuse slots** with patterns/strokes/labels for disambiguation.
-3. If you really must extend: add the same `--color-chart-N` var to
- **all four** SCSS blocks (HyperDX dark, HyperDX light, ClickStack
- dark, ClickStack light), append the hex to `CHART_PALETTE` /
- `CLICKSTACK_CHART_PALETTE`, append to `COLORS`, append a label to
- `COLOR_LABELS` in `ChartColors.stories.tsx`.
+3. If you really must extend: add `--color-chart-{newhue}` to the shared
+ `_chart-categorical-tokens.scss` partial — one edit covers both brands and
+ both schemes via the existing `@include` chain. Then append
+ `'chart-{newhue}'` to `CHART_PALETTE_TOKENS` in `common-utils/src/types.ts`,
+ add the hex to `CATEGORICAL_HEX_BY_TOKEN` in `utils.ts`, and add a label
+ entry in `ColorSwatchInput.tsx` → `TOKEN_LABELS` and
+ `ChartColors.stories.tsx` → `COLOR_LABELS`.
### A new semantic color (e.g. `--color-chart-pending`)
-1. Pick the hex per theme (HyperDX often uses brand variants;
- ClickStack uses Observable variants).
-2. Add `--color-chart-pending` and (if needed)
- `--color-chart-pending-highlight` to **all four** SCSS blocks.
-3. Add a hex to `CHART_PALETTE` and `CLICKSTACK_CHART_PALETTE`.
-4. Add a `getChartColorPending()` reader in `utils.ts` that calls
- `getSemanticChartColor('--color-chart-pending', hyperdxHex, clickstackHex)`.
-5. Update `ChartColors.stories.tsx` `SEMANTIC_CHART_COLORS` so the new
- color shows up in the design-tokens story.
-6. If it should override based on a label / status string, extend
+1. Pick the hex (unified across brands unless you have a deliberate per-brand split).
+2. Add `--color-chart-pending` and (if needed) `--color-chart-pending-highlight`
+ to the `@mixin chart-semantic-tokens` block in
+ `_chart-categorical-tokens.scss`. The mixin is `@include`'d in each theme's
+ `chart-tokens` mixin, which is `@include`'d in each scheme selector — one edit
+ covers dark and light for both brands. For a per-brand override only, declare
+ the var in that brand's `chart-tokens` mixin after the shared `@include`.
+3. Add `pending` (and optionally `pendingHighlight`) to
+ `SEMANTIC_CHART_PALETTE.hyperdx` and `.clickstack` in `utils.ts`.
+4. Append `'chart-pending'` to `CHART_PALETTE_TOKENS` (and
+ `SEMANTIC_PALETTE_TOKENS` slice will pick it up automatically) in
+ `common-utils/src/types.ts`.
+5. Add a `getChartColorPending()` reader in `utils.ts` that calls
+ `getSemanticChartColor('--color-chart-pending', 'pending')`.
+6. Update `ChartColors.stories.tsx` `SEMANTIC_CHART_COLORS` so the new color
+ shows up in the design-tokens story.
+7. Add a `'chart-pending': 'Pending'` entry to `TOKEN_LABELS` in
+ `ColorSwatchInput.tsx`.
+8. If it should override based on a label / status string, extend
`getLevelColor` / `logLevelColor` accordingly.
### A new heatmap palette
The current arrays were tuned to (a) be visible on the chart's dark/light
-background, (b) avoid red at the high end so error overlays stay
-readable, and (c) follow a perceptually monotonic luminance ramp.
-Don't add a new palette without those three properties verified — both
-"physically dim" and "perceptually noisy" gradients hurt readability.
+background, (b) avoid red at the high end so error overlays stay readable, and
+(c) follow a perceptually monotonic luminance ramp. Don't add a new palette
+without those three properties verified — both "physically dim" and
+"perceptually noisy" gradients hurt readability.
If you do need to add one (e.g. for a different chart type), keep it
-component-local like the existing ones, and re-use the same scheme-pick
-pattern (`useMantineColorScheme` + ternary).
+component-local like the existing ones, and re-use the same scheme-pick pattern
+(`useMantineColorScheme` + ternary).
## Anti-patterns
```tsx
- // hex string in a chart
- // CSS color keyword
- // raw Mantine palette for a status
-
-import { CHART_PALETTE } from '@/utils'; // CHART_PALETTE is not exported
+ // hex string in a chart
+ // CSS color keyword
+ // raw Mantine palette for a status
+ // palette object is module-private
```
Why each is wrong:
-- **Hex / keyword**: bypasses theme switching. ClickStack users will see
- HyperDX colors, dark/light won't react.
-- **Raw Mantine**: bypasses semantic mapping. `green.5` is not the same
- as `--color-chart-success` once we tweak the brand palette.
-- **Importing palette objects**: they're module-private. The exported
- surface is the reader functions and `COLORS` (the SSR fallback array,
- not for direct use in components).
+- **Hex / keyword**: bypasses theme switching. Dark/light won't react, and any
+ future per-theme tweak won't propagate.
+- **Raw Mantine**: bypasses semantic mapping. `green.5` is not the same as
+ `--color-chart-success` once we tweak the brand palette.
+- **Importing palette objects**: `CATEGORICAL_HEX_BY_TOKEN` and
+ `SEMANTIC_CHART_PALETTE` are module-private. The exported surface is the
+ reader functions, the token enums, and `COLORS` (the SSR fallback array —
+ prefer `getColorProps` / `getColorFromCSSToken` over indexing it directly).
## Pre-merge checklist for chart-touching PRs
-- [ ] Toggled HyperDX ↔ ClickStack theme — series colors change as
- expected (slot 1 swaps green ↔ blue).
-- [ ] Toggled dark ↔ light — categorical and semantic colors stay legible
- on both backgrounds; heatmap gradient flips palettes.
-- [ ] Status indicators use `getChartColor*()` / `var(--color-chart-*)`,
- not raw Mantine colors.
+- [ ] Toggled HyperDX ↔ ClickStack theme — semantic success and info stay
+ unified (chart-green / chart-blue); categorical palette stays identical.
+- [ ] Toggled dark ↔ light — categorical and semantic colors stay legible on
+ both backgrounds; heatmap gradient flips palettes.
+- [ ] Status indicators use `getChartColor*()` / `var(--color-chart-*)`, not raw
+ Mantine colors.
- [ ] No new hex strings in chart components — all colors flow through
`utils.ts` helpers or CSS vars.
-- [ ] If you added or changed a hex, changed it in **all four** places
- (HyperDX SCSS, ClickStack SCSS, `CHART_PALETTE` /
- `CLICKSTACK_CHART_PALETTE`, `COLORS`).
-- [ ] Storybook `Design Tokens / Chart Colors` still renders correctly.
+- [ ] If you added or changed a categorical hex, changed it in **both** places
+ (`_chart-categorical-tokens.scss` shared partial,
+ `CATEGORICAL_HEX_BY_TOKEN` in `utils.ts`).
+- [ ] If you added or changed a semantic hex, changed it in **both** places
+ (`chart-semantic-tokens` mixin in `_chart-categorical-tokens.scss`,
+ `SEMANTIC_CHART_PALETTE.{theme}` in `utils.ts`).
+- [ ] Storybook `Design tokens / Chart colors` still renders correctly.
## File reference summary
-| What | Where |
-| ------------------------------------------ | --------------------------------------------------------------------------- |
-| HyperDX chart vars (dark + light) | `packages/app/src/theme/themes/hyperdx/_tokens.scss` |
-| ClickStack chart vars (dark + light) | `packages/app/src/theme/themes/clickstack/_tokens.scss` |
-| JS palettes + reader functions | `packages/app/src/utils.ts` |
-| Multi-series wiring (`setLineColors` etc.) | `packages/app/src/ChartUtils.tsx` |
-| Heatmap palettes | `packages/app/src/components/DBHeatmapChart.tsx` (`darkPalette`, `lightPalette`) |
-| Storybook visual reference | `packages/app/src/theme/ChartColors.stories.tsx` |
-| Delta "all spans" reference color | `packages/app/src/components/deltaChartUtils.ts` (`ALL_SPANS_COLOR`) |
+| What | Where |
+| --------------------------------------------------- | -------------------------------------------------------------------------------- |
+| Shared chart vars (10 categorical hues + semantics) | `packages/app/src/theme/themes/_chart-categorical-tokens.scss` |
+| Per-theme `chart-tokens` mixin (`@include` chain) | `packages/app/src/theme/themes/hyperdx/_tokens.scss`, `clickstack/_tokens.scss` |
+| JS palette objects + reader functions | `packages/app/src/utils.ts` |
+| Palette token enum + legacy migration | `packages/common-utils/src/types.ts` |
+| Multi-series wiring (`setLineColors` etc.) | `packages/app/src/ChartUtils.tsx` |
+| Number-tile color picker | `packages/app/src/components/ColorSwatchInput.tsx` |
+| Heatmap palettes | `packages/app/src/components/DBHeatmapChart.tsx` (`darkPalette`, `lightPalette`) |
+| Storybook visual reference | `packages/app/src/theme/ChartColors.stories.tsx` |
+| Delta "all spans" reference color | `packages/app/src/components/deltaChartUtils.ts` (`ALL_SPANS_COLOR`) |
diff --git a/agent_docs/development.md b/agent_docs/development.md
index 3de31ccb92..305c0d9380 100644
--- a/agent_docs/development.md
+++ b/agent_docs/development.md
@@ -18,6 +18,14 @@ yarn dev # or equivalently: make dev
- `make dev-portal`: Start the dev portal manually (auto-started by `yarn dev`)
- `yarn lint`: Run linting across all packages
- `yarn dev:unit`: Run unit tests in watch mode (per package)
+- `yarn dev:mutation --mutate `: Run mutation tests (common-utils only). A
+ surviving mutant means the tests execute that code but don't assert on it.
+ Always scope it with `--mutate`; a whole-package run takes tens of minutes.
+ See [CONTRIBUTING.md](../CONTRIBUTING.md#mutation-tests)
+- `yarn seed:alerts --count N` (in `packages/api`): Fill this worktree's
+ dev-slot Mongo with N alerts for testing the alerts page at scale: half on
+ saved searches, half on dashboard tiles. Seeded dashboards and saved searches
+ are tagged `seeded`; `yarn seed:alerts --purge` removes them.
## Environment Configuration
@@ -33,8 +41,10 @@ multiple agents or developers working in parallel), use `make dev` instead of
### How It Works
-1. A deterministic slot (0-99) is computed from the worktree directory name (via
- `cksum`)
+1. `scripts/slots.sh` computes a deterministic slot (0-99) from the worktree
+ directory name (via `cksum`), or from its full path with `HDX_SLOT_FROM=path`
+ (see below). It is the only place slots and ports are computed: `yarn dev`,
+ `make dev-int`, `make dev-e2e`, and tools all use it.
2. Each service gets a unique port: `base + slot` (see table below)
3. Docker Compose runs with a unique project name (`hdx-dev-`)
4. Volume paths include the slot to prevent data corruption between worktrees
@@ -84,6 +94,36 @@ The portal auto-refreshes every 3 seconds and shows each worktree's:
HDX_DEV_SLOT=5 make dev
```
+### Worktrees With the Same Folder Name
+
+If your worktrees share a folder name (`a/hyperdx`, `b/hyperdx`), they get the
+same slot and clash. Set `HDX_SLOT_FROM=path` in your shell profile to hash each
+worktree's full path instead. It applies to the dev stack, integration tests,
+E2E tests, and the dev portal.
+
+```bash
+export HDX_SLOT_FROM=path
+```
+
+Your existing stacks move to new slots once, so their dev data (in
+`.volumes/*_`) starts empty. The default, `HDX_SLOT_FROM=name`, keeps
+today's slots.
+
+### Tools That Talk to the Dev Stack
+
+Run them through `scripts/with-slots.sh`. It sets this worktree's dev ports
+(`HDX_DEV_CH_HTTP_PORT`, `HYPERDX_API_PORT`, ...) and nothing else. Don't source
+`scripts/dev-env.sh` for this: it also clears `packages/app/.next` and registers
+the stack with the dev portal, which disrupts a running `yarn dev`.
+
+```bash
+# From a package script, e.g. packages/api's "seed:alerts"
+../../scripts/with-slots.sh ts-node scripts/seed-alerts.ts
+
+# Against another slot
+HDX_DEV_SLOT=5 scripts/with-slots.sh
+```
+
## Testing Strategy
### Testing Tools
@@ -246,62 +286,64 @@ yarn run lint
- **Controllers**: `packages/api/src/controllers/`
- **Pages**: `packages/app/pages/`
- **Components**: `packages/app/src/`
-- **Shared Utils**: `packages/common-utils/src/`
+- **Shared Utils**: `packages/common-utils/src/` — no root barrel, so imports
+ are deep `@hyperdx/common-utils/dist/...` paths. See
+ [`code_style.md`](code_style.md#where-shared-code-already-lives) for the
+ module map and the grep-first rule before you add a helper.
## Vercel Preview Deployments (Inline API)
-The `hyperdx-private` Vercel project deploys the `@hyperdx/app` Next.js app
-for preview environments. Normally `/api/*` requests proxy to a separately
-deployed API service (`SERVER_URL`). For PR previews we instead **inline the
-entire Express API** into the Next.js serverless function so a single Vercel
-deployment serves both the app and the API.
+The `hyperdx-private` Vercel project deploys the `@hyperdx/app` Next.js app for
+preview environments. Normally `/api/*` requests proxy to a separately deployed
+API service (`SERVER_URL`). For PR previews we instead **inline the entire
+Express API** into the Next.js serverless function so a single Vercel deployment
+serves both the app and the API.
### How it works
-1. `packages/api/src/serverless.ts` exposes the Express app from
- `api-app.ts` as a stateless `(req, res) => Promise` handler. It
- lazily connects to MongoDB on the first invocation and caches the
- connection across warm invocations.
-2. `packages/app/pages/api/[...all].ts` branches on the
- `HDX_PREVIEW_INLINE_API` env var. When `true`, it `require()`s the
- compiled serverless handler and dispatches directly. Otherwise it falls
- back to the existing http-proxy-middleware path used by Docker/standalone
- builds.
+1. `packages/api/src/serverless.ts` exposes the Express app from `api-app.ts` as
+ a stateless `(req, res) => Promise` handler. It lazily connects to
+ MongoDB on the first invocation and caches the connection across warm
+ invocations.
+2. `packages/app/pages/api/[...all].ts` branches on the `HDX_PREVIEW_INLINE_API`
+ env var. When `true`, it `require()`s the compiled serverless handler and
+ dispatches directly. Otherwise it falls back to the existing
+ http-proxy-middleware path used by Docker/standalone builds.
3. `packages/app/next.config.mjs` adds a webpack `externals` rule that marks
- `@hyperdx/api` as a CommonJS external **unless** `HDX_PREVIEW_INLINE_API`
- is `true`. This keeps production app builds (Docker fullstack image,
- standalone Next output) byte-for-byte equivalent to before — they never
- bundle passport-saml, mongoose, AWS SDK, etc.
+ `@hyperdx/api` as a CommonJS external **unless** `HDX_PREVIEW_INLINE_API` is
+ `true`. This keeps production app builds (Docker fullstack image, standalone
+ Next output) byte-for-byte equivalent to before — they never bundle
+ passport-saml, mongoose, AWS SDK, etc.
4. `vercel.json` declares the repo-root build commands so Vercel builds
`common-utils` → `api` → `app` in order.
### Required Vercel project env vars (Preview scope only)
-| Key | Notes |
-|---|---|
-| `HDX_PREVIEW_INLINE_API` | `true` — turns on the inline path |
-| `MONGO_URI` | Preview MongoDB connection string |
-| `EXPRESS_SESSION_SECRET` | Random 32+ char string |
-| `DISABLED_AUTH_METHODS` | `google,saml` (avoids per-preview OAuth callback URL config) |
-| `AI_PROVIDER`, `AI_API_KEY`, `AI_MODEL_NAME` | Optional, only if AI features should work |
-| `FRONTEND_URL` | Optional. Leave unset to use host-only cookies on the preview URL |
+| Key | Notes |
+| -------------------------------------------- | ----------------------------------------------------------------- |
+| `HDX_PREVIEW_INLINE_API` | `true` — turns on the inline path |
+| `MONGO_URI` | Preview MongoDB connection string |
+| `EXPRESS_SESSION_SECRET` | Random 32+ char string shared by all instances; unset generates one per process, invalidating sessions on restart and between instances |
+| `DISABLED_AUTH_METHODS` | `google,saml` (avoids per-preview OAuth callback URL config) |
+| `AI_PROVIDER`, `AI_API_KEY`, `AI_MODEL_NAME` | Optional, only if AI features should work |
+| `FRONTEND_URL` | Optional. Leave unset to use host-only cookies on the preview URL |
Production env vars on the same project must leave `HDX_PREVIEW_INLINE_API`
unset (or `false`) so production deployments keep using the proxy path.
### Limitations
-- **No background tasks.** Scheduled jobs (`packages/api/src/tasks/`,
- including `CheckAlertTask`) are not run in preview deployments.
+- **No background tasks.** Scheduled jobs (`packages/api/src/tasks/`, including
+ `CheckAlertTask`) are not run in preview deployments.
- **No alerts.** Alert evaluation depends on the cron task runner.
-- **No OPAMP server.** The OPAMP control-plane server (`APP_TYPE=opamp`) is
- not started in preview. The serverless entrypoint asserts
- `APP_TYPE=api` and refuses to boot otherwise.
+- **No OPAMP server.** The OPAMP control-plane server (`APP_TYPE=opamp`) is not
+ started in preview. The serverless entrypoint asserts `APP_TYPE=api` and
+ refuses to boot otherwise.
- **Function size.** The bundled function should stay under Vercel's 50 MB
unzipped limit. If it grows, mark heavy deps (`@aws-sdk/*`,
`@node-saml/passport-saml`, AI SDKs) as `serverExternalPackages` in
`next.config.mjs`.
-- **Cold-start latency.** First request after idle pays ~500–1500 ms for
- the initial Mongo connection.
-- **Shared preview MongoDB.** Unless you partition by database name, all
- preview deployments share the same Mongo state.
+- **Cold-start latency.** First request after idle pays ~500–1500 ms for the
+ initial Mongo connection.
+- **Shared preview MongoDB.** Unless you partition by database name, all preview
+ deployments share the same Mongo state.
diff --git a/agent_docs/evals.md b/agent_docs/evals.md
new file mode 100644
index 0000000000..ff918e3df4
--- /dev/null
+++ b/agent_docs/evals.md
@@ -0,0 +1,372 @@
+# MCP Eval Framework
+
+Read this before running evals, comparing MCP branches, or investigating eval
+results. For the full CLI reference, config schema, scenario details, and
+scoring formula, see [`packages/hdx-eval/README.md`](../packages/hdx-eval/README.md).
+
+## What It Does
+
+The eval framework spawns Claude Code as an SRE agent, gives it access to an
+MCP server, and asks it to solve observability scenarios (find a root cause,
+build a dashboard, etc.) against synthetic telemetry seeded into ClickHouse.
+Answers are graded with programmatic regex checks (40%) + an LLM judge (60%),
+minus a penalty for tool-call errors. The framework is MCP-agnostic — you can
+compare any two (or more) MCP servers side by side.
+
+### Available scenarios
+
+| Scenario | Type | What It Tests |
+|---|---|---|
+| `error-root-cause` | Investigation | Find a cascading DB timeout causing checkout failures |
+| `latency-spike` | Investigation | Identify a segmented p99 spike for enterprise tenants |
+| `noisy-signals` | Investigation | Find load-bearing log patterns among noise |
+| `segmented-regression` | Investigation | Detect a regression hidden by single-axis aggregation |
+| `service-health-check` | Investigation | Generate a peace-time health report |
+| `dashboard-build` | Dashboard creation | Build monitoring dashboards with diverse tile types |
+
+## Prerequisites
+
+| Requirement | Notes |
+|---|---|
+| `yarn dev` running | Each worktree needs `yarn dev` running with a fixed slot. This handles `yarn install`, `yarn build:common-utils`, Docker containers (ClickHouse, MongoDB), and the API server. See [Development Setup](development.md). |
+| `ANTHROPIC_API_KEY` or `AI_API_KEY` | Auto-loaded from `.env.local` at the monorepo root. Check `.env.local` in other worktrees for existing keys if missing. |
+| `claude` CLI | Installed globally (`which claude`). The harness spawns it in streaming JSON mode. |
+| `uv` | Only needed for `stdio`-type MCPs (e.g. raw ClickHouse MCP). Install via `curl -LsSf https://astral.sh/uv/install.sh \| sh`. |
+| Two git worktrees | One checked out to the branch under test, one to `main`. Use `git worktree list` to verify. |
+
+## Dual-Slot A/B Comparison (Primary Workflow)
+
+This is the standard workflow for comparing a feature branch against main.
+
+### Architecture
+
+Evals run from **one worktree** (the branch being tested). The eval harness
+talks to **two separate HyperDX API instances** running on different slots:
+
+```
+ ┌──────────────────────┐
+ │ Eval Harness │
+ │ (this worktree) │
+ │ │
+ │ eval.config.json │
+ │ ┌────────┬────────┐ │
+ │ │branch │ main │ │
+ │ │MCP │ MCP │ │
+ │ └───┬────┴───┬────┘ │
+ └──────┼────────┼──────┘
+ │ │
+ ┌────────────────┘ └────────────────┐
+ ▼ ▼
+ ┌─────────────────────┐ ┌─────────────────────┐
+ │ Slot 98 (main) │ │ Slot 99 (branch) │
+ │ API :30198 │ │ API :30199 │
+ │ CH :30598 │ │ CH :30599 │
+ │ Mongo:30498 │ │ Mongo:30499 │
+ │ MCP :30198/mcp │ │ MCP :30199/mcp │
+ └─────────────────────┘ └─────────────────────┘
+ main worktree branch worktree
+```
+
+Each slot runs its own ClickHouse instance. `setup-hyperdx` (step 2) creates
+the HyperDX account, Connection, and Sources on each instance, but does **not**
+seed telemetry data. The `run` command auto-seeds on first run, but only into
+the single ClickHouse specified in the top-level `clickhouse` config — so for
+dual-slot comparisons you must seed each instance explicitly (step 4b).
+
+### 1. Start both dev stacks
+
+`yarn dev` starts Docker infra (ClickHouse, MongoDB, OTel Collector) **and**
+the API + App servers in one command. Run it from each worktree with a fixed
+slot:
+
+```bash
+# From the main worktree (slot 98)
+HDX_DEV_SLOT=98 yarn dev
+
+# From the branch worktree (slot 99)
+HDX_DEV_SLOT=99 yarn dev
+```
+
+Each command prints a banner with its port assignments. Wait for both to
+finish starting, then verify:
+
+```bash
+curl -s -o /dev/null -w "Slot 98 API: %{http_code}\n" http://localhost:30198/api/v1/me
+curl -s -o /dev/null -w "Slot 99 API: %{http_code}\n" http://localhost:30199/api/v1/me
+# Expected: 404 (no auth token — but the server is responding)
+
+curl -s -o /dev/null -w "Slot 98 CH: %{http_code}\n" http://localhost:30598/
+curl -s -o /dev/null -w "Slot 99 CH: %{http_code}\n" http://localhost:30599/
+# Expected: 200
+```
+
+### 2. Run `setup-hyperdx` against each instance (if needed)
+
+> Run these commands from the worktree that has the eval framework code (the
+> branch under test).
+
+This registers an eval account, creates a ClickHouse Connection, and creates
+per-scenario Sources on each HyperDX instance. The command is **idempotent** —
+it skips anything that already exists, so it's safe to run every time.
+
+**Skip this step** if `eval.config.branch.json` and `eval.config.main.json`
+already exist and pass the check:
+
+```bash
+# Quick check: do saved configs exist and are the MCPs reachable?
+# NOTE: These cp commands temporarily overwrite eval.config.json with a
+# single-instance config. If you already have a combined two-MCP config,
+# you must re-run step 3 (jq merge) afterward.
+cp packages/hdx-eval/eval.config.branch.json packages/hdx-eval/eval.config.json 2>/dev/null && \
+ yarn workspace @hyperdx/hdx-eval dev setup-hyperdx --check && \
+ echo "Branch config OK" || echo "Branch config needs setup"
+
+cp packages/hdx-eval/eval.config.main.json packages/hdx-eval/eval.config.json 2>/dev/null && \
+ yarn workspace @hyperdx/hdx-eval dev setup-hyperdx --check && \
+ echo "Main config OK" || echo "Main config needs setup"
+```
+
+If either needs setup (or this is the first time):
+
+```bash
+# Setup main instance (slot 98)
+yarn workspace @hyperdx/hdx-eval dev setup-hyperdx --api-url http://localhost:30198
+cp packages/hdx-eval/eval.config.json packages/hdx-eval/eval.config.main.json
+
+# Setup branch instance (slot 99)
+yarn workspace @hyperdx/hdx-eval dev setup-hyperdx --api-url http://localhost:30199
+cp packages/hdx-eval/eval.config.json packages/hdx-eval/eval.config.branch.json
+```
+
+### 3. Build the combined eval config
+
+The eval runner needs a single `eval.config.json` with both MCPs so it can
+compare them in one batch. Use `jq` to merge:
+
+```bash
+cat packages/hdx-eval/eval.config.branch.json | jq \
+ --slurpfile main packages/hdx-eval/eval.config.main.json '
+ # Helper: rewrite mcp__hyperdx__ prefixes in toolPattern and deniedTools
+ def reprefix($new):
+ .toolPattern = ("mcp__" + $new + "__*")
+ | if .deniedTools then
+ .deniedTools = [.deniedTools[] | gsub("^mcp__hyperdx__"; "mcp__" + $new + "__")]
+ else . end;
+
+ . as $branch |
+ .mcps = {
+ "hyperdx-branch": ($branch.mcps.hyperdx + {"label": "HyperDX Branch", "enabled": true} | reprefix("hyperdx-branch")),
+ "hyperdx-main": ($main[0].mcps.hyperdx + {"label": "HyperDX Main", "enabled": true} | reprefix("hyperdx-main"))
+ } |
+ .hyperdxApi = $branch.hyperdxApi
+' > packages/hdx-eval/eval.config.json
+```
+
+> **Why the `reprefix` step?** The harness uses the config key (e.g.
+> `hyperdx-branch`) as the MCP server name, so Claude Code registers tools as
+> `mcp__hyperdx-branch__*`. The `toolPattern` and `deniedTools` from the
+> original single-MCP config still say `mcp__hyperdx__*`, which won't match.
+> The `reprefix` helper rewrites both to use the new key.
+
+Verify the config has both MCPs:
+
+```bash
+jq '.mcps | keys' packages/hdx-eval/eval.config.json
+# Expected: ["hyperdx-branch", "hyperdx-main"]
+```
+
+### 4. Verify MCP connectivity
+
+```bash
+BRANCH_KEY=$(jq -r '.hyperdxApi.accessKey' packages/hdx-eval/eval.config.branch.json)
+MAIN_KEY=$(jq -r '.mcps["hyperdx-main"].headers.Authorization' packages/hdx-eval/eval.config.json | sed 's/Bearer //')
+curl -s -o /dev/null -w "Branch MCP: %{http_code}\n" \
+ -H "Authorization: Bearer $BRANCH_KEY" http://localhost:30199/mcp
+curl -s -o /dev/null -w "Main MCP: %{http_code}\n" \
+ -H "Authorization: Bearer $MAIN_KEY" http://localhost:30198/mcp
+# Expected: 406 (SSE endpoint, wrong HTTP method — but proves auth works)
+```
+
+### 5. Seed both ClickHouse instances
+
+`setup-hyperdx` creates the HyperDX account and Sources but does **not** seed
+telemetry data. The `run` command's auto-seed only writes to the single
+ClickHouse in the top-level `clickhouse` config, so you must seed each
+instance explicitly:
+
+```bash
+# Seed the main instance (slot 98, CH on port 30598)
+HDX_DEV_SLOT=98 yarn workspace @hyperdx/hdx-eval dev seed
+
+# Seed the branch instance (slot 99, CH on port 30599)
+HDX_DEV_SLOT=99 yarn workspace @hyperdx/hdx-eval dev seed
+```
+
+The seed command is idempotent — it skips if data already exists. Replace
+`` with the scenario name (e.g. `error-root-cause`). You can seed
+multiple scenarios by running the command once per scenario.
+
+### 6. Run the comparison
+
+```bash
+yarn workspace @hyperdx/hdx-eval dev run \
+ --mcp hyperdx-branch,hyperdx-main \
+ --baseline hyperdx-main \
+ --runs 3 \
+ --max-turns 15 \
+ --timeout 600000 \
+ --judge-model claude-opus-4-7
+```
+
+The eval harness automatically loads `ANTHROPIC_API_KEY` (or `AI_API_KEY`)
+from `.env.local` at the monorepo root via `dotenvx`. If the key isn't there,
+either add it to `.env.local` or export it in your shell. Check `.env.local`
+files in other worktrees or `~/sites/hyperdx/.env.local` for existing keys.
+
+For single-MCP runs, the harness auto-seeds on first run if the scenario
+tables are empty. For dual-slot A/B runs, seed both instances explicitly
+(step 5 above). Subsequent runs reuse existing data and the saved anchor time.
+
+### Recommended flags
+
+| Flag | Recommended Value | Why |
+|---|---|---|
+| `--timeout` | `600000` (10 min) | The default 5 min causes false timeouts on complex scenarios like `dashboard-build`. |
+| `--concurrency` | omit (default 1) | Concurrent runs cause resource contention and false timeouts. Run sequentially. |
+| `--runs` | `3` minimum | Eval results have high variance. 3 runs is the minimum for meaningful comparison. |
+| `--baseline` | `hyperdx-main` | Sets main as the reference for delta computation in reports. |
+| `--judge-model` | `claude-opus-4-7` | Default. Use a strong model for judging. |
+| `--no-judge` | use for fast iteration | Skips the LLM judge (saves cost/time). Programmatic scores only. |
+
+## Other Use Cases
+
+The dual-slot A/B comparison is the primary workflow, but the eval framework
+supports other configurations with a single dev stack.
+
+### Quick score check (single MCP)
+
+Run one MCP against a scenario to get a baseline score without comparison.
+Useful for validating that a scenario works or checking score after a change.
+
+```bash
+# Single slot — just one `yarn dev` needed
+export HDX_DEV_SLOT=98
+yarn workspace @hyperdx/hdx-eval dev setup-hyperdx
+yarn workspace @hyperdx/hdx-eval dev run error-root-cause --mcp hyperdx --runs 1 --no-judge
+```
+
+### Model comparison (same MCP, different models)
+
+Compare how different models perform on the same MCP and data. Useful for
+evaluating model upgrades or cost/quality tradeoffs.
+
+```bash
+yarn workspace @hyperdx/hdx-eval dev run error-root-cause \
+ --mcp hyperdx --model claude-sonnet-4-20250514 --runs 3
+yarn workspace @hyperdx/hdx-eval dev run error-root-cause \
+ --mcp hyperdx --model claude-opus-4-6 --runs 3
+```
+
+### Non-HyperDX MCP (e.g. raw ClickHouse)
+
+The framework works with any MCP. The `clickhouse` stdio MCP is included by
+default in `eval.config.json` after setup. Use it to compare HyperDX's MCP
+against querying ClickHouse directly.
+
+```bash
+yarn workspace @hyperdx/hdx-eval dev run error-root-cause \
+ --mcp hyperdx,clickhouse --baseline clickhouse --runs 3
+```
+
+## Reading Results
+
+Results are written to `packages/hdx-eval/runs//`. Each batch
+contains:
+
+```
+_summary.md # Markdown comparison report
+_summary.json # Structured data for programmatic use
+/
+ /
+ /
+ 0.json # Full run trajectory (tool calls, tokens, timing)
+ 0.grade.json # Grade record (programmatic + judge scores)
+```
+
+Useful inspection commands:
+
+```bash
+# List all batches
+yarn workspace @hyperdx/hdx-eval dev runs-list
+
+# Show a specific run's tool calls and queries
+yarn workspace @hyperdx/hdx-eval dev runs-show --queries
+
+# Show the agent's final answer
+yarn workspace @hyperdx/hdx-eval dev runs-show --final-answer
+
+# Re-grade with only programmatic checks (fast, no API key needed)
+yarn workspace @hyperdx/hdx-eval dev grade --no-judge
+
+# Re-generate the comparison report with a different baseline
+yarn workspace @hyperdx/hdx-eval dev report --baseline hyperdx-main
+
+# Browse results in the web viewer
+yarn workspace @hyperdx/hdx-eval viewer
+```
+
+## Common Pitfalls
+
+### Containers bind to wrong ports
+
+If you run `docker compose up` without sourcing `scripts/dev-env.sh` first,
+containers bind to default ports (8123, 27017) instead of slot-specific ones.
+Always use the `bash -c 'export HDX_DEV_SLOT=... && source scripts/dev-env.sh
+&& docker compose ...'` pattern (see [Cleanup](#cleanup) below).
+
+### Access keys go stale after container restarts
+
+When MongoDB is recreated, the eval account's access key changes. Re-run
+`setup-hyperdx` against the affected instance and rebuild the combined config.
+
+### False timeouts
+
+Two common causes:
+1. **Default 5-minute timeout is too short** for complex scenarios. Use
+ `--timeout 600000` (10 minutes).
+2. **`--concurrency > 1`** causes resource contention between parallel Claude
+ processes. Run sequentially (omit `--concurrency`).
+
+### `eval.config.json` is gitignored
+
+The eval config files (`eval.config.json`, `eval.config.*.json`) are
+gitignored because they contain instance-specific access keys and Source IDs.
+They must be regenerated per environment using `setup-hyperdx`.
+
+### ClickHouse data missing after container recreation
+
+If Docker volumes are removed (e.g. `docker compose down -v`), seeded data is
+lost. The eval harness auto-seeds on the next run if it detects empty tables,
+but this adds time. Use `--reseed` explicitly to force a fresh seed.
+
+### Dev stack not starting
+
+If `yarn dev` fails:
+- Port already in use — another stack on the same slot is running. Check with
+ `lsof -i :301XX` and kill stale processes.
+- Docker not running — start Docker Desktop or OrbStack first.
+- Missing dependencies — run `yarn setup` from the repo root.
+
+## Cleanup
+
+```bash
+# If you started with `yarn dev`, Ctrl-C in each terminal stops everything.
+# If you started Docker separately:
+bash -c 'export HDX_DEV_SLOT=98 && source scripts/dev-env.sh 2>/dev/null && \
+ docker compose -p hdx-dev-98 -f docker-compose.dev.yml down'
+bash -c 'export HDX_DEV_SLOT=99 && source scripts/dev-env.sh 2>/dev/null && \
+ docker compose -p hdx-dev-99 -f docker-compose.dev.yml down'
+
+# Drop eval data from ClickHouse (optional)
+yarn workspace @hyperdx/hdx-eval dev drop
+```
diff --git a/agent_docs/naming_objects.md b/agent_docs/naming_objects.md
new file mode 100644
index 0000000000..4ee29ac077
--- /dev/null
+++ b/agent_docs/naming_objects.md
@@ -0,0 +1,91 @@
+# Naming objects
+
+Every user-named object (dashboard, saved search, dashboard tile, alert, and so
+on) gets its name through one control: `InlineNameInput`. It looks like plain
+title text until you hover or click it, then edits in place. There is no pencil
+button, no double-click, no separate "Save name" button, and no naming step
+inside a save modal.
+
+```tsx
+import {
+ InlineNameInput,
+ InlineNameInputControlled,
+} from '@/components/InlineNameInput/InlineNameInput';
+```
+
+## Pick the variant by how the object is saved
+
+| The object is... | Use | When the name is saved |
+| ------------------------------------------------------------------------ | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Already saved, and the page shows it (dashboard page, saved search page) | `InlineNameInput` with `onCommit` | Enter or blur commits. The typed name stays until `value` updates. Return a rejected promise from `onCommit` to revert it. Escape reverts, and an empty or unchanged name reverts. |
+| A draft in an editor (tile editor, new chart, new saved search) | `InlineNameInputControlled` with the form's `control` | With the rest of the form, when the user presses the editor's save button. |
+
+A rename on a saved object is an arrangement change, like moving a tile, so it
+saves without a confirmation. A name on a draft is part of the draft's content,
+so it waits for the explicit save.
+
+## Size and placement
+
+Sizes go from `xs` to `md`. Don't add larger ones.
+
+- `size="md"`: the page title, top-left of the page body under the breadcrumbs.
+ Pass `headingLevel` so the name is still the page heading for screen readers
+ and `getByRole('heading')`. Use `headingLevel={3}` to match the existing
+ dashboard and saved search pages.
+- `size="sm"` (default): editor headers, next to the close button and before the
+ editor's actions.
+- `size="xs"`: compact rows, such as a name inside a dense form or list.
+
+## Copy
+
+- `aria-label` is required because there is no visible label: "Dashboard name",
+ "Saved search name", "Tile name".
+- The placeholder is "Untitled" plus the object type in sentence case: "Untitled
+ dashboard", "Untitled search", "Untitled tile". Don't use "Name".
+- Don't add a "Name" label or helper text next to the field.
+
+## Examples
+
+Saved object, saves on commit:
+
+```tsx
+ setDashboard({ ...dashboard, name })}
+ placeholder="Untitled dashboard"
+ aria-label="Dashboard name"
+ size="md"
+ headingLevel={3}
+ data-testid="dashboard-name-input"
+/>
+```
+
+Draft, saves with the form:
+
+```tsx
+
+```
+
+## Don't
+
+- Don't build a click-to-edit title with a pencil `ActionIcon` and an edit mode.
+ That was `EditablePageName`, which this replaces.
+- Don't use a bordered `TextInput` for an object's own name in a page or editor
+ header.
+- Don't ask for the name in a modal before saving. Name inline, then save.
+- Don't show a success toast for a rename. Show an error toast if the save
+ fails.
+
+## Testing
+
+- The field is a textbox, so assert with `toHaveValue`, not `toHaveText`.
+- To rename in E2E: `fill()` the input, then press Enter. See
+ `DashboardPage.renameDashboard` in `packages/app/tests/e2e/page-objects/`.
+- Storybook: **Components/InlineNameInput** shows both variants and sizes.
diff --git a/agent_docs/observability.md b/agent_docs/observability.md
new file mode 100644
index 0000000000..a814e3643f
--- /dev/null
+++ b/agent_docs/observability.md
@@ -0,0 +1,365 @@
+# Observability & Instrumentation
+
+HyperDX is an observability product, so our own code should be a shining example
+of well-instrumented software. When you add or change a feature, instrument it
+as part of the change - not as an afterthought.
+
+This guide covers the conventions and the shared helpers. The principles are
+aligned with the
+[Instrumentation Score specification](https://github.com/instrumentation-score/spec/tree/main/rules).
+
+## TL;DR (the non-negotiables)
+
+1. **Every team-scoped operation carries team + user context.** Use
+ `setBusinessContext(...)` so `hyperdx.team.id` / `user.id` end up on
+ the trace. Auth middleware already does this for HTTP requests; background
+ jobs and other entry points must do it themselves.
+2. **If something is worth a log, put it on the span first.** The active span is
+ the wide event for the current unit of work — attach the fact there as an
+ attribute. If it also marks a countable event worth aggregating (an error, a
+ skip, a fired alert, a query), emit a counter or histogram alongside it.
+3. **Cardinality belongs on span _attributes_, not span _names_ or _metrics_.**
+ Never put raw queries, user input, IDs, or error messages into a span _name_
+ or a metric _attribute key/value_ — those must stay low-cardinality. Span
+ _attributes_ are the opposite: enrich them freely with high-cardinality
+ context (IDs, sizes, statuses), because that is what makes a trace queryable
+ after the fact.
+4. **Favor wide events for our own code — metrics stay first-class.** For the
+ instrumentation _in this repo_ we lean on richly-attributed spans: before
+ adding an instrument, ask whether the value belongs on the span for the work
+ already in flight, and put point-in-time / gauge-style values (sizes, counts,
+ depths, current state) there rather than in a gauge. This is an internal
+ engineering preference, not a rule against metrics — counters and histograms
+ remain first-class, feed alerts and SLOs, and many HyperDX deployments depend
+ heavily on them. See [Wide events over gauges](#wide-events-over-gauges).
+5. **Use the shared helpers** in
+ [`packages/api/src/utils/instrumentation.ts`](../packages/api/src/utils/instrumentation.ts).
+ Don't hand-roll tracer/meter lifecycle.
+6. **A new surface that queries ClickHouse declares its query attribution.**
+ One provider or one `withQueryAttribution` scope at the entry point, not a
+ change at each query. See [Query attribution](#query-attribution).
+
+## The shared helper library
+
+All manual instrumentation in `packages/api` goes through
+`@/utils/instrumentation`:
+
+| Helper | Purpose |
+| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
+| `withSpan(name, fn, opts?)` | Run an async unit of work in an active span. Records exceptions, sets OK/ERROR status, always ends the span. |
+| `setBusinessContext({ teamId, userId, email, ...extra })` | Attach standardized incident-remediation context to the trace + active span. |
+| `getStaticFeatureFlags()` | Returns the static (env/compile-time) feature-flag states as `feature_flag.*` attributes. |
+| `getCounter(name, opts?)` / `getHistogram(name, opts?)` | Memoized OTel instrument accessors. Always use these instead of `meter.create*`. |
+| `recordDuration(histogram, fn, attrs?)` | Run `fn`, recording its wall-clock duration on a histogram (even on throw). |
+
+The module also re-exports `SpanKind`, `SpanStatusCode`, and the relevant OTel
+types, so a single import is usually enough.
+
+### Tracing
+
+Rely on the HyperDX SDK auto-instrumentation (HTTP, Express, Mongo, etc.) for
+the common path. Add a **manual span only for a meaningful unit of work** that
+auto-instrumentation can't see - a background job step, a fan-out, an expensive
+computation, an external tool invocation.
+
+```ts
+import { withSpan } from '@/utils/instrumentation';
+
+await withSpan(
+ 'alerts.process_batch',
+ async span => {
+ span.setAttribute('hyperdx.alerts.batch.size', alerts.length);
+ return processBatch(alerts);
+ },
+ { attributes: { 'hyperdx.team.id': teamId } },
+);
+```
+
+Span rules to follow (from the Instrumentation Score spec):
+
+- **Bound span-name cardinality** (SPA-003): span names are constants or use
+ route templates - never interpolate IDs, queries, or URLs into the name. Put
+ those in attributes instead.
+- **Don't over-span** (SPA-001, SPA-005): keep `INTERNAL` spans limited and
+ meaningful. Never create a span per loop iteration or per trivial sub-call -
+ it bloats traces and hurts performance.
+- **Root spans are not `CLIENT`** (SPA-004): an entry point (HTTP handler, job
+ tick) should open a `SERVER`/`INTERNAL` span before issuing outbound `CLIENT`
+ calls. Auto-instrumentation handles this for HTTP; for headless workloads
+ (cron tasks), wrap the work in a top-level span.
+
+### Business context
+
+Attach team/user context as early as possible (at the auth boundary or the top
+of a job). `setBusinessContext` writes both to the whole trace (via the HyperDX
+SDK, requires `HDX_NODE_BETA_MODE=1`) and to the active span.
+
+```ts
+import {
+ getStaticFeatureFlags,
+ setBusinessContext,
+} from '@/utils/instrumentation';
+
+setBusinessContext({
+ teamId: user.team?.toString(),
+ userId: user._id?.toString(),
+ email: user.email,
+ ...getStaticFeatureFlags(),
+});
+```
+
+Standard attribute keys:
+
+| Key | Meaning |
+| -------------------------- | ---------------------------------------------------------- |
+| `hyperdx.team.id` | Owning team (multi-tenancy boundary). Set this everywhere. |
+| `user.id` | Acting user (OTel `user.*` semconv). |
+| `user.email` | Acting user's email (OTel `user.*` semconv). |
+| `hyperdx..` | Domain IDs, e.g. `hyperdx.alert.id`, `hyperdx.source.id`. |
+| `feature_flag.` | Evaluated feature/config flag state. |
+
+### Wide events over gauges
+
+> **Scope:** this is HyperDX's _internal_ instrumentation philosophy for the
+> code in this repo — not a recommendation that users pick wide events over
+> metrics. Metrics are a first-class HyperDX signal that many operators rely on;
+> nothing here discourages them. What follows is simply how _we_ prefer to
+> instrument our own services.
+
+We instrument in the spirit of
+[wide events](https://boristane.com/blog/observability-wide-events-101/): one
+richly-attributed span per unit of work beats a scatter of pre-aggregated
+metrics. Pre-aggregation only answers the questions you thought to ask in
+advance; a wide event lets you slice by any dimension _after_ the incident —
+"aborted uploads, but only from agents on this collector version, over 5 MB" is
+a trace query, not a metric you had the foresight to define.
+
+The practical consequence for our code: **default away from gauges.** A gauge is
+a point-in-time sample of a value, and that value almost always belongs to some
+unit of work — a request body size, a batch length, a queue depth at dequeue, a
+team count used to build a config. Prefer putting it on that operation's span as
+an attribute. There it keeps its correlations (which agent, which team, which
+outcome) and stays queryable across every other attribute on the event; a gauge
+sheds all of that the moment it's recorded.
+
+```ts
+// Don't: a bare gauge, stripped of the context that makes it useful.
+// meter.createObservableGauge('opamp.request.body_size')...
+
+// Do: attach the point-in-time value to the span for the work in flight, where
+// it can be sliced by agent, team, outcome, or any other span attribute.
+span.setAttribute('opamp.request.body_size_bytes', req.body.length);
+span.setAttribute('opamp.teams.count', teams.length);
+```
+
+Metrics still earn their place for **aggregate signals that must exist
+independently of any single event** — availability/latency SLIs, error rates,
+alert thresholds — because those stay correct under trace sampling and are cheap
+to alert on. That's the counters and histograms below. What we skip is the
+gauge: if a value is worth recording at a point in time, the span is where it
+belongs.
+
+The rare exception is an ambient value with no owning operation (a pool size or
+queue depth sampled by a background poller). Even then, prefer emitting a
+periodic wide event — a heartbeat span carrying the readings — over a raw gauge,
+so the readings stay sliceable; fall back to a gauge only when no such event
+exists.
+
+### Metrics
+
+When you write a log line that marks a countable event, add a metric too.
+Counters for occurrences, histograms for durations/sizes. For our own code we
+favor span attributes over gauges (see
+[Wide events over gauges](#wide-events-over-gauges)), but metrics themselves are
+first-class — reach for them for aggregate signals, alerts, and SLOs.
+
+```ts
+import {
+ getCounter,
+ getHistogram,
+ recordDuration,
+} from '@/utils/instrumentation';
+
+const queryErrors = getCounter('hyperdx.search.query_errors', {
+ description: 'Search query failures, labeled by error type.',
+});
+const queryDuration = getHistogram('hyperdx.search.query.duration_ms', {
+ description: 'Search query duration.',
+ unit: 'ms',
+});
+
+const result = await recordDuration(queryDuration, () => runQuery(cfg));
+// ...on a known error:
+queryErrors.add(1, { error_type: chType });
+```
+
+Metric conventions:
+
+- **Naming**: `hyperdx..` (snake_case event), e.g.
+ `hyperdx.alerts.evaluations`, `hyperdx.api.errors`.
+- **Attributes are low-cardinality** (MET-001): use fixed enums
+ (`{ outcome: 'fired' | 'resolved' | 'skipped_silenced' }`), status codes, or
+ bounded error-type strings. Never user IDs, team IDs, raw messages, or query
+ text.
+- **Always pass a `description`** (and `unit` for histograms).
+- **Define instruments at module scope** via `getCounter`/`getHistogram` so they
+ are created once and reused.
+
+### SLOs (operation metrics)
+
+To make a piece of functionality SLO-able, wrap it with `withOperationMetrics`
+(or call `recordOperationOutcome` directly when timing is managed manually, e.g.
+a streaming proxy). Both emit a standard pair of SLI signals tagged with a
+stable, low-cardinality `operation` name and `outcome` (`success` | `error`):
+
+| Metric | Type | Use as |
+| ------------------------------- | --------- | --------------- |
+| `hyperdx.operation.requests` | counter | availability SLI |
+| `hyperdx.operation.duration_ms` | histogram | latency SLI |
+
+```ts
+import { withOperationMetrics } from '@/utils/instrumentation';
+
+const chartConfig = await withOperationMetrics(
+ 'ai.assistant',
+ () => generateChart(prompt),
+ { source_kind: source.kind },
+);
+```
+
+Because every operation reports through the same two metrics, an SLO is just a
+filter on `operation` — availability = `outcome:success` / total, latency =
+the duration histogram. Keep `operation` a constant (e.g. `ai.assistant`,
+`clickhouse_proxy.query`); never interpolate IDs or user input into it. Reach
+for this on functionality with real failure modes worth a target (external
+dependencies, query proxies, AI calls) — not on thin CRUD that the HTTP
+auto-instrumentation and error middleware already cover.
+
+## Query attribution
+
+Every ClickHouse query HyperDX runs is tagged with what asked for it, so a row
+in the customer's `system.query_log` points back to a dashboard tile, a saved
+search, an alert, or an MCP tool. This is the one piece of our instrumentation
+that lands in *their* data rather than ours, so it stays small, carries no
+identity, and must never be able to fail a query.
+
+Two channels carry it, set by `BaseClickhouseClient.query`:
+
+| Where | Shape | What it answers |
+| --- | --- | --- |
+| `log_comment` | JSON payload, e.g. `{"v":1,"surface":"dashboard","dashboard":"…","tile":"…"}` | After the fact: `GROUP BY JSONExtractString(log_comment, 'tile')` over `system.query_log` |
+| `query_id` | `hdx--` | Right now: reading `system.processes` while a query is still running |
+
+The vocabulary lives in
+[`packages/common-utils/src/clickhouse/attribution.ts`](../packages/common-utils/src/clickhouse/attribution.ts).
+`QUERY_SURFACES` is a fixed, short list, because it goes in the `query_id` and
+people group by it. Anything naming one specific dashboard, tile or search goes
+in the id fields instead.
+
+### Declaring it
+
+Not at the call site that runs the query: there are roughly sixty of those, and
+`Metadata` accounts for two dozen on its own. Set it once where the reason for
+the query is known, and everything underneath picks it up.
+
+**Browser** — React context, read by `useClickhouseClient`:
+
+```tsx
+
+ {children}
+
+```
+
+Providers nest, and the innermost value wins, so a dashboard sets its id once
+and each tile adds only its own. A page that needs nothing finer than "which
+page" can use `withAppNavForSurface('…')` as its `getLayout` instead.
+
+Do **not** put attribution on the chart config: the config is the React Query
+cache key, so two surfaces asking the identical question would stop sharing a
+cache entry and query volume would double.
+
+**Server** — a scope around the request or job (`AsyncLocalStorage`, node only):
+
+```ts
+await withQueryAttribution({ surface: 'alert', alert: alert.id }, async () =>
+ processAlert(...),
+);
+```
+
+Wrap the whole request or job, not the query. One call at the top covers
+everything below it, including the materialized-view `EXPLAIN` probes.
+
+Order of precedence: the client's default, then the scope around the current
+request or job, then anything passed with an individual query.
+
+**Browser to API route** — for requests that reach ClickHouse through an API
+route instead of a ClickHouse client. PromQL is the case today: the browser
+calls `/api/v1/prometheus/query{,_range}`, and `/labels` and
+`/label/:name/values` for autocomplete and dashboard filters (tagged with
+`useMetadataQueryAttribution()`), over fetch. The caller sends
+`buildLogComment(attribution)` in the `x-hyperdx-query-attribution` header
+(`QUERY_ATTRIBUTION_HEADER`), and the router parses it with `parseLogComment`
+into a `withQueryAttribution` scope layered over the auth middleware's `api`
+scope. The browser's surface and ids win; `trace` and `label` stay the
+server's. Queries the route runs through a ClickHouse client (label lookups,
+and the `prometheusQuery()`/`prometheusQueryRange()` fallback) pick the scope
+up as usual. Requests proxied to ClickHouse's Prometheus HTTP API on 26.6+ are tagged
+by `clickhousePrometheusRequest`: `log_comment` as a URL setting, `query_id` as
+the `X-ClickHouse-Query-Id` header, because the handler rejects `query_id` as an
+unknown setting in the URL.
+
+### Rules
+
+- **Values from the browser are hints, not proof.** The client controls them.
+ `sanitizeField` drops control characters and caps the length, and the payload
+ only ever becomes JSON strings, never SQL.
+- **It must never throw.** A tag describes the work; it does not gate it. Read
+ fields with `?.` even where the type says they are required, and keep helpers
+ like `getActiveTraceId` from throwing. A half-populated object must not be
+ able to stop an alert from running.
+- **Set it where the reason is known, not just on the page.** A page-level
+ surface is a fallback. A modal or panel rendered outside a tile only gets the
+ page's surface, so give it its own (`chart-preview`) when it has ids to
+ report.
+- **Keep the payload small.** It rides in the URL query string on every browser
+ query. The cap is 1 KB; real payloads are 100 to 200 bytes. Today's fields
+ cannot fill the budget even at full length, so nothing is ever shed. If you
+ add a field, check that still holds: over budget, fields are dropped whole
+ from the end rather than truncated, so the value stays parseable JSON.
+
+## Where to look for examples
+
+- Generic span + status handling: `packages/api/src/utils/instrumentation.ts`
+- Span + metrics wrapper around a unit of work:
+ `packages/api/src/mcp/utils/tracing.ts`
+- Context at the auth boundary: `packages/api/src/middleware/auth.ts`
+- Error counter: `packages/api/src/middleware/error.ts`
+- Outcome counters in a background job:
+ `packages/api/src/tasks/checkAlerts/index.ts`
+- Query duration + error metrics:
+ `packages/api/src/routers/external-api/v2/search.ts` and `charts.ts`
+- Scheduled-task metrics pattern: `packages/api/src/tasks/metrics.ts`
+- Outcome counter + span on a non-HTTP entry point:
+ `packages/api/src/opamp/controllers/opampController.ts`
+- Duration + swallowed-error counters where errors never reach the error
+ middleware: `packages/api/src/routers/api/prometheus.ts`
+- Delivery attempt counter + duration around an outbound webhook:
+ `packages/api/src/tasks/checkAlerts/transports/generic.ts`
+- Per-event cap counter on a fan-out loop:
+ `packages/api/src/tasks/checkAlerts/template.ts`
+- Connection lifecycle-event counter: `packages/api/src/models/index.ts`
+- SLO operation metrics (`withOperationMetrics`) on an external dependency:
+ `packages/api/src/routers/api/ai.ts`
+- SLO operation metrics (`recordOperationOutcome`) on a streaming proxy:
+ `packages/api/src/routers/api/clickhouseProxy.ts`
+- End-to-end + sub-operation SLO metrics on a background job (alert evaluation
+ and its data fetch): `packages/api/src/tasks/checkAlerts/index.ts`
+ (`alerts.evaluate`, `alerts.query`)
+- Query attribution scope at the auth boundary:
+ `packages/api/src/middleware/auth.ts` (`nextWithQueryAttribution`)
+- Query attribution around one alert evaluation:
+ `packages/api/src/tasks/checkAlerts/index.ts` (`alertQueryAttribution`)
+- Query attribution per React subtree: `packages/app/src/queryAttribution.tsx`,
+ mounted per tile in `packages/app/src/DBDashboardPage.tsx`
diff --git a/agent_docs/page_layout.md b/agent_docs/page_layout.md
index 18d7274bb2..c1ca244760 100644
--- a/agent_docs/page_layout.md
+++ b/agent_docs/page_layout.md
@@ -35,8 +35,9 @@ Both live next to `withAppNav` in `packages/app/src/layout.tsx`, which wraps pag
| `leading` | Left cluster: source picker, badges, or other controls. When inputs are present, **do not** pair with `title` for the same page name. |
| `actions` | Right-aligned cluster: time range, Run/Save, sampling, refresh. |
| `children` | Full custom header when slots are not enough. Do not combine with `title` / `leading` / `actions` / `breadcrumbs` unless you use the breadcrumbs-only branch. |
+| `growing` | Opt in to block padding that only shows once the toolbar outgrows `min-height`. Sessions uses this for its multi-line search; title-only pages omit it. |
-Header styling is defined in `PageHeader.module.scss`: sticky top, bottom border, horizontal padding `var(--mantine-spacing-sm)` (same as Search `px="sm"`). Single-row headers keep `min-height: 60px`; stacked header (breadcrumbs + toolbar) grows with content.
+Header styling is defined in `PageHeader.module.scss`: sticky top, bottom border, horizontal padding `var(--mantine-spacing-sm)` (same as Search `px="sm"`). Single-row headers keep `min-height: 60px`; stacked header (breadcrumbs + toolbar) grows with content. Pass `growing` when the toolbar can outgrow that min-height (Sessions' multi-line search) so block padding appears only once the row is taller than the bar.
## PageLayout API
@@ -130,6 +131,21 @@ Sessions keeps a **single-row** query toolbar (source, where filter, time range,
- Kubernetes Dashboard, Clickhouse Dashboard — `PageLayout` with `breadcrumbs`, `leading`, `actions`, `padded` (no `title`)
- Client Sessions — `PageLayout` with custom `PageHeader` containing the full single-row query toolbar
+## Compact viewports
+
+A viewport under 900px is compact. `useCompactViewport` (`packages/app/src/components/CompactViewport/useCompactViewport.ts`) is the cutoff, and the nav collapses on the same width. Pages that switch layout use this hook.
+
+A page that is a side panel plus a main pane uses `ResponsiveSidePanel` (`packages/app/src/components/CompactViewport/ResponsiveSidePanel.tsx`):
+
+- Wide: the panel sits beside the main pane.
+- Compact: the main pane uses the full width. The panel opens in a drawer and does not take horizontal space. Pass a `title`; the drawer has no default. The panel stays unmounted until the first open, then stays mounted so closing the drawer does not drop its state.
+
+The caller owns whether the panel is open. Keep that state separate from any desktop collapse preference stored in localStorage. A phone session that wrote the desktop preference would hide the panel the next time the same browser opened the page on a desktop.
+
+Give the main pane `flex: 1` and `min-height: 0` inside a full-height column, so it keeps the space under the toolbar and scrolls inside itself. A hard `min-width` on the panel inside `overflow: hidden` clips the main pane on a phone.
+
+Search is the first page on this pattern. Other side-panel pages should adopt it the same way instead of a one-off media query.
+
## Knip
`packages/app` Knip entry roots are `pages/`, `scripts/`, and e2e tests. After you add or move a `PageLayout` import, run `yarn knip` from the repo root (or `yarn knip` in `packages/app` if configured) so new consumers stay wired.
diff --git a/agent_docs/tech_stack.md b/agent_docs/tech_stack.md
index 403e2cc602..1fb61f34e0 100644
--- a/agent_docs/tech_stack.md
+++ b/agent_docs/tech_stack.md
@@ -2,7 +2,7 @@
## Frontend (`packages/app`)
-- **Framework**: Next.js 14 with TypeScript
+- **Framework**: Next.js 16 with TypeScript
- **UI Components**: Mantine UI library (`@mantine/core`, `@mantine/dates`, `@mantine/hooks`)
- **State Management**: Jotai (global client state), TanStack Query (server state), URL params (filters)
- **Charts/Visualization**: Recharts, uPlot
diff --git a/agent_docs/themes.md b/agent_docs/themes.md
index ea01c96521..e162f9c9a6 100644
--- a/agent_docs/themes.md
+++ b/agent_docs/themes.md
@@ -256,6 +256,55 @@ function TopBar() {
Both hooks return memoized JSX elements so component identity stays
stable across renders.
+## Semantic component variants (Alert / Text / danger controls)
+
+On top of the raw `--color-*` vars, we expose **semantic component
+variants** so callouts and status text stay token-driven and consistent
+across brands and color modes. Reach for these instead of raw Mantine
+palette props (`color="yellow"`, `color="red"`, `c="green"`, …).
+
+- **``** — tinted `-subtle`
+ background with the title, icon, and **body text** in the semantic
+ color.
+- **``** — semantic inline status
+ text.
+- **``** — a *soft* tinted control
+ (`--color-bg-danger-subtle` + hover), not a solid red fill.
+ `warning`/`success` are intentionally not control variants.
+
+The variant → token mapping is centralized in
+**`packages/app/src/theme/themes/semanticVariants.ts`** (single source of
+truth: `SEMANTIC_ALERT_VARS`, `SEMANTIC_TEXT_COLORS`,
+`SEMANTIC_CONTROL_COLORS`), consumed by both brands' `mantineTheme.ts`
+(`Alert.extend`, `Text.extend`, `Button.extend`, `ActionIcon.extend`). It
+references only `--color-*` design tokens — never raw
+`--mantine-color-*`.
+
+### Semantic token families
+
+- **Semantic text**: `--color-text-{danger,warning,success,info}`
+ (+ `--color-text-success-hover`). Used by `Text` and `Alert`
+ foregrounds.
+- **Subtle backgrounds**: `--color-bg-{danger,warning,success,info}-subtle`
+ (+ `-subtle-hover`). Scheme-aware — pale tints in **light** (Mantine
+ `-0`/`-1`), deep tints in **dark** (inlined hex, since the numbered
+ scale's darkest green/yellow shades are mid-tone and can't hit AA).
+ Consumed by `Alert` and the `danger` soft control.
+
+**Accessibility**: every semantic text token clears **WCAG AA (4.5:1)**
+on its `-subtle` tint (including hover) in both schemes. In light mode
+`success`/`warning` text is darkened past the palette with a small
+`color-mix` because `green-9`/`yellow-9`/`orange-9` are mid-tone —
+preserve these `color-mix` adjustments when editing.
+
+**ClickStack `warning` is orange-based**: ClickStack remaps the `yellow`
+palette to its brand gold, so its warning tokens use Mantine `orange`
+(unremapped) for a true orange warning; HyperDX stays yellow-based.
+
+Storybook: `Components/Alert` (interactive `Playground`) and
+`Design tokens/Semantic variants` — use the Brand and Theme toolbar
+toggles to review all combinations.
+
## Per-brand differences
### HyperDX (default, green-first)
@@ -359,6 +408,12 @@ A few things that look like bugs but are intentional:
`theme/semanticColorsGrouped.ts` so the storybook renders it.
5. Consume it via `var(--color-…)` in your component.
+If the token backs a **semantic component variant** (Alert / Text /
+danger control), wire it through `theme/themes/semanticVariants.ts`
+rather than referencing it directly in each brand's `mantineTheme.ts`,
+and verify it in the `Design tokens/Semantic variants` story across both
+brands and schemes.
+
For chart-specific vars (`--color-chart-*`) there are additional steps —
see `data_viz_colors.md`.
@@ -431,8 +486,8 @@ Why each is wrong:
dev) — the change looks correct in both brands.
- [ ] Toggled light ↔ dark via preferences modal — change looks correct
in both modes.
-- [ ] Storybook `Design Tokens / Semantic Colors` and
- `Design Tokens / Chart Colors` still render and look correct.
+- [ ] Storybook `Design tokens / Semantic colors` and
+ `Design tokens / Chart colors` still render and look correct.
- [ ] No new `themeName === 'clickstack'` branches for *colors*; brand
branches are only for *content*, *layout*, or *feature-locked
preferences* (e.g. font picker).
@@ -457,7 +512,8 @@ Why each is wrong:
| ClickStack brand definition | `packages/app/src/theme/themes/clickstack/{index.ts,_tokens.scss,mantineTheme.ts,…}` |
| SSR fallback CSS vars | `packages/app/src/theme/themes/_base-tokens.scss` |
| Semantic var groups (storybook source) | `packages/app/src/theme/semanticColorsGrouped.ts` |
-| Storybook visual references | `packages/app/src/theme/{SemanticColors,ChartColors}.stories.tsx` |
+| Semantic component variants (source of truth)| `packages/app/src/theme/themes/semanticVariants.ts` |
+| Storybook visual references | `packages/app/src/theme/{SemanticColors,SemanticVariants,ChartColors}.stories.tsx` + `packages/app/src/components/Alert.stories.tsx` |
| Favicon assets | `packages/app/public/favicons/{hyperdx,clickstack}/` |
| Provider tests | `packages/app/src/theme/__tests__/ThemeProvider.test.tsx` |
| Theme validation tests | `packages/app/src/theme/__tests__/index.test.ts` |
diff --git a/docker-compose.ci.yml b/docker-compose.ci.yml
index bc65a426e5..bbc07b9799 100644
--- a/docker-compose.ci.yml
+++ b/docker-compose.ci.yml
@@ -22,7 +22,7 @@ services:
depends_on:
- ch-server
ch-server:
- image: clickhouse/clickhouse-server:26.5-alpine
+ image: clickhouse/clickhouse-server:26.8-alpine
environment:
# default settings
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml
index 95e882eca0..7f35762648 100644
--- a/docker-compose.dev.yml
+++ b/docker-compose.dev.yml
@@ -31,6 +31,7 @@ services:
OTEL_COLLECTOR_CORE_VERSION: ${OTEL_COLLECTOR_CORE_VERSION}
environment:
CLICKHOUSE_ENDPOINT: 'tcp://ch-server:9000?dial_timeout=10s'
+ # host:port (plain HTTP), http://host:port or https://host:port
CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT: 'ch-server:9363'
HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE: ${HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE}
HYPERDX_API_KEY: ${HYPERDX_API_KEY}
@@ -40,6 +41,10 @@ services:
# Uncomment to enable stdout logging for the OTel collector
OTEL_SUPERVISOR_LOGS: 'true'
HYPERDX_OTEL_EXPORTER_TABLES_TTL: '24h'
+ # Optional per-signal overrides; each falls back to HYPERDX_OTEL_EXPORTER_TABLES_TTL:
+ # HYPERDX_OTEL_EXPORTER_LOGS_TTL / _TRACES_TTL / _METRICS_TTL / _SESSIONS_TTL
+ # Apply TTL changes to already-existing tables too (default false):
+ # HYPERDX_OTEL_EXPORTER_RECONCILE_TABLE_TTL: 'true'
ENABLE_PROMQL: 'true'
volumes:
- ./docker/otel-collector/config.yaml:/etc/otelcol-contrib/config.yaml
@@ -51,6 +56,12 @@ services:
- '${HDX_DEV_OTEL_GRPC_PORT:-4317}:4317' # OTLP gRPC receiver
- '${HDX_DEV_OTEL_HTTP_PORT:-4318}:4318' # OTLP http receiver
- '${HDX_DEV_OTEL_METRICS_PORT:-8888}:8888' # metrics extension
+ # Uncomment to enable statsd ingestion (also requires adding the
+ # `statsd` receiver to a pipeline in custom.config.yaml - see
+ # packages/otel-collector/README.md). Left commented out by default:
+ # a fixed host port mapping fails the whole service's startup if
+ # something else on the host already owns 8125/udp.
+ # - '${HDX_DEV_OTEL_STATSD_PORT:-8125}:8125/udp' # statsd receiver
restart: always
networks:
- internal
@@ -101,7 +112,7 @@ services:
hdx.dev.service: clickhouse
hdx.dev.port: '${HDX_DEV_CH_HTTP_PORT:-8123}'
hdx.dev.url: 'http://localhost:${HDX_DEV_CH_HTTP_PORT:-8123}'
- image: clickhouse/clickhouse-server:26.5-alpine
+ image: clickhouse/clickhouse-server:26.8-alpine
ports:
- '${HDX_DEV_CH_HTTP_PORT:-8123}:8123' # http api
- '${HDX_DEV_CH_NATIVE_PORT:-9000}:9000' # native
diff --git a/docker-compose.yml b/docker-compose.yml
index abaa35aa7f..9f741fe8d7 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -42,6 +42,14 @@ services:
- '4317:4317' # OTLP gRPC receiver
- '4318:4318' # OTLP http receiver
- '8888:8888' # metrics extension
+ # Uncomment to enable statsd ingestion (also requires adding the
+ # `statsd` receiver to a pipeline via CUSTOM_OTELCOL_CONFIG_FILE - see
+ # packages/otel-collector/README.md). Left commented out by default:
+ # a fixed host port mapping fails the whole service's startup if
+ # something else on the host (e.g. a local StatsD daemon or Datadog
+ # Agent) already owns 8125/udp, which would otherwise break the
+ # default stack for anyone not using this feature.
+ # - '8125:8125/udp' # statsd receiver
restart: always
networks:
- internal
@@ -69,27 +77,39 @@ services:
# OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: 'http://otel-collector:4318/v1/logs'
OTEL_SERVICE_NAME: 'hdx-oss-app'
USAGE_STATS_ENABLED: ${USAGE_STATS_ENABLED:-true}
+ # Comma-separated hostnames or exact IP literals that may receive webhooks.
+ WEBHOOK_HOSTNAME_ALLOWLIST: ${WEBHOOK_HOSTNAME_ALLOWLIST:-}
+ # 32-byte base64 or hex key. Setting it encrypts stored Slack and OAuth
+ # tokens; unset means they are stored in plain text.
+ TOKEN_ENCRYPTION_KEY: ${TOKEN_ENCRYPTION_KEY:-}
+ EXPRESS_SESSION_SECRET: ${EXPRESS_SESSION_SECRET:-}
# Uncomment the next two lines to enable PromQL (Prometheus-compatible metrics)
# ENABLE_PROMQL: 'true'
# NEXT_PUBLIC_ENABLE_PROMQL: 'true'
+ # Uncomment to enable the alert detail page (in development)
+ # NEXT_PUBLIC_ENABLE_ALERT_DETAILS: 'true'
+ # Uncomment and set to append a label to the browser tab title, e.g.
+ # for operators running multiple instances (produces "HyperDX UK")
+ # NEXT_PUBLIC_INSTANCE_LABEL: 'UK'
DEFAULT_CONNECTIONS:
'[{"name":"Local
ClickHouse","host":"http://ch-server:8123","username":"default","password":""}]'
DEFAULT_SOURCES:
- '[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","metadataMaterializedViews":{"keyRollupTable":"otel_logs_key_rollup_15m","kvRollupTable":"otel_logs_kv_rollup_15m","granularity":"15
+ '[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","severityNumberExpression":"SeverityNumber","eventNameExpression":"EventName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","metadataMaterializedViews":{"kvRollupTable":"otel_logs_kv_rollup_15m","granularity":"15
minute"},"connection":"Local
- ClickHouse","traceSourceId":"Traces","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","metadataMaterializedViews":{"keyRollupTable":"otel_traces_key_rollup_15m","kvRollupTable":"otel_traces_kv_rollup_15m","granularity":"15
+ ClickHouse","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","spanEventsValueExpression":"Events","spanLinksValueExpression":"Links","metadataMaterializedViews":{"kvRollupTable":"otel_traces_kv_rollup_15m","granularity":"15
minute"},"connection":"Local
- ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","_id":"682586a8b1f81924e628e808","id":"682586a8b1f81924e628e808"},"connection":"Local
- ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","sessionSourceId":"Sessions"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local
- ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","metricSourceId":"Metrics"}]'
+ ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","summary":"otel_metrics_summary","exponential
+ histogram":"otel_metrics_exponential_histogram"},"connection":"Local
+ ClickHouse","logSourceId":"Logs"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","resourceAttributesExpression":"ResourceAttributes","connection":"Local
+ ClickHouse","traceSourceId":"Traces"}]'
networks:
- internal
depends_on:
- ch-server
- db
ch-server:
- image: clickhouse/clickhouse-server:26.5-alpine
+ image: clickhouse/clickhouse-server:26.8-alpine
# WARNING: Exposing the database port will make it accessible from outside the container,
# potentially allowing unauthorized access. If you uncomment the ports below,
# ensure to secure your database (e.g., with strong authentication, proper network rules, and firewalls).
diff --git a/docker/clickhouse/local/config.xml b/docker/clickhouse/local/config.xml
index aac116287c..9257ef6b0c 100644
--- a/docker/clickhouse/local/config.xml
+++ b/docker/clickhouse/local/config.xml
@@ -59,7 +59,7 @@
remote_write
default
-
+
@@ -67,12 +67,25 @@
remote_read
default
-
+
+
+
+
+ /prometheus/api/v1
+
+ prometheus_api_v1
+
+
+
+
+
system
diff --git a/docker/clickhouse/local/init-db-e2e.sh b/docker/clickhouse/local/init-db-e2e.sh
index 0ac945bf9e..d568410c89 100755
--- a/docker/clickhouse/local/init-db-e2e.sh
+++ b/docker/clickhouse/local/init-db-e2e.sh
@@ -55,6 +55,37 @@ ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)
TTL toDateTime(Timestamp) + toIntervalDay(30)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
+-- Distributed table over the e2e logs table whose DDL declares a column
+-- (\`ColumnMissingFromShard\`) that its underlying MergeTree table does NOT have.
+-- This is the real-world shape of a Distributed table that was ALTERed to add a
+-- column while one of its target tables was left behind: reading the declared
+-- columns works, but \`SELECT *\` (how the app loads full row details) fails with
+-- UNKNOWN_IDENTIFIER on the shard. The 'E2E Distributed Missing Column' fixture
+-- source points here so the E2E tests can assert the row-level error state and
+-- its Known Columns List hint. It routes to e2e_otel_logs, so it reuses that
+-- table's seeded rows and needs no seeding of its own.
+CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_otel_logs_distributed
+(
+ \`Timestamp\` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+ \`TraceId\` String CODEC(ZSTD(1)),
+ \`SpanId\` String CODEC(ZSTD(1)),
+ \`TraceFlags\` UInt8,
+ \`SeverityText\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`SeverityNumber\` UInt8,
+ \`ServiceName\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`Body\` String CODEC(ZSTD(1)),
+ \`ResourceSchemaUrl\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`ResourceAttributes\` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+ \`ScopeSchemaUrl\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`ScopeName\` String CODEC(ZSTD(1)),
+ \`ScopeVersion\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`ScopeAttributes\` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+ \`LogAttributes\` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+ \`EventName\` String CODEC(ZSTD(1)),
+ \`ColumnMissingFromShard\` String CODEC(ZSTD(1))
+)
+ENGINE = Distributed('hdx_cluster', '${DATABASE}', 'e2e_otel_logs', rand());
+
CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_otel_traces
(
\`Timestamp\` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
@@ -209,5 +240,250 @@ ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
TTL toDate(TimeUnix) + toIntervalDay(30)
-SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1
+SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
+
+-- Materialized view rollup for traces, used by E2E tests covering MV
+-- acceleration. The target table pre-aggregates Duration over 1-minute
+-- buckets grouped by the dimension columns ServiceName and StatusCode.
+-- The 'E2E Traces MV' fixture source references e2e_otel_traces_1m.
+CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_otel_traces_1m
+(
+ \`Timestamp\` DateTime,
+ \`ServiceName\` LowCardinality(String),
+ \`StatusCode\` LowCardinality(String),
+ \`count\` SimpleAggregateFunction(sum, UInt64),
+ \`avg__Duration\` AggregateFunction(avg, UInt64),
+ \`max__Duration\` SimpleAggregateFunction(max, UInt64),
+ \`quantile__Duration\` AggregateFunction(quantile(0.95), UInt64)
+)
+ENGINE = AggregatingMergeTree
+ORDER BY (Timestamp, ServiceName, StatusCode)
+SETTINGS index_granularity = 8192;
+
+CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.e2e_otel_traces_1m_mv TO ${DATABASE}.e2e_otel_traces_1m
+AS SELECT
+ toStartOfMinute(Timestamp) AS Timestamp,
+ ServiceName,
+ StatusCode,
+ count() AS count,
+ avgState(Duration) AS avg__Duration,
+ maxSimpleState(Duration) AS max__Duration,
+ quantileState(0.95)(Duration) AS quantile__Duration
+FROM ${DATABASE}.e2e_otel_traces
+GROUP BY
+ Timestamp,
+ ServiceName,
+ StatusCode;
+
+-- Table covering "interesting" filter key edge cases: a column whose name
+-- contains dots (\`__hdx_materialized_k8s.cluster.name\`),
+-- a column whose name contains a hyphen (\`service-name\`), a Map column accessed
+-- by a dotted key (ResourceAttributes['key.subKey.subSubKey']), a JSON column
+-- accessed by a nested path (ResourceAttributesJSON.key.subKey.subSubKey), a Map
+-- column whose NAME contains a hyphen (\`Map-Attributes\`['pod-name']), and a JSON
+-- column whose name AND nested keys contain hyphens (\`JSON-Attributes\`.\`key-1\`.\`key-2\`).
+-- Used by the filter-key edge case E2E tests to verify identifier escaping in filters.
+SET allow_experimental_json_type = 1;
+CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs_interesting_filter_keys
+(
+ \`Timestamp\` DateTime64(9),
+ \`TraceId\` String,
+ \`SpanId\` String,
+ \`SeverityText\` LowCardinality(String),
+ \`ServiceName\` LowCardinality(String),
+ \`Body\` String,
+ \`ResourceAttributes\` Map(LowCardinality(String), String),
+ \`ResourceAttributesJSON\` JSON,
+ \`__hdx_materialized_k8s.cluster.name\` LowCardinality(String),
+ \`service-name\` String,
+ \`Map-Attributes\` Map(LowCardinality(String), String),
+ \`JSON-Attributes\` JSON
+)
+ENGINE = MergeTree
+PARTITION BY toDate(Timestamp)
+PRIMARY KEY (ServiceName, Timestamp)
+ORDER BY (ServiceName, Timestamp);
+
+-- ---------------------------------------------------------------------------
+-- Metadata materialized-view source
+-- ---------------------------------------------------------------------------
+-- A standard otel_logs-schema base table (\`e2e_otel_logs_metadata_mv\`) plus a
+-- pair of "metadata" rollup tables that pre-aggregate facet keys and key/values
+-- per ColumnIdentifier in 15-minute buckets. The rollups are populated by
+-- materialized views that fire on insert into the base table. The
+-- 'E2E Metadata MV Logs' fixture source registers these rollups via its
+-- \`metadataMaterializedViews\` config, so the filter-key edge case tests exercise
+-- the rollup-backed facet path (keys/values read from the rollups instead of the
+-- base table) for the ServiceName native column and a LogAttributes map key.
+CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_otel_logs_metadata_mv
+(
+ \`Timestamp\` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+ \`TraceId\` String CODEC(ZSTD(1)),
+ \`SpanId\` String CODEC(ZSTD(1)),
+ \`TraceFlags\` UInt8,
+ \`SeverityText\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`SeverityNumber\` UInt8,
+ \`ServiceName\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`Body\` String CODEC(ZSTD(1)),
+ \`ResourceSchemaUrl\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`ResourceAttributes\` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+ \`ScopeSchemaUrl\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`ScopeName\` String CODEC(ZSTD(1)),
+ \`ScopeVersion\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`ScopeAttributes\` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+ \`LogAttributes\` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
+ \`EventName\` String CODEC(ZSTD(1)),
+ \`__hdx_materialized_k8s.cluster.name\` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.cluster.name'] CODEC(ZSTD(1)),
+ \`__hdx_materialized_k8s.container.name\` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.container.name'] CODEC(ZSTD(1)),
+ \`__hdx_materialized_k8s.deployment.name\` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.deployment.name'] CODEC(ZSTD(1)),
+ \`__hdx_materialized_k8s.namespace.name\` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.namespace.name'] CODEC(ZSTD(1)),
+ \`__hdx_materialized_k8s.node.name\` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.node.name'] CODEC(ZSTD(1)),
+ \`__hdx_materialized_k8s.pod.name\` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.pod.name'] CODEC(ZSTD(1)),
+ \`__hdx_materialized_k8s.pod.uid\` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.pod.uid'] CODEC(ZSTD(1)),
+ \`__hdx_materialized_deployment.environment.name\` LowCardinality(String) MATERIALIZED ResourceAttributes['deployment.environment.name'] CODEC(ZSTD(1))
+)
+ENGINE = MergeTree
+PARTITION BY toDate(Timestamp)
+ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)
+TTL toDateTime(Timestamp) + toIntervalDay(30)
+SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
+
+CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_otel_logs_kv_rollup_15m
+(
+ \`Timestamp\` DateTime,
+ \`ColumnIdentifier\` LowCardinality(String),
+ \`Key\` LowCardinality(String),
+ \`Value\` String,
+ \`count\` UInt64,
+ INDEX idx_count_minmax count TYPE minmax GRANULARITY 1,
+ INDEX idx_timestamp_minmax Timestamp TYPE minmax GRANULARITY 1
+)
+ENGINE = SummingMergeTree
+PARTITION BY toDate(Timestamp)
+ORDER BY (ColumnIdentifier, Key, Timestamp, Value)
+TTL Timestamp + toIntervalDay(30)
+SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
+
+CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.e2e_otel_logs_attr_kv_rollup_15m_mv TO ${DATABASE}.e2e_otel_logs_kv_rollup_15m
+(
+ \`Timestamp\` DateTime,
+ \`ColumnIdentifier\` String,
+ \`Key\` String,
+ \`Value\` String,
+ \`count\` UInt64
+)
+AS WITH elements AS
+ (
+ SELECT
+ 'ResourceAttributes' AS ColumnIdentifier,
+ toStartOfFifteenMinutes(Timestamp) AS Timestamp,
+ replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,
+ CAST(entry.2, 'String') AS Value
+ FROM ${DATABASE}.e2e_otel_logs_metadata_mv
+ ARRAY JOIN ResourceAttributes AS entry
+ UNION ALL
+ SELECT
+ 'LogAttributes' AS ColumnIdentifier,
+ toStartOfFifteenMinutes(Timestamp) AS Timestamp,
+ replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,
+ CAST(entry.2, 'String') AS Value
+ FROM ${DATABASE}.e2e_otel_logs_metadata_mv
+ ARRAY JOIN LogAttributes AS entry
+ UNION ALL
+ SELECT
+ 'ScopeAttributes' AS ColumnIdentifier,
+ toStartOfFifteenMinutes(Timestamp) AS Timestamp,
+ replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,
+ CAST(entry.2, 'String') AS Value
+ FROM ${DATABASE}.e2e_otel_logs_metadata_mv
+ ARRAY JOIN ScopeAttributes AS entry
+ UNION ALL
+ SELECT
+ 'NativeColumn' AS ColumnIdentifier,
+ toStartOfFifteenMinutes(Timestamp) AS Timestamp,
+ 'SeverityText' AS Key,
+ CAST(SeverityText, 'String') AS Value
+ FROM ${DATABASE}.e2e_otel_logs_metadata_mv
+ UNION ALL
+ SELECT
+ 'NativeColumn' AS ColumnIdentifier,
+ toStartOfFifteenMinutes(Timestamp) AS Timestamp,
+ 'ServiceName' AS Key,
+ CAST(ServiceName, 'String') AS Value
+ FROM ${DATABASE}.e2e_otel_logs_metadata_mv
+ UNION ALL
+ SELECT
+ 'NativeColumn' AS ColumnIdentifier,
+ toStartOfFifteenMinutes(Timestamp) AS Timestamp,
+ 'ScopeName' AS Key,
+ CAST(ScopeName, 'String') AS Value
+ FROM ${DATABASE}.e2e_otel_logs_metadata_mv
+ )
+SELECT
+ Timestamp,
+ ColumnIdentifier,
+ Key,
+ Value,
+ count() AS count
+FROM elements
+GROUP BY
+ Timestamp,
+ ColumnIdentifier,
+ Key,
+ Value;
+
+CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_otel_logs_key_rollup_15m
+(
+ \`Timestamp\` DateTime,
+ \`ColumnIdentifier\` LowCardinality(String),
+ \`Key\` LowCardinality(String),
+ \`count\` UInt64,
+ INDEX idx_count_minmax count TYPE minmax GRANULARITY 1,
+ INDEX idx_timestamp_minmax Timestamp TYPE minmax GRANULARITY 1
+)
+ENGINE = SummingMergeTree
+PARTITION BY toDate(Timestamp)
+ORDER BY (ColumnIdentifier, Key, Timestamp)
+TTL Timestamp + toIntervalDay(30)
+SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
+
+CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.e2e_otel_logs_key_rollup_15m_mv TO ${DATABASE}.e2e_otel_logs_key_rollup_15m
+(
+ \`Timestamp\` DateTime,
+ \`ColumnIdentifier\` LowCardinality(String),
+ \`Key\` LowCardinality(String),
+ \`count\` UInt64
+)
+AS SELECT
+ Timestamp,
+ ColumnIdentifier,
+ Key,
+ sum(count) AS count
+FROM ${DATABASE}.e2e_otel_logs_kv_rollup_15m
+GROUP BY
+ ColumnIdentifier,
+ Key,
+ Timestamp;
+
+-- ---------------------------------------------------------------------------
+-- Generic Logs source (HDX-5274)
+-- ---------------------------------------------------------------------------
+-- A log table with a custom ServiceName column, which is deliberately NOT
+-- part of the sort/primary key.
+CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_custom_service_name
+(
+ \`Timestamp\` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+ \`AppName\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`SeverityText\` LowCardinality(String) CODEC(ZSTD(1)),
+ \`Body\` String CODEC(ZSTD(1)),
+ \`LogAttributes\` Map(LowCardinality(String), String) CODEC(ZSTD(1))
+)
+ENGINE = MergeTree
+PARTITION BY toDate(Timestamp)
+ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp)
+TTL toDateTime(Timestamp) + toIntervalDay(30)
+SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
+
+-- PromQL fixture.
+CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_promql ENGINE = TimeSeries;
EOFSQL
diff --git a/docker/hyperdx/Dockerfile b/docker/hyperdx/Dockerfile
index fb8dcdf5fc..6a3321822a 100644
--- a/docker/hyperdx/Dockerfile
+++ b/docker/hyperdx/Dockerfile
@@ -8,9 +8,9 @@
# - Otel Collector (otelcol)
ARG NODE_VERSION=22.22
-ARG CLICKHOUSE_VERSION=26.5
-ARG OTEL_COLLECTOR_VERSION=0.149.0
-ARG OTEL_COLLECTOR_CORE_VERSION=1.55.0
+ARG CLICKHOUSE_VERSION=26.8
+ARG OTEL_COLLECTOR_VERSION=0.155.0
+ARG OTEL_COLLECTOR_CORE_VERSION=1.61.0
# base #############################################################################################
# == Otel Collector Image ==
@@ -52,7 +52,7 @@ COPY .yarnrc.yml yarn.lock package.json nx.json .prettierrc .prettierignore ./ts
# full source is copied in the build stages that need it.
COPY ./packages/common-utils/package.json ./packages/common-utils/package.json
COPY ./packages/api/jest.config.js ./packages/api/tsconfig.json ./packages/api/tsconfig.build.json ./packages/api/package.json ./packages/api/
-COPY ./packages/app/jest.config.js ./packages/app/tsconfig.json ./packages/app/tsconfig.build.json ./packages/app/package.json ./packages/app/next.config.mjs ./packages/app/mdx.d.ts ./packages/app/eslint.config.mjs ./packages/app/
+COPY ./packages/app/jest.config.js ./packages/app/tsconfig.json ./packages/app/tsconfig.build.json ./packages/app/package.json ./packages/app/next.config.mjs ./packages/app/mdx.d.ts ./packages/app/css.d.ts ./packages/app/eslint.config.mjs ./packages/app/
# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
RUN apk add --no-cache libc6-compat
@@ -86,6 +86,13 @@ COPY --from=app ./pages ./packages/app/pages
COPY --from=app ./public ./packages/app/public
COPY --from=app ./styles ./packages/app/styles
COPY --from=app ./types ./packages/app/types
+# Build inputs for public/whats-new.json, generated by next.config.mjs for the
+# in-app "What's new" section: the root changelog it parses and the parser
+# itself. WORKDIR is /app, and next.config resolves the changelog as
+# /app/packages/app/../../CHANGELOG.md. The build fails loudly if either is
+# missing (see next.config.mjs).
+COPY ./CHANGELOG.md ./CHANGELOG.md
+COPY --from=app ./scripts ./packages/app/scripts
ENV NEXT_TELEMETRY_DISABLED=1
ENV NEXT_OUTPUT_STANDALONE=true
@@ -129,7 +136,8 @@ COPY --chown=node:node --from=builder /app/packages/app/.next/standalone ./packa
COPY --chown=node:node --from=builder /app/packages/app/.next/static ./packages/app/packages/app/.next/static
COPY --chown=node:node --from=builder /app/packages/app/public ./packages/app/packages/app/public
-# Set up start script
+# Set up start scripts
+COPY --chown=node:node --from=hyperdx ./refresh-env.js /etc/local/refresh-env.js
COPY --chown=node:node --from=hyperdx ./entry.prod.sh /etc/local/entry.sh
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
@@ -151,10 +159,14 @@ ENV OTEL_RESOURCE_ATTRIBUTES="service.version=$CODE_VERSION"
COPY --from=ocb-builder --chmod=755 /build/output/otelcol-hyperdx /otelcontribcol
COPY --from=otel_collector_opampsupervisor_base --chmod=755 /usr/local/bin/opampsupervisor /usr/local/bin/opampsupervisor
-# Copy Node.js runtime from node base
+# Copy Node.js runtime from node base.
+# Deliberately NOT copying node's /usr/lib: it overwrites this image's
+# libapk.so, libssl, libcrypto and libz with whatever Alpine point release the
+# node image happens to be built on. When the two drift, apk stops linking
+# ("Error relocating /sbin/apk: ... symbol not found") and the build dies.
+# Node only needs libstdc++/libgcc from there; those are installed via apk below.
COPY --from=node_base --link /usr/local/bin /usr/local/bin
COPY --from=node_base --link /usr/local/lib /usr/local/lib
-COPY --from=node_base /usr/lib /usr/lib
COPY --from=node_base /usr/local/include /usr/local/include
# Set up Clickhouse
@@ -178,8 +190,11 @@ COPY --from=goose /bin/goose /usr/local/bin/goose
# Copy migrate binary (Go-based goose migration tool with full TLS support)
COPY --from=migrate-builder /migrate /usr/local/bin/migrate
-# Install MongoDB and other dependencies (consolidated into a single RUN command)
-RUN echo 'http://dl-cdn.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories && \
+# Install MongoDB and other dependencies (consolidated into a single RUN command).
+# libstdc++/libgcc are the Node.js runtime deps, installed before the v3.9
+# repositories are appended so they resolve from this image's own Alpine release.
+RUN apk add --no-cache libstdc++ libgcc && \
+ echo 'http://dl-cdn.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories && \
echo 'http://dl-cdn.alpinelinux.org/alpine/v3.9/community' >> /etc/apk/repositories && \
apk update && \
apk add --no-cache mongodb yaml-cpp=0.6.2-r2 ca-certificates && \
@@ -197,6 +212,7 @@ ENV NODE_ENV=production
# Set up App
COPY --from=prod /app /app
+COPY --from=hyperdx ./refresh-env.js /etc/local/refresh-env.js
COPY --from=hyperdx ./entry.local.base.sh /etc/local/entry.base.sh
# Copy VEX exception files for Docker Scout vulnerability suppression
@@ -208,7 +224,7 @@ WORKDIR /app
# Add hosts entry in entrypoint script instead of here
# Expose ports
-EXPOSE 8080 4317 4318 13133 8123 9000
+EXPOSE 8080 4317 4318 13133 8123 9000 8125/udp
# all-in-one no auth ############################################################################################
FROM all-in-one-base AS all-in-one-noauth
diff --git a/docker/hyperdx/clickhouseConfig.xml b/docker/hyperdx/clickhouseConfig.xml
index 50cab2ce7f..db8bbc1dc5 100644
--- a/docker/hyperdx/clickhouseConfig.xml
+++ b/docker/hyperdx/clickhouseConfig.xml
@@ -47,6 +47,16 @@
+
+
+ /prometheus/api/v1
+
+ prometheus_api_v1
+
+
+
+
+
system
diff --git a/docker/hyperdx/entry.local.base.sh b/docker/hyperdx/entry.local.base.sh
index 865cbc5d53..82ceb0aebf 100644
--- a/docker/hyperdx/entry.local.base.sh
+++ b/docker/hyperdx/entry.local.base.sh
@@ -18,8 +18,14 @@ export MONGO_URI="mongodb://db:27017/hyperdx"
export OPAMP_SERVER_URL="http://127.0.0.1:${OPAMP_PORT}"
export CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT="${CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT:-ch-server:9363}"
-export HYPERDX_IMAGE=$([[ "${IS_LOCAL_APP_MODE}" == "DANGEROUSLY_is_local_app_mode💀" ]] && echo "all-in-one" || echo "all-in-one-noauth")
-export EXPRESS_SESSION_SECRET="hyperdx is cool 👋"
+# These values must match the strings compared against in packages/api/src/config.ts
+# (IS_LOCAL_IMAGE / IS_ALL_IN_ONE_IMAGE). The auth variant must report
+# "all-in-one-auth" so that the INGESTION_API_KEY bootstrap path is enabled.
+export HYPERDX_IMAGE=$([[ "${IS_LOCAL_APP_MODE}" == "DANGEROUSLY_is_local_app_mode💀" ]] && echo "all-in-one-noauth" || echo "all-in-one-auth")
+# Keep the public demo default only when authentication is disabled.
+if [[ "${IS_LOCAL_APP_MODE}" == "DANGEROUSLY_is_local_app_mode💀" ]]; then
+ export EXPRESS_SESSION_SECRET="${EXPRESS_SESSION_SECRET:-hyperdx is cool 👋}"
+fi
# IS_LOCAL_APP_MODE should be set by the calling script
# Default to dangerous mode if not set
export IS_LOCAL_APP_MODE="${IS_LOCAL_APP_MODE}"
@@ -30,10 +36,10 @@ if [ -z "${DEFAULT_CONNECTIONS}" ]; then
DEFAULT_CONNECTIONS='[{"name":"Local ClickHouse","host":"http://localhost:8123","username":"default","password":""}]'
fi
export DEFAULT_CONNECTIONS
-export DEFAULT_SOURCES='[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","metadataMaterializedViews":{"keyRollupTable":"otel_logs_key_rollup_15m","kvRollupTable":"otel_logs_kv_rollup_15m","granularity":"15 minute"},"connection":"Local ClickHouse","traceSourceId":"Traces","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","metadataMaterializedViews":{"keyRollupTable":"otel_traces_key_rollup_15m","kvRollupTable":"otel_traces_kv_rollup_15m","granularity":"15 minute"},"connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","summary":"otel_metrics_summary","exponentialHistogram":"otel_metrics_exponential_histogram","_id":"682586a8b1f81924e628e808","id":"682586a8b1f81924e628e808"},"connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","sessionSourceId":"Sessions"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","metricSourceId":"Metrics"}]'
+export DEFAULT_SOURCES='[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","severityNumberExpression":"SeverityNumber","eventNameExpression":"EventName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","metadataMaterializedViews":{"kvRollupTable":"otel_logs_kv_rollup_15m","granularity":"15 minute"},"connection":"Local ClickHouse","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","spanEventsValueExpression":"Events","spanLinksValueExpression":"Links","metadataMaterializedViews":{"kvRollupTable":"otel_traces_kv_rollup_15m","granularity":"15 minute"},"connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","summary":"otel_metrics_summary","exponential histogram":"otel_metrics_exponential_histogram"},"connection":"Local ClickHouse","logSourceId":"Logs"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","resourceAttributesExpression":"ResourceAttributes","connection":"Local ClickHouse","traceSourceId":"Traces"}]'
# if BETA_CH_OTEL_JSON_SCHEMA_ENABLED is true then return empty default sources
if [ "$BETA_CH_OTEL_JSON_SCHEMA_ENABLED" = "true" ]; then
- export DEFAULT_SOURCES='[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"Traces","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","summary":"otel_metrics_summary","exponentialHistogram":"otel_metrics_exponential_histogram","_id":"682586a8b1f81924e628e808","id":"682586a8b1f81924e628e808"},"connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","sessionSourceId":"Sessions"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"Timestamp","name":"Sessions","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","metricSourceId":"Metrics"}]'
+ export DEFAULT_SOURCES='[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","severityNumberExpression":"SeverityNumber","eventNameExpression":"EventName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","spanEventsValueExpression":"Events","spanLinksValueExpression":"Links","connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","summary":"otel_metrics_summary","exponential histogram":"otel_metrics_exponential_histogram"},"connection":"Local ClickHouse","logSourceId":"Logs"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"Timestamp","name":"Sessions","resourceAttributesExpression":"ResourceAttributes","connection":"Local ClickHouse","traceSourceId":"Traces"}]'
fi
# Simulate Docker Service DNS
@@ -66,6 +72,8 @@ echo "ClickHouse is ready!"
# Start Otel Collector with entrypoint script for template rendering and log rotation
/otel-entrypoint.sh /usr/local/bin/opampsupervisor > /var/log/otel-collector.log 2>&1 &
+node /etc/local/refresh-env.js
+
# Start HyperDX app
./node_modules/.bin/concurrently \
"--kill-others-on-fail" \
diff --git a/docker/hyperdx/entry.prod.sh b/docker/hyperdx/entry.prod.sh
index bc9cd2130c..6f9888fdf8 100644
--- a/docker/hyperdx/entry.prod.sh
+++ b/docker/hyperdx/entry.prod.sh
@@ -12,6 +12,8 @@ echo ""
echo "Visit the HyperDX UI at $FRONTEND_URL"
echo ""
+node /etc/local/refresh-env.js
+
# Optionally include the dashboard provisioner task
EXTRA_NAMES=""
EXTRA_CMDS=""
diff --git a/docker/hyperdx/refresh-env.js b/docker/hyperdx/refresh-env.js
new file mode 100644
index 0000000000..47b7593808
--- /dev/null
+++ b/docker/hyperdx/refresh-env.js
@@ -0,0 +1,20 @@
+// Merge runtime NEXT_PUBLIC_* env vars into the existing __ENV.js written at
+// build time by configureRuntimeEnv, so build-time values are preserved and
+// runtime values (e.g. docker run -e) take precedence.
+const fs = require('fs');
+
+const path = './packages/app/packages/app/public/__ENV.js';
+
+let existing = {};
+try {
+ const content = fs.readFileSync(path, 'utf8');
+ const json = content.replace(/^window\.__ENV\s*=\s*/, '').replace(/;\s*$/, '');
+ existing = JSON.parse(json);
+} catch {}
+
+const runtime = Object.fromEntries(
+ Object.entries(process.env).filter(([k]) => k.startsWith('NEXT_PUBLIC_'))
+);
+
+const merged = { ...existing, ...runtime };
+fs.writeFileSync(path, 'window.__ENV = ' + JSON.stringify(merged) + ';');
diff --git a/docker/otel-collector/Dockerfile b/docker/otel-collector/Dockerfile
index 1d69a0890b..18355cd66e 100644
--- a/docker/otel-collector/Dockerfile
+++ b/docker/otel-collector/Dockerfile
@@ -1,6 +1,6 @@
## base #############################################################################################
-ARG OTEL_COLLECTOR_VERSION=0.149.0
-ARG OTEL_COLLECTOR_CORE_VERSION=1.55.0
+ARG OTEL_COLLECTOR_VERSION=0.155.0
+ARG OTEL_COLLECTOR_CORE_VERSION=1.61.0
FROM otel/opentelemetry-collector-opampsupervisor:${OTEL_COLLECTOR_VERSION} AS supervisor
FROM hairyhenderson/gomplate:v4.3.3-alpine AS gomplate
@@ -59,9 +59,8 @@ USER ${USER_UID}:${USER_GID}
COPY --from=supervisor --chmod=755 /usr/local/bin/opampsupervisor /opampsupervisor
COPY --from=ocb-builder --chmod=755 /build/output/otelcol-hyperdx /otelcontribcol
-# Copy entrypoint and log tail wrapper scripts
+# Copy entrypoint script
COPY --chmod=755 docker/otel-collector/entrypoint.sh /entrypoint.sh
-COPY --chmod=755 docker/otel-collector/log-tailer.sh /log-tailer.sh
## dev ##############################################################################################
FROM base AS dev
@@ -75,10 +74,12 @@ LABEL org.opencontainers.image.vendor="HyperDX" \
COPY --chown=10001:10001 docker/otel-collector/config.yaml /etc/otelcol-contrib/config.yaml
COPY --chown=10001:10001 docker/otel-collector/config.standalone.yaml /etc/otelcol-contrib/standalone-config.yaml
COPY --chown=10001:10001 docker/otel-collector/config.standalone.auth.yaml /etc/otelcol-contrib/standalone-auth-config.yaml
+COPY --chown=10001:10001 docker/otel-collector/config.standalone.oidc.yaml /etc/otelcol-contrib/standalone-oidc-config.yaml
+COPY --chown=10001:10001 docker/otel-collector/config.standalone.promql.yaml /etc/otelcol-contrib/standalone-promql-config.yaml
COPY --chown=10001:10001 docker/otel-collector/supervisor_docker.yaml.tmpl /etc/otel/supervisor.yaml.tmpl
COPY --chown=10001:10001 docker/otel-collector/schema /etc/otel/schema
-EXPOSE 4317 4318 13133
+EXPOSE 4317 4318 13133 8125/udp
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
CMD wget -q --spider http://localhost:13133/ || exit 1
@@ -97,10 +98,12 @@ LABEL org.opencontainers.image.vendor="HyperDX" \
COPY --chown=10001:10001 docker/otel-collector/config.yaml /etc/otelcol-contrib/config.yaml
COPY --chown=10001:10001 docker/otel-collector/config.standalone.yaml /etc/otelcol-contrib/standalone-config.yaml
COPY --chown=10001:10001 docker/otel-collector/config.standalone.auth.yaml /etc/otelcol-contrib/standalone-auth-config.yaml
+COPY --chown=10001:10001 docker/otel-collector/config.standalone.oidc.yaml /etc/otelcol-contrib/standalone-oidc-config.yaml
+COPY --chown=10001:10001 docker/otel-collector/config.standalone.promql.yaml /etc/otelcol-contrib/standalone-promql-config.yaml
COPY --chown=10001:10001 docker/otel-collector/supervisor_docker.yaml.tmpl /etc/otel/supervisor.yaml.tmpl
COPY --chown=10001:10001 docker/otel-collector/schema /etc/otel/schema
-EXPOSE 4317 4318 13133
+EXPOSE 4317 4318 13133 8125/udp
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
CMD wget -q --spider http://localhost:13133/ || exit 1
diff --git a/docker/otel-collector/config.standalone.auth.yaml b/docker/otel-collector/config.standalone.auth.yaml
index 32eece302a..0c596e8f54 100644
--- a/docker/otel-collector/config.standalone.auth.yaml
+++ b/docker/otel-collector/config.standalone.auth.yaml
@@ -4,8 +4,14 @@
extensions:
bearertokenauth/hyperdx:
scheme: ""
+ # The extension matches the full Authorization header value exactly, so
+ # list the `Bearer `-prefixed forms that RFC 6750 clients send alongside
+ # the bare token.
tokens:
- ${env:OTLP_AUTH_TOKEN}
+ - Bearer ${env:OTLP_AUTH_TOKEN}
+ - bearer ${env:OTLP_AUTH_TOKEN}
+ - BEARER ${env:OTLP_AUTH_TOKEN}
receivers:
otlp/hyperdx:
diff --git a/docker/otel-collector/config.standalone.oidc.yaml b/docker/otel-collector/config.standalone.oidc.yaml
new file mode 100644
index 0000000000..1df6484311
--- /dev/null
+++ b/docker/otel-collector/config.standalone.oidc.yaml
@@ -0,0 +1,27 @@
+# This configuration enables OIDC-based bearer token authentication for the
+# OTLP receiver. Only included when OIDC_ISSUER_URL environment variable is set.
+#
+# Clients must present a JWT as a bearer token (Authorization: Bearer ).
+# The collector verifies the token's signature against the issuer's published
+# JWKS (fetched from {issuer_url}/.well-known/openid-configuration), and
+# validates the issuer and audience claims. This is intended for deployments
+# that want short-lived, centrally-issued tokens instead of (or in addition
+# to) a long-lived static token -- see config.standalone.auth.yaml.
+
+extensions:
+ oidc/hyperdx:
+ issuer_url: ${env:OIDC_ISSUER_URL}
+ audience: ${env:OIDC_AUDIENCE}
+
+receivers:
+ otlp/hyperdx:
+ protocols:
+ grpc:
+ auth:
+ authenticator: oidc/hyperdx
+ http:
+ auth:
+ authenticator: oidc/hyperdx
+
+service:
+ extensions: [health_check, oidc/hyperdx]
diff --git a/docker/otel-collector/config.standalone.promql.yaml b/docker/otel-collector/config.standalone.promql.yaml
new file mode 100644
index 0000000000..f6f10a5125
--- /dev/null
+++ b/docker/otel-collector/config.standalone.promql.yaml
@@ -0,0 +1,27 @@
+# Enables remote-write of OTLP metrics to a Prometheus-compatible endpoint.
+# Only included when CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT is set.
+#
+# entrypoint.sh strips any http:// or https:// scheme from
+# CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT and exports it as
+# HYPERDX_PROMETHEUS_SCHEME / HYPERDX_PROMETHEUS_TLS_INSECURE. The defaults
+# keep plain HTTP if those are unset.
+#
+# Mirrors the OpAMP-managed behavior in
+# packages/api/src/opamp/controllers/opampController.ts (gated on
+# IS_PROMQL_ENABLED). When updating this file, keep it in sync with
+# buildOtelCollectorConfig().
+
+exporters:
+ prometheusremotewrite:
+ endpoint: ${env:HYPERDX_PROMETHEUS_SCHEME:-http}://${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}/write
+ tls:
+ insecure: ${env:HYPERDX_PROMETHEUS_TLS_INSECURE:-true}
+ resource_to_telemetry_conversion:
+ enabled: true
+
+service:
+ pipelines:
+ metrics/promql:
+ receivers: [otlp/hyperdx]
+ processors: [memory_limiter, batch]
+ exporters: [prometheusremotewrite]
diff --git a/docker/otel-collector/config.standalone.yaml b/docker/otel-collector/config.standalone.yaml
index 2e0e92b175..81eb44173e 100644
--- a/docker/otel-collector/config.standalone.yaml
+++ b/docker/otel-collector/config.standalone.yaml
@@ -6,13 +6,13 @@ receivers:
protocols:
grpc:
include_metadata: true
- endpoint: "0.0.0.0:4317"
+ endpoint: '0.0.0.0:4317'
http:
cors:
- allowed_origins: ["*"]
- allowed_headers: ["*"]
+ allowed_origins: ['*']
+ allowed_headers: ['*']
include_metadata: true
- endpoint: "0.0.0.0:4318"
+ endpoint: '0.0.0.0:4318'
connectors:
routing/logs:
@@ -31,7 +31,7 @@ exporters:
username: ${env:CLICKHOUSE_USER}
ttl: 720h
logs_table_name: hyperdx_sessions
- timeout: 5s
+ timeout: ${env:HYPERDX_OTEL_EXPORTER_TIMEOUT:-5s}
create_schema: ${env:HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA:-false}
json: ${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}
retry_on_failure:
@@ -45,7 +45,7 @@ exporters:
password: ${env:CLICKHOUSE_PASSWORD}
username: ${env:CLICKHOUSE_USER}
ttl: 720h
- timeout: 5s
+ timeout: ${env:HYPERDX_OTEL_EXPORTER_TIMEOUT:-5s}
create_schema: ${env:HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA:-false}
json: ${env:HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE:-false}
retry_on_failure:
@@ -53,12 +53,6 @@ exporters:
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
- prometheusremotewrite:
- endpoint: http://${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}/write
- tls:
- insecure: true
- resource_to_telemetry_conversion:
- enabled: true
service:
# Pipeline `processors:` lists live in the base config.yaml so users can
@@ -71,10 +65,6 @@ service:
metrics:
receivers: [otlp/hyperdx]
exporters: [clickhouse]
- metrics/promql:
- receivers: [otlp/hyperdx]
- processors: [memory_limiter, batch]
- exporters: [prometheusremotewrite]
logs/in:
receivers: [otlp/hyperdx]
exporters: [routing/logs]
diff --git a/docker/otel-collector/config.yaml b/docker/otel-collector/config.yaml
index 98cc78e800..fce203c602 100644
--- a/docker/otel-collector/config.yaml
+++ b/docker/otel-collector/config.yaml
@@ -3,14 +3,16 @@ processors:
transform:
log_statements:
- context: log
- error_mode: ignore
+ # OTTL has no JSON validity check, so ParseJSON failures on non-JSON
+ # brace-delimited bodies are expected and would otherwise be logged.
+ error_mode: silent
statements:
# JSON parsing: Extends log attributes with the fields from structured log body content, either as an OTEL map or
# as a string containing JSON content.
- - set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{.*\\}))")) where
- IsString(log.body)
+ - set(log.cache, ExtractPatterns(log.body, "(?P<0>(\\{\\s*\".*\\}))"))
+ where IsString(log.body)
- merge_maps(log.attributes, ParseJSON(log.cache["0"]), "upsert")
- where IsMap(log.cache)
+ where IsMap(log.cache) and IsString(log.cache["0"])
- flatten(log.attributes) where IsMap(log.cache)
- merge_maps(log.attributes, log.body, "upsert") where IsMap(log.body)
- context: log
@@ -163,6 +165,14 @@ service:
port: 8888
logs:
level: ${HYPERDX_LOG_LEVEL}
+ # The collector's zap logger defaults to stderr. Under the supervisor's
+ # `passthrough_logs`, stdout is re-emitted at Info and stderr at Error,
+ # so leaving the default would surface every collector log line as an
+ # error. Writing to stdout keeps the supervisor-assigned level sane; the
+ # collector's real severity is still in the wrapped record's own `level`
+ # field. In standalone mode both streams land in the container log, so
+ # this is a no-op there.
+ output_paths: [stdout]
extensions: [health_check]
# Pipeline receivers and exporters are configured dynamically in
# opampController.ts. Pipeline `processors:` lists are declared here in
diff --git a/docker/otel-collector/entrypoint.sh b/docker/otel-collector/entrypoint.sh
index 272473e418..058b94ec41 100644
--- a/docker/otel-collector/entrypoint.sh
+++ b/docker/otel-collector/entrypoint.sh
@@ -24,6 +24,43 @@ if [ "$HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE" = "true" ]; then
export HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA=true
fi
+# CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT accepts a bare host:port (plain HTTP,
+# the original format) or an http:// or https:// URL. Both the standalone
+# config and the OpAMP-managed config (opampController.ts) need it as
+# host:port, because it is also a Prometheus scrape target, so the scheme is
+# split off here and handed to the prometheusremotewrite exporter through
+# HYPERDX_PROMETHEUS_SCHEME and HYPERDX_PROMETHEUS_TLS_INSECURE. Runs before
+# migrations so a bad value fails fast without touching ClickHouse.
+prometheus_endpoint_error() {
+ echo "ERROR: CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT $1. Use host:port, http://host:port or https://host:port." >&2
+ exit 1
+}
+
+if [ -n "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" ]; then
+ HYPERDX_PROMETHEUS_SCHEME=http
+ HYPERDX_PROMETHEUS_TLS_INSECURE=true
+ case "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" in
+ https://*)
+ HYPERDX_PROMETHEUS_SCHEME=https
+ HYPERDX_PROMETHEUS_TLS_INSECURE=false
+ ;;
+ http://*) ;;
+ # Only the scheme is echoed back: the rest of the URL may carry credentials.
+ *://*) prometheus_endpoint_error "has an unsupported scheme '${CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT%%://*}'" ;;
+ esac
+
+ PROMETHEUS_HOST_PORT="${CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT#*://}"
+ PROMETHEUS_HOST_PORT="${PROMETHEUS_HOST_PORT%/}"
+ case "$PROMETHEUS_HOST_PORT" in
+ "") prometheus_endpoint_error "has no host" ;;
+ */*) prometheus_endpoint_error "must not include a path (/write is appended automatically)" ;;
+ esac
+
+ export CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT="$PROMETHEUS_HOST_PORT"
+ export HYPERDX_PROMETHEUS_SCHEME HYPERDX_PROMETHEUS_TLS_INSECURE
+ echo "CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT scheme: $HYPERDX_PROMETHEUS_SCHEME"
+fi
+
# Run ClickHouse schema migrations if not using legacy schema creation
if [ "$HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA" != "true" ]; then
# Run Go-based migrate tool with TLS support
@@ -45,12 +82,37 @@ if [ -z "$OPAMP_SERVER_URL" ]; then
# Build collector arguments with multiple config files
COLLECTOR_ARGS="--config /etc/otelcol-contrib/config.yaml --config /etc/otelcol-contrib/standalone-config.yaml"
- # Add bearer token auth config if OTLP_AUTH_TOKEN is specified (only used in standalone mode)
- if [ -n "$OTLP_AUTH_TOKEN" ]; then
+ # Add OIDC-based bearer token auth config if OIDC_ISSUER_URL is specified,
+ # otherwise fall back to static bearer token auth if OTLP_AUTH_TOKEN is
+ # specified (only used in standalone mode). These are mutually exclusive:
+ # both configure the same receiver's auth.authenticator, so enabling both
+ # would just make whichever config file is loaded last win.
+ if [ -n "$OIDC_ISSUER_URL" ]; then
+ # oidcauthextension requires a non-empty audience unless ignore_audience
+ # is set (which we don't expose here, since silently skipping the
+ # audience check would weaken the auth rather than just fail loudly).
+ # Fail fast with a clear message instead of letting the collector crash
+ # on the extension's own less obvious "no audience provided" error.
+ if [ -z "$OIDC_AUDIENCE" ]; then
+ echo "ERROR: OIDC_ISSUER_URL is set but OIDC_AUDIENCE is not. Both are required to enable OIDC authentication." >&2
+ exit 1
+ fi
+ echo "OIDC_ISSUER_URL is configured, enabling OIDC-based bearer token authentication"
+ COLLECTOR_ARGS="$COLLECTOR_ARGS --config /etc/otelcol-contrib/standalone-oidc-config.yaml"
+ elif [ -n "$OTLP_AUTH_TOKEN" ]; then
echo "OTLP_AUTH_TOKEN is configured, enabling bearer token authentication"
COLLECTOR_ARGS="$COLLECTOR_ARGS --config /etc/otelcol-contrib/standalone-auth-config.yaml"
fi
+ # Enable prometheus remote-write of OTLP metrics only when an endpoint is
+ # configured. Mirrors the OpAMP-managed IS_PROMQL_ENABLED gating in
+ # packages/api/src/opamp/controllers/opampController.ts so the standalone
+ # collector does not attempt to export to a missing endpoint.
+ if [ -n "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" ]; then
+ echo "CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT is set, enabling prometheus remote-write"
+ COLLECTOR_ARGS="$COLLECTOR_ARGS --config /etc/otelcol-contrib/standalone-promql-config.yaml"
+ fi
+
# Add custom config file if specified
if [ -n "$CUSTOM_OTELCOL_CONFIG_FILE" ]; then
echo "Including custom config: $CUSTOM_OTELCOL_CONFIG_FILE"
@@ -68,20 +130,10 @@ else
# Supervisor mode - run with OpAMP supervisor
echo "Running in supervisor mode (OPAMP_SERVER_URL: $OPAMP_SERVER_URL)"
- if [ "$OTEL_SUPERVISOR_LOGS" = "true" ]; then
- # Start log tailer process in background for agent.log
- # Arguments: log_file_path [check_interval_seconds]
- /log-tailer.sh /etc/otel/supervisor-data/agent.log 1 &
-
- # Create a agent log file for the supervisor and collector child process. Normally
- # this file would be created as a standard file but we just want a FIFO pipe that
- # will pass data over to the tail process in the entrypoint script. This avoids
- # the need to the supervisor to store and forward the logs in its memory while also
- # eliminating the need for volume based storage.
- if [ ! -e /etc/otel/supervisor-data/agent.log ]; then
- mkfifo /etc/otel/supervisor-data/agent.log || echo "Failed to create FIFO" >&2
- fi
- fi
+ # OTEL_SUPERVISOR_LOGS=true is consumed by supervisor.yaml.tmpl, which sets
+ # the supervisor's native `agent::passthrough_logs`. With this the supervisor
+ # forwards the collector's stdout/stderr through its own logger, keeping a
+ # single writer on the container's stdout.
# Render the supervisor config template using gomplate
# Write to supervisor-data directory which has proper permissions for otel user
diff --git a/docker/otel-collector/log-tailer.sh b/docker/otel-collector/log-tailer.sh
deleted file mode 100644
index 2dcd4adada..0000000000
--- a/docker/otel-collector/log-tailer.sh
+++ /dev/null
@@ -1,23 +0,0 @@
-#!/bin/sh
-# Generic log tailer script that follows a log file and echoes new lines to stdout
-# Usage: log-tailer.sh [sleep_interval]
-
-# Parse arguments
-LOG_FILE="${1}"
-SLEEP_INTERVAL="${2:-1}"
-
-# Validate required argument
-if [ -z "$LOG_FILE" ]; then
- echo "Error: Log file path is required" >&2
- echo "Usage: $0 [sleep_interval]" >&2
- exit 1
-fi
-
-while true; do
- # Use tail -F to follow the file by name, not by descriptor
- # This handles rotation and truncation gracefully
- # -n 0: Start from the end (don't output existing content)
- # -F: Follow by name and retry if file is inaccessible
- # -s: Sleep interval between checks
- tail -n 0 -F -s "$SLEEP_INTERVAL" "$LOG_FILE" || true
-done
diff --git a/docker/otel-collector/schema/seed/00002_otel_logs.sql b/docker/otel-collector/schema/seed/00002_otel_logs.sql
index c329f59783..2664576ec8 100644
--- a/docker/otel-collector/schema/seed/00002_otel_logs.sql
+++ b/docker/otel-collector/schema/seed/00002_otel_logs.sql
@@ -40,5 +40,5 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)
-TTL toDateTime(Timestamp) + ${TABLES_TTL}
+TTL toDateTime(Timestamp) + ${LOGS_TTL}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1, enable_block_number_column = 1, enable_block_offset_column = 1;
diff --git a/docker/otel-collector/schema/seed/00002_otel_logs_compat.sql b/docker/otel-collector/schema/seed/00002_otel_logs_compat.sql
index e4c18b17ff..2aa6f1c82f 100644
--- a/docker/otel-collector/schema/seed/00002_otel_logs_compat.sql
+++ b/docker/otel-collector/schema/seed/00002_otel_logs_compat.sql
@@ -38,5 +38,5 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)
-TTL toDateTime(Timestamp) + ${TABLES_TTL}
+TTL toDateTime(Timestamp) + ${LOGS_TTL}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1, enable_block_number_column = 1, enable_block_offset_column = 1;
diff --git a/docker/otel-collector/schema/seed/00003_otel_metrics.sql b/docker/otel-collector/schema/seed/00003_otel_metrics.sql
index b5e0b1d2a0..03938e811a 100644
--- a/docker/otel-collector/schema/seed/00003_otel_metrics.sql
+++ b/docker/otel-collector/schema/seed/00003_otel_metrics.sql
@@ -9,16 +9,16 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_gauge
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
- `MetricName` String CODEC(ZSTD(1)),
+ `MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
- `StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
- `TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+ `StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
+ `TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Value` Float64 CODEC(ZSTD(1)),
`Flags` UInt32 CODEC(ZSTD(1)),
`Exemplars.FilteredAttributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
- `Exemplars.TimeUnix` Array(DateTime64(9)) CODEC(ZSTD(1)),
+ `Exemplars.TimeUnix` Array(DateTime) CODEC(ZSTD(1)),
`Exemplars.Value` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.SpanId` Array(String) CODEC(ZSTD(1)),
`Exemplars.TraceId` Array(String) CODEC(ZSTD(1)),
@@ -27,12 +27,13 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_gauge
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
+ INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
+ INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
-ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
-TTL toDateTime(TimeUnix) + ${TABLES_TTL}
+ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
+TTL toDateTime(TimeUnix) + ${METRICS_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementBegin
@@ -46,16 +47,16 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_sum
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
- `MetricName` String CODEC(ZSTD(1)),
+ `MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
- `StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
- `TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+ `StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
+ `TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Value` Float64 CODEC(ZSTD(1)),
`Flags` UInt32 CODEC(ZSTD(1)),
`Exemplars.FilteredAttributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
- `Exemplars.TimeUnix` Array(DateTime64(9)) CODEC(ZSTD(1)),
+ `Exemplars.TimeUnix` Array(DateTime) CODEC(ZSTD(1)),
`Exemplars.Value` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.SpanId` Array(String) CODEC(ZSTD(1)),
`Exemplars.TraceId` Array(String) CODEC(ZSTD(1)),
@@ -66,12 +67,13 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_sum
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
+ INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
+ INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
-ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
-TTL toDateTime(TimeUnix) + ${TABLES_TTL}
+ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
+TTL toDateTime(TimeUnix) + ${METRICS_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementEnd
@@ -86,18 +88,18 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_histogram
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
- `MetricName` String CODEC(ZSTD(1)),
+ `MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
- `StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
- `TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+ `StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
+ `TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Count` UInt64 CODEC(Delta(8), ZSTD(1)),
`Sum` Float64 CODEC(ZSTD(1)),
`BucketCounts` Array(UInt64) CODEC(ZSTD(1)),
`ExplicitBounds` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.FilteredAttributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
- `Exemplars.TimeUnix` Array(DateTime64(9)) CODEC(ZSTD(1)),
+ `Exemplars.TimeUnix` Array(DateTime) CODEC(ZSTD(1)),
`Exemplars.Value` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.SpanId` Array(String) CODEC(ZSTD(1)),
`Exemplars.TraceId` Array(String) CODEC(ZSTD(1)),
@@ -110,12 +112,13 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_histogram
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
+ INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
+ INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
-ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
-TTL toDateTime(TimeUnix) + ${TABLES_TTL}
+ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
+TTL toDateTime(TimeUnix) + ${METRICS_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementEnd
@@ -130,12 +133,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_exponential_histogram
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
- `MetricName` String CODEC(ZSTD(1)),
+ `MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
- `StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
- `TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+ `StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
+ `TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Count` UInt64 CODEC(Delta(8), ZSTD(1)),
`Sum` Float64 CODEC(ZSTD(1)),
`Scale` Int32 CODEC(ZSTD(1)),
@@ -145,7 +148,7 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_exponential_histogram
`NegativeOffset` Int32 CODEC(ZSTD(1)),
`NegativeBucketCounts` Array(UInt64) CODEC(ZSTD(1)),
`Exemplars.FilteredAttributes` Array(Map(LowCardinality(String), String)) CODEC(ZSTD(1)),
- `Exemplars.TimeUnix` Array(DateTime64(9)) CODEC(ZSTD(1)),
+ `Exemplars.TimeUnix` Array(DateTime) CODEC(ZSTD(1)),
`Exemplars.Value` Array(Float64) CODEC(ZSTD(1)),
`Exemplars.SpanId` Array(String) CODEC(ZSTD(1)),
`Exemplars.TraceId` Array(String) CODEC(ZSTD(1)),
@@ -158,12 +161,13 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_exponential_histogram
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
+ INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
+ INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
-ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
-TTL toDateTime(TimeUnix) + ${TABLES_TTL}
+ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
+TTL toDateTime(TimeUnix) + ${METRICS_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementEnd
@@ -178,12 +182,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_summary
`ScopeDroppedAttrCount` UInt32 CODEC(ZSTD(1)),
`ScopeSchemaUrl` String CODEC(ZSTD(1)),
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
- `MetricName` String CODEC(ZSTD(1)),
+ `MetricName` LowCardinality(String) CODEC(ZSTD(1)),
`MetricDescription` String CODEC(ZSTD(1)),
`MetricUnit` String CODEC(ZSTD(1)),
`Attributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
- `StartTimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
- `TimeUnix` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
+ `StartTimeUnix` DateTime CODEC(Delta, ZSTD(1)),
+ `TimeUnix` DateTime CODEC(Delta, ZSTD(1)),
`Count` UInt64 CODEC(Delta(8), ZSTD(1)),
`Sum` Float64 CODEC(ZSTD(1)),
`ValueAtQuantiles.Quantile` Array(Float64) CODEC(ZSTD(1)),
@@ -194,12 +198,13 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_summary
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_attr_key mapKeys(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1
+ INDEX idx_attr_value mapValues(Attributes) TYPE bloom_filter(0.01) GRANULARITY 1,
+ INDEX idx_time_minmax TimeUnix TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(TimeUnix)
-ORDER BY (ServiceName, MetricName, Attributes, toUnixTimestamp64Nano(TimeUnix))
-TTL toDateTime(TimeUnix) + ${TABLES_TTL}
+ORDER BY (ServiceName, MetricName, toStartOfHour(TimeUnix), cityHash64(Attributes), TimeUnix)
+TTL toDateTime(TimeUnix) + ${METRICS_TTL}
SETTINGS ttl_only_drop_parts = 1;
-- +goose StatementEnd
diff --git a/docker/otel-collector/schema/seed/00004_hyperdx_sessions.sql b/docker/otel-collector/schema/seed/00004_hyperdx_sessions.sql
index 89077b8328..d82bccfd55 100644
--- a/docker/otel-collector/schema/seed/00004_hyperdx_sessions.sql
+++ b/docker/otel-collector/schema/seed/00004_hyperdx_sessions.sql
@@ -30,6 +30,6 @@ ENGINE = MergeTree
PARTITION BY toDate(TimestampTime)
PRIMARY KEY (ServiceName, TimestampTime)
ORDER BY (ServiceName, TimestampTime, Timestamp)
-TTL TimestampTime + ${TABLES_TTL}
+TTL TimestampTime + ${SESSIONS_TTL}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
diff --git a/docker/otel-collector/schema/seed/00005_otel_traces.sql b/docker/otel-collector/schema/seed/00005_otel_traces.sql
index 699e1d30d7..ce964efb37 100644
--- a/docker/otel-collector/schema/seed/00005_otel_traces.sql
+++ b/docker/otel-collector/schema/seed/00005_otel_traces.sql
@@ -27,18 +27,18 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_traces
`SampleRate` UInt64 MATERIALIZED greatest(toUInt64OrZero(SpanAttributes['SampleRate']), 1) CODEC(T64, ZSTD(1)),
`ResourceAttributeItems` Array(String) ALIAS arrayMap((arr) -> concat(arr.1, '=', arr.2), ResourceAttributes::Array(Tuple(String, String))),
`SpanAttributeItems` Array(String) ALIAS arrayMap((arr) -> concat(arr.1, '=', arr.2), SpanAttributes::Array(Tuple(String, String))),
- INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_rum_session_id __hdx_materialized_rum.sessionId TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
+ INDEX idx_trace_id TraceId TYPE text(tokenizer = 'array'),
+ INDEX idx_rum_session_id __hdx_materialized_rum.sessionId TYPE text(tokenizer = 'array'),
+ INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE text(tokenizer = 'array'),
INDEX idx_res_attr_items ResourceAttributeItems TYPE text(tokenizer = 'array'),
- INDEX idx_span_attr_key mapKeys(SpanAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
+ INDEX idx_span_attr_key mapKeys(SpanAttributes) TYPE text(tokenizer = 'array'),
INDEX idx_span_attr_items SpanAttributeItems TYPE text(tokenizer = 'array'),
INDEX idx_duration Duration TYPE minmax GRANULARITY 1,
- INDEX idx_lower_span_name lower(SpanName) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
+ INDEX idx_lower_span_name SpanName TYPE text(tokenizer = 'splitByNonAlpha', preprocessor=lower(SpanName))
)
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SpanName, toDateTime(Timestamp))
-TTL toDate(Timestamp) + ${TABLES_TTL}
+TTL toDate(Timestamp) + ${TRACES_TTL}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
diff --git a/docker/otel-collector/schema/seed/00005_otel_traces_compat.sql b/docker/otel-collector/schema/seed/00005_otel_traces_compat.sql
index e2279b6c15..480be6b093 100644
--- a/docker/otel-collector/schema/seed/00005_otel_traces_compat.sql
+++ b/docker/otel-collector/schema/seed/00005_otel_traces_compat.sql
@@ -41,5 +41,5 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_traces
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SpanName, toDateTime(Timestamp))
-TTL toDate(Timestamp) + ${TABLES_TTL}
+TTL toDate(Timestamp) + ${TRACES_TTL}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
diff --git a/docker/otel-collector/schema/seed/00006_otel_logs_rollups.sql b/docker/otel-collector/schema/seed/00006_otel_logs_rollups.sql
index dfb448ce22..82db4d1be6 100644
--- a/docker/otel-collector/schema/seed/00006_otel_logs_rollups.sql
+++ b/docker/otel-collector/schema/seed/00006_otel_logs_rollups.sql
@@ -1,6 +1,6 @@
-- +goose Up
--- Map Attributes Key Value Rollup (all map columns in one table)
+-- Key Value Rollup (many columns in one table)
CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs_kv_rollup_15m
(
`Timestamp` DateTime,
@@ -14,61 +14,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs_kv_rollup_15m
ENGINE = SummingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ColumnIdentifier, Key, Timestamp, Value)
-TTL Timestamp + ${TABLES_TTL}
+TTL Timestamp + ${LOGS_TTL}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
--- Key Only Rollup (derived from KV rollup)
-CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_logs_key_rollup_15m
-(
- `Timestamp` DateTime,
- `ColumnIdentifier` LowCardinality(String),
- `Key` LowCardinality(String),
- `count` UInt64,
- INDEX idx_count_minmax count TYPE minmax GRANULARITY 1,
- INDEX idx_timestamp_minmax Timestamp TYPE minmax GRANULARITY 1
-)
-ENGINE = SummingMergeTree
-PARTITION BY toDate(Timestamp)
-ORDER BY (ColumnIdentifier, Key, Timestamp)
-TTL Timestamp + ${TABLES_TTL}
-SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
-
-CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.otel_logs_key_rollup_15m_mv TO ${DATABASE}.otel_logs_key_rollup_15m
-AS SELECT
- Timestamp,
- ColumnIdentifier,
- Key,
- sum(count) as count
-FROM ${DATABASE}.otel_logs_kv_rollup_15m
-GROUP BY ColumnIdentifier, Key, Timestamp;
-
-- Single MV: CTE with UNION ALL across all columns, then aggregate
CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.otel_logs_attr_kv_rollup_15m_mv TO ${DATABASE}.otel_logs_kv_rollup_15m
AS WITH elements AS (
- SELECT
- 'ResourceAttributes' AS ColumnIdentifier,
- toStartOfFifteenMinutes(Timestamp) AS Timestamp,
- replaceRegexpAll(entry.1, '\\[\\d+\\]', '[*]') AS Key,
- CAST(entry.2 AS String) AS Value
- FROM ${DATABASE}.otel_logs
- ARRAY JOIN ResourceAttributes AS entry
- UNION ALL
- SELECT
- 'LogAttributes' AS ColumnIdentifier,
- toStartOfFifteenMinutes(Timestamp) AS Timestamp,
- replaceRegexpAll(entry.1, '\\[\\d+\\]', '[*]') AS Key,
- CAST(entry.2 AS String) AS Value
- FROM ${DATABASE}.otel_logs
- ARRAY JOIN LogAttributes AS entry
- UNION ALL
- SELECT
- 'ScopeAttributes' AS ColumnIdentifier,
- toStartOfFifteenMinutes(Timestamp) AS Timestamp,
- replaceRegexpAll(entry.1, '\\[\\d+\\]', '[*]') AS Key,
- CAST(entry.2 AS String) AS Value
- FROM ${DATABASE}.otel_logs
- ARRAY JOIN ScopeAttributes AS entry
- UNION ALL
SELECT
'NativeColumn' AS ColumnIdentifier,
toStartOfFifteenMinutes(Timestamp) AS Timestamp,
diff --git a/docker/otel-collector/schema/seed/00007_otel_traces_rollups.sql b/docker/otel-collector/schema/seed/00007_otel_traces_rollups.sql
index 2ca574dbb2..80afc569cf 100644
--- a/docker/otel-collector/schema/seed/00007_otel_traces_rollups.sql
+++ b/docker/otel-collector/schema/seed/00007_otel_traces_rollups.sql
@@ -1,6 +1,6 @@
-- +goose Up
--- Key Value Rollup (all columns in one table)
+-- Key Value Rollup (many columns in one table)
CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_traces_kv_rollup_15m
(
`Timestamp` DateTime,
@@ -14,54 +14,12 @@ CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_traces_kv_rollup_15m
ENGINE = SummingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ColumnIdentifier, Key, Timestamp, Value)
-TTL Timestamp + ${TABLES_TTL}
+TTL Timestamp + ${TRACES_TTL}
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
--- Key Only Rollup (derived from KV rollup)
-CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_traces_key_rollup_15m
-(
- `Timestamp` DateTime,
- `ColumnIdentifier` LowCardinality(String),
- `Key` LowCardinality(String),
- `count` UInt64,
- INDEX idx_count_minmax count TYPE minmax GRANULARITY 1,
- INDEX idx_timestamp_minmax Timestamp TYPE minmax GRANULARITY 1
-)
-ENGINE = SummingMergeTree
-PARTITION BY toDate(Timestamp)
-ORDER BY (ColumnIdentifier, Key, Timestamp)
-TTL Timestamp + ${TABLES_TTL}
-SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
-
--- Key only rollup MV
-CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.otel_traces_key_rollup_15m_mv TO ${DATABASE}.otel_traces_key_rollup_15m
-AS SELECT
- Timestamp,
- ColumnIdentifier,
- Key,
- sum(count) AS count
-FROM ${DATABASE}.otel_traces_kv_rollup_15m
-GROUP BY ColumnIdentifier, Key, Timestamp;
-
-- Single MV: CTE with UNION ALL across all columns, then aggregate
CREATE MATERIALIZED VIEW IF NOT EXISTS ${DATABASE}.otel_traces_kv_rollup_15m_mv TO ${DATABASE}.otel_traces_kv_rollup_15m
AS WITH elements AS (
- SELECT
- 'ResourceAttributes' AS ColumnIdentifier,
- toStartOfFifteenMinutes(Timestamp) AS Timestamp,
- replaceRegexpAll(entry.1, '\\[\\d+\\]', '[*]') AS Key,
- CAST(entry.2 AS String) AS Value
- FROM ${DATABASE}.otel_traces
- ARRAY JOIN ResourceAttributes AS entry
- UNION ALL
- SELECT
- 'SpanAttributes' AS ColumnIdentifier,
- toStartOfFifteenMinutes(Timestamp) AS Timestamp,
- replaceRegexpAll(entry.1, '\\[\\d+\\]', '[*]') AS Key,
- CAST(entry.2 AS String) AS Value
- FROM ${DATABASE}.otel_traces
- ARRAY JOIN SpanAttributes AS entry
- UNION ALL
SELECT
'NativeColumn' AS ColumnIdentifier,
toStartOfFifteenMinutes(Timestamp) AS Timestamp,
diff --git a/docker/otel-collector/schema/seed/00008_otel_metrics_timeseries.sql b/docker/otel-collector/schema/seed/00008_otel_metrics_timeseries.sql
index 50323f3d05..48b8372b44 100644
--- a/docker/otel-collector/schema/seed/00008_otel_metrics_timeseries.sql
+++ b/docker/otel-collector/schema/seed/00008_otel_metrics_timeseries.sql
@@ -1,4 +1,4 @@
-- +goose Up
-CREATE TABLE IF NOT EXISTS ${DATABASE}.otel_metrics_ts
+CREATE TABLE IF NOT EXISTS ${DATABASE}.metrics_ts
ENGINE = TimeSeries
SETTINGS allow_experimental_time_series_table = 1;
diff --git a/docker/otel-collector/supervisor_docker.yaml.tmpl b/docker/otel-collector/supervisor_docker.yaml.tmpl
index f91e3cc59f..9439ae7c17 100644
--- a/docker/otel-collector/supervisor_docker.yaml.tmpl
+++ b/docker/otel-collector/supervisor_docker.yaml.tmpl
@@ -18,6 +18,10 @@ capabilities:
agent:
executable: /otelcontribcol
+{{- if eq (getenv "OTEL_SUPERVISOR_LOGS") "true" }}
+ # Stream the collector's stdout/stderr through te supervisor's own logger.
+ passthrough_logs: true
+{{- end }}
config_files:
- /etc/otelcol-contrib/config.yaml
{{- if getenv "CUSTOM_OTELCOL_CONFIG_FILE" }}
diff --git a/docs/alert-webhook-template-variables.md b/docs/alert-webhook-template-variables.md
new file mode 100644
index 0000000000..2a78f461a6
--- /dev/null
+++ b/docs/alert-webhook-template-variables.md
@@ -0,0 +1,118 @@
+# Alert webhook template variables
+
+**Generic** and **incident.io** webhook bodies are Handlebars templates. These
+variables are available inside them, so a receiver can route, filter or dedupe
+a HyperDX alert without parsing the human-readable message body.
+
+| Variable | Type | Notes |
+| --- | --- | --- |
+| `{{title}}` | string | Alert title. |
+| `{{body}}` | string | Rendered message body (markdown). |
+| `{{link}}` | string | Deep link back into HyperDX. |
+| `{{state}}` | string | Raw internal alert state. |
+| `{{eventId}}` | string | Unique id for this firing. |
+| `{{startTime}}` / `{{endTime}}` | number | Evaluation window, Unix ms. |
+| `{{startTimeISO}}` / `{{endTimeISO}}` | string | The same window, ISO-8601. |
+| `{{alertId}}` | string | Stable id of the alert itself — the key to dedupe on. |
+| `{{status}}` | string | `firing`, `resolved`, `no_data`, `pending` or `error`. |
+| `{{alertType}}` | string | `search`, `dashboard_chart` or `inline_query`. |
+| `{{comparator}}` | string | `>=`, `>`, `<`, `<=`, `=`, `!=`, `between`, `outside`. |
+| `{{threshold}}` | number | The configured threshold. For `between`/`outside`, the lower bound. |
+| `{{thresholdMax}}` | number | Upper bound of a `between`/`outside` condition. Always unset for every other comparator, even if the alert was once a range, so guard it (see below). |
+| `{{value}}` | number | The value that triggered or resolved the alert. |
+| `{{groupKey}}` | string | The breaching group, for a grouped alert. |
+| `{{var}}` | object | The breaching group's values by group-by column or SQL alias. Use one value at a time, e.g. `{{var.customer_id}}`, or `{{var.[service.name]}}` for a name with dots. Empty when not grouped, and on the resolve of a group that has no rows left. |
+| `{{sourceQuery}}` | string | The condition behind the alert — search expression or SQL (see below). |
+| `{{teamId}}` | string | Team the alert belongs to. |
+| `{{note}}` | string | The alert's freeform note — commonly a runbook link. |
+
+Strings are JSON-escaped, so they are safe to drop into a quoted slot.
+Numbers (`startTime`, `endTime`, `threshold`, `thresholdMax`, `value`) are
+emitted raw for unquoted slots. Every enriched variable is optional and renders
+as an empty string when the alert doesn't carry it — an alert with no group has
+an empty `{{groupKey}}`, for instance.
+
+`{{sourceQuery}}` reads whichever fields the alert's condition lives in: a
+saved-search alert reports the search's, a dashboard-tile alert its tile's, and
+an inline alert its own. For a raw SQL chart that is the whole SQL template.
+Otherwise the condition is spread over several inputs and they are joined with
+`AND`, each bracketed when there is more than one — for example
+`(ServiceName: "checkout") AND (SeverityText: "error")`. An alert with no
+condition at all reports an empty string.
+
+It reports only what the alert query actually applies, so it will not always
+match everything the chart shows. A builder chart contributes its chart-level
+`where` and the `aggCondition` of the series the alert reads — the last one,
+which is the series that produces the value. A chart's pinned `filters` and
+`having` are **not** included, because a tile or inline alert does not apply
+them. A saved search contributes its `where` and its pinned `filters`, which it
+does apply.
+
+Treat the result as opaque text rather than something to parse or re-run:
+nothing in the variable set says which dialect you got, and the parts of a
+builder condition can each be Lucene or SQL, so the joined string is not
+necessarily valid in either. It is truncated at 2000 characters. PromQL charts
+do not support alerts, so no webhook carries a PromQL expression.
+
+An empty string is not valid JSON in an unquoted numeric slot, so guard any
+number that may be absent. Compare against `undefined` rather than using
+`{{#if}}`, which treats a legitimate bound of `0` as absent:
+
+```
+{
+ "threshold": {{threshold}}{{#unless (eq thresholdMax undefined)}},
+ "threshold_max": {{thresholdMax}}{{/unless}}
+}
+```
+
+Keep a newline or space between `{{/unless}}` and a closing `}` — Handlebars
+reads `}}}` as a triple-stache and fails to compile the template.
+
+The template is Handlebars, not JSON: it only has to be JSON once rendered.
+Write string arguments inside `{{ … }}` with plain or single quotes —
+`{{#if (eq state "ALERT")}}` or `{{#if (eq state 'ALERT')}}` — never JSON-escaped
+as `\"ALERT\"`, which Handlebars cannot parse. The webhook form shows where a
+body fails to compile. The API rejects a body that doesn't compile, or that
+calls a helper other than `eq` and the Handlebars built-ins, when a webhook is
+created, updated, or tested.
+
+The **Send test** button on the webhook form fills every variable with a sample
+value, so a template that uses them can be checked before an alert fires. The
+sample is a `between` alert, so `{{thresholdMax}}` is populated there — a test
+send will not catch a template that breaks when an optional number is absent,
+which is what the guard above is for.
+
+## incident.io
+
+incident.io accepts only `firing` or `resolved` in `status`, so the default
+body maps HyperDX's state onto those two and carries `{{status}}` in
+`metadata.hyperdx_status`, alongside the alert id, condition and evaluation
+window. Only a firing and a resolve are delivered today, so that key reads
+`firing` or `resolved` as well. `metadata` is free-form, but incident.io only
+surfaces the keys its alert source is configured to parse.
+
+`deduplication_key` is `{{eventId}}`, which is stable per alert, group and
+channel across a firing and its resolve, so incident.io closes the alert it
+opened rather than opening a second one. It is derived from those three, so
+changing the alert's group-by or deleting and recreating the webhook starts a
+new key and leaves anything already open unresolved. Dedupe on `{{alertId}}`
+instead if every group of a grouped alert should collapse into one incident.
+
+## Example
+
+Routing by severity and deduping on the alert rather than the firing:
+
+```
+{
+ "alert_id": "{{alertId}}",
+ "dedup_key": "{{alertId}}-{{groupKey}}",
+ "status": "{{status}}",
+ "summary": "{{title}}",
+ "urgency": "{{#if (eq alertType 'dashboard_chart')}}low{{else}}high{{/if}}",
+ "value": {{value}},
+ "threshold": {{threshold}},
+ "window": { "start": "{{startTimeISO}}", "end": "{{endTimeISO}}" },
+ "runbook": "{{note}}",
+ "link": "{{link}}"
+}
+```
diff --git a/gascity/smoke/echo.py b/gascity/smoke/echo.py
new file mode 100644
index 0000000000..98e3d357c4
--- /dev/null
+++ b/gascity/smoke/echo.py
@@ -0,0 +1,38 @@
+# Header/body capture server for the HyperDX smoke test. Answers every request
+# with 200 and a tiny ClickHouse-ish body; appends one JSON line per request to
+# /captures/requests.jsonl (method, path, headers, body prefix).
+import json, os, sys
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+OUT = '/captures/requests.jsonl'
+
+class H(BaseHTTPRequestHandler):
+ def _handle(self):
+ if 'chunked' in (self.headers.get('transfer-encoding') or '').lower():
+ body = b''
+ while True:
+ size = int(self.rfile.readline().strip().split(b';')[0] or b'0', 16)
+ if size == 0:
+ self.rfile.readline()
+ break
+ body += self.rfile.read(size)
+ self.rfile.readline()
+ else:
+ n = int(self.headers.get('content-length') or 0)
+ body = self.rfile.read(n) if n else b''
+ rec = {'method': self.command, 'path': self.path,
+ 'headers': {k.lower(): v for k, v in self.headers.items()},
+ 'body_prefix': body[:1048576].decode('latin-1')}
+ with open(OUT, 'a') as f:
+ f.write(json.dumps(rec) + '\n')
+ payload = b'{"meta":[{"name":"1","type":"UInt8"}],"data":[{"1":1}],"rows":1}'
+ self.send_response(200)
+ self.send_header('Content-Type', 'application/json')
+ self.send_header('Content-Length', str(len(payload)))
+ self.end_headers()
+ self.wfile.write(payload)
+ do_GET = do_POST = do_PUT = do_DELETE = _handle
+ def log_message(self, *a):
+ pass
+
+ThreadingHTTPServer(('0.0.0.0', int(sys.argv[1]) if len(sys.argv) > 1 else 8123), H).serve_forever()
diff --git a/gascity/smoke/smoke.sh b/gascity/smoke/smoke.sh
new file mode 100755
index 0000000000..26aee005ab
--- /dev/null
+++ b/gascity/smoke/smoke.sh
@@ -0,0 +1,341 @@
+#!/usr/bin/env bash
+# Local smoke test for a gascity/hyperdx image: proxy-auth trust boundary,
+# secret hygiene (logs, ClickHouse proxy, OTel export), and FerretDB state.
+#
+# smoke.sh infra-up # network + FerretDB (prod-pinned) + capture servers
+# smoke.sh run # start HyperDX as container (fake secrets)
+# smoke.sh bootstrap # first-team registration (prod already has a team)
+# smoke.sh auth # positive/negative auth matrix + leak checks
+# smoke.sh telemetry # SMOKE_TELEMETRY=off run: self-telemetry stays off
+# smoke.sh seed # create a dashboard, saved search, webhook, alert
+# smoke.sh verify-seed # read the seeded state back
+# smoke.sh failclosed # unsafe proxy-auth configs must not start
+# smoke.sh rotation # current/previous secret overlap window
+# smoke.sh stop | infra-down
+#
+# `run` knobs: SMOKE_LOG_LEVEL (default info; use trace for the most verbose
+# request logs), SMOKE_TELEMETRY=off (no HYPERDX_API_KEY and no OTLP endpoint,
+# the production posture), SMOKE_SECRET / SMOKE_SECRET_PREVIOUS (rotation).
+set -euo pipefail
+
+NET=hdx-smoke
+PG_IMG=ghcr.io/ferretdb/postgres-documentdb@sha256:4a2364db8c91fd82855a4eab326202e247aa9ed6a6e8a85c76c1693a5a9ab408
+FDB_IMG=ghcr.io/ferretdb/ferretdb@sha256:7449f6ce1e75b187c328c9ce65ec4e8963767b34b4d604f9b9b6605c6d8d3ee7
+HERE=$(cd "$(dirname "$0")" && pwd)
+CAP=${CAP:-/tmp/hdx-smoke-captures}
+# Fake, test-only values. Same length for the wrong secret so the
+# timingSafeEqual branch (not the length short-circuit) is exercised too.
+SECRET=smoke-secret-0123456789abcdef0123456789abcdef
+WRONG_SAME_LEN=smoke-secret-ffffffffffffffffffffffffffffffff
+NEW_SECRET=smoke-rotated-secret-fedcba9876543210fedcba98
+# Browser credentials that must never reach stdout or the connection host.
+GATE_COOKIE=FAKE-GATE-SESSION-COOKIE
+BEARER=FAKE-BEARER-ACCESS-KEY-0123456789
+# No host port publishing (shared host, contested ports): talk to the
+# container's bridge IP directly. TARGET is the HyperDX container name.
+TARGET=${TARGET:-hdx-target}
+cip() { docker inspect -f "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}" "$1"; }
+HOST=hdx.smoke.test
+APP="http://$HOST:8080"; API="http://$HOST:8000"; IP=127.0.0.1
+set_urls() { IP=$(cip "$TARGET"); }
+pass=0; fail=0
+ok() { echo "PASS $*"; pass=$((pass + 1)); }
+ko() { echo "FAIL $*"; fail=$((fail + 1)); }
+expect() { # expect
+ if [ "$2" = "$3" ]; then ok "$1 (got $3)"; else ko "$1 (want $2, got $3)"; fi
+}
+curl() { command curl --resolve "$HOST:8080:$IP" --resolve "$HOST:8000:$IP" "$@"; }
+code() { curl -s -o /dev/null -w '%{http_code}' "$@"; }
+
+infra_up() {
+ docker network inspect $NET >/dev/null 2>&1 || docker network create $NET >/dev/null
+ mkdir -p $CAP && chmod 0777 $CAP
+ docker run -d --name hdx-pg --network $NET \
+ -e POSTGRES_USER=hyperdx -e POSTGRES_PASSWORD=pgpass -e POSTGRES_DB=postgres \
+ -e PGDATA=/var/lib/postgresql/data/pgdata $PG_IMG >/dev/null
+ for _ in $(seq 1 60); do
+ docker exec hdx-pg pg_isready -U hyperdx -d postgres >/dev/null 2>&1 && break; sleep 2
+ done
+ docker run -d --name hdx-ferretdb --network $NET \
+ -e FERRETDB_POSTGRESQL_URL=postgres://hyperdx:pgpass@hdx-pg:5432/postgres $FDB_IMG >/dev/null
+ # capture servers: a fake ClickHouse (connection host) and a fake OTLP endpoint
+ docker run -d --name hdx-echo-ch --network $NET --user "$(id -u):$(id -g)" -v "$HERE":/srv:ro \
+ -v $CAP:/captures python:3.12-alpine sh -c \
+ 'cp /srv/echo.py /tmp/e.py && sed -i s#requests.jsonl#ch.jsonl# /tmp/e.py && python /tmp/e.py 8123' >/dev/null
+ docker run -d --name hdx-echo-otlp --network $NET --user "$(id -u):$(id -g)" -v "$HERE":/srv:ro \
+ -v $CAP:/captures python:3.12-alpine sh -c \
+ 'cp /srv/echo.py /tmp/e.py && sed -i s#requests.jsonl#otlp.jsonl# /tmp/e.py && python /tmp/e.py 4318' >/dev/null
+ sleep 3
+}
+
+run() { # run [extra docker args...]
+ local img=$1 name=$2; shift 2
+ local telemetry=(-e HYPERDX_API_KEY=smoke-dummy-ingest-key
+ -e OTEL_EXPORTER_OTLP_ENDPOINT=http://hdx-echo-otlp:4318)
+ [ "${SMOKE_TELEMETRY:-capture}" = off ] && telemetry=()
+ local previous=()
+ [ -n "${SMOKE_SECRET_PREVIOUS:-}" ] &&
+ previous=(-e "PROXY_AUTH_SHARED_SECRET_PREVIOUS=$SMOKE_SECRET_PREVIOUS")
+ docker run -d --name "$name" --network $NET \
+ --user 1000:1000 --cap-drop ALL --security-opt no-new-privileges \
+ -e HYPERDX_API_PORT=8000 -e HYPERDX_APP_PORT=8080 \
+ -e "HYPERDX_LOG_LEVEL=${SMOKE_LOG_LEVEL:-info}" \
+ -e USAGE_STATS_ENABLED=false \
+ -e PROXY_AUTH_ENABLED=true -e PROXY_AUTH_ALLOWED_EMAIL_DOMAINS=gascity.com \
+ -e PROXY_AUTH_HEADER=x-auth-request-email \
+ -e "PROXY_AUTH_SHARED_SECRET=${SMOKE_SECRET:-$SECRET}" "${previous[@]}" \
+ -e HYPERDX_APP_URL=http://$HOST -e FRONTEND_URL=http://$HOST:8080 \
+ -e SERVER_URL=http://127.0.0.1:8000 \
+ -e MONGO_URI=mongodb://hyperdx:pgpass@hdx-ferretdb:27017/hyperdx \
+ "${telemetry[@]}" \
+ -e 'DEFAULT_CONNECTIONS=[{"name":"Smoke ClickHouse","host":"http://hdx-echo-ch:8123","username":"ro","password":"ro-pass"}]' \
+ -e 'DEFAULT_SOURCES=[{"from":{"databaseName":"otel","tableName":"otel_logs"},"kind":"log","name":"Logs","timestampValueExpression":"Timestamp","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Smoke ClickHouse"}]' \
+ "$@" "$img" >/dev/null
+ TARGET=$name; set_urls
+ for _ in $(seq 1 90); do
+ [ "$(code $API/health)" = 200 ] && [ "$(code $APP/)" != 000 ] && break
+ sleep 2
+ done
+ sleep 3
+ echo "started $name ($img) at $APP: api /health=$(code $API/health)"
+}
+
+# gcurl : a request as the gate would forward it
+gcurl() { local e=$1 s=$2; shift 2; curl -s -H "x-auth-request-email: $e" -H "x-hdx-proxy-auth-secret: $s" "$@"; }
+gcode() { local e=$1 s=$2; shift 2; gcurl "$e" "$s" -o /dev/null -w '%{http_code}' "$@"; }
+
+bootstrap() {
+ set_urls
+ local c
+ c=$(code -X POST $APP/api/register/password -H 'content-type: application/json' \
+ -d '{"email":"owner@gascity.com","password":"Sm0ke-Test-Passw0rd!","confirmPassword":"Sm0ke-Test-Passw0rd!"}')
+ echo "register first team owner: $c"
+}
+
+auth() {
+ TARGET=$1; set_urls
+ local name=$1 base=$APP/api jar body
+ jar=$(mktemp); rm -f $jar
+ : > $CAP/ch.jsonl; : > $CAP/otlp.jsonl
+ expect "T1 no headers -> 401" 401 "$(code $base/me)"
+ expect "T2 identity header without secret -> 401" 401 \
+ "$(code -H 'x-auth-request-email: alice@gascity.com' $base/me)"
+ expect "T3 wrong secret (same length) -> 401" 401 "$(gcode alice@gascity.com $WRONG_SAME_LEN $base/me)"
+ expect "T4 wrong secret (other length) -> 401" 401 "$(gcode alice@gascity.com short $base/me)"
+ expect "T4b empty secret header -> 401" 401 \
+ "$(code -H 'x-auth-request-email: alice@gascity.com' -H 'x-hdx-proxy-auth-secret;' $base/me)"
+ body=$(gcurl Alice@GasCity.com $SECRET -c $jar $base/me)
+ expect "T5 gate identity + secret -> /me is alice" alice@gascity.com \
+ "$(echo "$body" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("email"))' 2>/dev/null || echo none)"
+ # header names as the edge proxy sends them (canonical case); T14 then
+ # checks that this spelling is consumed before logging as well
+ body=$(curl -s -H 'X-Auth-Request-Email: alice@gascity.com' -H "X-Hdx-Proxy-Auth-Secret: $SECRET" $base/me)
+ expect "T5b canonical-case gate headers -> /me is alice" alice@gascity.com \
+ "$(echo "$body" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("email"))' 2>/dev/null || echo none)"
+ expect "T5c trailing-dot domain -> 401" 401 "$(gcode alice@gascity.com. $SECRET $base/me)"
+ body=$(gcurl alice+x@gascity.com $SECRET $base/me)
+ expect "T5d plus-address is its own identity, never linked to alice" alice+x@gascity.com \
+ "$(echo "$body" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("email"))' 2>/dev/null || echo none)"
+ expect "T6 bare session cookie (no gate headers) -> 401" 401 "$(code -b $jar $base/me)"
+ expect "T6b session cookie + wrong secret -> 401" 401 "$(gcode alice@gascity.com $WRONG_SAME_LEN -b $jar $base/me)"
+ expect "T7 domain outside allowlist -> 403" 403 "$(gcode mallory@evil.com $SECRET $base/me)"
+ expect "T8 multi-valued identity header -> 401" 401 "$(gcode 'a@gascity.com,b@gascity.com' $SECRET $base/me)"
+ local c8; c8=$(gcode $'a\x01b@gascity.com' $SECRET $base/me)
+ case $c8 in 400|401) ok "T8b control char in identity rejected (got $c8; 400 = Node's header parser)";;
+ *) ko "T8b control char in identity (got $c8)";; esac
+ local sid_alice sid_bob
+ sid_alice=$(awk '$6 == "connect.sid" {print $7}' $jar)
+ body=$(gcurl bob@gascity.com $SECRET -b $jar -c $jar.bob $base/me)
+ expect "T9 alice's cookie + gate says bob -> /me is bob" bob@gascity.com \
+ "$(echo "$body" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("email"))' 2>/dev/null || echo none)"
+ sid_bob=$(awk '$6 == "connect.sid" {print $7}' $jar.bob)
+ if [ -n "$sid_alice" ] && [ -n "$sid_bob" ] && [ "$sid_alice" != "$sid_bob" ]; then
+ ok "T9b identity switch issues a new session id (no fixation)"
+ else ko "T9b identity switch kept the session id (alice=${sid_alice:0:12} bob=${sid_bob:0:12})"; fi
+ expect "T10 password login does not bypass the gate" 401 "$(
+ curl -s -o /dev/null -c $jar.pw -X POST -H 'content-type: application/json' \
+ -d '{"email":"owner@gascity.com","password":"Sm0ke-Test-Passw0rd!"}' $base/login/password
+ code -b $jar.pw $base/me)"
+ expect "T11 direct API port (8000) enforces the same gate" 401 \
+ "$(code -H 'x-auth-request-email: alice@gascity.com' $API/me)"
+ expect "T11b direct API port with gate identity" 200 "$(gcode alice@gascity.com $SECRET $API/me)"
+ local loc
+ loc=$(gcurl alice@gascity.com $SECRET -o /dev/null -w '%{redirect_url}' $base/logout)
+ expect "T12 logout bounces to the gate sign-out" /oauth2/sign_out "${loc#$APP}"
+ # ClickHouse proxy: what reaches the (user-configurable) connection host?
+ local cid
+ cid=$(gcurl alice@gascity.com $SECRET -b $jar $base/connections | python3 -c \
+ 'import json,sys; d=json.load(sys.stdin); d=d.get("data",d) if isinstance(d,dict) else d; print(d[0].get("id") or d[0].get("_id"))')
+ # with the SSO gate's cookie and a bearer token, as a browser behind
+ # oauth2-proxy (or a script holding an access key) could send them
+ gcurl alice@gascity.com $SECRET -o /dev/null -X POST \
+ -H "cookie: _oauth2_proxy=$GATE_COOKIE" -H "authorization: Bearer $BEARER" \
+ -H "x-hyperdx-connection-id: $cid" -H 'content-type: text/plain' \
+ --data 'SELECT 1 FORMAT JSON' "$base/clickhouse-proxy?query=SELECT%201" || true
+ sleep 1
+ echo "INFO CH proxy captured $(wc -l < $CAP/ch.jsonl) request(s) at the connection host"
+ if grep -q "$SECRET" $CAP/ch.jsonl; then ko "T13 shared secret forwarded to the connection host"
+ else ok "T13 shared secret not forwarded to the connection host"; fi
+ if grep -q "$GATE_COOKIE" $CAP/ch.jsonl; then ko "T13b SSO gate cookie forwarded to the connection host"
+ else ok "T13b SSO gate cookie not forwarded to the connection host"; fi
+ if grep -q "$BEARER" $CAP/ch.jsonl; then ko "T13c bearer token forwarded to the connection host"
+ else ok "T13c bearer token not forwarded to the connection host"; fi
+ # error paths must not log the secret either (T14 scans their output)
+ expect "T14e malformed JSON body (error handler path) -> 400" 400 \
+ "$(gcode alice@gascity.com $SECRET -b $jar -X POST -H 'content-type: application/json' \
+ --data '{"name":' $base/dashboards)"
+ expect "T14f connection-test failure (console.error path) -> 500" 500 \
+ "$(gcode alice@gascity.com $SECRET -b $jar -X POST -H 'content-type: application/json' \
+ --data '{"host":"http://hdx-smoke-no-such-host.invalid:8123"}' $base/clickhouse-proxy/test)"
+ # Request logs (stdout -> pod logs -> otel_logs in prod)
+ sleep 2
+ docker logs "$name" > $CAP/stdout.$name.log 2>&1 || true
+ local nl; nl=$(grep -c "$SECRET" $CAP/stdout.$name.log || true)
+ if [ "$nl" -gt 0 ]; then ko "T14 shared secret written to container stdout ($nl lines)"
+ else ok "T14 shared secret absent from container stdout"; fi
+ nl=$(grep -c 'connect.sid=' $CAP/stdout.$name.log || true)
+ if [ "$nl" -gt 0 ]; then ko "T14b session cookie written to container stdout ($nl lines)"
+ else ok "T14b session cookie absent from container stdout"; fi
+ nl=$(grep -c "$GATE_COOKIE" $CAP/stdout.$name.log || true)
+ if [ "$nl" -gt 0 ]; then ko "T14c SSO gate cookie written to container stdout ($nl lines)"
+ else ok "T14c SSO gate cookie absent from container stdout"; fi
+ nl=$(grep -c "$BEARER" $CAP/stdout.$name.log || true)
+ if [ "$nl" -gt 0 ]; then ko "T14d bearer token written to container stdout ($nl lines)"
+ else ok "T14d bearer token absent from container stdout"; fi
+ echo "INFO container stdout: $(wc -l < $CAP/stdout.$name.log) lines," \
+ "$(grep -c '"req":{' $CAP/stdout.$name.log || true) request-log lines"
+ sleep 8 # let the OTLP batch processors flush
+ if grep -q "$SECRET" $CAP/otlp.jsonl 2>/dev/null; then
+ ko "T15 shared secret exported via OTLP ($(grep -c "$SECRET" $CAP/otlp.jsonl) export requests)"
+ else ok "T15 shared secret absent from OTLP export"; fi
+ if grep -q -e "$GATE_COOKIE" -e "$BEARER" -e 'connect.sid=' $CAP/otlp.jsonl 2>/dev/null; then
+ ko "T15b browser credential (gate cookie, bearer or session cookie) exported via OTLP"
+ else ok "T15b browser credentials absent from OTLP export"; fi
+ echo "INFO OTLP export requests captured so far: $(wc -l < $CAP/otlp.jsonl 2>/dev/null || echo 0)"
+ echo "SUMMARY $name: $pass passed, $fail failed"
+ rm -f $jar $jar.pw $jar.bob
+}
+
+# Production posture (SMOKE_TELEMETRY=off at `run`): no HYPERDX_API_KEY and no
+# OTLP endpoint, so neither the API, the alert task nor the browser starts the
+# bundled OpenTelemetry SDK (the Next standalone server ships no
+# instrumentation hook at all).
+telemetry() {
+ TARGET=$1; set_urls
+ local name=$1 body nl
+ : > $CAP/otlp.jsonl
+ gcurl alice@gascity.com $SECRET -o /dev/null $APP/api/me
+ body=$(curl -s $APP/api/config)
+ expect "TEL1 /api/config hands out no apiKey" absent \
+ "$(echo "$body" | python3 -c 'import json,sys; print("present" if json.load(sys.stdin).get("apiKey") else "absent")' 2>/dev/null || echo unparsable)"
+ sleep 8
+ docker logs "$name" > $CAP/stdout.$name.log 2>&1 || true
+ nl=$(grep 'OpenTelemetry SDK initialization skipped' $CAP/stdout.$name.log |
+ sed -n 's/^\[\([A-Z-]*\)\].*/\1/p' | sort -u | tr '\n' ' ' || true)
+ expect "TEL2 SDK initialization skipped in the API and the alert task" "ALERT-TASK API " "$nl"
+ nl=$(grep -c -i 'health check \(passed\|failed\)' $CAP/stdout.$name.log || true)
+ expect "TEL3 no SDK health checks against an export endpoint" 0 "$nl"
+ expect "TEL4 nothing exported over OTLP" 0 "$(wc -l < $CAP/otlp.jsonl)"
+ echo "SUMMARY $name telemetry: $pass passed, $fail failed"
+}
+
+# Rotation window (attended): current=new + previous=old accepts both, then
+# clearing previous rejects the old value. previous == current must refuse to
+# start without printing the value.
+rotation() {
+ local img=$1 out
+ SMOKE_SECRET=$NEW_SECRET SMOKE_SECRET_PREVIOUS=$SECRET run "$img" hdx-rot
+ expect "R1 previous (old) secret accepted during overlap" 200 "$(gcode alice@gascity.com $SECRET $API/me)"
+ expect "R2 current (new) secret accepted during overlap" 200 "$(gcode alice@gascity.com $NEW_SECRET $API/me)"
+ expect "R3 any other secret rejected during overlap" 401 "$(gcode alice@gascity.com $WRONG_SAME_LEN $API/me)"
+ docker rm -f hdx-rot >/dev/null
+ SMOKE_SECRET=$NEW_SECRET run "$img" hdx-rot
+ expect "R4 old secret rejected once previous is cleared" 401 "$(gcode alice@gascity.com $SECRET $API/me)"
+ expect "R5 new secret accepted once previous is cleared" 200 "$(gcode alice@gascity.com $NEW_SECRET $API/me)"
+ docker rm -f hdx-rot >/dev/null
+ out=$(start_refused "$img" -e PROXY_AUTH_ENABLED=true \
+ -e "PROXY_AUTH_SHARED_SECRET=$NEW_SECRET" -e "PROXY_AUTH_SHARED_SECRET_PREVIOUS=$NEW_SECRET")
+ if echo "$out" | grep -q 'PROXY_AUTH_SHARED_SECRET_PREVIOUS must differ'; then
+ ok "R6 previous == current refuses to start"
+ else ko "R6 previous == current started (or failed differently)"; fi
+ if echo "$out" | grep -q "$NEW_SECRET"; then ko "R6b the refusal printed the secret"
+ else ok "R6b the refusal does not print the secret"; fi
+ echo "SUMMARY rotation: $pass passed, $fail failed"
+}
+
+seed() {
+ set_urls
+ local base=$APP/api src wh ss
+ src=$(gcurl alice@gascity.com $SECRET $base/sources | python3 -c \
+ 'import json,sys; d=json.load(sys.stdin); print(d[0].get("id") or d[0].get("_id"))')
+ gcurl alice@gascity.com $SECRET -o /dev/null -w 'dashboard %{http_code}\n' -X POST \
+ -H 'content-type: application/json' $base/dashboards \
+ -d '{"name":"smoke-dashboard","tiles":[],"tags":["smoke"]}'
+ ss=$(gcurl alice@gascity.com $SECRET -X POST -H 'content-type: application/json' $base/saved-search \
+ -d "{\"name\":\"smoke-search\",\"select\":\"\",\"where\":\"SeverityText:error\",\"whereLanguage\":\"lucene\",\"source\":\"$src\",\"tags\":[]}" \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("id") or d.get("_id"))')
+ echo "saved search $ss"
+ wh=$(gcurl alice@gascity.com $SECRET -X POST -H 'content-type: application/json' $base/webhooks \
+ -d '{"name":"smoke-hook","service":"generic","url":"https://example.com/hook","body":"{}"}' \
+ | python3 -c 'import json,sys; d=json.load(sys.stdin); d=d.get("data",d); print(d.get("id") or d.get("_id"))')
+ echo "webhook $wh"
+ gcurl alice@gascity.com $SECRET -o /dev/null -w 'alert %{http_code}\n' -X POST \
+ -H 'content-type: application/json' $base/alerts \
+ -d "{\"source\":\"saved_search\",\"savedSearchId\":\"$ss\",\"interval\":\"5m\",\"threshold\":1,\"thresholdType\":\"above\",\"channel\":{\"type\":\"webhook\",\"webhookId\":\"$wh\"}}"
+}
+
+verify_seed() {
+ set_urls
+ local base=$APP/api
+ echo "dashboards: $(gcurl alice@gascity.com $SECRET $base/dashboards | python3 -c 'import json,sys; d=json.load(sys.stdin); d=d.get("data",d) if isinstance(d,dict) else d; print([x.get("name") for x in d])')"
+ echo "saved searches: $(gcurl alice@gascity.com $SECRET $base/saved-search | python3 -c 'import json,sys; d=json.load(sys.stdin); d=d.get("data",d) if isinstance(d,dict) else d; print([x.get("name") for x in d])')"
+ echo "alerts: $(gcurl alice@gascity.com $SECRET $base/alerts | python3 -c 'import json,sys; d=json.load(sys.stdin); d=d.get("data",d) if isinstance(d,dict) else d; print([(x.get("displayName"), x.get("tags"), x.get("state")) for x in d])')"
+ echo "me: $(gcurl alice@gascity.com $SECRET $base/me | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get("email"), "accessKey" in d)')"
+}
+
+# start_refused [docker args...]: run a config that must refuse to
+# start; print up to 20 kB of its output. Waits at most 90 s, then removes the
+# container either way, so an image that wrongly starts is not left running.
+start_refused() {
+ local img=$1; shift
+ docker rm -f hdx-refuse >/dev/null 2>&1 || true
+ docker run -d --name hdx-refuse --network $NET --user 1000:1000 \
+ -e MONGO_URI=mongodb://hyperdx:pgpass@hdx-ferretdb:27017/hyperdx \
+ -e FRONTEND_URL=http://localhost:8080 "$@" "$img" >/dev/null
+ timeout 90 docker wait hdx-refuse >/dev/null 2>&1 || true
+ docker logs hdx-refuse 2>&1 | head -c 20000 || true
+ docker rm -f hdx-refuse >/dev/null 2>&1 || true
+}
+
+failclosed() {
+ local img=$1 out
+ out=$(start_refused "$img" -e PROXY_AUTH_ENABLED=true)
+ if echo "$out" | grep -q 'requires PROXY_AUTH_SHARED_SECRET'; then
+ ok "T16 refuses to start with PROXY_AUTH_ENABLED and no secret"
+ else ko "T16 started (or failed differently) without a secret"; fi
+ # the SDK's advanced network capture records request headers (the secret
+ # included) before the app can consume them
+ out=$(start_refused "$img" -e PROXY_AUTH_ENABLED=true \
+ -e "PROXY_AUTH_SHARED_SECRET=$SECRET" -e HDX_NODE_ADVANCED_NETWORK_CAPTURE=true)
+ if echo "$out" | grep -q 'incompatible with HDX_NODE_ADVANCED_NETWORK_CAPTURE'; then
+ ok "T16b refuses to start with advanced network capture on"
+ else ko "T16b started (or failed differently) with advanced network capture on"; fi
+ if echo "$out" | grep -q "$SECRET"; then ko "T16c the capture refusal printed the secret"
+ else ok "T16c the capture refusal does not print the secret"; fi
+}
+
+case "${1:-}" in
+ infra-up) infra_up ;;
+ infra-down) docker rm -f hdx-pg hdx-ferretdb hdx-echo-ch hdx-echo-otlp >/dev/null 2>&1 || true
+ docker network rm $NET >/dev/null 2>&1 || true ;;
+ run) shift; run "$@" ;;
+ stop) docker rm -f "$2" >/dev/null ;;
+ bootstrap) TARGET=$2; bootstrap ;;
+ auth) auth "$2" ;;
+ telemetry) telemetry "$2" ;;
+ seed) TARGET=$2; seed ;;
+ verify-seed) TARGET=$2; verify_seed ;;
+ failclosed) failclosed "$2" ;;
+ rotation) rotation "$2" ;;
+ *) sed -n '2,19p' "$0"; exit 2 ;;
+esac
diff --git a/knip.json b/knip.json
index 7868b33e02..7d88c3d288 100644
--- a/knip.json
+++ b/knip.json
@@ -1,6 +1,9 @@
{
"$schema": "https://unpkg.com/knip@6/schema.json",
"workspaces": {
+ ".": {
+ "entry": ["scripts/__tests__/*.test.mjs"]
+ },
"packages/app": {
"entry": ["pages/**/*.{ts,tsx}", "scripts/*.js", "tests/e2e/**/*.ts"],
"project": ["src/**/*.{ts,tsx}", "pages/**/*.{ts,tsx}"],
@@ -8,7 +11,8 @@
"ignoreDependencies": [
"eslint-config-next",
"@chromatic-com/storybook",
- "@storybook/addon-themes"
+ "@storybook/addon-themes",
+ "react-is"
]
},
"packages/api": {
@@ -25,16 +29,32 @@
},
"packages/common-utils": {
"entry": ["src/**/*.ts", "!src/__tests__/**", "!src/**/*.test.*"],
- "project": ["src/**/*.ts"]
+ "project": ["src/**/*.ts"],
+ "ignoreDependencies": ["@stryker-mutator/core"],
+ "ignoreBinaries": ["stryker"]
+ },
+ "packages/hdx-eval": {
+ "project": ["src/**/*.ts"],
+ "ignoreDependencies": ["tsx"],
+ "ignoreBinaries": ["claude"]
}
},
- "ignore": ["scripts/dev-portal/**", ".github/scripts/**"],
- "ignoreBinaries": ["make", "migrate", "playwright"],
+ "ignore": [
+ "scripts/dev-portal/**",
+ ".github/scripts/**",
+ ".github/actions/**",
+ "docker/hyperdx/**",
+ ".claude/**"
+ ],
+ "ignoreBinaries": ["make", "migrate", "stryker"],
"ignoreDependencies": [
"@dotenvx/dotenvx",
"concurrently",
"dotenv",
- "babel-plugin-react-compiler"
+ "babel-plugin-react-compiler",
+ "@stryker-mutator/core",
+ "jscpd"
],
+ "include": ["nsExports"],
"exclude": ["enumMembers", "duplicates"]
}
diff --git a/nx.json b/nx.json
index dce58ff153..88bf780a5f 100644
--- a/nx.json
+++ b/nx.json
@@ -1,4 +1,5 @@
{
+ "analytics": false,
"affected": {
"defaultBase": "main"
},
@@ -8,19 +9,61 @@
},
"targetDefaults": {
"build": {
+ "cache": true,
"dependsOn": ["^build"]
- }
- },
- "tasksRunnerOptions": {
- "default": {
- "runner": "nx/tasks-runners/default",
- "options": {
- "cacheableOperations": ["build", "dev:build", "lint", "test"]
- }
+ },
+ "ci:build": {
+ "cache": true,
+ "dependsOn": ["^ci:build"],
+ // The `runtime` input pairs with `.nvmrc` in sharedGlobals: the file
+ // catches a declared version bump, this catches an actual interpreter
+ // that disagrees with it (local shells, a runner image change).
+ "inputs": ["production", "^production", { "runtime": "node --version" }],
+ "outputs": ["{projectRoot}/dist"]
+ },
+ "ci:lint": {
+ "cache": true,
+ "dependsOn": ["^ci:build"],
+ "inputs": ["default", "^production", { "runtime": "node --version" }]
+ },
+ // Unit tests import the compiled `dist` of workspace dependencies (e.g.
+ // `@hyperdx/common-utils/dist/types`), so they need the same `^ci:build`
+ // that `ci:lint` has. Without it, `nx run-many -t ci:unit` can run jest
+ // against a stale or absent dist — which fails intermittently, because the
+ // nx cache usually leaves a usable dist behind and occasionally does not.
+ "ci:unit": {
+ "cache": true,
+ "dependsOn": ["^ci:build"],
+ "inputs": ["default", "^production", { "runtime": "node --version" }]
+ },
+ // `outputs` is not optional here. Without it nx caches only the terminal
+ // output, so a cache hit replays a convincing build log and leaves `dist`
+ // exactly as it found it — stale, or missing entirely. That bites because
+ // `ci:build` writes the same `dist` from a different command (tsup alone,
+ // no `tsc --emitDeclarationOnly`), so running unit tests and then `yarn dev`
+ // leaves you on whichever dist won the race, with no sign anything is off.
+ "dev:build": {
+ "cache": true,
+ "outputs": ["{projectRoot}/dist"]
+ },
+ "lint": {
+ "cache": true
+ },
+ "test": {
+ "cache": true
}
},
"namedInputs": {
- "default": ["{projectRoot}/**/*", "sharedGlobals"],
+ "default": [
+ "{projectRoot}/**/*",
+ "sharedGlobals",
+ // `packages/api/openapi.json` is regenerated by `docgen` inside the api's
+ // own `ci:lint`, so leaving it in would make that task mutate its own
+ // input set. It has to be excluded here rather than on `ci:lint` itself:
+ // `@hyperdx/app` depends on the api, so the file also reaches app's lint
+ // hash via `^production`, which expands the *dependency's* named inputs.
+ "!{projectRoot}/openapi.json"
+ ],
"production": [
"default",
"!{projectRoot}/**/?(*.)+(spec|test).[jt]s?(x)?(.snap)",
@@ -28,6 +71,13 @@
"!{projectRoot}/jest.config.[jt]s",
"!{projectRoot}/eslint.config.mjs"
],
- "sharedGlobals": []
+ "sharedGlobals": [
+ "{workspaceRoot}/tsconfig.base.json",
+ "{workspaceRoot}/.prettierrc",
+ "{workspaceRoot}/.prettierignore",
+ "{workspaceRoot}/package.json",
+ "{workspaceRoot}/yarn.lock",
+ "{workspaceRoot}/.nvmrc"
+ ]
}
}
diff --git a/package.json b/package.json
index e2133c7e01..6659ab07d9 100644
--- a/package.json
+++ b/package.json
@@ -24,9 +24,10 @@
"eslint-plugin-security": "^3.0.1",
"eslint-plugin-simple-import-sort": "^12.1.1",
"husky": "^8.0.3",
- "knip": "^6.0.1",
+ "jscpd": "5.1.2",
+ "knip": "^6.19.0",
"lint-staged": "^13.1.2",
- "nx": "21.3.11",
+ "nx": "23.1.1",
"prettier": "3.3.3",
"tslib": "^2.6.0",
"typescript-eslint": "^8.46.0"
@@ -47,13 +48,17 @@
"dev:compose": "sh -c '. ./scripts/dev-env.sh && docker compose -p \"$HDX_DEV_PROJECT\" -f docker-compose.dev.yml'",
"knip": "knip",
"knip:ci": "knip --reporter json",
+ "ratchet": "node scripts/ci/ratchet.mjs",
+ "ratchet:update": "node scripts/ci/ratchet.mjs --update",
+ "dupes": "node scripts/ci/dupes.mjs",
+ "dupes:all": "node scripts/ci/dupes.mjs --all",
"lint": "npx nx run-many -t ci:lint",
"lint:fix": "npx nx run-many -t lint:fix",
"version": "make version",
- "release": "npx changeset tag && npx changeset publish"
+ "release": "npx changeset publish --no-git-tag && npx changeset tag"
},
"lint-staged": {
- "packages/api/src/routers/external-api/**/*.ts": [
+ "packages/api/src/{routers/external-api/**/*.ts,utils/swagger.ts}": [
"prettier --write --ignore-unknown",
"eslint --flag v10_config_lookup_from_file --fix --quiet",
"sh -c 'cd packages/api && yarn run lint:openapi && git add openapi.json'"
@@ -68,18 +73,22 @@
},
"packageManager": "yarn@4.13.0",
"resolutions": {
- "@types/react": "19.0.7",
- "@types/react-dom": "19.0.3",
+ "@types/react": "19.2.7",
+ "@types/react-dom": "19.2.3",
"@types/express": "4.17.21",
"@types/express-serve-static-core": "4.17.43",
"express": "^4.20.0",
"send": "^0.19.0",
"serve-static": "^1.16.0",
"cookie": "^0.7.0",
- "brace-expansion": "^2.0.2",
+ "@stryker-mutator/core/minimatch": "^9.0.5",
"diff": "^5.2.2",
"on-headers": "^1.1.0",
- "fast-xml-parser": "^4.5.4",
- "systeminformation": "^5.24.0"
+ "fast-xml-parser": "^4.5.6",
+ "systeminformation": "^5.24.0",
+ "@asyncapi/specs": "6.11.1"
+ },
+ "resolutionsNotes": {
+ "@stryker-mutator/core/minimatch": "Dev tooling, not a CVE override. Added when a blanket `brace-expansion` pin forced v2 (CJS) while minimatch v10 needs v5's named exports. That blanket pin is gone — it broke `nx affected` the same way — so brace-expansion now resolves per consumer and Stryker would most likely work on minimatch v10. The pin stays because no CI job runs `yarn dev:mutation`, so nothing would catch it if that guess is wrong. Drop it once someone has run the mutation suite on v10. See CONTRIBUTING.md#mutation-tests."
}
}
diff --git a/packages/api/.env.development b/packages/api/.env.development
index 8ba0a6c0c8..d97b0ff888 100644
--- a/packages/api/.env.development
+++ b/packages/api/.env.development
@@ -1,6 +1,7 @@
# Ports come from scripts/dev-env.sh (worktree isolation) or root .env (defaults)
HYPERDX_LOG_LEVEL=debug
-EXPRESS_SESSION_SECRET="hyperdx is cool 👋"
+# Development only: keep sessions valid across nodemon restarts.
+EXPRESS_SESSION_SECRET="local-development-only"
FRONTEND_URL="http://localhost:${HYPERDX_APP_PORT}"
HDX_NODE_ADVANCED_NETWORK_CAPTURE=1
HDX_NODE_BETA_MODE=1
@@ -21,7 +22,8 @@ USAGE_STATS_ENABLED=false
NODE_OPTIONS="--max-http-header-size=131072"
ENABLE_SWAGGER=true
DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://localhost:${HDX_DEV_CH_HTTP_PORT}","username":"default","password":""}]
-DEFAULT_SOURCES=[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","metadataMaterializedViews":{"keyRollupTable":"otel_logs_key_rollup_15m","kvRollupTable":"otel_logs_kv_rollup_15m","granularity":"15 minute"},"connection":"Local ClickHouse","traceSourceId":"Traces","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","metadataMaterializedViews":{"keyRollupTable":"otel_traces_key_rollup_15m","kvRollupTable":"otel_traces_kv_rollup_15m","granularity":"15 minute"},"connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","_id":"682586a8b1f81924e628e808","id":"682586a8b1f81924e628e808"},"connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","sessionSourceId":"Sessions"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"JSON Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"JSON Traces","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"JSON Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"JSON Logs","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"JSON Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum"},"connection":"Local ClickHouse","logSourceId":"JSON Logs","traceSourceId":"JSON Traces"}]
+DEFAULT_SOURCES=[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","severityNumberExpression":"SeverityNumber","eventNameExpression":"EventName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","metadataMaterializedViews":{"kvRollupTable":"otel_logs_kv_rollup_15m","granularity":"15 minute"},"connection":"Local ClickHouse","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","spanEventsValueExpression":"Events","spanLinksValueExpression":"Links","metadataMaterializedViews":{"kvRollupTable":"otel_traces_kv_rollup_15m","granularity":"15 minute"},"connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","summary":"otel_metrics_summary","exponential histogram":"otel_metrics_exponential_histogram"},"connection":"Local ClickHouse","logSourceId":"Logs"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","resourceAttributesExpression":"ResourceAttributes","connection":"Local ClickHouse","traceSourceId":"Traces"},{"from":{"databaseName":"otel_json","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"JSON Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","severityNumberExpression":"SeverityNumber","eventNameExpression":"EventName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"JSON Traces","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"JSON Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","spanEventsValueExpression":"Events","spanLinksValueExpression":"Links","connection":"Local ClickHouse","logSourceId":"JSON Logs","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"JSON Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","summary":"otel_metrics_summary","exponential histogram":"otel_metrics_exponential_histogram"},"connection":"Local ClickHouse","logSourceId":"JSON Logs"},{"kind":"promql","name":"PromQL","connection":"Local ClickHouse","from":{"databaseName":"default","tableName":"metrics_ts"},"timestampValueExpression":"timestamp"}]
+
INGESTION_API_KEY="super-secure-ingestion-api-key"
HYPERDX_API_KEY=$INGESTION_API_KEY
ANTHROPIC_API_KEY="your-anthropic-api-key-here"
diff --git a/packages/api/.env.e2e b/packages/api/.env.e2e
index 41fbd2f1cf..9db633c4d6 100644
--- a/packages/api/.env.e2e
+++ b/packages/api/.env.e2e
@@ -21,3 +21,6 @@ OPAMP_PORT=${HDX_E2E_OPAMP_PORT:-20320}
# Disable usage stats for e2e tests
USAGE_STATS_ENABLED=false
+
+# Allow PromQL queries in e2e tests.
+ENABLE_PROMQL=true
diff --git a/packages/api/.env.test b/packages/api/.env.test
index 0b67f560d1..45ee1e9b43 100644
--- a/packages/api/.env.test
+++ b/packages/api/.env.test
@@ -6,7 +6,6 @@ CLICKHOUSE_HOST=http://localhost:${HDX_CI_CH_PORT:-18123}
CLICKHOUSE_PASSWORD=api
CLICKHOUSE_USER=api
RUN_SCHEDULED_TASKS_EXTERNALLY=true
-EXPRESS_SESSION_SECRET="hyperdx is cool 👋"
FRONTEND_URL=http://app:8080
MONGO_URI=mongodb://localhost:${HDX_CI_MONGO_PORT:-39999}/hyperdx-test
NODE_ENV=test
@@ -16,4 +15,4 @@ OPAMP_PORT=${HDX_CI_OPAMP_PORT:-14320}
# Note: the logger module is mocked in jest.setup.ts to suppress expected
# operational noise (validation errors, MCP tool errors, etc.) during tests.
HYPERDX_LOG_LEVEL=error
-ENABLE_PROMQL=true
\ No newline at end of file
+ENABLE_PROMQL=true
diff --git a/packages/api/CHANGELOG.md b/packages/api/CHANGELOG.md
index 5f4e8b8e17..fee0b4d08f 100644
--- a/packages/api/CHANGELOG.md
+++ b/packages/api/CHANGELOG.md
@@ -1,5 +1,1286 @@
# @hyperdx/api
+## 2.42.0
+
+### Minor Changes
+
+- 9c2c7031: feat: Support PromQL dashboard tiles in the external API
+
+ `/api/v2/dashboards` now returns PromQL tiles and accepts them on create and update, as `configType: "promql"` with a `sourceId` and a list of `expressions`. Previously PromQL tiles were left out of responses, so saving a dashboard back through the API deleted them. MCP dashboard tools still hide PromQL tiles until they support editing them.
+
+- eb72cfa1: feat(mcp): add `clickstack_service_health`, a one-call per-service health report for a trace source (throughput, error rate, p50/p95/p99, top error groups), compared with yesterday and ranked worst first
+- 95aef70f: Add an optional `eventNameExpression` to log sources, pointing at the
+ OpenTelemetry log EventName. Source auto-detection sets it when an OTel log
+ table has a string `EventName` column. The field is stored and editable only:
+ nothing reads it yet. HDX-5787
+- bc47e485: Add an optional `severityNumberExpression` to log sources, pointing at the
+ numeric OpenTelemetry SeverityNumber (1-24). Source auto-detection sets it when
+ an OTel log table has an integer `SeverityNumber` column. The field is stored
+ and editable only: log levels still come from the severity text.
+- cfa865bd: feat: expose group-by values to webhook body templates as `{{var.}}`, and keep them on resolve notifications
+
+### Patch Changes
+
+- 9891a791: feat: Add ad hoc dashboard filter schemas and types
+- 27cccf7a: fix: Reject creating or updating PromQL sources through the external API when PromQL is not enabled
+
+ `POST` and `PUT /api/v2/sources` now return a 400 for `kind: "promql"` unless `ENABLE_PROMQL` is set, since PromQL sources cannot be queried without it.
+
+- 8d064df4: fix: Reject external API PromQL tiles that reference a non-PromQL source, and keep PromQL tiles when an MCP agent updates a dashboard
+
+ `/api/v2/dashboards` now returns a 400 when a PromQL tile's `sourceId` is not a PromQL source. MCP `clickstack_save_dashboard` updates no longer delete the PromQL tiles that MCP hides from agents.
+
+- 0000fa4b: feat: Add a line interpolation display setting (smooth, linear, step)
+- ee9f84b2: feat(mcp): record the response size of each MCP tool call
+- 24d324c2: fix(mcp): store whereLanguage as lucene when clickstack_save_saved_search creates a search without one, so its alerts no longer parse a Lucene filter as SQL
+- 748315d7: feat(mcp): advertise a title, description, website URL, and SVG icon in the MCP server's initialize response
+- cdfae457: fix(mcp): `clickstack_sql` now binds the bucket-size parameters its description
+ advertises. `{intervalSeconds:Int64}`, `{intervalMilliseconds:Int64}`,
+ `$__interval_s`, `$__timeInterval` and `$__timeInterval_ms` used to fail with
+ "Substitution intervalSeconds is not set", including the tool's own time-series
+ example.
+- b00d4100: feat(mcp): `clickstack_sql` accepts an optional `sourceId`, so `$__sourceTable`
+ expands to that source's table. Unknown sources, other teams' sources, and a
+ source on a different connection than `connectionId` are rejected, as is
+ `$__sourceTable` without a `sourceId`.
+- a06b17fb: fix(mcp): stop `clickstack_sql` callers from guessing table and timestamp column
+ names. `clickstack_list_sources` now returns each source's database-qualified
+ `table` (and `database` for metric sources), the `sql` examples use ``
+ and `` placeholders instead of `otel_logs` and `TimestampTime`,
+ and unknown-table errors from `clickstack_sql` point at `clickstack_list_sources`.
+- e494a46d: feat(mcp): declare readOnlyHint, destructiveHint, idempotentHint, and openWorldHint on every MCP tool
+- 03f9fd5a: fix(mcp): describe every MCP tool input, drop descriptions that restate schema constraints, and keep tool descriptions under Claude Code's 2 KB limit
+- 5cabc115: feat: Support distribution heatmaps calculated from promql data
+- a4b63953: fix: render percentile aggregations (`p50`, `p90`, `p95`, `p99`) as `quantile`
+ at the matching level. A chart config with `aggFn: 'p95'` rendered as the
+ nonexistent ClickHouse function `p95If`. The AI assistant now returns
+ percentiles with a level instead of a bare `quantile` (median), and v1 metric
+ chart conversion keeps the percentile level.
+- ca8e3fda: fix: support https for the PromQL remote-write endpoint and ClickHouse scraping
+
+ `CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT` now accepts an `http://` or `https://` URL as well as a bare `host:port`. With `https://`, the collector's `prometheusremotewrite` exporter sends metrics over TLS and verifies the server certificate against the system CAs. Previously the exporter always used plain HTTP, so an HTTPS-only endpoint could not receive metrics. This applies to both standalone and OpAMP-managed collectors.
+
+ A bare `host:port` keeps plain HTTP, so existing setups are unchanged. The value must not include a path (`/write` is still appended automatically); the collector now exits at startup with a clear error if it does, or if the scheme is anything other than `http` or `https`.
+
+ The OpAMP-managed collector scraped the endpoint over plain HTTP regardless of its scheme. ClickHouse is now scraped in its own `clickhouse` job (previously the `otelcol` job) using the endpoint's scheme, with TLS verification on. The collector's own metrics endpoint stays on plain HTTP. A bare `host:port` or `http://` endpoint behaves as before. Scraped ClickHouse metrics now carry `job="clickhouse"` instead of `job="otelcol"`.
+
+- c94d12c3: fix: Resolve PromQL queries through the source instead of a caller-supplied connection and table
+
+ The `/api/v1/prometheus/*` routes now take `sourceId` instead of `connectionId`, and read the connection and TimeSeries table from that PromQL source. Caller-supplied `database`/`table` params are ignored for ClickHouse and no longer forwarded to external Prometheus endpoints. Saved PromQL chart configs now require `source` and no longer persist `connection`, so provisioned dashboards with PromQL tiles must set `source`.
+
+- be5304dd: Forward the bytes consumed by the ClickHouse proxy's body parsers without
+ changing JSON values, form fields or text encodings. Keep request framing in
+ sync with inflated bytes and preserve unparsed multipart streams.
+- af6e7f6b: Add a description field to the tile editor. The description is now stored on the
+ tile's chart config next to its name, and is kept for series tiles in the
+ external Dashboards API.
+- 02080f20: Add an optional tile description, accepted by the external Dashboards API and
+ shown as an info tooltip next to the tile name.
+- ce6654dd: refactor: Store the time display setting as a time zone instead of a UTC switch, converting existing settings, and remove the `isUTC` option from `formatDate`
+- 28168ae6: fix: Floor the time filter start to the precision of a coarse timestamp column (DateTime, DateTime64 below ms) when a finer column also filters the range, so a sub-second start no longer drops the rest of that tick; make the external API v2 search `maxResults` cap configurable via `EXTERNAL_API_SEARCH_MAX_RESULTS` (defaults to 2000, the previous hardcoded value)
+- ecddea2a: fix: reject generic and incident.io webhook body templates that don't compile,
+ or that call a helper delivery doesn't register (e.g. a typo for `eq`), with a
+ 400 on create, update, and test (internal API, External API v2, and MCP
+ `clickstack_save_webhook`). Add `POST /api/v2/webhooks/validate` to check a
+ webhook config without saving it, e.g. at Terraform plan time.
+- Updated dependencies [92058236]
+- Updated dependencies [3e9185d1]
+- Updated dependencies [180a1f1d]
+- Updated dependencies [a682ce13]
+- Updated dependencies [9891a791]
+- Updated dependencies [43b09a7d]
+- Updated dependencies [0000fa4b]
+- Updated dependencies [eb72cfa1]
+- Updated dependencies [03f9fd5a]
+- Updated dependencies [5cabc115]
+- Updated dependencies [a4b63953]
+- Updated dependencies [3bedfab3]
+- Updated dependencies [c94d12c3]
+- Updated dependencies [95aef70f]
+- Updated dependencies [bc47e485]
+- Updated dependencies [85675fc8]
+- Updated dependencies [af6e7f6b]
+- Updated dependencies [02080f20]
+- Updated dependencies [dab6eb89]
+- Updated dependencies [ce6654dd]
+- Updated dependencies [28168ae6]
+- Updated dependencies [807cd9a5]
+- Updated dependencies [cfa865bd]
+ - @hyperdx/common-utils@0.32.0
+
+## 2.41.0
+
+### Minor Changes
+
+- 76d33d73: The API no longer falls back to the public session secret `hyperdx is cool 👋`.
+ If `EXPRESS_SESSION_SECRET` is unset, the API generates a random secret and
+ reports it once at startup when authentication is enabled. Authenticated
+ deployments that explicitly set the old public value also get a random secret
+ and the same warning; the public value remains available only in the no-auth
+ local demo. Set it to a random string (`openssl rand -hex 32`) in any deployment
+ with authentication enabled — without it, restarting the API signs every user
+ out, and replicas do not share sessions.
+- d8b1cdfa: feat: Attribute PromQL queries to the dashboard, tile or search that issued them, through both the `prometheusQuery`/`prometheusQueryRange` table functions and ClickHouse's Prometheus HTTP API
+- ab703243: feat: attribute ClickHouse queries to the dashboard, tile, search or alert that
+ issued them
+
+ A row in `system.query_log` previously couldn't be attributed to the part of
+ HyperDX that produced it. Now every query HyperDX runs has a small JSON
+ `log_comment`, plus a `query_id` starting with `hdx-`.
+
+ Attribute load after the fact:
+
+ ```sql
+ SELECT
+ JSONExtractString(log_comment, 'surface') AS surface,
+ JSONExtractString(log_comment, 'dashboard') AS dashboard,
+ JSONExtractString(log_comment, 'tile') AS tile,
+ count(),
+ sum(read_rows)
+ FROM system.query_log
+ WHERE type = 'QueryFinish' AND log_comment != '' AND query_id like 'hdx-%'
+ GROUP BY ALL
+ ORDER BY sum(read_rows) DESC
+ ```
+
+ Or spot a query while it is still running, in `system.processes`, where the
+ `query_id` names the surface with no JSON parsing needed.
+
+ The payload records which part of the product asked (a dashboard, a search, an
+ alert, an MCP tool, the chart explorer, session replay, field lookups, and so
+ on), the dashboard and tile or saved search id, the source id, and on the server
+ the trace id of the request. Field lookups and autocomplete are labelled too, so
+ they can be told apart from a user's chart queries.
+
+ The exception is queries that skip settings processing: the `system.settings`,
+ server version and Cloud-detection probes, and the onboarding connection check.
+ They get a `query_id` but an empty `log_comment`.
+
+- 68082b67: feat: add stacked line (stacked area) time-series display type
+
+### Patch Changes
+
+- 14003f08: feat: upgrade the AI SDK to v7 and trace AI calls with GenAI semantic conventions
+
+ AI assistant calls now emit OpenTelemetry spans that follow the GenAI semantic conventions (`invoke_agent` / `chat` / `execute_tool` spans with `gen_ai.*` attributes) instead of the legacy `ai.*` span shape. Team and user attribution moves from `ai.telemetry.metadata.*` to `hyperdx.team.id` and `user.id` (plus `gen_ai.conversation.id` when a session is known). Saved searches, alerts or dashboards keyed on the old `ai.*` span names or attributes need updating.
+
+- 07c93656: fix: cast FixedString columns before implicit hasToken search
+
+ A bare search term against a FixedString column such as OTel TraceId compiled to
+ hasToken(lower(TraceId), ...), which ClickHouse rejects. Those haystacks are now
+ CAST to String first.
+
+- 6873416d: feat: Add a series mode to builder heatmaps
+- 0b855534: fix: fall back to the Map key scan when the text index read fails
+
+ `getMapKeys` returned an empty list whenever its `mergeTreeTextIndex` query
+ failed, so the rollup and bounded `mapKeys` scan paths never ran. ClickHouse
+ refuses `mergeTreeTextIndex` with `ACCESS_DENIED` on any table with a row
+ policy, which left search autocomplete with no Map keys on those tables. A
+ failed text index read now falls through to the next strategy. After a row
+ policy denial it skips the key rollup table, which the policy doesn't cover,
+ and reads keys only from the policy-filtered source table.
+
+- c2e21cf5: fix(mcp): return partial `clickstack_describe_source` results on timeout
+
+ The describe deadline is raised from 10s to 30s to match the other MCP query
+ tools. When it is hit, the tool now returns the schema and samples gathered so
+ far, flagged `partial` with `skippedStages`, instead of failing outright. It
+ only returns a timeout error if the column schema never loaded.
+
+- cf5b831f: perf(mcp): speed up `clickstack_list_metrics` and give the metric discovery
+ tools a 30s budget
+
+ `clickstack_list_metrics` now scans metric names only (about 9x fewer bytes read
+ per kind), fetches unit and description just for the returned page, and scans
+ kinds in parallel. When the time budget runs out it returns the kinds that
+ finished, reports the rest in `partialFailure`, and returns a `nextCursor` that
+ resumes at the timed-out kind instead of failing the whole call. It also no
+ longer drops later kinds when one kind exactly fills a page.
+ `clickstack_list_metrics` and `clickstack_describe_metric` now allow 30s instead
+ of 10s, matching the other MCP query tools.
+
+- 7aeb6215: refactor(mcp): share wall-clock timeout handling across MCP tools via
+ `runWithTimeout`, and source the 30s ClickHouse query cap from one constant.
+- 6ea5f52c: feat: only provision a default PromQL source when PromQL is enabled
+
+ `DEFAULT_SOURCES` entries of kind `promql` are now skipped unless
+ `ENABLE_PROMQL=true`, so a single sources list can include a PromQL source
+ without creating it on deployments that have PromQL turned off.
+
+- ff5befa8: Support "Minimum auto granularity" on PromQL sources. It floors the step of charts on auto granularity, as it does for metric sources, and `$__rate_interval` now uses it as the source's scrape interval (still 15s when unset) instead of always assuming 15s. A granularity picked on a tile is never changed.
+- 1be9de8f: feat: Add a promql variable format that escapes based on single/multi-select
+- 837c6837: feat: Support series mode heatmap in API and MCP
+- 0d2425d5: feat: Allow configuring dashboard filters as single-select
+- 1ce8b88e: fix: rank Map keys by frequency when a text index serves key discovery
+
+ The two `mergeTreeTextIndex` paths in `getMapKeys` grouped keys and applied
+ `LIMIT maxKeys` with no `ORDER BY`, so on a column with more keys than the
+ limit the filter sidebar showed an arbitrary subset. They now order by
+ `sum(cardinality) DESC, key`, which keeps the most frequent keys, as the key
+ rollup path already does with `sum(count) DESC`.
+
+- e88cf3d5: fix(mcp): time-limit the `clickstack_trace_top_time_consuming_operations` tool.
+ Its queries now use the MCP request timeout and `max_execution_time` ceiling,
+ which a source's `querySettings` can no longer override. A timeout returns
+ guidance to shorten the window or narrow `parentFilter` instead of an "invalid
+ SQL" hint, and the client-side `Timeout error.` now gets the timeout hint in
+ every MCP tool. Sources with a multi-column `timestampValueExpression`
+ (including a leading Date column such as `EventDate, EventTime`) now return
+ correct breakdowns.
+- Updated dependencies [07c93656]
+- Updated dependencies [6873416d]
+- Updated dependencies [0b855534]
+- Updated dependencies [0846f3b2]
+- Updated dependencies [e61461be]
+- Updated dependencies [ff5befa8]
+- Updated dependencies [d8b1cdfa]
+- Updated dependencies [1be9de8f]
+- Updated dependencies [ab703243]
+- Updated dependencies [0d2425d5]
+- Updated dependencies [68082b67]
+- Updated dependencies [1ce8b88e]
+ - @hyperdx/common-utils@0.31.0
+
+## 2.40.0
+
+### Minor Changes
+
+- d91e66b8: Make the external API v2 rate limit (`/api/v2/*`) configurable via `EXTERNAL_API_RATE_LIMIT_MAX`. Defaults to 100 requests/minute, matching the previous hardcoded value.
+
+### Patch Changes
+
+- 1e355ac4: fix: don't scan the whole table to discover Map keys
+
+ `getMapKeys` only applied a time predicate when the caller passed both a date
+ range and a timestamp expression; otherwise the raw `sampledKeys` scan ran with
+ no `WHERE` and touched every part of the table. It now defaults a missing date
+ range to the last 24 hours and skips the raw scan entirely when there is no
+ timestamp expression to filter on. The chart, alert and dashboard-filter
+ editors pass the source and date range they already have so Map keys keep
+ autocompleting there.
+
+- 793fe19e: feat: Substitute dashboard variables in markdown tiles
+- ec4f5087: fix: allow grouping gauge and sum metric charts by materialized and alias columns
+
+ Grouping or selecting a MATERIALIZED or ALIAS column on a gauge or sum metric chart failed with `Unknown expression identifier`, because the intermediate query didn't carry those columns through.
+
+- 08d9a908: feat: Plot several PromQL expressions on one chart
+- 19182e7e: fix: only split a query at a standalone SETTINGS keyword
+
+ `extractSettingsClauseFromEnd` cut the query at the first "settings" anywhere in
+ it, including inside a string or an identifier. A multi-series metric chart with
+ a metric such as `app.settings.reloads` produced SQL with an unterminated
+ string, and a SQL filter on a column such as `AppSettings` was rewritten to
+ reference `App`. The keyword now has to stand alone outside quotes.
+
+- d9e2c58b: Add a per-source floor for auto granularity. A metric source can now set "Minimum auto granularity" (Team Settings → Sources → your Metrics source) so that auto-inferred time buckets never go below it — useful when the underlying metric is reported on a fixed interval (e.g. a 60s scrape), since a short selected date range can otherwise auto-infer a smaller bucket than that interval and render a sparse/steppy series (alternating real-sample/empty buckets). Mirrors Grafana's per-datasource "Min interval" setting. Unset (the default) preserves the existing unfloored behavior, and an explicit (non-auto) granularity chosen on a tile is never affected.
+- a5e7841c: fix(mcp): time-bound the trace waterfall span/log fetches so they prune
+ partitions instead of scanning the full retention window. The
+ `clickstack_trace_waterfall` tool now threads the search window into its span
+ and correlated-log queries, adds a `max_execution_time` ceiling, and probes a
+ trace's `[min, max]` span extent so an explicit `traceId` older than the
+ default window — or a trace that ran longer than an hour — still resolves in
+ full. The probe also runs for auto-picked traces, so a picked trace whose root
+ predates the window is no longer truncated into a partial tree. The fetch window
+ is width-clamped to the recent tail so a reused or sentinel `traceId` (e.g. an
+ all-zero id from an uninstrumented emitter) can't widen the scan back toward the
+ retention edge or stitch unrelated occurrences into one tree. Empty-result hints
+ now name the recoverable action (pass an explicit `startTime`) instead of the
+ misleading "widen the window", and a probe that fails is surfaced rather than
+ silently swallowed. ClickHouse query timeouts are also reclassified from `user`
+ to `server` errors.
+- Updated dependencies [1e355ac4]
+- Updated dependencies [793fe19e]
+- Updated dependencies [ec4f5087]
+- Updated dependencies [8a9fcd2f]
+- Updated dependencies [08d9a908]
+- Updated dependencies [a8a72c11]
+- Updated dependencies [8cfb2672]
+- Updated dependencies [4570d5ee]
+- Updated dependencies [78ed5921]
+- Updated dependencies [23423c13]
+- Updated dependencies [19182e7e]
+- Updated dependencies [d9e2c58b]
+ - @hyperdx/common-utils@0.30.0
+
+## 2.39.1
+
+## 2.39.0
+
+### Minor Changes
+
+- ff1e77ce: Backfill alert `displayName` and `tags` from the referenced saved search or dashboard on API startup. Alerts that already have a display name or tags are skipped.
+- e31e5d8d: Offer dashboard tile alerts for Terraform import. `clickhouse_clickstack_alert`
+ gained `source = "tile"` with `dashboard_id`/`tile_id` in provider 3.28.0, so
+ the bulk export and the per-alert menu now include tile alerts instead of
+ skipping every alert that is not a saved-search one. A file carrying a tile
+ alert asks for `>= 3.28.0` and explains the hand edit its generated config
+ needs; an export without one still installs on 3.25.x. A tile alert is withheld
+ when its tile has a blank or duplicated name — the provider's `tile_ids` map is
+ keyed by tile name and omits those, so the alert could only be pinned to a
+ literal id the next dashboard apply can re-mint — or when its dashboard is
+ provisioned, since ProvisionDashboardsTask rewrites those tiles wholesale. Both
+ decisions are made server-side, on the import manifest and on the alerts
+ listing, because neither response carries a dashboard's sibling tile names.
+- f007c37f: Add a context-aware getting-started checklist to the sidebar for recently-created teams. After the setup steps (connect ClickHouse, add data) complete, a second phase tracks product-usage milestones persisted per user on `user.onboardingData`: exploring data, building a dashboard, setting up an alert, and using the MCP server. Completion is recorded server-side so it counts from the UI, the external REST API v2, or an MCP tool; the card can be dismissed and reappears if a new task is added to the registry.
+- b2174306: feat: run PromQL against ClickHouse through its Prometheus HTTP API
+
+ ClickHouse-backed `/query` and `/query_range` now proxy to the connection's `/prometheus/api/v1/*` endpoint (the `prometheus_api_v1` HTTP handler, ClickHouse 26.8+) instead of calling the `prometheusQuery`/`prometheusQueryRange` table functions. The `database` and `table` query parameters are forwarded so one handler serves any TimeSeries table. ClickHouse holds the Prometheus API forward-compatible while the TimeSeries engine is in preview; the table functions and inner-table schema are not. Bundled ClickHouse images move to 26.8.
+
+- 972634d2: Report the whole alert condition in the `{{sourceQuery}}` webhook template
+ variable. It read only a chart's top-level `where`, so an alert defined by a
+ per-series `aggCondition` — a common shape — still rendered empty. The variable
+ now reports every part of the condition the alert query actually applies: a
+ chart's `where` plus the `aggCondition` of the series the alert reads, and a
+ saved search's `where` plus its pinned filters. A chart's pinned filters are
+ deliberately excluded, since a tile or inline alert does not apply them. The
+ value is truncated at 2000 characters.
+
+ Editing an alert off a `between` or `outside` comparator now clears the stored
+ `thresholdMax` instead of leaving the old bound on the document, where it was
+ also served by the alerts APIs and would advertise a range that no longer
+ fires. Webhook templates already guarded against this on read.
+
+ The webhook form's variable list and the API's fallback body template both
+ derive from one list in common-utils, which `buildWebhookTemplateVariables` is
+ typed against, so a variable cannot be added without appearing in both places.
+ The "Send test" payload carries a sample value for every variable, so a body
+ template can be checked before an alert fires.
+
+ The documented guard for an optional number is now
+ `{{#unless (eq thresholdMax undefined)}}` rather than `{{#if thresholdMax}}`,
+ which treats a legitimate bound of `0` as absent.
+
+### Patch Changes
+
+- 4d18cb09: fix: fetch a grouped alert's example log lines once per window
+
+ A saved-search alert puts a handful of example log lines into the notification it sends, and fetching them takes a second query. That query was being made while building each message, so an alert grouped by service asked ClickHouse for the same lines once per breaching service — ten services meant ten identical queries over the same data, because the query only filters by the saved search and the time window, never by the group. It now runs once and every notification for that window shares the answer. An alert catching up on skipped ticks still fetches lines for each window it backfills, since those genuinely differ. Ungrouped alerts are unaffected; they only ever asked once.
+
+- c8cc8e5e: feat: Include alert tags in the tags API response
+- 482d2cb0: feat: Paginate the alerts page
+- 5311d63c: fix: give incident.io webhooks a body incident.io accepts
+
+ An incident.io webhook saved without a body was sent the generic `{"text": ...}` payload, which has neither of the two fields incident.io requires, so every delivery was rejected and no alert was ever raised. It now gets an incident.io payload carrying a deduplication key that is stable across a firing and its resolve, so incident.io closes the alert it opened, plus the alert id, status, condition and evaluation window in `metadata` for routing. The webhook body editor and its list of template variables are also available when incident.io is the selected service, not only for Generic, so the payload can be tailored to an alert source's configured fields.
+
+- 6b391715: Accept metric selects without a value expression on the external dashboards API. A tile that aggregates a metric names its value with `metricName` and has no expression to give, so `/api/v2/dashboards/validate` was rejecting dashboards the editor itself writes, and Terraform could not import them.
+- 84c67f4b: fix: show only the delivery time in an alert's notification duration
+
+ The notification duration on an alert's evaluation list was timing everything an alert does once it decides to fire: building the message title and links, querying the log lines that go in the body, rendering the template, and then delivering it. That made the column read in seconds while the webhook underneath it answered in milliseconds — the column and its own per-target breakdown disagreed, and the figure was dominated by work that has nothing to do with how fast the notification target responded. It now times the delivery alone. Evaluations already recorded keep their old figure and will read high.
+
+- cfacdbe5: feat: relative date ranges for dashboards can now be saved
+- 806d242e: feat: page and filter GET /alerts server-side
+- b19fa12a: refactor(api): backport alertConfigHasGroupBy helper
+- 78a33ba4: feat: Allow configuring dashboard filters as required
+- edb693a6: Apply saved-search pinned filters to alert notification sample queries so sample
+ log lines come from the same row set the alert counted.
+- 0a371980: fix: Keep `/api/sources` responses stable for sources whose stored `metadataMaterializedViews` has no nested `_id`
+- bef61fbc: feat: Scope tags endpoint by resource type
+- 6c85ca02: feat: add a pluggable token encryption service for stored third-party tokens. Set `TOKEN_ENCRYPTION_KEY` to a 32-byte key (base64 or hex) to encrypt them with AES-256-GCM; without it they are stored unencrypted.
+- Updated dependencies [482d2cb0]
+- Updated dependencies [e31e5d8d]
+- Updated dependencies [5311d63c]
+- Updated dependencies [96ac6b1b]
+- Updated dependencies [84c67f4b]
+- Updated dependencies [f007c37f]
+- Updated dependencies [cfacdbe5]
+- Updated dependencies [806d242e]
+- Updated dependencies [f7ae72c2]
+- Updated dependencies [b1e48b99]
+- Updated dependencies [78a33ba4]
+- Updated dependencies [3876d6b9]
+- Updated dependencies [bef61fbc]
+- Updated dependencies [b4840573]
+- Updated dependencies [972634d2]
+ - @hyperdx/common-utils@0.29.0
+
+## 2.38.0
+
+### Minor Changes
+
+- 74c28e7f: feat: Alerts now persist their own `displayName` and `tags`
+- 9c4f94f2: Add enriched template variables to Generic and incident.io webhook bodies:
+ `{{alertId}}`, `{{status}}`, `{{alertType}}`, `{{comparator}}`, `{{threshold}}`,
+ `{{value}}`, `{{groupKey}}`, `{{sourceQuery}}`, `{{teamId}}`, `{{note}}`, and
+ ISO-8601 `{{startTimeISO}}` / `{{endTimeISO}}` alongside the existing Unix-ms
+ `{{startTime}}` / `{{endTime}}`.
+
+ Receivers can now route, filter and dedupe on an alert's identity and condition
+ without parsing the rendered message body. Existing templates are unaffected —
+ every new variable is additive and renders empty when an alert doesn't carry it.
+
+- f0d1cef5: Support inline chart alerts (source `inline` + `chartConfig`) in the external
+ API v2 and the MCP `clickstack_save_alert` / `clickstack_get_alert` tools.
+ Inline alerts can now be created, updated, listed, and deleted through
+ `/api/v2/alerts` using the same tile-config dialect as v2 dashboards, with the
+ same validation rules as the internal API: display-type allowlist, metric
+ formula validation, the formula source-kind gate, raw SQL template validation,
+ and team-scoped source/connection ownership. Passing a `chartConfig` to a
+ `tile` or `saved_search` alert is now rejected instead of silently dropped,
+ and reading an inline alert whose config carries internal-only fields omits
+ `chartConfig` rather than returning a lossy approximation.
+
+ Also fixes three pre-existing issues on the external v2 dashboards path: a
+ gauge tile saved through `clickstack_save_dashboard` with `isDelta: true` was
+ persisted as a non-delta (the flag was dropped converting MCP tiles to the
+ internal shape); a tile config carrying an unrecognized `configType` (e.g.
+ `promql`) was silently stored as a builder config and skipped the formula and
+ number-select rules, and is now rejected; and validation errors on alert
+ bodies no longer collapse to `Invalid input` — the schema reports the failing
+ branch's own message.
+
+- d9c5c455: fix: preserve a Connection host path prefix when proxying PromQL.
+ `proxyToPrometheus` joined absolute Prometheus paths (`/api/v1/query_range`,
+ `/api/v1/query`, `/api/v1/query_exemplars`, `/api/v1/label/.../values`) with
+ `new URL(path, host)`, which replaces the host pathname instead of appending to
+ it. VictoriaMetrics cluster `vmselect` URLs such as
+ `http://vmselect:8481/select/0/prometheus` were rewritten to
+ `/api/v1/query_range` and rejected. The join now keeps the existing pathname.
+
+ This is a behavior change for Connections whose host already included a path
+ that was never meant as a Prometheus API prefix — for example
+ `http://prom:9090/graph` copied from the Prometheus UI. That previously happened
+ to work because the absolute API path replaced `/graph`; requests now go to
+ `/graph/api/v1/query_range` and will 404. Trim stray paths from existing
+ Connection hosts before upgrading. Root-mounted hosts (`http://prom:9090` or
+ `http://prom:9090/`) are unchanged.
+
+ Query parameters on the Connection host are now only a fallback for a fixed set
+ of real Prometheus API params (`query`, `time`, `start`, `end`, `step`,
+ `match`/`match[]`, `limit`, `timeout`, `stats`): a request value for one of
+ these (including repeatable ones such as `match[]`) always wins and replaces a
+ same-named host value outright, rather than being dropped. Any other host query
+ key the request never mentions -- for example `?extra_label=namespace%3Dprod`
+ pinning a VictoriaMetrics tenant scope -- is left as-is and is never overridable
+ by the request, since a param name outside that fixed set is not forwarded at
+ all regardless of what the host carries. This also means a host copied with a
+ stray query string (not just a stray path) now forwards its non-Prometheus keys
+ upstream as a fallback on every request -- trim those too if they weren't
+ intended as Prometheus API params.
+
+ This is also a behavior change for a direct API caller (e.g. curl or Terraform)
+ that previously relied on sending an arbitrary, non-Prometheus query param
+ through this endpoint: that param is now silently dropped rather than forwarded,
+ regardless of whether the Connection host carries anything under the same name.
+
+### Patch Changes
+
+- 25a3b015: Fix `{{sourceQuery}}` returning empty for inline-query and dashboard-tile
+ alerts. It read only the saved search's filter, so alerts backed by a chart
+ config — where the query lives on the alert or the tile — advertised a variable
+ that never rendered. It now resolves the query from whichever config backs the
+ alert: the builder `where` or the raw `sqlTemplate`.
+
+ Add `{{thresholdMax}}`, the upper bound of a `between` / `outside` condition.
+ Receivers previously saw only the lower bound and could not reconstruct the
+ range that fired. It renders empty for every other comparator.
+
+ Test Webhook now sends a sample value for every template variable. It carried
+ only the original seven, so a body using an enriched variable rendered it empty
+ — and because `threshold`, `thresholdMax` and `value` are emitted unquoted, a
+ body like `{"value": {{value}}}` was sent as `{"value": }` and rejected,
+ failing the test for a template that works on a real firing.
+
+- 808b3453: Accept `Bearer `-prefixed Authorization header values on the OTel ingest endpoint in OpAMP-managed mode. The collector's bearer-token authenticator matches the full header value exactly and was configured with only the bare API key, so RFC 6750 clients that send `Authorization: Bearer ` were rejected. The generated collector config now also accepts `Bearer`, `bearer`, and `BEARER` prefixed forms of each ingestion API key.
+- bf4443df: feat: Support autocomplete for PromQL label filters
+- 55db91fa: feat: Support static filters in MCP
+- 89a897ec: Fixed single-series histogram charts failing with "Unknown expression or function identifier" when sorted by a group-by column or expression. The histogram translation packs group values into a single `group` Array, so the table default ORDER BY (the raw group-by text) referenced source columns that no longer exist in scope; matched sort items now address the packed array positionally.
+- f1062a7b: Fixed multi-series metric charts failing with "Unknown expression or function identifier" when sorted by an expression group-by (e.g. `ResourceAttributes['service.name']`). Table tiles default their ORDER BY to the group-by text, so any multi-series metric table grouped by a resource/attribute-derived expression failed to render. Such sort expressions are now evaluated inside each per-series branch through internal companion columns instead of being re-evaluated in the composed outer query, where the source columns no longer exist.
+- 4184a898: feat: Support dashboard filters based on Prometheus label values
+- 27360036: feat: Support series filter (matcher) in PromQL label dashboard filters
+- 7ed8dc8c: feat: Evaluate Prometheus `match[]` series selectors on the labels and label values endpoints for ClickHouse TimeSeries connections
+- bcf0257c: feat: Accept optional time bounds on Prometheus label values endpoint
+- cff6388c: feat: Add static filters to schemas and APIs
+- Updated dependencies [74c28e7f]
+- Updated dependencies [b917308d]
+- Updated dependencies [55db91fa]
+- Updated dependencies [89a897ec]
+- Updated dependencies [f1062a7b]
+- Updated dependencies [4184a898]
+- Updated dependencies [27360036]
+- Updated dependencies [c7965927]
+- Updated dependencies [cff6388c]
+ - @hyperdx/common-utils@0.28.1
+
+## 2.37.0
+
+### Minor Changes
+
+- 0558f77e: Record and show which notification target an evaluation's delivery time went to. `webhookDurationMs` was a single number covering the whole delivery, and because targets are dispatched concurrently the slowest one sets it — so a multi-target alert reported a figure with no way to tell which webhook was responsible, or that the other targets were fine.
+
+ Each dispatch is now timed individually and aggregated per target across the evaluation, since a grouped alert notifies the same target once per firing group and again on resolve. One entry per distinct target carries its webhook id, display name, summed duration, how many dispatches it took, and how many failed. The evaluation history's "Notification duration" cell expands in place to show the breakdown.
+
+ Stored per evaluation rather than per dispatch: a 50-group alert notifying 10 targets would otherwise write 500 entries onto every history row. The array is capped at `ALERT_NOTIFICATION_TARGETS_LIMIT` and sorted slowest-first, so the cap drops the least interesting rows. Records written before this change keep rendering their total with nothing to expand.
+
+- df4a7a55: Add a new `inline` alert source that persists its own chart config directly on the alert, so alerts no longer require a saved search (logs) or a dashboard tile (metrics). The config is the same shape a dashboard tile stores — builder configs on log/trace/metric sources plus raw SQL (Line/Stacked Bar/Number display types); PromQL is rejected. The internal alerts API accepts and returns the new source, and the check-alerts task evaluates inline alerts through the same code path as tile alerts (including group-by and multi-window behavior). Notifications for inline alerts link to the chart explorer seeded with the alert's config over the alerting window, and default their title to the config's name. Backend only — the creation/edit UI and external API v2 support land separately.
+
+### Patch Changes
+
+- 3c81bb96: Build alert notifications for an alert's configured channels directly, instead of encoding them as `@webhook-` mention strings and parsing them back out. That round-trip carried only `type` and `webhookId`, and it appended the channels _after_ whatever the user wrote in the message body — so a body containing `MAX_NOTIFICATIONS_PER_EVENT` mentions consumed every slot of the per-event cap and the alert's own configured channel, the one target it was set up to notify, was silently never reached. Configured channels are now queued first and are exempt from that cap, which only ever meant to bound ad hoc mentions; `channels` is already bounded by `MAX_ALERT_CHANNELS`. Mentions written into the message body are unchanged, still capped, and still deduplicated against the configured channels so naming one twice notifies it once. This also removes the lossiness that prevented a channel from carrying any field beyond its webhook id through to delivery, which downstream forks with richer channel types (e.g. an `email` channel, or a Slack-app channel that also needs a Slack channel id) could not work around. `getDefaultExternalActions` is removed, as nothing needs the mention-string form of a configured channel any more.
+- f11038ef: feat: Persist variable-keyed dashboard filter value state
+- 892cc653: feat(mcp): improve metric discovery, add quiet-saturation eval scenario
+- b52a6fa8: Advertise the MCP quantile `level` field as a string enum so Gemini-backed clients can use the server at all. `z.union([z.literal(0.5), ...])` renders as `{ "type": "number", "enum": [0.5, 0.9, 0.95, 0.99] }`, and Gemini's function declarations only accept `enum` alongside `type: "string"` — so a client that forwards MCP tool schemas to the provider had its entire tool list rejected because of this one field, surfacing as a generic "trouble connecting to the model provider" error that named neither the tool nor the property. Affected `clickstack_timeseries`, `clickstack_table`, `clickstack_save_dashboard` and `clickstack_patch_dashboard`. Only the advertised wire type changes: numeric input is still accepted for callers working from a cached schema, the value is coerced back to a number before any consumer sees it, and out-of-set values are still rejected. The external REST API's own `level` contract is untouched. A new test asserts that no advertised tool schema carries a non-string `enum` or an array-form `items`, complementing the draft-2020-12 metaschema check.
+- 5fc33413: feat: Support dashboard variables in the MCP server
+- Updated dependencies [0558f77e]
+- Updated dependencies [f11038ef]
+- Updated dependencies [df4a7a55]
+- Updated dependencies [f9f7d5bc]
+- Updated dependencies [82852c3a]
+- Updated dependencies [de9038e7]
+- Updated dependencies [5fc33413]
+- Updated dependencies [7662fae8]
+- Updated dependencies [93b51b13]
+- Updated dependencies [64326d09]
+ - @hyperdx/common-utils@0.28.0
+
+## 2.36.0
+
+### Minor Changes
+
+- 8723d7af: Alerts can be configured with multiple notification channels (up to 10 webhooks) via the new `channels` field on the v2 external API, internal API, and the MCP `clickstack_save_alert` tool. The legacy singular `channel` field is still accepted on input and mirrored in responses, so existing integrations keep working unchanged.
+
+ Note that alert updates are a full replace, not a merge. A client that sends only the legacy `channel` field when updating an alert that has several channels will reduce it to that one channel — fetch the alert and resend the complete `channels` array to preserve them.
+
+- a4b2ad00: The API now recovers from a MongoDB that is unreachable at startup, and exposes a Mongo-aware readiness endpoint. Previously a failed initial connect was never retried: the process kept listening while every Mongo-backed request timed out, `/health` reported 200, and Kubernetes kept the pod Ready indefinitely — cascading into OpAMP 500s and crash-looping collectors. The initial connection is now retried with capped exponential backoff until it succeeds, and both the API and OpAMP servers expose `GET /ready`, which returns 503 unless MongoDB is connected (point Kubernetes readiness probes at it; `/health` remains a pure liveness check).
+- dc29d57f: Chart formulas are now supported across every API surface that persists or accepts chart configs. The external dashboards API v2 and the MCP `save_dashboard` / `patch_dashboard` tools accept `formulas` (letter-ref arithmetic over the tile's select items, e.g. `A / (A + B) * 100`) and `showOperandSeries` on line, stacked bar, table and number builder tiles, round-trip them through GET/PUT, and validate the expressions on write — unknown series refs, malformed syntax, combining formulas with `asRatio`, multiple formulas on a number tile, and formulas on formula-incapable source kinds (anything other than metric, log, or trace) are all rejected with actionable errors. MCP `query_tile` computes formula columns for both metric and log/trace event tiles, the query-guide prompt documents the feature, and the OpenAPI spec includes the new `Formula` schema. The CLI's dashboard tile pipeline now delegates its number/table config transforms to the shared common-utils implementations, so formula tiles render with operand-hiding behavior identical to the web.
+
+### Patch Changes
+
+- d205a776: Allow the ClickHouse exporter request timeout to be configured with
+ `HYPERDX_OTEL_EXPORTER_TIMEOUT` in both OpAMP-managed and standalone collector
+ modes. The default remains 5 seconds.
+- 90da4097: Disable mongoose autoIndex in check-alerts worker to prevent MongoExpiredSessionError
+- 43f68566: Allow editing and deleting alerts directly from the alert details page. An
+ "Edit alert" action opens a modal for changing the alert's threshold,
+ evaluation interval, schedule, group-by (saved-search alerts), notification
+ webhook, and note, and a Delete action (with confirmation) removes the alert
+ and returns to the alerts list. Alert API responses now include the
+ notification channel's webhook id and the alert's name/message template so
+ edits round-trip these fields.
+- 40ec0858: feat: Add dashboard variable properties to external dashboards API
+- b0b13806: Align alert firing/recovery chart markers with the evaluated data: markers are now drawn at the start of the newest evaluated bucket (matching the evaluation history table and the plotted data point) instead of at the evaluation time, which sat one bucket to the right.
+- c592207b: Fix MCP tool schemas being rejected by strict JSON Schema draft 2020-12 clients. The number-tile `colorRules` `between` rule declared its `value` as a Zod tuple, which `zod-to-json-schema` renders in the draft-07 tuple form (`items: [ ... ]`). Draft 2020-12 requires `items` to be a schema rather than an array, so `clickstack_save_dashboard` and `clickstack_patch_dashboard` failed validation — and clients that forward MCP tool schemas straight to an LLM provider (e.g. the Anthropic API) rejected the entire tool list with `tools.N.custom.input_schema: JSON schema is invalid`, making the MCP server unusable. `value` is now a fixed-length array, which validates identically and serializes to the same `[min, max]` wire format. A new test validates every MCP tool's input schema against the 2020-12 metaschema so this cannot regress.
+- e153f46d: Tile alerts on metric charts with formulas now evaluate the formula value instead of the last raw operand series: the alert task previously dropped `formulas`/`showOperandSeries` when rebuilding the tile's chart config, so an alert on a formula tile compared the threshold against a raw operand (e.g. bytes) rather than the derived value. Grouped ratio tile alerts also now honor `ratioMode` (`share_of_total` previously evaluated as `per_group`).
+- 7294944a: fix: route per-query SQL debug logging through an injectable logger (#2416)
+
+ `BaseClickhouseClient` dumped raw SQL to the console on every ClickHouse query,
+ unconditionally and outside the pino logger, flooding API logs with query spam.
+
+ Query logging now goes through an optional per-client `customLogger` on
+ `ClickhouseClientOptions`, logged at `debug`, and is silent when no logger is
+ passed. The API injects a pino-backed logger, so query logging follows the
+ existing `HYPERDX_LOG_LEVEL` setting instead of writing to `console.debug`. The
+ browser client defaults to a console logger that pretty-prints the SQL as a
+ single multi-line block, so query SQL stays visible and readable in devtools in
+ all builds instead of wrapping into one long line.
+
+ The API's log level now defaults to `info` (was `debug`), so SQL logging is
+ silent in production unless `HYPERDX_LOG_LEVEL=debug` is set. Dev and CI env
+ files already pin their levels explicitly and are unaffected. The default also
+ now applies when `HYPERDX_LOG_LEVEL` is set but empty — which is what Compose
+ passes when the variable is unset in the environment, and which previously made
+ pino throw at startup.
+
+- e60a7d30: fix: populate the span StatusMessage on failed MCP tool calls so the error text is visible in the trace
+- 9f640a61: Add a Rotate action for the personal API access key in Team Settings → API & Agents. Previously the personal access key — the bearer token for the external API v2 and the MCP server — was generated once at account creation and could never be changed, so a leaked key could only be remediated by deleting the user. Rotating immediately revokes the previous key, so MCP / AI agent configs, external API v2 clients, Terraform / IaC providers, and CI scripts using the old key must be updated with the new one. Browser sessions are unaffected.
+- 08e5b62f: Stop the external dashboards API returning aggregation parameters the aggregation cannot carry: a `level` left over from a quantile agg, or a `valueExpression` left over on a count. Both are ignored when rendering, but the input schema rejects them, so a GET body could not be PUT back and importing a dashboard into Terraform failed with "Level can only be used with quantile aggregation function".
+- Updated dependencies [be26530f]
+- Updated dependencies [c349a5dd]
+- Updated dependencies [68d2ed20]
+- Updated dependencies [2eedfb26]
+- Updated dependencies [1ce61c0c]
+- Updated dependencies [43f68566]
+- Updated dependencies [b1d8dc14]
+- Updated dependencies [40ec0858]
+- Updated dependencies [b0b13806]
+- Updated dependencies [a94d6da8]
+- Updated dependencies [8be68100]
+- Updated dependencies [c592207b]
+- Updated dependencies [9c7742fa]
+- Updated dependencies [dc29d57f]
+- Updated dependencies [7294944a]
+- Updated dependencies [3ecf73c2]
+- Updated dependencies [3ecf73c2]
+- Updated dependencies [e153f46d]
+- Updated dependencies [9f640a61]
+- Updated dependencies [ea127077]
+ - @hyperdx/common-utils@0.27.0
+
+## 2.35.0
+
+### Minor Changes
+
+- fd54ac78: Persist alert evaluation errors (query errors, timeouts, webhook failures) as
+ ERROR-state AlertHistory records instead of only a latest-only snapshot,
+ upserted per evaluation window so retries collapse into a single row. Query
+ timeouts are classified separately (QUERY_TIMEOUT, including timeouts wrapped
+ by the ClickHouse query client) with an actionable message. ERROR rows are
+ excluded from scheduling/backfill computations so failed windows are still
+ retried and backfilled, and once a failed window recovers (via a same-window
+ retry or a later tick's backfill) its stale ERROR row is removed. Evaluation
+ analytics (query/webhook durations, backfilled buckets) are recorded on every
+ history row.
+- 05a3fd81: Add the AlertHistory evaluations read model and GET /alerts/:id/evaluations
+ endpoint: per-window evaluation history scoped to a time range (clamped to the
+ retention window) with per-group breakdown for group-by alerts, evaluation
+ analytics fields, deduped error surfacing for ERROR-state windows, and
+ cursor-based pagination that always advances across gaps. Adds read-side
+ schema/type support for ERROR-state AlertHistory rows and evaluation analytics.
+- 4fa4975a: Add a `clickstack_query_tiles` MCP tool that validates many dashboard tiles in
+ a single call. It accepts a dashboard ID and an optional list of tile IDs
+ (default: every non-markdown tile), runs the tile queries with bounded
+ concurrency, and returns a compact per-tile success/failure summary
+ (status, row count, errors, and raw-SQL macro warnings) plus an aggregate
+ count. A tile that fails to query is reported inline without failing the whole
+ call, so an agent can validate an entire dashboard in one or two calls instead
+ of one `clickstack_query_tile` call per tile. The `clickstack_save_dashboard`
+ guidance now points at the batch tool for post-save validation.
+- d201b71f: Add an optional `serviceVersionExpression` to log and trace sources, identifying
+ the running release of a service. Defaults to the OpenTelemetry
+ `service.version` resource attribute; teams whose release identifier lives
+ elsewhere, such as a container image tag under GitOps, can point it there
+ instead of changing instrumentation.
+
+### Patch Changes
+
+- b9430a62: feat: Add broadcast and variable settings to dashboard filters
+- b6196031: Treat a session whose user no longer exists as logged out instead of failing the
+ request. Deleting a team member left that person's browser holding a session
+ cookie pointing at a user document that was gone, and `deserializeUser` reported
+ the missing user as an error rather than as an unauthenticated session. Because
+ `passport.session()` runs ahead of every router, each request carrying the
+ cookie came back `500 Something went wrong :(` regardless of path or method,
+ including public routes such as `POST /team/setup/:token` and `GET /logout`, so
+ a removed person could neither accept a fresh invite nor clear their own
+ session. The stale id is now dropped from the session and the request continues
+ unauthenticated, so protected routes answer 401 and the browser is sent back to
+ the login page.
+- de783063: Clear the remaining small api ESLint warnings and enforce their rules. Merges
+ the duplicate Express `declare global` namespace blocks in the auth middleware
+ (the `namespace` + empty-interface augmentation pattern is required, so it
+ carries a scoped disable with a comment), and scopes `n/no-process-exit` off for
+ the process entry points (`src/index.ts`, `src/tasks/index.ts`) where exiting
+ with a status code is intended. `@typescript-eslint/no-namespace`,
+ `no-empty-object-type`, and `n/no-process-exit` are promoted to `error` and the
+ api `--max-warnings` ceiling is lowered. Behavior is unchanged.
+- 018a6486: Clean up ESLint warnings and tighten lint enforcement. Resolved all
+ `no-unused-vars` and `@typescript-eslint/ban-ts-comment` warnings (removing dead
+ code and converting `@ts-ignore` to described `@ts-expect-error`), then promoted
+ those rules to `error` in the api/app/common-utils/cli/hdx-eval configs, disabled
+ the noisy `@typescript-eslint/no-empty-function` rule in app, and lowered each
+ package's `--max-warnings` ceiling so the counts can't regress. Behavior is
+ unchanged.
+- 582f3940: Show password requirements on the Join Team page and align the checklist with the server policy. When a user accepts a team invite and sets their password, the same live password policy checklist used on the auth/register page is now displayed, so users no longer have to guess the required length, casing, number, and special-character rules. The checklist previously diverged from the server in two ways that could show all-green checks for a password the server rejects: its special-character rule used a broader pattern than the backend (so a password whose only special character was e.g. `~`, a backtick, or a space passed the checklist but failed on submit), and it never surfaced the 72-character maximum (so an over-long password passed the checklist but failed on submit). The length rule now enforces both the minimum and maximum, and the password policy checks (length bounds, casing, number, and the accepted special-character set) live in a single shared module in `@hyperdx/common-utils` used by both the frontend checklist and the backend `passwordSchema`, so they can no longer drift. Finally, when the server rejects a password the Join Team page now shows the specific reason(s) it failed (e.g. "Password must include at least one special character (!@#$%^&\*(),.?\":{}|<>;-+=)") instead of a generic "Password is invalid", so users are told exactly what to change — including which special characters are accepted.
+- f891eb19: fix(mcp): steer agents toward builder query tools instead of raw SQL (HDX-4892). Telemetry showed agents (notebook investigations) using `clickstack_sql` for ~73% of data queries — usually for single-source aggregations, top-N, and time-series that the builder tools express more reliably (raw SQL also had ~2x the error rate). Reworded the `clickstack_sql` description to mark it a last resort, added a reciprocal "prefer me over SQL" nudge to `clickstack_table`, `clickstack_timeseries`, and `clickstack_search`, and added a server-level `instructions` tool-selection policy so the guidance is surfaced on `initialize` rather than only via the opt-in `query_guide` prompt.
+- 4c5ccfc4: Add MCP tool annotations (readOnlyHint, destructiveHint) to every MCP tool so
+ clients can distinguish read-only query tools from mutating ones. Read/query
+ tools are marked read-only; save/patch and delete tools are marked destructive
+ since they can overwrite or remove existing resources. Hints that would be
+ redundant against the MCP spec defaults are omitted (e.g. destructiveHint is
+ left off read-only tools, where it has no meaning).
+- 6662379e: feat: expose summary metrics through the mcp
+- f34cfaed: Remove the non-functional `GET /ext/silence-alert/:token` endpoint and its dead code path.
+- 711b905d: Guide dashboard MCP agents to filter builder tiles (table, line, stacked_bar,
+ number, pie, bar) with the per-series `where` on each select item, which the
+ chart editor surfaces as the tile's visible "Where" box. The dashboard prompt
+ and the select-item `where` tool description now steer toward it, and the save
+ and patch tools reject a tile-config-level `where`/`whereLanguage` on these
+ types with an actionable message (the editor does not render a tile-level filter
+ for them, so it would be invisible and uneditable). Search, heatmap, and
+ event_patterns tiles keep their tile-level `where`.
+- 908b27ed: Reject source writes that reference malformed, missing, or another team's
+ connection.
+- Updated dependencies [fd54ac78]
+- Updated dependencies [05a3fd81]
+- Updated dependencies [b9430a62]
+- Updated dependencies [546dd442]
+- Updated dependencies [cab98c7c]
+- Updated dependencies [018a6486]
+- Updated dependencies [8508b6c7]
+- Updated dependencies [2d33b83b]
+- Updated dependencies [0ed72ddf]
+- Updated dependencies [aedb514f]
+ - @hyperdx/common-utils@0.26.0
+
+## 2.34.0
+
+### Minor Changes
+
+- 3d61cf92: Cap high-cardinality time-chart series to protect the browser from rendering
+ thousands of lines at once. Time charts now materialize and draw a bounded
+ number of series per tile, with escape hatches to reveal the rest on demand: a
+ "+N more" affordance in the hover and pinned tooltips, and a "load all series"
+ action that lifts the cap for a chart. Tooltips also cap how many rows they
+ render per frame so a wide bucket can't mount thousands of popovers. The
+ external dashboards API exposes the per-tile series limit as a three-state value
+ across tile types — omit for the default cap, 0 for unlimited, or a positive N
+ for the top N
+- fa73b84c: feat(mcp): add `clickstack_emerging_signals` MCP tool — a two-window Drain pattern novelty detector that set-differences mined log/event patterns between an earlier baseline window and a current window to surface what is newly emerging or has disappeared. Shares the Drain sample-and-mine pipeline with `clickstack_event_patterns` via an extracted `mineWindowPatterns` helper, and keys patterns across windows with a `normalizeTemplate` helper.
+- 97ca34df: feat: Allow configuring a `series` table for accelerating metrics
+- f9c52445: feat: add /v1/prometheus/query_exemplars, and harden the Prometheus proxy
+
+ Adds a `query_exemplars` route that proxies to Prometheus's native
+ `/api/v1/query_exemplars` for Prometheus-backed connections, and answers with an
+ empty success for ClickHouse-backed ones, where exemplars are read from the metric
+ table instead.
+
+ Three fixes to the shared proxy while adding a route to it:
+
+ - Responses now carry `X-Content-Type-Options: nosniff`, set before anything can
+ return so the proxy's own error bodies get it too, and the upstream content-type
+ is never forwarded — every response is relabelled `application/json`. The
+ connection host is member-configured, so its response body is untrusted output on
+ our own origin, and an allowlist is easy to slip past: `application/json,
+text/html` clears a prefix-anchored JSON test while the browser keeps the last
+ media type.
+ - A client that navigates away mid-body no longer counts as a backend error.
+ - Proxy failures increment `prometheusQueryErrors`. `proxyToPrometheus` handles its
+ own failures and returns normally, so the callers' `catch` never ran and all four
+ proxied endpoints reported zero errors while still recording duration. Counted on
+ 5xx only, so a user's malformed PromQL does not read as a backend fault.
+ - The exemplar window is bounded by narrowing rather than rejecting, so a wide
+ dashboard range still works.
+
+- 1af1998c: Add Terraform helpers for adopting existing HyperDX resources with the ClickHouse provider. An "Export to Terraform" button on dashboards, saved searches, and saved-search alerts shows a ready-to-paste `import {}` block plus collapsible provider setup, and a team settings section ("API & Agents") downloads an import file covering dashboards, alerts, saved searches, sources, connections, and webhooks.
+
+ Dashboards carrying a tile the provider cannot represent, and PromQL sources, are excluded from the export and reported as skipped — in the UI and in the generated file. The provider reads a dashboard back through the external API v2, which either drops such a tile or substitutes an empty line chart, and writes tiles back whole, so importing one would destroy that tile on the next apply.
+
+ Import-only by design: resource configuration is generated by `terraform plan -generate-config-out`, which reads through the provider, rather than by HyperDX — the external API's dashboard serialisation is a field allowlist, so generating `dashboard_json` from it could silently drop tile settings on apply. Tile alerts are excluded because the provider models only saved-search alerts.
+
+ Terraform addresses are derived from each resource's id, not its name, so renaming a resource in HyperDX and re-exporting does not produce a destroy-and-recreate plan. The generator lives in `@hyperdx/common-utils` so the API can produce the same artefact the UI does. The manifest endpoint caps each listing at 1000 rows and reports which types were capped, so a very large team is told its export is partial rather than silently receiving one.
+
+ Also redacts `Authorization` and `Cookie` headers from API request logs.
+
+### Patch Changes
+
+- 3f87fe4b: Return 404 when updating a missing alert.
+- 94d028c8: Return a not-found response when updating a missing dashboard.
+- a794562d: Clear stale source-specific alert references when changing alert source.
+- 2d78083a: fix: Expose `seriesLimit` via external API and MCP
+- 1c3be6f0: Key the external API and MCP rate limiters on the access key, not the raw
+ `Authorization` header
+
+ `validateUserAccessKey` accepts any text before `Bearer `, so a single access
+ key authenticates under unlimited header spellings. The limiter bucketed on the
+ header value, so varying that prefix handed each request a fresh quota. Requests
+ that carry no usable access key now fall back to the client IP.
+
+- 1af1998c: Scope `DELETE /team/invitation/:id` to the caller's team. It previously deleted by id alone, so any authenticated user could revoke another team's pending invitation if they knew its id. Unknown or out-of-team ids now return 404.
+- Updated dependencies [3d61cf92]
+- Updated dependencies [ed9d9a67]
+- Updated dependencies [c97789a0]
+- Updated dependencies [2468b256]
+- Updated dependencies [97ca34df]
+- Updated dependencies [6a35df06]
+- Updated dependencies [d1c669dc]
+- Updated dependencies [1af1998c]
+- Updated dependencies [b082f700]
+- Updated dependencies [347f0a69]
+ - @hyperdx/common-utils@0.25.0
+
+## 2.33.0
+
+### Minor Changes
+
+- 874a5e95: feat(mcp): add source and webhook management tools so the ingest → dashboard flow can be automated end to end. New MCP tools: `clickstack_save_source` / `clickstack_delete_source` and `clickstack_save_webhook` / `clickstack_delete_webhook` (save creates when `id` is omitted and updates when provided). Webhook logic is now shared via `createWebhook` / `updateWebhook` / `deleteWebhook` controllers: `createWebhook` is used by the internal API, External API v2, and MCP; `updateWebhook` / `deleteWebhook` are shared by External API v2 and MCP (the internal API retains its own masked-secret update/delete flow).
+
+ `clickstack_describe_source` now returns a round-trippable `config` block — the exact flat shape `clickstack_save_source` accepts, including fields the curated summary previously omitted (correlation IDs `logSourceId`/`traceSourceId`/`metricSourceId`/`sessionSourceId`, `defaultTableSelectExpression`, `parentSpanIdExpression`, `spanKindExpression`, materialized views, etc.). This closes the read/write asymmetry that made a faithful source clone impossible: an agent can read a source's full config back and pass it straight into `clickstack_save_source` to clone or read-modify-write it.
+
+ fix(alerts): a generic/incidentio webhook persisted without a body (the body default is only applied by the UI form, not the API/MCP create paths) no longer crashes `sendGenericWebhook` on `Handlebars.compile(undefined)`. It now falls back to the default body template so the alert still fires.
+
+### Patch Changes
+
+- 017c296e: fix: Fix DataCloneError from MCP grouped bar/pie query
+- 0e280949: fix: MCP endpoint (`/api/mcp`) now returns 405 for GET and DELETE instead of aborting spec-compliant clients. The stateless Streamable HTTP transport doesn't offer a server-initiated SSE stream or client-terminable sessions, so it now responds `405 Method Not Allowed` (with `Allow: POST`) for those methods, which official MCP SDK clients (e.g. Claude Code) treat as "not offered, continue" rather than a failed connection.
+- 1b658f3c: fix: Handle per-connection failures in alerts task without exiting
+- Updated dependencies [fa1a0687]
+ - @hyperdx/common-utils@0.24.1
+
+## 2.32.0
+
+### Patch Changes
+
+- 01508d1d: fix: Improve and standardize webhook URL validation
+
+ ### `WEBHOOK_HOSTNAME_ALLOWLIST`
+
+ Use this optional setting to permit webhook delivery to a hostname or private/reserved IP address that the SSRF validator would otherwise block. Values are comma-separated. A hostname entry also permits its subdomains, while an IPv4 or IPv6 entry matches only that exact address. The allowlist does not bypass protocol validation, Slack hostname validation, or the exact host-and-port block for configured ClickHouse and MongoDB services.
+
+ For example, this permits `localhost`, any `*.hooks.localhost` hostname, the exact IPv4 address `10.0.0.1`, and the exact IPv6 address `fd00::1`:
+
+ ```env
+ WEBHOOK_HOSTNAME_ALLOWLIST=localhost,hooks.localhost,10.0.0.1,fd00::1
+ ```
+
+- 641175d8: Add MCP client name and version attributes to tool-invocation spans.
+- 00eef721: feat: Implement quantile for exponential histogram metrics
+- eadea332: feat: surface OpenTelemetry span links in the trace view. Trace sources gain an
+ optional `spanLinksValueExpression` field (auto-detected from the OTel `Links`
+ column), and the span detail panel shows a new "Span Links" section. Each link
+ has an "Open trace" action that opens the linked trace in place in the same
+ panel, with a breadcrumb trail you can step back through, and shows the link's
+ trace state and attributes as chips.
+- 5dd6facb: feat: Add exponential histogram metrics support to MCP
+- Updated dependencies [ad27a513]
+- Updated dependencies [00eef721]
+- Updated dependencies [00eef721]
+- Updated dependencies [7a4ad986]
+- Updated dependencies [eadea332]
+- Updated dependencies [9cb69915]
+- Updated dependencies [7d806fb8]
+- Updated dependencies [f5f9cd19]
+ - @hyperdx/common-utils@0.24.0
+
+## 2.31.0
+
+### Patch Changes
+
+- 1705b37a: fix: Block webhook URLs targeting known-bad IP ranges
+- 3d02a56a: fix: Disable redirects when delivering alert webhooks
+- 758ab638: Fix: Prevent grouped alerts from getting permanently stuck in the ALERT state by resetting history state to OK when thresholds are no longer exceeded.
+- Updated dependencies [ff05b3df]
+- Updated dependencies [1705b37a]
+- Updated dependencies [73819932]
+- Updated dependencies [7accfd2e]
+ - @hyperdx/common-utils@0.23.0
+
+## 2.30.1
+
+## 2.30.0
+
+### Minor Changes
+
+- c29d0df23: feat: Add categorical bar chart display type
+- 727d3274: Add an opt-in Datadog receiver (gated behind `ENABLE_DATADOG_RECEIVER`) so a
+ Datadog Agent can ship traces, metrics, and logs to HyperDX. The contrib
+ `datadogreceiver` is compiled into the collector binary and, when enabled, the
+ OpAMP controller attaches it (listening on `0.0.0.0:8126`) to the traces,
+ metrics, and logs pipelines. When collector authentication is enforced, the
+ receiver validates the `DD-API-KEY` header against team API keys.
+- 880fb668c: feat: add event patterns as a first-class dashboard tile type
+
+ Event patterns can now be created, edited, and saved as dashboard tiles with a dedicated "Pattern Expression" editor. Supported across the UI, MCP server, and External API v2.
+
+- 232e87139: feat(dashboards): overlay alert firing/recovery markers on tile charts
+
+ Adds an optional "alert annotations" overlay to dashboard timeseries tiles.
+ When enabled via the dashboard menu ("Show alert annotations"), tiles that have
+ an alert draw a red vertical marker at the moment the alert fired and a green
+ marker when it recovered, so alert events can be correlated with the chart in
+ one view. The overlay is off by default and its state lives in the URL
+ (`?alertAnnotations=true`), not on the saved dashboard. Backed by a new
+ team-scoped `GET /api/alerts/:id/history` endpoint that returns only alert state
+ transitions within the requested time range, so annotations honor the
+ dashboard's selected window.
+
+- 617355378: External API v2: make error responses consistent and add concurrency safety.
+
+ - `/api/v2/alerts` `403`/`404` responses now return a JSON `{ message }` body
+ (previously an empty plaintext body), matching the documented `Error` schema
+ and the saved-search/webhook routers.
+ - `DELETE /api/v2/alerts/:id` now returns `404` for an alert that does not exist
+ (previously always `200`). The `404` was already part of the documented
+ contract; delete is no longer idempotent for a missing alert.
+ - `PUT /api/v2/webhooks/:id` can now return `409` when the webhook's destination
+ (`url`/`service`) was changed concurrently between read and write. Clients
+ should re-read and retry.
+
+- 617355378: External API v2: add offset/limit pagination to the alerts, saved-searches, and
+ webhooks list endpoints. Each now accepts `limit` (1–1000, default 1000) and
+ `offset` (>=0, default 0) query params and returns a `meta: { total, limit,
+offset }` block alongside `data`. Results are sorted by `_id` so paging is
+ stable across requests.
+
+ Backward compatible: the default `limit` is the maximum (1000), so callers that
+ don't paginate keep receiving all their records (up to the cap) as before. Use
+ `limit`/`offset` to page through larger result sets.
+
+ Behavior change: `/api/v2/alerts` and `/api/v2/webhooks` were previously
+ unbounded and now hard-cap a single page at 1000 records. A team that exceeds
+ 1000 alerts or webhooks will only see the first 1000 unless the client reads
+ the total and pages with `offset`; the full set is still reachable, but a
+ pre-`meta` client that never paginated would silently process only the first
+ page. To make the truncation detectable without parsing the body, each list
+ response now also sets an `X-Total-Count` header with the full count (matching
+ `meta.total`), and the server logs a warning when a default-limit page is
+ truncated.
+
+- 617355378: External API v2: add bearer-auth CRUD for saved searches and webhooks so
+ providers can manage them as resources. Adds a new
+ `/api/v2/saved-searches` router (list/get/create/update/delete, team-scoped,
+ validates `sourceId` ownership) and upgrades `/api/v2/webhooks` from
+ list-only to full CRUD (POST/PUT/DELETE). Webhook `headers` and `queryParams`
+ are write-only — accepted on create/update but never returned on read — so
+ auth tokens and other secrets do not leak.
+- abf5b537: Adds a POST /api/v2/dashboards/validate endpoint to the external v2 API. It
+- ba598baba: feat: Add a custom ORDER BY input for Bar and Pie charts
+- c29d0df23: feat: Allow specifying a limit on pie and bar chart series
+- 0c7254360: Adding consecutive-window configuration to alerts, so that you can specify a condition like "only fire this alert after some condition is met for N consecutive windows." This helps prevent flaky alerts (and pages), and cut down on alert noise in many cases.
+
+ Also adds a `PENDING` alert state for alarms that _will_ fire if current trends continue.
+
+- bdf9352a2: Add Create, Read (by ID), Update, and Delete routes to the external API v2 sources router. Granularity fields in write requests accept the same short format the API returns (e.g. `5m`, `15s`).
+- 27e80e965: feat(alerts): implement webhook retries and exponential backoff
+
+### Patch Changes
+
+- 73e6e876e: Expose consecutive-window alerting on the external API.
+- 1aaa9388a: fix: honor INGESTION_API_KEY in the all-in-one auth image
+
+ The all-in-one entrypoint set `HYPERDX_IMAGE` to values (`all-in-one` /
+ `all-in-one-noauth`) that did not match the strings `config.ts` compares against
+ (`all-in-one-auth` / `all-in-one-noauth`). As a result `IS_ALL_IN_ONE_IMAGE` was
+ never true in the auth image, so the `INGESTION_API_KEY` env var was silently
+ ignored and the OpAMP-delivered collector config only accepted the team's
+ UI-generated key.
+
+ The entrypoint now reports `all-in-one-auth` for the auth variant and
+ `all-in-one-noauth` for the no-auth variant, so a pre-shared `INGESTION_API_KEY`
+ is added to the collector's accepted bearer tokens. This lets demo/bootstrap
+ stacks specify a known ingestion key up front instead of retrieving the
+ generated key from the UI — the all-in-one equivalent of the standalone
+ collector's `OTLP_AUTH_TOKEN`.
+
+- ec11fae92: fix: allow creating and editing Sources in Local App Mode
+
+ In Local App Mode (`IS_LOCAL_APP_MODE`) the auth middleware injects a plain
+ string team id onto the request instead of a Mongoose `ObjectId`. The sources
+ router's create/update handlers called `teamId.toJSON()`, which only exists on
+ `ObjectId`, causing an HTTP 500 (`TypeError: teamId.toJSON is not a function`)
+ when saving a Source. Use `teamId.toString()` instead, which works for both
+ string and `ObjectId` team ids.
+
+- 328e7b437: Fix: Block webhook deletion when one or more alerts still reference it, prompting the user to reassign or remove those alerts first.
+- 60cf52842: fix(mcp): guide agents to size dashboard tiles correctly (HDX-4661)
+- bfc6fb5c: Classify MCP tool errors as `user` (bad input, not-found) or `server` (infrastructure failure, timeout) so alerting rules can filter on `error_category=server` without noise from agent input mistakes. Adds `error_category` attribute to spans and the `hyperdx.mcp.tool.errors` metric counter. ClickHouse errors are auto-classified by inspecting the error type and walking the cause chain for TCP-level codes.
+- d16db2557: Add `backgroundChart` support to number tiles in the MCP dashboard tools (`clickstack_save_dashboard` and `clickstack_patch_dashboard`). Builder number tiles can now carry an optional background trend sparkline (`type` line or area, with an optional palette-token `color`), matching the dashboard editor and the v2 REST API. Raw SQL number tiles do not support it.
+- 5081c8cbb: feat: include the source Section in MCP source tools
+
+ The `clickstack_list_sources` and `clickstack_describe_source` MCP tools now
+ return the optional Section label on each source, so agents see the same source
+ grouping that the source selector shows. Sources without a section are
+ unchanged.
+
+- 476add172: Improve API telemetry quality. The OpAMP message handler span is now a wide event
+ carrying agent correlation and self-description context (instance UID, sequence
+ number and gap, raw + decoded capability flags, new-vs-existing, service name and
+ version, OS type, host arch, health/last-error/uptime, remote config apply
+ status/error, last-applied and sent config hashes for drift detection, effective
+ config presence/size, teams count, request/response sizes). A new
+ `hyperdx.opamp.remote_config_applications` counter tracks whether pushed configs
+ actually applied on agents. The shared error middleware now recognizes body-parser
+ errors: client disconnects (`request.aborted` / `ECONNABORTED`) are classified as
+ operational, logged at debug instead of error, and kept out of error tracking, and
+ `hyperdx.api.errors` gains a bounded `error_type` dimension so aborts, oversized
+ bodies, and malformed payloads are distinguishable.
+- a01717e47: Bumped node version in .nvmrc to 22.23.1
+- bb7ae21e8: Upgrade the TypeScript devDependency from 5.9 to 6.0 across all packages.
+- Updated dependencies [c29d0df23]
+- Updated dependencies [ba598baba]
+- Updated dependencies [c29d0df23]
+- Updated dependencies [3f1e1fe4]
+- Updated dependencies [0c7254360]
+- Updated dependencies [617355378]
+- Updated dependencies [e2145678d]
+- Updated dependencies [bb7ae21e8]
+ - @hyperdx/common-utils@0.22.0
+
+## 2.29.0
+
+### Minor Changes
+
+- 9119de5f: Add unique MongoDB index on accessKey field in User model to eliminate full collection scans during API key authentication. This could cause startup failures if any existing users share duplicate accessKey values.
+- 9f23b7e58: Feat: Added a V2 endpoint for team management.
+- 5cd709020: Add UI support for configuring an external Prometheus-compatible endpoint on a
+ connection. Modify Connections model to now have a boolean
+ `isPrometheusEndpoint` field and use host for storing the host.
+- b798f91f: Add connection management endpoints to the external API (`/api/v2/connections`), supporting list, get, create, update, and delete with Bearer token authentication. Passwords are write-only and never returned by the API.
+- 63469fe0e: feat(mcp): first-class metric source support
+
+ - Two new tools: `clickstack_list_metrics` paginates the metric-name catalog with optional kind / namePattern (ILIKE) / time-window filters and opaque cursor pagination; `clickstack_describe_metric` returns per-metric kind(s), unit, description, attribute keys, and sampled values (with kind auto-detection).
+ - `clickstack_describe_source` is metric-aware: picks a representative metric table (gauge → sum → histogram), runs column / map-key / value-sampling against it, and adds a per-kind metric-name sample.
+ - `clickstack_timeseries` and `clickstack_table` accept `metricType` (gauge / sum / histogram), `metricName`, and `isDelta` on each select item, plus `aggFn:"increase"` for Sum counters. `valueExpression` defaults to `"Value"` for metric sources. Surfaces the renderer's 20-group top-N cap on `increase + groupBy` as a neutral hint.
+ - Dashboard prompt's "use raw SQL for metric tiles" workaround is replaced with positive discovery-workflow guidance and one worked example per supported kind.
+ - `summary` and `"exponential histogram"` kinds remain out of scope (no query renderer support yet).
+
+- f126d5b1: Support number-tile color authoring through the external dashboards API. The v2 REST API and OpenAPI spec now accept `color` (a palette token) and `colorRules` (ordered conditional color rules, last match wins) on builder number tiles, and `color` on raw SQL number tiles, matching what the in-product number-tile editor persists. Color rules accept the numeric and equality operators the editor offers (`gt`, `gte`, `lt`, `lte`, `between`, `eq`, `neq`). Existing dashboards keep working: tiles saved before the palette was renamed to hue names are normalized to the current token names on read.
+- ebfc2e80a: Extend observability instrumentation to the remaining API surfaces using the
+ shared helpers. Add custom metrics and tracing to previously log-only paths:
+ OpAMP message handling (message outcomes, agent status reports, remote configs
+ sent), the Prometheus proxy router (query duration + swallowed-error counters
+ labeled by endpoint and backend), alert webhook/notification delivery (delivery
+ attempts and duration labeled by service and outcome), and MongoDB connection
+ lifecycle events.
+
+ Add a reusable SLO primitive (`withOperationMetrics` / `recordOperationOutcome`)
+ that emits standard availability + latency SLIs (`hyperdx.operation.requests`
+ and `hyperdx.operation.duration_ms`, labeled by `operation` and `outcome`) so
+ SLOs can be defined per piece of application functionality. Apply it to the AI
+ assistant generation call, the ClickHouse proxy (query passthrough +
+ connection test), and alert processing — both the end-to-end alert evaluation
+ (`alerts.evaluate`, excluding scheduling skips) and its ClickHouse data fetch
+ (`alerts.query`) — paths whose failures previously surfaced only as logs or
+ failures-only counters with no latency or denominator.
+
+- bbc29859d: Improve API observability instrumentation. Add a centralized tracing + metrics
+ helper library (`withSpan`, `setBusinessContext`, `getStaticFeatureFlags`,
+ memoized `getCounter`/`getHistogram`, `recordDuration`), attach consistent
+ team/user/feature-flag context to traces across all auth paths (session,
+ access-key, local mode), and add custom metrics for previously log-only hot
+ paths: API errors, alert evaluation outcomes/query/process failures, and
+ external API search/charts query duration and errors.
+- 17e1eb19d: feat: Add an "external link" row-click action for dashboard table tiles
+
+### Patch Changes
+
+- 998ea5d0: feat: Add option to fit time chart y-axis lower bound
+- 0497ca5dd: Bump http-proxy-middleware to v4, replacing http-proxy with httpxy
+- ee907386: fix: Add sourceId to MCP Raw SQL Tile schema
+- 9a7e392a: fix: Add missing numberFormats, compareToPreviousPeriod fields to MCP Schemas
+- cdd7ca07: fix(mcp): reduce describe_source timeouts by using rollup tables for map key discovery
+- d11991b0c: fix: enforce password complexity on team invite acceptance
+- 8261b461: fix: inline parametric aggregate function arguments instead of passing as query parameters
+- 973d1201b: fix: polish promql experience across the app
+- 8164492f: fix(mcp): improve alias field descriptions and examples for readable chart legends
+- a19ba549: feat(mcp): add patch_dashboard, get_dashboard_tile, search_dashboards tools
+
+ Add three new MCP dashboard tools for granular operations:
+
+ - `hyperdx_get_dashboard_tile` — retrieve a single tile by tileId
+ - `hyperdx_patch_dashboard` — update name/tags and/or replace one tile
+ without resubmitting the full dashboard
+ - `hyperdx_search_dashboards` — search by name and/or tags
+
+ Fix empty parameter schema on patch/search tools caused by Zod
+ `.refine()` wrapping. Document Lucene substring matching limitations
+ prominently in tool descriptions and query guide prompt.
+
+ **Breaking (minor):** Tile `name` on `hyperdx_save_dashboard` now requires
+ at least 1 character (`.min(1)`). Previously empty string `""` was accepted
+ and silently persisted as a blank title. Callers sending `name: ""` will
+ now receive a validation error.
+
+- 7e7159a5: fix(mcp): improve error hints and fix readonly mode for query safety settings
+
+ Switch MCP ClickHouse safety settings from readonly=1 to readonly=2 so
+ max_execution_time and max_result_rows are actually applied (readonly=1
+ silently rejects all setting changes).
+
+ Improve DateTime64 cast error hint to recommend parseDateTime64BestEffort()
+ which works on both DateTime and DateTime64 columns, replacing
+ toDateTime64() which only works on DateTime64.
+
+ Add error hint for unknown column/identifier errors directing agents to
+ call describe_source before retrying.
+
+- f34a31fdc: Support number-tile color in the MCP dashboard tools. `save_dashboard` and `patch_dashboard` now accept a static `color` and conditional `colorRules` on builder number tiles, and a static `color` on raw SQL number tiles, matching the external REST dashboards API.
+- f6bda8c5: refactor(mcp): simplify ObjectId validation with shared helpers and schema-level checks
+
+ Add `mcpError()` and `validateObjectId()` utilities to reduce boilerplate
+ across MCP tool handlers. Move ObjectId validation into Zod input schemas
+ for always-required ID fields, eliminating inline checks entirely. Remaining
+ conditional checks use the new one-liner helper.
+
+- f326ccf8: fix(mcp): quote multi-word aliases in orderBy and steer event-pattern usage
+
+ Quote resolved aliases that are not bare identifiers (e.g. `"P95 Latency"`)
+ in `resolveOrderBy` output, in both the direct alias-match and aggFn-match
+ paths. Previously an unquoted multi-word alias produced SQL-invalid
+ `ORDER BY` output. Incoming orderBy values are stripped of surrounding
+ double-quote/backtick quoting before matching, so agents that already quote
+ the alias resolve correctly without being double-quoted.
+
+ Also document the alias-quoting requirement in the `orderBy` schema
+ descriptions, and update the `clickstack_event_patterns` tool description to
+ steer agents toward it (over `clickstack_search` / `clickstack_table`) when
+ exploring what messages, errors, or events exist.
+
+- 750b8afe: feat(mcp): add denoise option to clickstack_search tool
+
+ Add a `denoise` boolean parameter to the MCP `clickstack_search` tool that
+ automatically filters out high-frequency repetitive event patterns from
+ search results, mirroring the web app's "Denoise Results" feature.
+
+ When enabled, the tool samples 10k random events, mines patterns using
+ the Drain algorithm, identifies noisy patterns (>10% of sample), and
+ filters them out of result rows. Returns filtered rows plus metadata
+ listing removed patterns with estimated counts.
+
+ Extracts shared denoise constants (`DENOISE_SAMPLE_SIZE`,
+ `DENOISE_NOISE_THRESHOLD`) into `@hyperdx/common-utils` so the web app
+ and MCP server use the same values.
+
+- caba7c255: fix: Nudge agents towards macros in raw SQL tiles
+- f113ea36: fix(mcp): add ClickHouse safety settings (max_execution_time, max_result_rows, readonly) for MCP query execution
+- 634101c33: chore: upgrade moduleResolution to NodeNext and simplify clickhouseProxy static import
+- ba626ef96: Add `backgroundChart` support to number tiles in the external dashboards API (`/api/v2/dashboards`). Builder number tiles can now carry an optional background trend sparkline (`type` line or area, with an optional palette-token `color`) over the v2 REST API, matching the dashboard editor. Raw SQL number tiles do not support it.
+- 60a91e43: fix(mcp): remove max_result_rows from MCP safety settings
+
+ Remove the hardcoded max_result_rows=100000 setting from MCP query
+ execution. Some ClickHouse connections impose profile constraints that
+ cap max_result_rows below our default, causing SETTING_CONSTRAINT_VIOLATION
+ errors. The remaining safety settings (max_execution_time=30, readonly=2)
+ and trimToolResponse provide sufficient protection.
+
+ Add a SETTING_CONSTRAINT_VIOLATION error hint so constrained settings
+ surface actionable guidance instead of raw ClickHouse errors.
+
+- e03971b0: refactor(theme): rename chart palette tokens from chart-1..10 to hue-named
+ (chart-blue, chart-orange, ...) and unify the categorical palette across HyperDX
+ and ClickStack
+
+ Stored configs from the initial color picker (#2265) keep working.
+ `ChartPaletteTokenSchema` stays strict (a plain `z.enum`, so its `z.input`
+ matches `z.output` — wrapping it in `z.preprocess` would poison
+ `validateRequest`'s `req.body` inference all the way up to
+ `Dashboard.tiles[i].config.color`). Migration of legacy `chart-1` .. `chart-10`
+ happens at five complementary points so no entry or wire-format path can slip
+ through, all composing over a single shared walker
+ (`walkRawDashboardTileColors` in `common-utils`) so the per-tile traversal
+ stays in lockstep:
+
+ - **Fetch-time / write-time (React)**: `normalizeDashboardTileColors` in
+ `packages/app/src/dashboard.ts` heals dashboards on read
+ (`useDashboards` / `fetchLocalDashboards` / `fetchDashboards`) and on write
+ (`useUpdateDashboard` / `useCreateDashboard`). Unresolvable color strings
+ (stale hexes, hand-edited values, forward-rolled future tokens) are
+ preserved so the user's chosen value survives a render pass — the strict
+ server-side schema surfaces a clear error on next save instead of the
+ normalizer quietly dropping the field.
+ - **JSON import**: `DBDashboardImportPage` runs
+ `normalizeRawDashboardTileColors` on the parsed JSON _before_ the strict
+ `DashboardTemplateSchema.safeParse`, so templates exported from a
+ pre-rename deploy import cleanly.
+ - **Server-side GET response healing**: `getDashboards` / `getDashboard` in
+ `packages/api/src/controllers/dashboard.ts` rewrite legacy tile colors on
+ the way out. Pre-rename Mongo docs are served on the wire as
+ hue-named tokens so non-React HTTP clients (CI scripts, stale bundle
+ tabs during a rolling deploy, the external API) can round-trip
+ GET → PATCH without ever resurrecting `chart-N` through the strict
+ schema.
+ - **Server-side write shim**: the dashboards POST / PATCH routes mount
+ a request-body preprocessor that rewrites legacy tile colors before
+ `validateRequest` runs `ChartPaletteTokenSchema`. Catches non-React
+ HTTP callers (stale-bundle tabs during a rolling deploy, CI scripts,
+ MCP, the upcoming external-API parity work) for a one-release
+ deprecation window without weakening the schema's input/output equality.
+ The dashboard provisioner task applies the same shim before parsing
+ on-disk template files.
+ - **Render-time (belt-and-suspenders)**: `DBNumberChart` and
+ `ColorSwatchInput` also call `resolveChartPaletteToken` for tiles
+ constructed in memory between fetch and save (`ChartEditor` form
+ state, unit-test fixtures, hand-rolled `Tile` literals).
+
+ The migration preserves the HyperDX slot ordering from #2265 (slot 1 = brand
+ green, slot 2 = blue, etc.).
+
+ **ClickStack legacy color caveat:** Pre-rename ClickStack used a different slot
+ ordering than HyperDX (`--color-chart-1` was brand blue `#437eef`, not brand
+ green). The migration map uses HyperDX slot ordering, so any ClickStack
+ dashboard saved via #2265 with `color: 'chart-1'` will flip from blue to
+ Observable green after migration. We chose this trade-off deliberately over
+ branching the legacy map by active theme: `LEGACY_CHART_PALETTE_TOKEN_MAP` lives
+ in `common-utils` (shared with the API), and migration is one-shot persisted on
+ next save — theme-branching would couple common-utils to browser DOM state and
+ still produce wrong results for users whose active theme changed since the
+ original pick. Affected users can manually re-pick the desired hue via the (now
+ hue-labeled) color picker.
+
+ The categorical palette is based on Observable 10, with `chart-blue` swapped to
+ `#437eef` to match the brand link color
+ (`--click-global-color-text-link-default`); all other hues are straight from
+ Observable 10. The palette resolves identically on both themes — picking
+ `chart-blue` always renders the brand blue. Brand identity for charts moves
+ entirely into the semantic layer: `--color-chart-success` and `--color-chart-info`
+ resolve to categorical `chart-green` (`#3ca951`) and `chart-blue` (`#437eef`) on
+ both HyperDX and ClickStack, so success fills, info-level logs, and the
+ matching multi-series slots all read consistently across brands.
+
+ Internally, JS (`CATEGORICAL_HEX_BY_TOKEN` in `packages/app/src/utils.ts`) is
+ the source of truth for categorical hues — `getColorFromCSSVariable` and
+ `getColorFromCSSToken` skip `getComputedStyle` for categorical tokens since the
+ palette is unified across themes. The matching `--color-chart-{hue}` CSS vars in
+ `_tokens.scss` remain as a stylesheet-author affordance (inline `var()` use,
+ devtools inspection) and a hook for any future per-brand override. Semantic
+ tokens still resolve through `getComputedStyle` because they genuinely vary per
+ theme.
+
+- adac913d: refactor(mcp): rename all MCP tool prefixes from `hyperdx_` to `clickstack_`
+
+ Rename the MCP server name from `hyperdx` to `clickstack` and update all 19
+ tool names (e.g. `hyperdx_search` → `clickstack_search`), along with
+ descriptions, prompts, error messages, and test references.
+
+- 1a64796c1: Removing relative imports and using path aliases
+- 03f9dd70: feat: add an optional Section field to data sources
+
+ Sources can now carry an optional free-text Section label, set from the source
+ settings form. The value is persisted and returned by GET /api/v2/sources, so
+ external API consumers can read it. This lays the groundwork for grouping and
+ searching sources by section in the source selector.
+
+- 6e0880a75: feat: Add Known Columns List setting for distributed tables
+- fc3ef2dc: fix(alerts): populate `{{attributes.*}}` template variables for tile/chart alerts from group-by fields
+- 81e524c2: feat(charts): cap group-by time charts to a top-N series limit to prevent browser memory exhaustion on high-cardinality group-bys. The cap defaults to 100 (the number of series rendered) and is configurable per team via a new "Time Chart Series Limit" setting; series beyond the cap remain available in the series selector.
+- 55a255a0a: refactor(metrics): unify AttributesHash to variadic cityHash64 across Map and
+ JSON metric schemas
+
+ Sum / Gauge / Histogram metric queries now compute AttributesHash as
+ `cityHash64(ScopeAttributes, ResourceAttributes, Attributes)` for both
+ Map(LowCardinality(String), String) and JSON attribute columns. Previously
+ the Map-schema path wrapped the three maps in `mapConcat()` before hashing,
+ and the JSON-schema path used the variadic form; the schema-detection
+ ClickHouse round-trip and the `attrHashExpr` helper / `isJsonSchema`
+ plumbing are gone.
+
+ Compatibility:
+
+ - Per-row AttributesHash values change for every Map-schema metric row,
+ but the hash is recomputed inside CTEs on every query — no materialized
+ view, projection, ALIAS column, or cache persists it, so no downstream
+ consumer is affected (audit: OSS only).
+ - Cross-scope same-key behaviour shifts: two rows that carry the same
+ logical key in different attribute scopes (e.g. `host` in
+ `ResourceAttributes` for one emission and `host` in `Attributes` for the
+ next) now hash distinctly and land in separate series. Previously the
+ mapConcat path collapsed them into one series. This only matters when an
+ OTel collector processor promotes attributes across scopes mid-stream;
+ most SDKs emit attributes in stable scopes. The new behaviour is captured
+ by an integration test in `packages/api/src/clickhouse/__tests__`.
+
+ HDX-4466.
+
+- 9bbf68079: fix: bug preventing deletion of nested subdocuments like metadataMVs
+- Updated dependencies [1d44098e5]
+- Updated dependencies [998ea5d0]
+- Updated dependencies [ee907386]
+- Updated dependencies [5c46215f8]
+- Updated dependencies [45954c318]
+- Updated dependencies [5cd709020]
+- Updated dependencies [5a1dde4d3]
+- Updated dependencies [ae39bc436]
+- Updated dependencies [8261b461]
+- Updated dependencies [bf6e1f29]
+- Updated dependencies [973d1201b]
+- Updated dependencies [677e3f71]
+- Updated dependencies [89949b1b]
+- Updated dependencies [747352f3]
+- Updated dependencies [750b8afe]
+- Updated dependencies [caba7c255]
+- Updated dependencies [f40cf686b]
+- Updated dependencies [17e1eb19d]
+- Updated dependencies [e03971b0]
+- Updated dependencies [adac913d]
+- Updated dependencies [1a64796c1]
+- Updated dependencies [c74744a5]
+- Updated dependencies [03f9dd70]
+- Updated dependencies [6e0880a75]
+- Updated dependencies [81e524c2]
+- Updated dependencies [da3caab43]
+- Updated dependencies [55a255a0a]
+ - @hyperdx/common-utils@0.21.0
+
## 2.28.0
### Minor Changes
diff --git a/packages/api/eslint.config.mjs b/packages/api/eslint.config.mjs
index 5c926d4224..e5dfaa9697 100644
--- a/packages/api/eslint.config.mjs
+++ b/packages/api/eslint.config.mjs
@@ -37,15 +37,15 @@ export default [
rules: {
...nodePlugin.configs.recommended.rules,
...securityPlugin.configs['recommended-legacy'].rules,
- '@typescript-eslint/ban-ts-comment': 'warn',
+ '@typescript-eslint/ban-ts-comment': 'error',
'@typescript-eslint/no-empty-interface': 'off',
- '@typescript-eslint/no-empty-object-type': 'warn',
+ '@typescript-eslint/no-empty-object-type': 'error',
'@typescript-eslint/no-explicit-any': 'off',
'@typescript-eslint/no-floating-promises': 'error',
'@typescript-eslint/no-unsafe-type-assertion': 'warn',
- '@typescript-eslint/no-namespace': 'warn',
+ '@typescript-eslint/no-namespace': 'error',
'@typescript-eslint/no-unused-vars': [
- 'warn',
+ 'error',
{
argsIgnorePattern: '^_',
varsIgnorePattern: '^_',
@@ -57,7 +57,7 @@ export default [
tryExtensions: ['.js', '.ts', '.json'],
},
],
- 'n/no-process-exit': 'warn',
+ 'n/no-process-exit': 'error',
'n/no-missing-import': 'off',
'n/no-unpublished-import': [
'error',
@@ -74,6 +74,18 @@ export default [
'prettier/prettier': 'error',
'simple-import-sort/imports': 'error',
'simple-import-sort/exports': 'error',
+ 'no-restricted-imports': [
+ 'error',
+ {
+ patterns: [
+ {
+ regex: '^\\.\\.(/|$)',
+ message:
+ 'Use the @/ path alias instead of parent-relative imports (../).',
+ },
+ ],
+ },
+ ],
},
languageOptions: {
parser: tseslint.parser,
@@ -102,4 +114,13 @@ export default [
},
},
},
+ {
+ // Process entry points and the CLI task runner: exiting with a status code
+ // on startup failure, task completion, or a last-resort uncaught-exception
+ // handler is the intended behavior here.
+ files: ['src/index.ts', 'src/tasks/index.ts'],
+ rules: {
+ 'n/no-process-exit': 'off',
+ },
+ },
];
diff --git a/packages/api/jest.config.js b/packages/api/jest.config.js
index 8fc45f0f9d..d34b88a93a 100644
--- a/packages/api/jest.config.js
+++ b/packages/api/jest.config.js
@@ -1,6 +1,15 @@
const { createJsWithTsPreset } = require('ts-jest');
-const tsJestTransformCfg = createJsWithTsPreset();
+const tsJestTransformCfg = createJsWithTsPreset({
+ tsconfig: {
+ // TypeScript 6 requires an explicit rootDir when compiling a subset of
+ // files (ts-jest compiles per-file), otherwise it errors with TS5011.
+ rootDir: './src',
+ // The AI SDK (v7+) ships ESM-only builds that Jest's CJS loader can't
+ // parse, so they are transpiled like source (see transformIgnorePatterns).
+ allowJs: true,
+ },
+});
/** @type {import("jest").Config} **/
module.exports = {
@@ -10,9 +19,12 @@ module.exports = {
testEnvironment: 'node',
verbose: true,
rootDir: './src',
- testMatch: ['**/__tests__/*.test.ts?(x)'],
+ testMatch: ['**/__tests__/**/*.test.ts?(x)'],
+ testPathIgnorePatterns: ['/node_modules/', '.*\\.int\\.test\\.tsx?$'],
testTimeout: 30000,
moduleNameMapper: {
'@/(.*)$': '/$1',
},
+ // Keep in sync with jest.int.config.js.
+ transformIgnorePatterns: ['/node_modules/(?!(ai|@ai-sdk|@workflow/serde)/)'],
};
diff --git a/packages/api/jest.int.config.js b/packages/api/jest.int.config.js
new file mode 100644
index 0000000000..0e8fe29158
--- /dev/null
+++ b/packages/api/jest.int.config.js
@@ -0,0 +1,32 @@
+const { createJsWithTsPreset } = require('ts-jest');
+
+const base = require('./jest.config.js');
+
+// http-proxy-middleware v4 and its deps ship ESM-only builds Jest's CJS
+// loader can't parse. jest.setup.ts blanket-mocks the package; the
+// clickhouse-proxy int tests need the real thing, so transpile it here
+// (`allowJs` for the plain-JS sources, `.mjs` for httpxy's entry).
+const esmDepsTransformCfg = createJsWithTsPreset({
+ tsconfig: {
+ rootDir: './src',
+ allowJs: true,
+ },
+});
+
+/** @type {import("jest").Config} **/
+module.exports = {
+ ...base,
+ transform: {
+ ...esmDepsTransformCfg.transform,
+ '^.+\\.mjs$': esmDepsTransformCfg.transform['^.+\\.[tj]sx?$'],
+ },
+ moduleFileExtensions: ['js', 'mjs', 'json', 'ts', 'tsx', 'node'],
+ // Superset of the unit config's list (AI SDK packages) plus the proxy deps.
+ transformIgnorePatterns: [
+ '/node_modules/(?!(ai|@ai-sdk|@workflow/serde|http-proxy-middleware|httpxy|is-plain-obj)/)',
+ ],
+ testMatch: ['**/__tests__/**/*.int.test.ts?(x)'],
+ // Override the unit config's ignore list: it excludes `.int.test.ts`, which
+ // would otherwise hide every integration test from this suite.
+ testPathIgnorePatterns: ['/node_modules/'],
+};
diff --git a/packages/api/jest.setup.ts b/packages/api/jest.setup.ts
index 7549dd086e..3e270f01e9 100644
--- a/packages/api/jest.setup.ts
+++ b/packages/api/jest.setup.ts
@@ -1,4 +1,21 @@
-jest.retryTimes(1, { logErrorsBeforeRetry: true });
+// Overridable via JEST_RETRY_TIMES (e.g. the nightly zero-retry flake hunt).
+// Honours an explicit 0 so retries can be forced off; an unset/blank/negative/
+// non-integer value falls back to the default of 1 rather than coercing to 0.
+const rawJestRetries = process.env.JEST_RETRY_TIMES;
+const parsedJestRetries = rawJestRetries ? Number(rawJestRetries) : NaN;
+jest.retryTimes(
+ Number.isInteger(parsedJestRetries) && parsedJestRetries >= 0
+ ? parsedJestRetries
+ : 1,
+ { logErrorsBeforeRetry: true },
+);
+
+// http-proxy-middleware v4 is ESM-only and Jest's CJS module loader cannot
+// load ESM packages. Auto-mock it; the clickhouse-proxy int tests opt out via
+// jest.unmock() (jest.int.config.js transpiles the package for them).
+jest.mock('http-proxy-middleware', () => ({
+ createProxyMiddleware: jest.fn(() => jest.fn()),
+}));
// Suppress noisy console output during test runs.
// - debug/info: ClickHouse query logging, server startup messages
@@ -13,7 +30,11 @@ jest.mock('@/utils/slack', () => ({
postMessageToWebhook: jest.fn().mockResolvedValue(null),
}));
-// Mock global fetch for generic webhook calls
+// Mock global fetch for generic webhook calls. The real one is kept on
+// `globalThis.realFetch` so a test that needs to reach a live service (e.g.
+// ClickHouse's Prometheus API) can `mockFetch.mockImplementation(realFetch)`
+// for its duration.
+Object.assign(globalThis, { realFetch: global.fetch });
global.fetch = jest.fn().mockResolvedValue({
ok: true,
text: jest.fn().mockResolvedValue(''),
diff --git a/packages/api/openapi.json b/packages/api/openapi.json
index 6018c8300c..f6ea5912f6 100644
--- a/packages/api/openapi.json
+++ b/packages/api/openapi.json
@@ -24,6 +24,10 @@
"name": "Charts",
"description": "Endpoints for querying chart data"
},
+ {
+ "name": "Connections",
+ "description": "Endpoints for managing ClickHouse connections"
+ },
{
"name": "Sources",
"description": "Endpoints for managing data sources"
@@ -88,9 +92,88 @@
"type": "string",
"enum": [
"saved_search",
- "tile"
+ "tile",
+ "inline"
],
- "description": "Alert source type."
+ "description": "Alert source type. \"saved_search\" alerts monitor a saved search, \"tile\" alerts monitor a dashboard tile, and \"inline\" alerts carry their own chart configuration (see AlertChartConfig) without requiring a saved search or dashboard.\n"
+ },
+ "AlertChartConfigOverlay": {
+ "type": "object",
+ "description": "Fields an alert's chart config carries on top of the dashboard tile config dialect. Unlike a tile (whose name lives on the tile, not its config), an inline alert's config is standalone.\n",
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "Display name for the alert query. Used in notification titles and as an alert-name fallback when the alert itself has no name.\n",
+ "example": "Error Rate Query"
+ },
+ "where": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "Chart-level filter applied on top of every select item's own \"where\" (combined via AND). Builder variants only; rejected on Raw SQL variants (filter inside the sqlTemplate instead).\n",
+ "example": "ServiceName:api"
+ },
+ "whereLanguage": {
+ "$ref": "#/components/schemas/QueryLanguage",
+ "description": "Language of the chart-level \"where\" filter."
+ }
+ }
+ },
+ "AlertLineChartConfig": {
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/LineChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/AlertChartConfigOverlay"
+ }
+ ]
+ },
+ "AlertBarChartConfig": {
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/BarChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/AlertChartConfigOverlay"
+ }
+ ]
+ },
+ "AlertStackedLineChartConfig": {
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/StackedLineChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/AlertChartConfigOverlay"
+ }
+ ]
+ },
+ "AlertNumberChartConfig": {
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/NumberChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/AlertChartConfigOverlay"
+ }
+ ]
+ },
+ "AlertChartConfig": {
+ "description": "The chart configuration an inline alert evaluates, in the same dialect as dashboard tile configs plus the alert-only fields in AlertChartConfigOverlay. Only the display types the alert evaluator supports are accepted: line, stacked_bar, stacked_line, and number, in both builder and Raw SQL (configType \"sql\") variants. Raw SQL templates must reference the evaluation window via the time-filter and interval macros (e.g. $__timeFilter and $__timeInterval), and the referenced source/connection must belong to the team.\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/AlertLineChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/AlertBarChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/AlertStackedLineChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/AlertNumberChartConfig"
+ }
+ ]
},
"AlertState": {
"type": "string",
@@ -98,7 +181,8 @@
"ALERT",
"OK",
"INSUFFICIENT_DATA",
- "DISABLED"
+ "DISABLED",
+ "PENDING"
],
"description": "Current alert state."
},
@@ -113,6 +197,7 @@
"type": "string",
"enum": [
"QUERY_ERROR",
+ "QUERY_TIMEOUT",
"WEBHOOK_ERROR",
"INVALID_ALERT",
"UNKNOWN"
@@ -198,6 +283,15 @@
"propertyName": "type"
}
},
+ "AlertChannels": {
+ "type": "array",
+ "description": "Notification channels to trigger when the alert fires or resolves. Between 1 and 10 channels; duplicates are rejected.\n",
+ "minItems": 1,
+ "maxItems": 10,
+ "items": {
+ "$ref": "#/components/schemas/AlertChannel"
+ }
+ },
"Alert": {
"type": "object",
"properties": {
@@ -225,6 +319,10 @@
"nullable": true,
"example": "ServiceName"
},
+ "chartConfig": {
+ "$ref": "#/components/schemas/AlertChartConfig",
+ "description": "Chart configuration for inline alerts. Required when source is \"inline\" and rejected otherwise. Returned on single-alert responses (GET by ID, POST, PUT); the list endpoint omits it.\n"
+ },
"threshold": {
"type": "number",
"description": "Threshold value for triggering the alert. For between and not_between threshold types, this is the lower bound.",
@@ -267,13 +365,39 @@
},
"channel": {
"$ref": "#/components/schemas/AlertChannel",
- "description": "Alert notification channel configuration."
+ "description": "First notification channel, mirrored from \"channels\" for pre-multi-channel clients."
+ },
+ "channels": {
+ "$ref": "#/components/schemas/AlertChannels",
+ "description": "All notification channels to trigger when the alert fires or resolves."
},
"name": {
"type": "string",
- "description": "Human-friendly alert name.",
+ "description": "Alert name template (Handlebars), rendered as the notification title. When omitted, a default title is generated from the display name, value and threshold.",
+ "nullable": true,
+ "example": "Errors for {{group}} hit {{value}}"
+ },
+ "displayName": {
+ "type": "string",
+ "description": "Display name shown in the alerts list and in notification titles. Defaults to the name of the referenced saved search, dashboard tile, or inline chart when omitted or null.",
+ "nullable": true,
+ "minLength": 1,
+ "maxLength": 512,
+ "example": "Checkout error spike"
+ },
+ "tags": {
+ "type": "array",
+ "items": {
+ "type": "string",
+ "maxLength": 32
+ },
+ "maxItems": 50,
+ "description": "Tags for the alert. Defaults to the tags of the referenced saved search or dashboard when omitted or null; inline alerts have no parent to inherit from and default to an empty list.",
"nullable": true,
- "example": "Test Alert"
+ "example": [
+ "checkout",
+ "p1"
+ ]
},
"message": {
"type": "string",
@@ -288,6 +412,13 @@
"minLength": 1,
"maxLength": 4096,
"example": "Threshold raised from 50 to 100 on 2026-01-15. See [runbook](https://wiki.example.com/runbook)."
+ },
+ "numConsecutiveWindows": {
+ "type": "integer",
+ "minimum": 1,
+ "nullable": true,
+ "description": "Fire the alert only after its condition has been met for this many consecutive evaluation windows. While the condition is met but fewer than this many consecutive windows have violated, the alert is in the PENDING state.",
+ "example": 3
}
}
},
@@ -298,7 +429,27 @@
},
{
"type": "object",
+ "required": [
+ "displayName",
+ "tags"
+ ],
"properties": {
+ "displayName": {
+ "type": "string",
+ "description": "The display name for the alert in the UI. Derived from the saved search name, dashboard tile name, or inline chartConfig name when not explicitly set.",
+ "example": "Checkout error spike"
+ },
+ "tags": {
+ "type": "array",
+ "items": {
+ "type": "string"
+ },
+ "description": "The tags for the alert. Derived from the tags of the referenced saved search or dashboard when not explicitly set; inline alerts default to an empty list.",
+ "example": [
+ "checkout",
+ "p1"
+ ]
+ },
"id": {
"type": "string",
"description": "Unique alert identifier.",
@@ -352,11 +503,11 @@
},
{
"type": "object",
+ "description": "At least one of \"channel\" or \"channels\" must be provided. Sending both is allowed only when \"channel\" matches the first entry of \"channels\", so a response body can be echoed back unchanged. Responses always include both, with \"channel\" mirroring the first entry of \"channels\".\n",
"required": [
"threshold",
"interval",
- "thresholdType",
- "channel"
+ "thresholdType"
]
}
]
@@ -368,11 +519,11 @@
},
{
"type": "object",
+ "description": "At least one of \"channel\" or \"channels\" must be provided. Sending both is allowed only when \"channel\" matches the first entry of \"channels\", so a response body can be echoed back unchanged. Responses always include both, with \"channel\" mirroring the first entry of \"channels\". Updates replace the alert's configuration rather than merging it: sending only the legacy \"channel\" field for an alert that has several channels reduces it to that one channel. Fetch the alert and resend the complete \"channels\" array to preserve them.\n",
"required": [
"threshold",
"interval",
- "thresholdType",
- "channel"
+ "thresholdType"
]
}
]
@@ -388,6 +539,10 @@
},
"AlertsListResponse": {
"type": "object",
+ "required": [
+ "data",
+ "meta"
+ ],
"properties": {
"data": {
"type": "array",
@@ -395,6 +550,10 @@
"items": {
"$ref": "#/components/schemas/AlertResponse"
}
+ },
+ "meta": {
+ "$ref": "#/components/schemas/PaginationMeta",
+ "description": "Pagination metadata for this result page."
}
}
},
@@ -581,6 +740,186 @@
}
}
},
+ "Connection": {
+ "type": "object",
+ "required": [
+ "id",
+ "name",
+ "host",
+ "username"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "description": "Unique connection ID.",
+ "example": "507f1f77bcf86cd799439012"
+ },
+ "name": {
+ "type": "string",
+ "description": "Display name for the connection.",
+ "example": "Production ClickHouse"
+ },
+ "host": {
+ "type": "string",
+ "description": "ClickHouse HTTP endpoint URL.",
+ "example": "https://clickhouse.example.com:8443"
+ },
+ "username": {
+ "type": "string",
+ "description": "ClickHouse username.",
+ "example": "default"
+ },
+ "hyperdxSettingPrefix": {
+ "type": "string",
+ "description": "Optional prefix for HyperDX-specific ClickHouse settings. Must only contain alphanumeric characters and underscores.",
+ "nullable": true,
+ "example": "hyperdx_"
+ },
+ "isPrometheusEndpoint": {
+ "type": "boolean",
+ "description": "Optional. When true, `host` is treated as a Prometheus-compatible API endpoint (e.g. Prometheus or Thanos) and PromQL queries are proxied to it. When false or omitted, `host` is a ClickHouse HTTP endpoint.",
+ "example": false
+ },
+ "createdAt": {
+ "type": "string",
+ "format": "date-time",
+ "description": "Creation timestamp",
+ "example": "2025-01-01T00:00:00.000Z"
+ },
+ "updatedAt": {
+ "type": "string",
+ "format": "date-time",
+ "description": "Last update timestamp",
+ "example": "2025-06-15T10:30:00.000Z"
+ }
+ }
+ },
+ "CreateConnectionRequest": {
+ "type": "object",
+ "required": [
+ "name",
+ "host",
+ "username"
+ ],
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "Display name for the connection.",
+ "example": "Production ClickHouse"
+ },
+ "host": {
+ "type": "string",
+ "description": "ClickHouse HTTP endpoint URL.",
+ "example": "https://clickhouse.example.com:8443"
+ },
+ "username": {
+ "type": "string",
+ "description": "ClickHouse username.",
+ "example": "default"
+ },
+ "password": {
+ "type": "string",
+ "writeOnly": true,
+ "description": "ClickHouse password. Never returned by the API.",
+ "example": "my-secret-password"
+ },
+ "hyperdxSettingPrefix": {
+ "type": "string",
+ "description": "Optional prefix for HyperDX-specific ClickHouse settings. Must only contain alphanumeric characters and underscores.",
+ "nullable": true,
+ "example": "hyperdx_"
+ },
+ "isPrometheusEndpoint": {
+ "type": "boolean",
+ "description": "Optional. When true, `host` is treated as a Prometheus-compatible API endpoint (e.g. Prometheus or Thanos) and PromQL queries are proxied to it. When false or omitted, `host` is a ClickHouse HTTP endpoint.",
+ "example": false
+ }
+ }
+ },
+ "UpdateConnectionRequest": {
+ "type": "object",
+ "required": [
+ "name",
+ "host",
+ "username"
+ ],
+ "properties": {
+ "name": {
+ "type": "string",
+ "description": "Display name for the connection.",
+ "example": "Production ClickHouse"
+ },
+ "host": {
+ "type": "string",
+ "description": "ClickHouse HTTP endpoint URL.",
+ "example": "https://clickhouse.example.com:8443"
+ },
+ "username": {
+ "type": "string",
+ "description": "ClickHouse username.",
+ "example": "default"
+ },
+ "password": {
+ "type": "string",
+ "writeOnly": true,
+ "description": "ClickHouse password. If omitted or empty, the existing password is kept.",
+ "example": "my-new-secret-password"
+ },
+ "hyperdxSettingPrefix": {
+ "type": "string",
+ "description": "Optional prefix for HyperDX-specific ClickHouse settings. Set to null or an empty string to clear the existing value. If omitted, the existing value is kept.",
+ "nullable": true,
+ "example": "hyperdx_"
+ },
+ "isPrometheusEndpoint": {
+ "type": "boolean",
+ "description": "Optional. When true, `host` is treated as a Prometheus-compatible API endpoint. When false or omitted, `host` is a ClickHouse HTTP endpoint. Omit to keep the existing value unchanged.",
+ "example": false
+ }
+ }
+ },
+ "ConnectionResponseEnvelope": {
+ "type": "object",
+ "properties": {
+ "data": {
+ "$ref": "#/components/schemas/Connection",
+ "description": "The connection object."
+ }
+ }
+ },
+ "ConnectionsListResponse": {
+ "type": "object",
+ "properties": {
+ "data": {
+ "type": "array",
+ "description": "List of connection objects.",
+ "items": {
+ "$ref": "#/components/schemas/Connection"
+ }
+ },
+ "meta": {
+ "type": "object",
+ "description": "Present only when one or more stored connections could not be serialized and were omitted from `data`.",
+ "properties": {
+ "skipped": {
+ "type": "integer",
+ "description": "Number of connections omitted from the response because they failed serialization.",
+ "example": 1
+ },
+ "skippedIds": {
+ "type": "array",
+ "description": "IDs of the connections that were omitted.",
+ "items": {
+ "type": "string"
+ },
+ "example": [
+ "507f1f77bcf86cd799439012"
+ ]
+ }
+ }
+ }
+ }
+ },
"NumberFormatOutput": {
"type": "string",
"enum": [
@@ -620,6 +959,17 @@
"description": "Query language for the where clause."
},
"SavedFilterValue": {
+ "description": "A single saved dashboard filter selection. Either a rendered SQL condition, or a selection addressed by the name of the dashboard variable it belongs to.\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/SqlSavedFilterValue"
+ },
+ {
+ "$ref": "#/components/schemas/VariableSavedFilterValue"
+ }
+ ]
+ },
+ "SqlSavedFilterValue": {
"type": "object",
"required": [
"condition"
@@ -631,16 +981,54 @@
"sql"
],
"default": "sql",
- "description": "Filter type. Currently only \"sql\" is supported.",
+ "description": "Filter type.",
"example": "sql"
},
"condition": {
"type": "string",
+ "maxLength": 10000,
"description": "SQL filter condition. For example use expressions in the form \"column IN ('value')\".",
"example": "ServiceName IN ('hdx-oss-dev-api')"
}
}
},
+ "VariableSavedFilterValue": {
+ "type": "object",
+ "required": [
+ "type",
+ "name",
+ "values"
+ ],
+ "properties": {
+ "type": {
+ "type": "string",
+ "enum": [
+ "variable"
+ ],
+ "description": "Filter type.",
+ "example": "variable"
+ },
+ "name": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 1024,
+ "description": "The variableName of the dashboard variable this selection belongs to. Only allowed for variable-enabled filters.\n",
+ "example": "service"
+ },
+ "values": {
+ "type": "array",
+ "maxItems": 1000,
+ "description": "Selected values",
+ "items": {
+ "type": "string",
+ "maxLength": 10000
+ },
+ "example": [
+ "hdx-oss-dev-api"
+ ]
+ }
+ }
+ },
"MetricDataType": {
"type": "string",
"enum": [
@@ -656,6 +1044,7 @@
"type": "string",
"enum": [
"stacked_bar",
+ "stacked_line",
"line"
],
"description": "Visual representation type for the time series."
@@ -779,26 +1168,216 @@
}
}
},
- "TimeChartSeries": {
+ "ChartPaletteToken": {
+ "type": "string",
+ "enum": [
+ "chart-blue",
+ "chart-orange",
+ "chart-red",
+ "chart-cyan",
+ "chart-green",
+ "chart-pink",
+ "chart-purple",
+ "chart-light-blue",
+ "chart-brown",
+ "chart-gray",
+ "chart-success",
+ "chart-warning",
+ "chart-error"
+ ],
+ "description": "Palette token used to color a number tile. Tokens reflow across light and dark themes, so raw hex values are not accepted.\n",
+ "example": "chart-red"
+ },
+ "BackgroundChart": {
"type": "object",
"required": [
- "type",
- "sourceId",
- "aggFn",
- "where",
- "whereLanguage",
- "groupBy"
+ "type"
],
+ "description": "Optional background trend sparkline drawn behind a number tile's value, derived from a time-bucketed version of the tile's query. Builder number tiles only (raw SQL number tiles have no time dimension to bucket).\n",
"properties": {
"type": {
"type": "string",
"enum": [
- "time"
+ "line",
+ "area"
],
- "description": "Series type discriminator. Must be \"time\" for time-series charts.",
- "example": "time"
+ "description": "Sparkline shape.",
+ "example": "line"
},
- "sourceId": {
+ "color": {
+ "$ref": "#/components/schemas/ChartPaletteToken",
+ "description": "Optional palette-token override for the sparkline. When unset the sparkline inherits the tile's static color.\n"
+ }
+ }
+ },
+ "NumericColorCondition": {
+ "type": "object",
+ "required": [
+ "operator",
+ "value",
+ "color"
+ ],
+ "description": "Color rule comparing the displayed value against a single numeric bound.",
+ "properties": {
+ "operator": {
+ "type": "string",
+ "enum": [
+ "gt",
+ "gte",
+ "lt",
+ "lte"
+ ],
+ "description": "Numeric comparison operator.",
+ "example": "gt"
+ },
+ "value": {
+ "type": "number",
+ "description": "Numeric bound the displayed value is compared against. Only finite numbers are accepted (Infinity and NaN are rejected).\n",
+ "example": 100
+ },
+ "color": {
+ "$ref": "#/components/schemas/ChartPaletteToken",
+ "description": "Color applied when the rule matches."
+ },
+ "label": {
+ "type": "string",
+ "maxLength": 40,
+ "description": "Optional label describing the rule.",
+ "example": "High"
+ }
+ }
+ },
+ "BetweenColorCondition": {
+ "type": "object",
+ "required": [
+ "operator",
+ "value",
+ "color"
+ ],
+ "description": "Color rule matching when the displayed value falls within an inclusive range.",
+ "properties": {
+ "operator": {
+ "type": "string",
+ "enum": [
+ "between"
+ ],
+ "description": "Range comparison operator.",
+ "example": "between"
+ },
+ "value": {
+ "type": "array",
+ "minItems": 2,
+ "maxItems": 2,
+ "items": {
+ "type": "number"
+ },
+ "description": "Inclusive [min, max] range. Both bounds must be finite numbers.\n",
+ "example": [
+ 100,
+ 500
+ ]
+ },
+ "color": {
+ "$ref": "#/components/schemas/ChartPaletteToken",
+ "description": "Color applied when the rule matches."
+ },
+ "label": {
+ "type": "string",
+ "maxLength": 40,
+ "description": "Optional label describing the rule.",
+ "example": "Warning"
+ }
+ }
+ },
+ "EqualityColorCondition": {
+ "type": "object",
+ "required": [
+ "operator",
+ "value",
+ "color"
+ ],
+ "description": "Color rule matching when the displayed value equals (eq) or does not equal (neq) a number or string.",
+ "properties": {
+ "operator": {
+ "type": "string",
+ "enum": [
+ "eq",
+ "neq"
+ ],
+ "description": "Equality comparison operator.",
+ "example": "eq"
+ },
+ "value": {
+ "oneOf": [
+ {
+ "type": "number"
+ },
+ {
+ "type": "string",
+ "maxLength": 200
+ }
+ ],
+ "description": "A finite number, or a string up to 200 characters, to compare for equality.\n",
+ "example": "OK"
+ },
+ "color": {
+ "$ref": "#/components/schemas/ChartPaletteToken",
+ "description": "Color applied when the rule matches."
+ },
+ "label": {
+ "type": "string",
+ "maxLength": 40,
+ "description": "Optional label describing the rule.",
+ "example": "Healthy"
+ }
+ }
+ },
+ "NumberTileColorCondition": {
+ "description": "A single conditional color rule for a number tile. Rules are evaluated in order and the last matching rule wins. When no rule matches, the static color applies, then the default text color. The number-tile editor surfaces numeric and equality operators only.\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/NumericColorCondition"
+ },
+ {
+ "$ref": "#/components/schemas/BetweenColorCondition"
+ },
+ {
+ "$ref": "#/components/schemas/EqualityColorCondition"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "operator",
+ "mapping": {
+ "gt": "#/components/schemas/NumericColorCondition",
+ "gte": "#/components/schemas/NumericColorCondition",
+ "lt": "#/components/schemas/NumericColorCondition",
+ "lte": "#/components/schemas/NumericColorCondition",
+ "between": "#/components/schemas/BetweenColorCondition",
+ "eq": "#/components/schemas/EqualityColorCondition",
+ "neq": "#/components/schemas/EqualityColorCondition"
+ }
+ }
+ },
+ "TimeChartSeries": {
+ "type": "object",
+ "required": [
+ "type",
+ "sourceId",
+ "aggFn",
+ "where",
+ "whereLanguage",
+ "groupBy"
+ ],
+ "properties": {
+ "type": {
+ "type": "string",
+ "enum": [
+ "time"
+ ],
+ "description": "Series type discriminator. Must be \"time\" for time-series charts.",
+ "example": "time"
+ },
+ "sourceId": {
"type": "string",
"description": "ID of the data source to query",
"example": "65f5e4a3b9e77c001a567890"
@@ -1187,6 +1766,30 @@
}
}
},
+ "Formula": {
+ "type": "object",
+ "required": [
+ "expression"
+ ],
+ "description": "A derived series computed from the chart's select items via a letter-ref arithmetic expression (metric, log, and trace sources only). \"A\" refers to select[0], \"B\" to select[1], and so on. The grammar supports + - * /, parentheses, and numeric constants; expressions are parsed and validated, never passed through as raw SQL. Division by zero or a missing operand yields NULL (rendered as a gap).\n",
+ "properties": {
+ "expression": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "Arithmetic expression over the select items by position, e.g. \"A / (A + B) * 100\" for a success-rate percentage.\n",
+ "example": "A / (A + B) * 100"
+ },
+ "alias": {
+ "type": "string",
+ "description": "Display label for the formula series in chart legends and column headers. Falls back to the raw expression text when unset.\n",
+ "example": "Success rate %"
+ },
+ "numberFormat": {
+ "$ref": "#/components/schemas/NumberFormat",
+ "description": "Per-series number formatting options for the formula series.\n"
+ }
+ }
+ },
"LineBuilderChartConfig": {
"type": "object",
"required": [
@@ -1239,6 +1842,21 @@
"description": "Fill missing time buckets with zero instead of leaving gaps.",
"default": true
},
+ "fitYAxisToData": {
+ "type": "boolean",
+ "description": "Set the y-axis lower bound to the minimum of the displayed data instead of zero, making small fluctuations between series easier to see.\n",
+ "default": false
+ },
+ "lineInterpolation": {
+ "type": "string",
+ "enum": [
+ "smooth",
+ "linear",
+ "step"
+ ],
+ "example": "step",
+ "description": "How points are connected. Defaults to \"smooth\" when omitted.\n"
+ },
"numberFormat": {
"$ref": "#/components/schemas/NumberFormat",
"description": "Number formatting options for displayed values."
@@ -1247,6 +1865,25 @@
"type": "boolean",
"description": "Overlay the equivalent previous time period for comparison.",
"default": false
+ },
+ "seriesLimit": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Maximum number of series rendered (top-N by value). Omit to use the default render cap, set 0 for unlimited, or a positive N to keep the top N series.\n",
+ "example": 5
+ },
+ "formulas": {
+ "type": "array",
+ "maxItems": 10,
+ "description": "Derived series computed from the select items via letter-ref arithmetic (\"A\" = select[0], \"B\" = select[1], ...). Metric, log, and trace sources only. Cannot be combined with asRatio.\n",
+ "items": {
+ "$ref": "#/components/schemas/Formula"
+ }
+ },
+ "showOperandSeries": {
+ "type": "boolean",
+ "description": "Only meaningful with formulas. When false, only the formula series are returned; the raw operand series are hidden.\n",
+ "default": true
}
}
},
@@ -1305,6 +1942,112 @@
"numberFormat": {
"$ref": "#/components/schemas/NumberFormat",
"description": "Number formatting options for displayed values."
+ },
+ "seriesLimit": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Maximum number of series rendered (top-N by value). Omit to use the default render cap, set 0 for unlimited, or a positive N to keep the top N series.",
+ "example": 5
+ },
+ "formulas": {
+ "type": "array",
+ "maxItems": 10,
+ "description": "Derived series computed from the select items via letter-ref arithmetic (\"A\" = select[0], \"B\" = select[1], ...). Metric, log, and trace sources only. Cannot be combined with asRatio.\n",
+ "items": {
+ "$ref": "#/components/schemas/Formula"
+ }
+ },
+ "showOperandSeries": {
+ "type": "boolean",
+ "description": "Only meaningful with formulas. When false, only the formula series are returned; the raw operand series are hidden.\n",
+ "default": true
+ }
+ }
+ },
+ "StackedLineBuilderChartConfig": {
+ "type": "object",
+ "required": [
+ "displayType",
+ "sourceId",
+ "select"
+ ],
+ "description": "Builder configuration for a stacked-line time-series chart.",
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "stacked_line"
+ ],
+ "description": "Display type discriminator. Must be \"stacked_line\" for stacked-line charts.",
+ "example": "stacked_line"
+ },
+ "sourceId": {
+ "type": "string",
+ "description": "ID of the data source to query.",
+ "example": "65f5e4a3b9e77c001a111111"
+ },
+ "select": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 20,
+ "description": "One or more aggregated values to plot. When asRatio is true, exactly two select items are required.\n",
+ "items": {
+ "$ref": "#/components/schemas/SelectItem"
+ }
+ },
+ "groupBy": {
+ "type": "string",
+ "description": "Field expression to group results by (creates separate stacked bands per group value).",
+ "example": "service",
+ "maxLength": 10000
+ },
+ "asRatio": {
+ "type": "boolean",
+ "description": "Plot select[0] / select[1] as a ratio. Requires exactly two select items.",
+ "default": false
+ },
+ "alignDateRangeToGranularity": {
+ "type": "boolean",
+ "description": "Align the date range boundaries to the query granularity interval.",
+ "default": true
+ },
+ "fillNulls": {
+ "type": "boolean",
+ "description": "Fill missing time buckets with zero instead of leaving gaps.",
+ "default": true
+ },
+ "lineInterpolation": {
+ "type": "string",
+ "enum": [
+ "smooth",
+ "linear",
+ "step"
+ ],
+ "example": "step",
+ "description": "How points are connected. Defaults to \"linear\" when omitted.\n"
+ },
+ "numberFormat": {
+ "$ref": "#/components/schemas/NumberFormat",
+ "description": "Number formatting options for displayed values."
+ },
+ "seriesLimit": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Maximum number of series rendered (top-N by value). Omit to use the default render cap, set 0 for unlimited, or a positive N to keep the top N series.",
+ "example": 5
+ },
+ "formulas": {
+ "type": "array",
+ "maxItems": 10,
+ "description": "Derived series computed from the select items via letter-ref arithmetic (\"A\" = select[0], \"B\" = select[1], ...). Metric, log, and trace sources only. Cannot be combined with asRatio.\n",
+ "items": {
+ "$ref": "#/components/schemas/Formula"
+ }
+ },
+ "showOperandSeries": {
+ "type": "boolean",
+ "description": "Only meaningful with formulas. When false, only the formula series are returned; the raw operand series are hidden.\n",
+ "default": true
}
}
},
@@ -1375,6 +2118,19 @@
"onClick": {
"$ref": "#/components/schemas/OnClick",
"description": "Optional link-out configuration applied when a user clicks a row."
+ },
+ "formulas": {
+ "type": "array",
+ "maxItems": 10,
+ "description": "Derived columns computed from the select items via letter-ref arithmetic (\"A\" = select[0], \"B\" = select[1], ...). Metric, log, and trace sources only. Cannot be combined with asRatio.\n",
+ "items": {
+ "$ref": "#/components/schemas/Formula"
+ }
+ },
+ "showOperandSeries": {
+ "type": "boolean",
+ "description": "Only meaningful with formulas. When false, only the formula columns are returned; the raw operand columns are hidden.\n",
+ "default": true
}
}
},
@@ -1403,15 +2159,39 @@
"select": {
"type": "array",
"minItems": 1,
- "maxItems": 1,
- "description": "Exactly one aggregated value to display as a single number.",
+ "maxItems": 20,
+ "description": "Exactly one aggregated value to display as a single number — unless \"formulas\" is set, in which case the select items are the formula's operands and the (single) formula value is displayed instead.\n",
"items": {
"$ref": "#/components/schemas/SelectItem"
}
},
+ "formulas": {
+ "type": "array",
+ "maxItems": 1,
+ "description": "A single derived value computed from the select items via letter-ref arithmetic (\"A\" = select[0], \"B\" = select[1], ...). Metric, log, and trace sources only. Number tiles display the formula value and always hide the operand series.\n",
+ "items": {
+ "$ref": "#/components/schemas/Formula"
+ }
+ },
"numberFormat": {
"$ref": "#/components/schemas/NumberFormat",
"description": "Number formatting options for displayed values."
+ },
+ "color": {
+ "$ref": "#/components/schemas/ChartPaletteToken",
+ "description": "Optional static color applied to the displayed number."
+ },
+ "colorRules": {
+ "type": "array",
+ "maxItems": 10,
+ "description": "Ordered conditional color rules evaluated against the displayed value (last match wins). Falls back to color, then the default text color when no rule matches.\n",
+ "items": {
+ "$ref": "#/components/schemas/NumberTileColorCondition"
+ }
+ },
+ "backgroundChart": {
+ "$ref": "#/components/schemas/BackgroundChart",
+ "description": "Optional background trend sparkline drawn behind the value.\n"
}
}
},
@@ -1452,51 +2232,118 @@
"description": "Field expression to group results by (one slice per group value).",
"example": "service"
},
- "numberFormat": {
- "$ref": "#/components/schemas/NumberFormat",
- "description": "Number formatting options for displayed values."
- }
- }
- },
- "HeatmapSelectItem": {
- "type": "object",
- "required": [
- "valueExpression"
- ],
- "description": "Single select item for a heatmap tile. The value being bucketed is provided in valueExpression and the count contributing to each bucket in countExpression. The heatmap-specific fields (countExpression, heatmapScaleType) are persisted on the select item, not the chart config. The chart-level discriminator is the HeatmapChartConfig's `displayType: \"heatmap\"`; no aggregation function or alias is exposed on this select item because the heatmap aggregation function is fixed internally and the HeatmapSeriesEditor does not render an alias input.\n",
- "properties": {
- "valueExpression": {
+ "orderBy": {
"type": "string",
- "minLength": 1,
"maxLength": 10000,
- "description": "SQL expression for the value being bucketed on the y-axis. Must be non-empty.",
- "example": "Duration"
+ "description": "Optional custom SQL ORDER BY expression (raw SQL). Overrides the default value-descending ordering and, when combined with \"limit\", controls which slices are kept.\n",
+ "example": "\"Count\" DESC"
},
- "countExpression": {
- "type": "string",
- "maxLength": 10000,
- "description": "SQL expression for the count contributing to each bucket. Defaults to \"count()\" in the editor when omitted.\n",
- "example": "count()"
+ "numberFormat": {
+ "$ref": "#/components/schemas/NumberFormat",
+ "description": "Number formatting options for displayed values."
},
- "heatmapScaleType": {
- "type": "string",
- "enum": [
- "log",
- "linear"
- ],
- "description": "Scale type used to bucket values on the y-axis.",
- "example": "log"
+ "limit": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Maximum number of slices (SQL LIMIT). Without a custom \"orderBy\" the query keeps the groups with the largest aggregated values; with an \"orderBy\" it keeps the first slices in that order. Omit or set 0 to fetch all groups.\n",
+ "example": 10
}
}
},
- "HeatmapChartConfig": {
+ "CategoricalBarBuilderChartConfig": {
"type": "object",
"required": [
"displayType",
"sourceId",
"select"
],
- "description": "Builder configuration for a heatmap tile. Heatmap is builder-only (no Raw SQL variant) and currently supports trace sources. The row-level filter lives at the chart-config level (where / whereLanguage), matching the HeatmapSeriesEditor in the UI.\n",
+ "description": "Builder configuration for a categorical bar chart tile. Each bar represents one group value. Distinct from stacked_bar, which is a time-series chart.\n",
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "bar"
+ ],
+ "description": "Display type discriminator. Must be \"bar\" for categorical bar charts.",
+ "example": "bar"
+ },
+ "sourceId": {
+ "type": "string",
+ "description": "ID of the data source to query.",
+ "example": "65f5e4a3b9e77c001a111111"
+ },
+ "select": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 1,
+ "description": "Exactly one aggregated value used to size each bar.",
+ "items": {
+ "$ref": "#/components/schemas/SelectItem"
+ }
+ },
+ "groupBy": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "Field expression to group results by (one bar per group value).",
+ "example": "service"
+ },
+ "orderBy": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "Optional custom SQL ORDER BY expression (raw SQL). Overrides the default value-descending ordering and, when combined with \"limit\", controls which bars are kept.\n",
+ "example": "\"Count\" DESC"
+ },
+ "numberFormat": {
+ "$ref": "#/components/schemas/NumberFormat",
+ "description": "Number formatting options for displayed values."
+ },
+ "limit": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Maximum number of bars (SQL LIMIT). Without a custom \"orderBy\" the query keeps the groups with the largest aggregated values; with an \"orderBy\" it keeps the first bars in that order. Omit or set 0 to fetch all groups.\n",
+ "example": 10
+ }
+ }
+ },
+ "HeatmapSelectItem": {
+ "type": "object",
+ "required": [
+ "valueExpression"
+ ],
+ "description": "Single select item for a heatmap tile. The value being bucketed is provided in valueExpression and the count contributing to each bucket in countExpression. The heatmap-specific fields (countExpression, heatmapScaleType) are persisted on the select item, not the chart config. The chart-level discriminator is the HeatmapChartConfig's `displayType: \"heatmap\"`; no aggregation function or alias is exposed on this select item because the heatmap aggregation function is fixed internally and the HeatmapSeriesEditor does not render an alias input.\n",
+ "properties": {
+ "valueExpression": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 10000,
+ "description": "SQL expression for the value being bucketed on the y-axis. Must be non-empty.",
+ "example": "Duration"
+ },
+ "countExpression": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "SQL expression for the count contributing to each bucket. Defaults to \"count()\" in the editor when omitted.\n",
+ "example": "count()"
+ },
+ "heatmapScaleType": {
+ "type": "string",
+ "enum": [
+ "log",
+ "linear"
+ ],
+ "description": "Scale type used to bucket values on the y-axis.",
+ "example": "log"
+ }
+ }
+ },
+ "HeatmapDistributionChartConfig": {
+ "type": "object",
+ "required": [
+ "displayType",
+ "sourceId",
+ "select"
+ ],
+ "description": "Builder configuration for a distribution-mode heatmap tile, which buckets a numeric value on the y-axis. Requires a trace source. The row-level filter lives at the chart-config level (where / whereLanguage), matching the HeatmapSeriesEditor in the UI.\n",
"properties": {
"displayType": {
"type": "string",
@@ -1506,6 +2353,14 @@
"description": "Display type discriminator. Must be \"heatmap\" for heatmap tiles.",
"example": "heatmap"
},
+ "heatmapMode": {
+ "type": "string",
+ "enum": [
+ "distribution"
+ ],
+ "description": "Heatmap mode. \"distribution\" is the default when omitted.",
+ "example": "distribution"
+ },
"sourceId": {
"type": "string",
"description": "ID of the data source to query.",
@@ -1538,6 +2393,78 @@
}
}
},
+ "HeatmapSeriesChartConfig": {
+ "type": "object",
+ "required": [
+ "displayType",
+ "heatmapMode",
+ "sourceId",
+ "select"
+ ],
+ "description": "Builder configuration for a series-mode heatmap tile, which shows one aggregated series per groupBy value as a row on the y-axis. Requires a trace, log, or metric source. Any filter lives on the select item (where / whereLanguage).\n",
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "heatmap"
+ ],
+ "description": "Display type discriminator. Must be \"heatmap\" for heatmap tiles.",
+ "example": "heatmap"
+ },
+ "heatmapMode": {
+ "type": "string",
+ "enum": [
+ "series"
+ ],
+ "description": "Heatmap mode. Must be \"series\" for series heatmaps.",
+ "example": "series"
+ },
+ "sourceId": {
+ "type": "string",
+ "description": "ID of the data source to query.",
+ "example": "65f5e4a3b9e77c001a111111"
+ },
+ "select": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 1,
+ "description": "Exactly one aggregated value used to color each cell.",
+ "items": {
+ "$ref": "#/components/schemas/SelectItem"
+ }
+ },
+ "groupBy": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "Field expression to group results by (one row per group value).",
+ "example": "ServiceName"
+ },
+ "numberFormat": {
+ "$ref": "#/components/schemas/NumberFormat",
+ "description": "Number formatting options for displayed values."
+ }
+ }
+ },
+ "HeatmapChartConfig": {
+ "description": "Heatmap tile. There is no Raw SQL variant. For the builder variant, omit heatmapMode (or set it to \"distribution\") for the distribution heatmap, or set it to \"series\" for the series heatmap. Set configType to \"promql\" for the PromQL variant.\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/HeatmapDistributionChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/HeatmapSeriesChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/HeatmapPromqlChartConfig"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "configType",
+ "mapping": {
+ "promql": "#/components/schemas/HeatmapPromqlChartConfig"
+ }
+ }
+ },
"SearchChartConfig": {
"type": "object",
"required": [
@@ -1580,6 +2507,47 @@
}
}
},
+ "EventPatternsChartConfig": {
+ "type": "object",
+ "required": [
+ "displayType",
+ "sourceId"
+ ],
+ "description": "Configuration for an event pattern mining tile. Clusters log or trace events by recurring message shapes.",
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "event_patterns"
+ ],
+ "description": "Display type discriminator. Must be \"event_patterns\" for pattern mining tiles.",
+ "example": "event_patterns"
+ },
+ "sourceId": {
+ "type": "string",
+ "description": "ID of the data source to mine patterns from.",
+ "example": "65f5e4a3b9e77c001a111111"
+ },
+ "select": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "Column or expression to mine patterns from. Leave empty to use the source default (Body for logs, SpanName for traces).",
+ "default": "",
+ "example": "Body"
+ },
+ "where": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "Filter condition for the pattern mining query (syntax depends on whereLanguage).",
+ "default": "",
+ "example": "level:error"
+ },
+ "whereLanguage": {
+ "$ref": "#/components/schemas/QueryLanguage",
+ "description": "Query language for the where clause."
+ }
+ }
+ },
"MarkdownChartConfig": {
"type": "object",
"required": [
@@ -1676,6 +2644,21 @@
"type": "boolean",
"description": "Expand date range boundaries to the query granularity interval.",
"default": true
+ },
+ "fitYAxisToData": {
+ "type": "boolean",
+ "description": "Set the y-axis lower bound to the minimum of the displayed data instead of zero, making small fluctuations between series easier to see.\n",
+ "default": false
+ },
+ "lineInterpolation": {
+ "type": "string",
+ "enum": [
+ "smooth",
+ "linear",
+ "step"
+ ],
+ "example": "step",
+ "description": "How points are connected. Defaults to \"smooth\" when omitted.\n"
}
}
}
@@ -1715,6 +2698,50 @@
}
]
},
+ "StackedLineRawSqlChartConfig": {
+ "description": "Raw SQL configuration for a stacked-line time-series chart.",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/RawSqlChartConfigBase"
+ },
+ {
+ "type": "object",
+ "required": [
+ "displayType"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "stacked_line"
+ ],
+ "description": "Display as a stacked-line time-series chart.",
+ "example": "stacked_line"
+ },
+ "fillNulls": {
+ "type": "boolean",
+ "description": "Fill missing time buckets with zero instead of leaving gaps.",
+ "default": true
+ },
+ "alignDateRangeToGranularity": {
+ "type": "boolean",
+ "description": "Expand date range boundaries to the query granularity interval.",
+ "default": true
+ },
+ "lineInterpolation": {
+ "type": "string",
+ "enum": [
+ "smooth",
+ "linear",
+ "step"
+ ],
+ "example": "step",
+ "description": "How points are connected. Defaults to \"linear\" when omitted.\n"
+ }
+ }
+ }
+ ]
+ },
"TableRawSqlChartConfig": {
"description": "Raw SQL configuration for a table chart.",
"allOf": [
@@ -1762,6 +2789,18 @@
],
"description": "Display as a single big-number chart.",
"example": "number"
+ },
+ "color": {
+ "$ref": "#/components/schemas/ChartPaletteToken",
+ "description": "Optional static color applied to the displayed number."
+ },
+ "colorRules": {
+ "type": "array",
+ "maxItems": 10,
+ "description": "Ordered conditional color rules evaluated against the displayed value (last match wins). Falls back to color, then the default text color when no rule matches.\n",
+ "items": {
+ "$ref": "#/components/schemas/NumberTileColorCondition"
+ }
}
}
}
@@ -1791,347 +2830,954 @@
}
]
},
- "LineChartConfig": {
- "description": "Line chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate).\n",
- "oneOf": [
+ "CategoricalBarRawSqlChartConfig": {
+ "description": "Raw SQL configuration for a categorical bar chart.",
+ "allOf": [
{
- "$ref": "#/components/schemas/LineBuilderChartConfig"
+ "$ref": "#/components/schemas/RawSqlChartConfigBase"
},
{
- "$ref": "#/components/schemas/LineRawSqlChartConfig"
+ "type": "object",
+ "required": [
+ "displayType"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "bar"
+ ],
+ "description": "Display as a categorical bar chart.",
+ "example": "bar"
+ }
+ }
}
+ ]
+ },
+ "PromqlRangeExpression": {
+ "type": "object",
+ "required": [
+ "expression"
],
- "discriminator": {
- "propertyName": "configType",
- "mapping": {
- "sql": "#/components/schemas/LineRawSqlChartConfig"
+ "description": "A PromQL expression evaluated over the dashboard time range.",
+ "properties": {
+ "expression": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "PromQL expression. Supports dashboard variables and the $__interval, $__rate_interval, and $__range macros.\n",
+ "example": "sum by (job) (rate(http_requests_total[$__rate_interval]))"
+ },
+ "alias": {
+ "type": "string",
+ "maxLength": 10000,
+ "description": "Prefixed to every series name this expression produces.",
+ "example": "Requests"
}
}
},
- "BarChartConfig": {
- "description": "Stacked-bar chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate).\n",
- "oneOf": [
+ "PromqlInstantExpression": {
+ "description": "A PromQL expression on a table, number, pie, or bar tile, which can be evaluated at a single instant instead of over a range.\n",
+ "allOf": [
{
- "$ref": "#/components/schemas/BarBuilderChartConfig"
+ "$ref": "#/components/schemas/PromqlRangeExpression"
},
{
- "$ref": "#/components/schemas/BarRawSqlChartConfig"
- }
- ],
- "discriminator": {
- "propertyName": "configType",
- "mapping": {
- "sql": "#/components/schemas/BarRawSqlChartConfig"
+ "type": "object",
+ "properties": {
+ "queryType": {
+ "type": "string",
+ "enum": [
+ "range",
+ "instant"
+ ],
+ "description": "Whether to evaluate over the time range or at its end. Defaults to \"range\" when omitted.\n",
+ "example": "instant"
+ }
+ }
}
- }
+ ]
},
- "OnClickFilterTemplate": {
+ "PromqlReducibleExpression": {
+ "description": "A PromQL expression on a number, pie, or bar tile, which reduces each series of a range query to a single value.\n",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/PromqlInstantExpression"
+ },
+ {
+ "type": "object",
+ "properties": {
+ "reducer": {
+ "type": "string",
+ "enum": [
+ "lastNotNull",
+ "min",
+ "max",
+ "mean",
+ "sum",
+ "count"
+ ],
+ "description": "How each series' samples are reduced to one value. Defaults to \"lastNotNull\" when omitted. It also applies to instant queries, where each series has a single sample, so e.g. \"count\" turns every value into 1.\n",
+ "example": "max"
+ }
+ }
+ }
+ ]
+ },
+ "PromqlChartConfigBase": {
"type": "object",
- "description": "A templated filter applied to the link-out destination. The rendered template value is combined with the expression as `expression IN (...)` on the destination search or dashboard. Multiple templates sharing the same expression are merged into a single IN clause.\n",
"required": [
- "kind",
- "expression",
- "template"
+ "configType",
+ "sourceId"
],
+ "description": "Shared fields for PromQL chart configs. Set configType to \"promql\" and provide sourceId + expressions. The connection is read from the source. PromQL is a preview feature and must be enabled on the deployment (ENABLE_PROMQL) before PromQL tiles can be queried.\n",
"properties": {
- "kind": {
+ "configType": {
"type": "string",
"enum": [
- "expressionTemplate"
+ "promql"
],
- "description": "Filter template kind. Currently only \"expressionTemplate\" is supported.",
- "example": "expressionTemplate"
+ "description": "Must be \"promql\" to use the PromQL chart config variant.",
+ "example": "promql"
},
- "expression": {
+ "sourceId": {
"type": "string",
- "minLength": 1,
- "description": "The column/expression to filter the destination by (e.g. \"ServiceName\").",
- "example": "ServiceName"
+ "description": "ID of the PromQL source to query. Must be a source of kind \"promql\".",
+ "example": "65f5e4a3b9e77c001a333333"
},
- "template": {
+ "granularity": {
"type": "string",
- "minLength": 1,
- "description": "Value template rendered against the clicked row; supports row column variables in `{{column}}` form (e.g. `{{ServiceName}}`).\n",
- "example": "{{ServiceName}}"
+ "pattern": "^(\\d+ (second|minute|hour|day)|auto)$",
+ "description": "Resolution of range queries, used as the Prometheus step and the $__interval macro. Defaults to \"auto\", which picks one from the time range.\n",
+ "example": "5 minute"
+ },
+ "numberFormat": {
+ "$ref": "#/components/schemas/NumberFormat",
+ "description": "Number formatting options for displayed values."
}
}
},
- "OnClickTarget": {
- "description": "Identifies the source (for type=search) or dashboard (for type=dashboard) to link out to. Set mode to \"id\" to resolve a concrete ID, or \"template\" to resolve by rendered name at click time.\n",
- "oneOf": [
+ "PromqlTimeChartConfigBase": {
+ "description": "Shared fields for PromQL time-series chart configs.",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/PromqlChartConfigBase"
+ },
{
"type": "object",
"required": [
- "mode",
- "id"
+ "expressions"
],
"properties": {
- "mode": {
- "type": "string",
- "enum": [
- "id"
- ],
- "description": "Target is a single dashboard or log/trace source",
- "example": "id"
+ "expressions": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 10,
+ "description": "PromQL expressions to plot.",
+ "items": {
+ "$ref": "#/components/schemas/PromqlRangeExpression"
+ }
},
- "id": {
+ "legendTemplate": {
"type": "string",
- "description": "ID of the target source (for search) or dashboard (for dashboard).",
- "example": "65f5e4a3b9e77c001a567890"
+ "maxLength": 1024,
+ "description": "Template for series names, using the series' Prometheus labels in double braces.\n",
+ "example": "{{job}} - {{instance}}"
+ },
+ "fillNulls": {
+ "type": "boolean",
+ "description": "Fill missing time buckets with zero instead of leaving gaps.",
+ "default": true
+ },
+ "alignDateRangeToGranularity": {
+ "type": "boolean",
+ "description": "Expand date range boundaries to the query granularity interval.",
+ "default": true
+ },
+ "seriesLimit": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Maximum number of series rendered (top-N by value). Omit to use the default render cap, set 0 for unlimited, or a positive N to keep the top N series.\n",
+ "example": 5
}
}
+ }
+ ]
+ },
+ "LinePromqlChartConfig": {
+ "description": "PromQL configuration for a line time-series chart.",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/PromqlTimeChartConfigBase"
},
{
"type": "object",
"required": [
- "mode",
- "template"
+ "displayType"
],
"properties": {
- "mode": {
+ "displayType": {
"type": "string",
"enum": [
- "template"
+ "line"
],
- "description": "Target is matched by name against the template.",
- "example": "template"
+ "description": "Display as a line time-series chart.",
+ "example": "line"
},
- "template": {
+ "compareToPreviousPeriod": {
+ "type": "boolean",
+ "description": "Overlay the equivalent previous time period for comparison.",
+ "default": false
+ },
+ "lineInterpolation": {
"type": "string",
- "minLength": 1,
- "description": "Name template rendered against the clicked row; supports `{{column}}` variables.\n",
- "example": "{{ServiceName}}"
+ "enum": [
+ "smooth",
+ "linear",
+ "step"
+ ],
+ "example": "step",
+ "description": "How points are connected. Defaults to \"smooth\" when omitted.\n"
}
}
}
- ],
- "discriminator": {
- "propertyName": "mode"
- }
+ ]
},
- "OnClickSearch": {
- "type": "object",
- "required": [
- "type",
- "target"
- ],
- "description": "Link-out that navigates to the HyperDX search view.",
- "properties": {
- "type": {
- "type": "string",
- "enum": [
- "search"
- ],
- "description": "OnClick variant discriminator. Must be \"search\" for search link-outs.",
- "example": "search"
- },
- "target": {
- "$ref": "#/components/schemas/OnClickTarget",
- "description": "The source to navigate to."
- },
- "whereTemplate": {
- "type": "string",
- "description": "Optional WHERE clause template applied to the destination search.",
- "example": "ServiceName = '{{ServiceName}}'"
- },
- "whereLanguage": {
- "$ref": "#/components/schemas/QueryLanguage",
- "description": "Language of the rendered whereTemplate."
+ "BarPromqlChartConfig": {
+ "description": "PromQL configuration for a stacked-bar time-series chart.",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/PromqlTimeChartConfigBase"
},
- "filters": {
- "type": "array",
- "description": "Optional dashboard filter templates rendered against the clicked row.",
- "items": {
- "$ref": "#/components/schemas/OnClickFilterTemplate"
+ {
+ "type": "object",
+ "required": [
+ "displayType"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "stacked_bar"
+ ],
+ "description": "Display as a stacked-bar time-series chart.",
+ "example": "stacked_bar"
+ }
}
}
- }
+ ]
},
- "OnClickDashboard": {
- "type": "object",
- "required": [
- "type",
- "target"
- ],
- "description": "Link-out that navigates to a HyperDX dashboard.",
- "properties": {
- "type": {
- "type": "string",
- "enum": [
- "dashboard"
- ],
- "description": "OnClick variant discriminator. Must be \"dashboard\" for dashboard link-outs.",
- "example": "dashboard"
- },
- "target": {
- "$ref": "#/components/schemas/OnClickTarget",
- "description": "The dashboard to navigate to."
- },
- "whereTemplate": {
- "type": "string",
- "description": "Optional WHERE clause template applied to the destination dashboard.",
- "example": "ServiceName = '{{ServiceName}}'"
- },
- "whereLanguage": {
- "$ref": "#/components/schemas/QueryLanguage",
- "description": "Language of the rendered whereTemplate."
+ "StackedLinePromqlChartConfig": {
+ "description": "PromQL configuration for a stacked-line time-series chart.",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/PromqlTimeChartConfigBase"
},
- "filters": {
- "type": "array",
- "description": "Optional dashboard filter templates rendered against the clicked row.",
- "items": {
- "$ref": "#/components/schemas/OnClickFilterTemplate"
+ {
+ "type": "object",
+ "required": [
+ "displayType"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "stacked_line"
+ ],
+ "description": "Display as a stacked-line time-series chart.",
+ "example": "stacked_line"
+ },
+ "lineInterpolation": {
+ "type": "string",
+ "enum": [
+ "smooth",
+ "linear",
+ "step"
+ ],
+ "example": "step",
+ "description": "How points are connected. Defaults to \"linear\" when omitted.\n"
+ }
}
}
- }
+ ]
},
- "OnClick": {
- "description": "Link-out configuration applied when a user clicks a row of a table tile. Only table tiles (builder or raw SQL) currently support onClick. When target.mode is \"id\", the referenced source (type=search) or dashboard (type=dashboard) must already exist for the team.\n",
- "oneOf": [
+ "TablePromqlChartConfig": {
+ "description": "PromQL configuration for a table chart, with one row per sample: a column per label, a Value column, and a Timestamp column when the expression is a range query. An instant query gives one row per series.\n",
+ "allOf": [
{
- "$ref": "#/components/schemas/OnClickSearch"
+ "$ref": "#/components/schemas/PromqlChartConfigBase"
},
{
- "$ref": "#/components/schemas/OnClickDashboard"
- }
- ],
- "discriminator": {
- "propertyName": "type",
- "mapping": {
- "search": "#/components/schemas/OnClickSearch",
- "dashboard": "#/components/schemas/OnClickDashboard"
+ "type": "object",
+ "required": [
+ "displayType",
+ "expressions"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "table"
+ ],
+ "description": "Display as a table chart.",
+ "example": "table"
+ },
+ "expressions": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 10,
+ "description": "PromQL expressions. Only the first is queried; the rest are kept but ignored.\n",
+ "items": {
+ "$ref": "#/components/schemas/PromqlInstantExpression"
+ }
+ }
+ }
}
- }
+ ]
},
- "TableChartConfig": {
- "description": "Table chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate).\n",
- "oneOf": [
+ "NumberPromqlChartConfig": {
+ "description": "PromQL configuration for a single big-number chart.",
+ "allOf": [
{
- "$ref": "#/components/schemas/TableBuilderChartConfig"
+ "$ref": "#/components/schemas/PromqlChartConfigBase"
},
{
- "$ref": "#/components/schemas/TableRawSqlChartConfig"
- }
- ],
- "discriminator": {
- "propertyName": "configType",
- "mapping": {
- "sql": "#/components/schemas/TableRawSqlChartConfig"
+ "type": "object",
+ "required": [
+ "displayType",
+ "expressions"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "number"
+ ],
+ "description": "Display as a single big-number chart.",
+ "example": "number"
+ },
+ "expressions": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 10,
+ "description": "PromQL expressions. Only the first is queried; the rest are kept but ignored. The tile displays the first value.\n",
+ "items": {
+ "$ref": "#/components/schemas/PromqlReducibleExpression"
+ }
+ },
+ "color": {
+ "$ref": "#/components/schemas/ChartPaletteToken",
+ "description": "Optional static color applied to the displayed number."
+ },
+ "colorRules": {
+ "type": "array",
+ "maxItems": 10,
+ "description": "Ordered conditional color rules evaluated against the displayed value (last match wins). Falls back to color, then the default text color when no rule matches.\n",
+ "items": {
+ "$ref": "#/components/schemas/NumberTileColorCondition"
+ }
+ },
+ "backgroundChart": {
+ "$ref": "#/components/schemas/BackgroundChart",
+ "description": "Optional trend sparkline drawn behind the number. Only rendered when the expression uses a range query.\n"
+ }
+ }
}
- }
+ ]
},
- "NumberChartConfig": {
- "description": "Single big-number chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate).\n",
+ "PiePromqlChartConfig": {
+ "description": "PromQL configuration for a pie chart, with one slice per series.",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/PromqlChartConfigBase"
+ },
+ {
+ "type": "object",
+ "required": [
+ "displayType",
+ "expressions"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "pie"
+ ],
+ "description": "Display as a pie chart.",
+ "example": "pie"
+ },
+ "expressions": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 10,
+ "description": "PromQL expressions. Only the first is queried; the rest are kept but ignored.\n",
+ "items": {
+ "$ref": "#/components/schemas/PromqlReducibleExpression"
+ }
+ },
+ "legendTemplate": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "Template for slice names, using the series' Prometheus labels in double braces.\n",
+ "example": "{{job}}"
+ },
+ "limit": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Maximum number of slices (top-N by value), at most 500. Omit or set 0 to show up to 500.\n",
+ "example": 10
+ }
+ }
+ }
+ ]
+ },
+ "CategoricalBarPromqlChartConfig": {
+ "description": "PromQL configuration for a categorical bar chart, with one bar per series.",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/PromqlChartConfigBase"
+ },
+ {
+ "type": "object",
+ "required": [
+ "displayType",
+ "expressions"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "bar"
+ ],
+ "description": "Display as a categorical bar chart.",
+ "example": "bar"
+ },
+ "expressions": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 10,
+ "description": "PromQL expressions. Only the first is queried; the rest are kept but ignored.\n",
+ "items": {
+ "$ref": "#/components/schemas/PromqlReducibleExpression"
+ }
+ },
+ "legendTemplate": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "Template for bar names, using the series' Prometheus labels in double braces.\n",
+ "example": "{{job}}"
+ },
+ "limit": {
+ "type": "integer",
+ "minimum": 0,
+ "description": "Maximum number of bars (top-N by value), at most 500. Omit or set 0 to show up to 500.\n",
+ "example": 10
+ }
+ }
+ }
+ ]
+ },
+ "HeatmapPromqlChartConfig": {
+ "description": "PromQL configuration for a heatmap.",
+ "allOf": [
+ {
+ "$ref": "#/components/schemas/PromqlChartConfigBase"
+ },
+ {
+ "type": "object",
+ "required": [
+ "displayType",
+ "expressions"
+ ],
+ "properties": {
+ "displayType": {
+ "type": "string",
+ "enum": [
+ "heatmap"
+ ],
+ "description": "Display as a heatmap.",
+ "example": "heatmap"
+ },
+ "expressions": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 10,
+ "description": "PromQL expressions. Only the first is queried; the rest are kept but ignored.\n",
+ "items": {
+ "$ref": "#/components/schemas/PromqlRangeExpression"
+ }
+ },
+ "heatmapMode": {
+ "type": "string",
+ "enum": [
+ "distribution",
+ "series",
+ "histogram"
+ ],
+ "description": "\"distribution\" buckets the values of all series over time, \"series\" draws one row per series, and \"histogram\" draws one row per Prometheus histogram bucket (series need an \"le\" label). Defaults to \"distribution\" when omitted.\n",
+ "example": "histogram"
+ },
+ "heatmapScaleType": {
+ "type": "string",
+ "enum": [
+ "log",
+ "linear"
+ ],
+ "description": "Y-axis scale of a distribution heatmap.",
+ "example": "log"
+ },
+ "legendTemplate": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "Template for row names of a series heatmap, using the series' Prometheus labels in double braces.\n",
+ "example": "{{job}}"
+ }
+ }
+ }
+ ]
+ },
+ "LineChartConfig": {
+ "description": "Line chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate), or to \"promql\" for the PromQL variant (requires sourceId and expressions).\n",
"oneOf": [
{
- "$ref": "#/components/schemas/NumberBuilderChartConfig"
+ "$ref": "#/components/schemas/LineBuilderChartConfig"
},
{
- "$ref": "#/components/schemas/NumberRawSqlChartConfig"
+ "$ref": "#/components/schemas/LineRawSqlChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/LinePromqlChartConfig"
}
],
"discriminator": {
"propertyName": "configType",
"mapping": {
- "sql": "#/components/schemas/NumberRawSqlChartConfig"
+ "sql": "#/components/schemas/LineRawSqlChartConfig",
+ "promql": "#/components/schemas/LinePromqlChartConfig"
}
}
},
- "PieChartConfig": {
- "description": "Pie chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate).\n",
+ "BarChartConfig": {
+ "description": "Stacked-bar chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate), or to \"promql\" for the PromQL variant (requires sourceId and expressions).\n",
"oneOf": [
{
- "$ref": "#/components/schemas/PieBuilderChartConfig"
+ "$ref": "#/components/schemas/BarBuilderChartConfig"
},
{
- "$ref": "#/components/schemas/PieRawSqlChartConfig"
+ "$ref": "#/components/schemas/BarRawSqlChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/BarPromqlChartConfig"
}
],
"discriminator": {
"propertyName": "configType",
"mapping": {
- "sql": "#/components/schemas/PieRawSqlChartConfig"
+ "sql": "#/components/schemas/BarRawSqlChartConfig",
+ "promql": "#/components/schemas/BarPromqlChartConfig"
}
}
},
- "TileConfig": {
- "description": "Tile chart configuration. displayType is the primary discriminant and determines which variant group applies. For displayTypes that support both builder and Raw SQL modes (line, stacked_bar, table, number, pie), configType is the secondary discriminant: omit it for the builder variant or set it to \"sql\" for the Raw SQL variant. The heatmap, search, and markdown displayTypes only have a builder variant.\n",
+ "StackedLineChartConfig": {
+ "description": "Stacked-line chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate), or to \"promql\" for the PromQL variant (requires sourceId and expressions).\n",
"oneOf": [
{
- "$ref": "#/components/schemas/LineChartConfig"
- },
- {
- "$ref": "#/components/schemas/BarChartConfig"
- },
- {
- "$ref": "#/components/schemas/TableChartConfig"
- },
- {
- "$ref": "#/components/schemas/NumberChartConfig"
- },
- {
- "$ref": "#/components/schemas/PieChartConfig"
- },
- {
- "$ref": "#/components/schemas/HeatmapChartConfig"
+ "$ref": "#/components/schemas/StackedLineBuilderChartConfig"
},
{
- "$ref": "#/components/schemas/SearchChartConfig"
+ "$ref": "#/components/schemas/StackedLineRawSqlChartConfig"
},
{
- "$ref": "#/components/schemas/MarkdownChartConfig"
+ "$ref": "#/components/schemas/StackedLinePromqlChartConfig"
}
],
"discriminator": {
- "propertyName": "displayType",
+ "propertyName": "configType",
"mapping": {
- "line": "#/components/schemas/LineChartConfig",
- "stacked_bar": "#/components/schemas/BarChartConfig",
- "table": "#/components/schemas/TableChartConfig",
- "number": "#/components/schemas/NumberChartConfig",
- "pie": "#/components/schemas/PieChartConfig",
- "heatmap": "#/components/schemas/HeatmapChartConfig",
- "search": "#/components/schemas/SearchChartConfig",
- "markdown": "#/components/schemas/MarkdownChartConfig"
+ "sql": "#/components/schemas/StackedLineRawSqlChartConfig",
+ "promql": "#/components/schemas/StackedLinePromqlChartConfig"
}
}
},
- "DashboardContainerTab": {
+ "OnClickFilterTemplate": {
"type": "object",
- "description": "A single tab inside a dashboard container. Tiles join a tab via tabId.",
+ "description": "A templated filter applied to the link-out destination. The rendered template value is combined with the expression as `expression IN (...)` on the destination search or dashboard. Multiple templates sharing the same expression are merged into a single IN clause.\n",
"required": [
- "id",
- "title"
+ "kind",
+ "expression",
+ "template"
],
"properties": {
- "id": {
+ "kind": {
+ "type": "string",
+ "enum": [
+ "expressionTemplate"
+ ],
+ "description": "Filter template kind. Currently only \"expressionTemplate\" is supported.",
+ "example": "expressionTemplate"
+ },
+ "expression": {
"type": "string",
"minLength": 1,
- "maxLength": 256,
- "description": "Unique identifier for the tab within its container.",
- "example": "errors"
+ "description": "The column/expression to filter the destination by (e.g. \"ServiceName\").",
+ "example": "ServiceName"
},
- "title": {
+ "template": {
"type": "string",
"minLength": 1,
- "maxLength": 256,
- "description": "Display title for the tab.",
- "example": "Errors"
+ "description": "Value template rendered against the clicked row; supports row column variables in `{{column}}` form (e.g. `{{ServiceName}}`).\n",
+ "example": "{{ServiceName}}"
}
}
},
- "DashboardContainer": {
+ "OnClickTarget": {
+ "description": "Identifies the source (for type=search) or dashboard (for type=dashboard) to link out to. Set mode to \"id\" to resolve a concrete ID, or \"template\" to resolve by rendered name at click time.\n",
+ "oneOf": [
+ {
+ "type": "object",
+ "required": [
+ "mode",
+ "id"
+ ],
+ "properties": {
+ "mode": {
+ "type": "string",
+ "enum": [
+ "id"
+ ],
+ "description": "Target is a single dashboard or log/trace source",
+ "example": "id"
+ },
+ "id": {
+ "type": "string",
+ "description": "ID of the target source (for search) or dashboard (for dashboard).",
+ "example": "65f5e4a3b9e77c001a567890"
+ }
+ }
+ },
+ {
+ "type": "object",
+ "required": [
+ "mode",
+ "template"
+ ],
+ "properties": {
+ "mode": {
+ "type": "string",
+ "enum": [
+ "template"
+ ],
+ "description": "Target is matched by name against the template.",
+ "example": "template"
+ },
+ "template": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Name template rendered against the clicked row; supports `{{column}}` variables.\n",
+ "example": "{{ServiceName}}"
+ }
+ }
+ }
+ ],
+ "discriminator": {
+ "propertyName": "mode"
+ }
+ },
+ "OnClickSearch": {
"type": "object",
- "description": "A grouping container for tiles on a dashboard. Tiles join a container via containerId.",
"required": [
- "id",
- "title",
- "collapsed"
+ "type",
+ "target"
],
+ "description": "Link-out that navigates to the HyperDX search view.",
"properties": {
- "id": {
+ "type": {
"type": "string",
- "minLength": 1,
+ "enum": [
+ "search"
+ ],
+ "description": "OnClick variant discriminator. Must be \"search\" for search link-outs.",
+ "example": "search"
+ },
+ "target": {
+ "$ref": "#/components/schemas/OnClickTarget",
+ "description": "The source to navigate to."
+ },
+ "whereTemplate": {
+ "type": "string",
+ "description": "Optional WHERE clause template applied to the destination search.",
+ "example": "ServiceName = '{{ServiceName}}'"
+ },
+ "whereLanguage": {
+ "$ref": "#/components/schemas/QueryLanguage",
+ "description": "Language of the rendered whereTemplate."
+ },
+ "filters": {
+ "type": "array",
+ "description": "Optional dashboard filter templates rendered against the clicked row.",
+ "items": {
+ "$ref": "#/components/schemas/OnClickFilterTemplate"
+ }
+ }
+ }
+ },
+ "OnClickDashboard": {
+ "type": "object",
+ "required": [
+ "type",
+ "target"
+ ],
+ "description": "Link-out that navigates to a HyperDX dashboard.",
+ "properties": {
+ "type": {
+ "type": "string",
+ "enum": [
+ "dashboard"
+ ],
+ "description": "OnClick variant discriminator. Must be \"dashboard\" for dashboard link-outs.",
+ "example": "dashboard"
+ },
+ "target": {
+ "$ref": "#/components/schemas/OnClickTarget",
+ "description": "The dashboard to navigate to."
+ },
+ "whereTemplate": {
+ "type": "string",
+ "description": "Optional WHERE clause template applied to the destination dashboard.",
+ "example": "ServiceName = '{{ServiceName}}'"
+ },
+ "whereLanguage": {
+ "$ref": "#/components/schemas/QueryLanguage",
+ "description": "Language of the rendered whereTemplate."
+ },
+ "filters": {
+ "type": "array",
+ "description": "Optional dashboard filter templates rendered against the clicked row.",
+ "items": {
+ "$ref": "#/components/schemas/OnClickFilterTemplate"
+ }
+ }
+ }
+ },
+ "OnClickExternal": {
+ "type": "object",
+ "required": [
+ "type",
+ "urlTemplate"
+ ],
+ "description": "Link-out that navigates to an arbitrary external URL (e.g. a Grafana or Langfuse dashboard). The rendered URL must be an absolute http(s) URL.\n",
+ "properties": {
+ "type": {
+ "type": "string",
+ "enum": [
+ "external"
+ ],
+ "description": "OnClick variant discriminator. Must be \"external\" for external link-outs.",
+ "example": "external"
+ },
+ "urlTemplate": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Handlebars template rendered against the clicked row; supports `{{column}}` variables. The rendered value must be an absolute http(s) URL.\n",
+ "example": "https://example.com/d/abc?var-service={{ServiceName}}"
+ }
+ }
+ },
+ "OnClick": {
+ "description": "Link-out configuration applied when a user clicks a row of a table tile. Only table tiles (builder or raw SQL) currently support onClick. When target.mode is \"id\", the referenced source (type=search) or dashboard (type=dashboard) must already exist for the team.\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/OnClickSearch"
+ },
+ {
+ "$ref": "#/components/schemas/OnClickDashboard"
+ },
+ {
+ "$ref": "#/components/schemas/OnClickExternal"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "type",
+ "mapping": {
+ "search": "#/components/schemas/OnClickSearch",
+ "dashboard": "#/components/schemas/OnClickDashboard",
+ "external": "#/components/schemas/OnClickExternal"
+ }
+ }
+ },
+ "TableChartConfig": {
+ "description": "Table chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate), or to \"promql\" for the PromQL variant (requires sourceId and expressions).\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/TableBuilderChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/TableRawSqlChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/TablePromqlChartConfig"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "configType",
+ "mapping": {
+ "sql": "#/components/schemas/TableRawSqlChartConfig",
+ "promql": "#/components/schemas/TablePromqlChartConfig"
+ }
+ }
+ },
+ "NumberChartConfig": {
+ "description": "Single big-number chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate), or to \"promql\" for the PromQL variant (requires sourceId and expressions).\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/NumberBuilderChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/NumberRawSqlChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/NumberPromqlChartConfig"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "configType",
+ "mapping": {
+ "sql": "#/components/schemas/NumberRawSqlChartConfig",
+ "promql": "#/components/schemas/NumberPromqlChartConfig"
+ }
+ }
+ },
+ "PieChartConfig": {
+ "description": "Pie chart. Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate), or to \"promql\" for the PromQL variant (requires sourceId and expressions).\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/PieBuilderChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/PieRawSqlChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/PiePromqlChartConfig"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "configType",
+ "mapping": {
+ "sql": "#/components/schemas/PieRawSqlChartConfig",
+ "promql": "#/components/schemas/PiePromqlChartConfig"
+ }
+ }
+ },
+ "CategoricalBarChartConfig": {
+ "description": "Categorical bar chart (one bar per group value; not a time series). Omit configType for the builder variant (requires sourceId and select). Set configType to \"sql\" for the Raw SQL variant (requires connectionId and sqlTemplate), or to \"promql\" for the PromQL variant (requires sourceId and expressions).\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/CategoricalBarBuilderChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/CategoricalBarRawSqlChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/CategoricalBarPromqlChartConfig"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "configType",
+ "mapping": {
+ "sql": "#/components/schemas/CategoricalBarRawSqlChartConfig",
+ "promql": "#/components/schemas/CategoricalBarPromqlChartConfig"
+ }
+ }
+ },
+ "TileConfig": {
+ "description": "Tile chart configuration. displayType is the primary discriminant and determines which variant group applies. configType is the secondary discriminant: omit it for the builder variant, set it to \"sql\" for the Raw SQL variant, or set it to \"promql\" for the PromQL variant. Line, stacked_bar, stacked_line, table, number, pie, and bar support all three. Heatmap supports builder and PromQL. Search, event_patterns, and markdown only have a builder variant.\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/LineChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/BarChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/StackedLineChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/TableChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/NumberChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/PieChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/CategoricalBarChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/HeatmapChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/SearchChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/EventPatternsChartConfig"
+ },
+ {
+ "$ref": "#/components/schemas/MarkdownChartConfig"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "displayType",
+ "mapping": {
+ "line": "#/components/schemas/LineChartConfig",
+ "stacked_bar": "#/components/schemas/BarChartConfig",
+ "stacked_line": "#/components/schemas/StackedLineChartConfig",
+ "table": "#/components/schemas/TableChartConfig",
+ "number": "#/components/schemas/NumberChartConfig",
+ "pie": "#/components/schemas/PieChartConfig",
+ "bar": "#/components/schemas/CategoricalBarChartConfig",
+ "heatmap": "#/components/schemas/HeatmapChartConfig",
+ "search": "#/components/schemas/SearchChartConfig",
+ "event_patterns": "#/components/schemas/EventPatternsChartConfig",
+ "markdown": "#/components/schemas/MarkdownChartConfig"
+ }
+ }
+ },
+ "DashboardContainerTab": {
+ "type": "object",
+ "description": "A single tab inside a dashboard container. Tiles join a tab via tabId.",
+ "required": [
+ "id",
+ "title"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 256,
+ "description": "Unique identifier for the tab within its container.",
+ "example": "errors"
+ },
+ "title": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 256,
+ "description": "Display title for the tab.",
+ "example": "Errors"
+ }
+ }
+ },
+ "DashboardContainer": {
+ "type": "object",
+ "description": "A grouping container for tiles on a dashboard. Tiles join a container via containerId.",
+ "required": [
+ "id",
+ "title",
+ "collapsed"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "minLength": 1,
"maxLength": 256,
"description": "Unique identifier for the container within the dashboard.",
"example": "service-health"
@@ -2229,6 +3875,12 @@
"maxLength": 256,
"description": "References a tab inside the tile's container by id. Requires containerId to be set, and the container to declare a matching tab.",
"example": "errors"
+ },
+ "description": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "Short explanation of the tile, shown as a tooltip next to its name.",
+ "example": "Share of API requests that returned a 5xx response."
}
}
},
@@ -2289,11 +3941,33 @@
]
},
"FilterInput": {
- "type": "object",
- "description": "Dashboard filter key that can be added to a dashboard",
- "required": [
- "type",
- "name",
+ "description": "A drop-down filter on a dashboard. Depending on type, filters can either broadcast\nselected value(s) to every tile (isBroadcastEnabled), expose selected\nvalues as a variable (isVariableEnabled) or both. Typically a filter should\ndo just one of these things. Any filter may additionally be marked\nrequired via minSelections, which blocks the tiles that read it until it\nhas a selected value - or every tile on the dashboard, with\nisGlobalRequirement.\n",
+ "oneOf": [
+ {
+ "$ref": "#/components/schemas/QueryExpressionFilterInput"
+ },
+ {
+ "$ref": "#/components/schemas/StaticListFilterInput"
+ },
+ {
+ "$ref": "#/components/schemas/PrometheusLabelFilterInput"
+ }
+ ],
+ "discriminator": {
+ "propertyName": "type",
+ "mapping": {
+ "QUERY_EXPRESSION": "#/components/schemas/QueryExpressionFilterInput",
+ "STATIC_LIST": "#/components/schemas/StaticListFilterInput",
+ "PROMETHEUS_LABEL": "#/components/schemas/PrometheusLabelFilterInput"
+ }
+ }
+ },
+ "QueryExpressionFilterInput": {
+ "type": "object",
+ "description": "A filter whose dropdown values are queried from ClickHouse: \"expression\"\nnames the column and \"sourceId\" the source to read them from. Its\nselection can be broadcast into matching tiles' WHERE clauses.\n",
+ "required": [
+ "type",
+ "name",
"expression",
"sourceId"
],
@@ -2303,7 +3977,7 @@
"enum": [
"QUERY_EXPRESSION"
],
- "description": "Filter type. Must be \"QUERY_EXPRESSION\".",
+ "description": "Filter type discriminator. Must be \"QUERY_EXPRESSION\".",
"example": "QUERY_EXPRESSION"
},
"name": {
@@ -2315,7 +3989,7 @@
"expression": {
"type": "string",
"minLength": 1,
- "description": "Key expression used when applying this dashboard filter key",
+ "description": "SQL expression used when querying values for this filter, and when applying this dashboard filter to tiles.",
"example": "environment"
},
"sourceId": {
@@ -2355,10 +4029,236 @@
"items": {
"type": "string"
},
- "description": "Optional list of source IDs this filter applies to. Omit or provide\nan empty array to apply the filter to ALL tiles regardless of source.\nA non-empty array restricts the filter to only tiles whose source ID\nis in the list; tiles using other sources are not affected by the\nselected filter value(s).\n",
+ "description": "Optional list of source IDs this filter applies to. Omit or provide\nan empty array to apply the filter to ALL tiles regardless of source.\nA non-empty array restricts the filter to only tiles whose source ID\nis in the list; tiles using other sources are not affected by the\nselected filter value(s). Scopes the broadcast condition only, so a\nnon-empty array is rejected when isBroadcastEnabled is false, and is\nomitted from responses for such a filter.\n",
"example": [
"65f5e4a3b9e77c001a111111"
]
+ },
+ "isBroadcastEnabled": {
+ "type": "boolean",
+ "description": "Whether the selected value is applied as a filter condition on every\nbuilder tile this filter applies to (see appliesToSourceIds), and every\nraw sql tile using the $__filters macro. Omitting the field means enabled.\n",
+ "default": true,
+ "example": false
+ },
+ "isVariableEnabled": {
+ "type": "boolean",
+ "description": "Whether the selected value is exposed to tile queries as a dashboard\nvariable named by variableName. Tiles may reference it as `$variableName`\nor using the (preferred) `$__filter($)` and\n`$__conditionalAll(, $)` macros.\n",
+ "default": false,
+ "example": true
+ },
+ "variableName": {
+ "type": "string",
+ "maxLength": 64,
+ "pattern": "^[a-zA-Z][a-zA-Z0-9_]*$",
+ "description": "Token tiles reference this filter's selected value by, as\n`$variableName`. Must start with a letter and may contain only\nletters, numbers, and underscores. Defaults to the display name with\nwhitespace replaced by underscores and remaining illegal characters\nremoved, so a variable-enabled filter whose name derives nothing\nusable must send this field explicitly. Variable names must be unique\nacross a dashboard's variable-enabled filters. Names the variable\nonly, so the field is rejected when isVariableEnabled is not true, and\nis omitted from responses for such a filter.\n",
+ "example": "environment"
+ },
+ "minSelections": {
+ "type": "integer",
+ "enum": [
+ 0,
+ 1
+ ],
+ "default": 0,
+ "description": "Minimum number of values that must be selected before tiles load.\nSet to 1 to make the filter required. Only 0 and 1 are accepted.\nOmit the field (or send 0) for the default optional behavior.\n",
+ "example": 1
+ },
+ "isGlobalRequirement": {
+ "type": "boolean",
+ "default": false,
+ "description": "Widens a required filter's block to every tile on the dashboard. False\n(the default) blocks only the tiles that read the filter: those\nreferencing its variableName, and those its broadcast applies to.\nIgnored unless minSelections is 1.\n",
+ "example": true
+ },
+ "maxSelections": {
+ "type": "integer",
+ "enum": [
+ 1
+ ],
+ "description": "Maximum number of values that can be selected at once. Set to 1 to make\nthe filter single-select. Omit the field for the default multi-select\nbehavior.\n",
+ "example": 1
+ }
+ }
+ },
+ "StaticListFilterInput": {
+ "type": "object",
+ "description": "A filter whose dropdown offers a static custom \"options\" list. It carries no\nexpression, so it cannot be broadcast, and supports only variable mode.\n",
+ "required": [
+ "type",
+ "name",
+ "options"
+ ],
+ "properties": {
+ "type": {
+ "type": "string",
+ "enum": [
+ "STATIC_LIST"
+ ],
+ "description": "Discriminator. Must be \"STATIC_LIST\".",
+ "example": "STATIC_LIST"
+ },
+ "name": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Display name for the dashboard filter key",
+ "example": "Environment"
+ },
+ "options": {
+ "type": "array",
+ "minItems": 1,
+ "maxItems": 1000,
+ "items": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 10000
+ },
+ "description": "The values this filter's dropdown offers. Must be non-empty and\nfree of duplicates.\n",
+ "example": [
+ "prod",
+ "staging",
+ "dev"
+ ]
+ },
+ "isBroadcastEnabled": {
+ "type": "boolean",
+ "enum": [
+ false
+ ],
+ "default": false,
+ "description": "Must be false — there is no expression to broadcast. Omit it and it\nis false.\n",
+ "example": false
+ },
+ "isVariableEnabled": {
+ "type": "boolean",
+ "enum": [
+ true
+ ],
+ "default": true,
+ "description": "Must be true if provided, and is true when omitted. Tiles reference\nthe selection as `$variableName`, or with the `$__filter(, $)`\nand `$__conditionalAll(, $)` macros. The\none-argument `$__filter($)` form renders the filter's\nown expression, which this kind of filter does not have, so it\nreports that the expression must be passed explicitly.\n",
+ "example": true
+ },
+ "variableName": {
+ "type": "string",
+ "maxLength": 64,
+ "pattern": "^[a-zA-Z][a-zA-Z0-9_]*$",
+ "description": "Token tiles reference this filter's selected value by, as\n`$variableName`. Must start with a letter and may contain only\nletters, numbers, and underscores. Defaults to the display name with\nwhitespace replaced by underscores and remaining illegal characters\nremoved, so a variable-enabled filter whose name derives nothing\nusable must send this field explicitly. Variable names must be unique\nacross a dashboard's variable-enabled filters. Names the variable\nonly, so the field is rejected when isVariableEnabled is not true, and\nis omitted from responses for such a filter.\n",
+ "example": "environment"
+ },
+ "minSelections": {
+ "type": "integer",
+ "enum": [
+ 0,
+ 1
+ ],
+ "default": 0,
+ "description": "Minimum number of values that must be selected before tiles load.\nSet to 1 to make the filter required. Only 0 and 1 are accepted.\nOmit the field (or send 0) for the default optional behavior.\n",
+ "example": 1
+ },
+ "isGlobalRequirement": {
+ "type": "boolean",
+ "default": false,
+ "description": "Widens a required filter's block to every tile on the dashboard. False\n(the default) blocks only the tiles that read the filter: those\nreferencing its variableName, and those its broadcast applies to.\nIgnored unless minSelections is 1.\n",
+ "example": true
+ },
+ "maxSelections": {
+ "type": "integer",
+ "enum": [
+ 1
+ ],
+ "description": "Maximum number of values that can be selected at once. Set to 1 to make\nthe filter single-select. Omit the field for the default multi-select\nbehavior.\n",
+ "example": 1
+ }
+ }
+ },
+ "PrometheusLabelFilterInput": {
+ "type": "object",
+ "description": "A filter whose dropdown lists the values for a Prometheus label,\nread from a PromQL source over the dashboard's time range.\nSupports variable mode only. PromQL is a preview feature and must be\nenabled on the deployment (ENABLE_PROMQL) before these filters can\nlist values.\n",
+ "required": [
+ "type",
+ "name",
+ "sourceId",
+ "label"
+ ],
+ "properties": {
+ "type": {
+ "type": "string",
+ "enum": [
+ "PROMETHEUS_LABEL"
+ ],
+ "description": "Discriminator. Must be \"PROMETHEUS_LABEL\".",
+ "example": "PROMETHEUS_LABEL"
+ },
+ "name": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Display name for the dashboard filter",
+ "example": "Pod"
+ },
+ "sourceId": {
+ "type": "string",
+ "description": "Id of the PromQL source the label values are read from.\n",
+ "example": "65f5e4a3b9e77c001a123456"
+ },
+ "label": {
+ "type": "string",
+ "minLength": 1,
+ "maxLength": 1024,
+ "description": "Label whose values populate the dropdown. Use \"__name__\" to list\nmetric names.\n",
+ "example": "pod"
+ },
+ "match": {
+ "type": "string",
+ "minLength": 1,
+ "description": "Optional Prometheus series selector narrowing which series the\nlabel values are read from.\n",
+ "example": "up{job=\"api\"}"
+ },
+ "isBroadcastEnabled": {
+ "type": "boolean",
+ "enum": [
+ false
+ ],
+ "default": false,
+ "description": "Must be false or omitted.",
+ "example": false
+ },
+ "isVariableEnabled": {
+ "type": "boolean",
+ "enum": [
+ true
+ ],
+ "default": true,
+ "description": "Must be true if provided, and is true when omitted. Tiles reference\nthe selection as `$variableName`.\n",
+ "example": true
+ },
+ "variableName": {
+ "type": "string",
+ "maxLength": 64,
+ "pattern": "^[a-zA-Z][a-zA-Z0-9_]*$",
+ "description": "Token that tiles reference this filter's selected value by, as\n`$variableName`. Must start with a letter and may contain only\nletters, numbers, and underscores. Defaults to the display name with\nwhitespace replaced by underscores and remaining illegal characters\nremoved. Variable names must be unique across a dashboard's\nvariable-enabled filters.\n",
+ "example": "pod"
+ },
+ "minSelections": {
+ "type": "integer",
+ "enum": [
+ 0,
+ 1
+ ],
+ "default": 0,
+ "description": "Minimum number of values that must be selected before tiles load.\nSet to 1 to make the filter required. Only 0 and 1 are accepted.\nOmit the field (or send 0) for the default optional behavior.\n",
+ "example": 1
+ },
+ "isGlobalRequirement": {
+ "type": "boolean",
+ "default": false,
+ "description": "Widens a required filter's block to every tile on the dashboard. False\n(the default) blocks only the tiles that read the filter: those\nreferencing its variableName, and those its broadcast applies to.\nIgnored unless minSelections is 1.\n",
+ "example": true
+ },
+ "maxSelections": {
+ "type": "integer",
+ "enum": [
+ 1
+ ],
+ "description": "Maximum number of values that can be selected at once. Set to 1 to make\nthe filter single-select. Omit the field for the default multi-select\nbehavior.\n",
+ "example": 1
}
}
},
@@ -2418,7 +4318,7 @@
},
"filters": {
"type": "array",
- "description": "Dashboard filter keys added to the dashboard and applied to all tiles",
+ "description": "Dropdown filters added to the dashboard. Each one broadcasts\nits selected value as a condition, acts as a variable which\ncan be referenced in tile queries, or both.\n",
"items": {
"$ref": "#/components/schemas/Filter"
}
@@ -2488,7 +4388,7 @@
},
"filters": {
"type": "array",
- "description": "Dashboard filter keys to add to the dashboard and apply across all tiles",
+ "description": "Dropdown filters added to the dashboard. Each one broadcasts\nits selected value as a condition, acts as a variable which\ncan be referenced in tile queries, or both.\n",
"items": {
"$ref": "#/components/schemas/FilterInput"
}
@@ -2559,7 +4459,7 @@
},
"filters": {
"type": "array",
- "description": "Dashboard filter keys on the dashboard, applied across all tiles",
+ "description": "Dropdown filters added to the dashboard. Each one broadcasts\nits selected value as a condition, acts as a variable which\ncan be referenced in tile queries, or both.\n",
"items": {
"$ref": "#/components/schemas/Filter"
}
@@ -2622,34 +4522,62 @@
}
}
},
- "SearchRequest": {
+ "SavedSearchFilter": {
"type": "object",
"required": [
+ "condition"
+ ],
+ "description": "A single pinned filter applied to the search. New or changed filters must use the SQL predicate form the UI produces so they render as a sidebar facet: ` IN ('', ...)` (or `NOT IN` / `BETWEEN`). Conditions not in this form are rejected on create, and on update unless they are echoed back unchanged from the stored saved search (so a read-modify-write of a legacy filter still succeeds). Note: existing saved searches created in the UI may also return a Lucene text filter (`type: lucene`) or a structured comparison (`type: sql_ast` with `operator`/`left`/`right`); reads return these stored shapes verbatim, and they are preserved on unchanged update.",
+ "properties": {
+ "type": {
+ "type": "string",
+ "enum": [
+ "sql"
+ ],
+ "default": "sql",
+ "description": "Always `sql`. Only SQL predicate filters render in the sidebar.",
+ "example": "sql"
+ },
+ "condition": {
+ "type": "string",
+ "maxLength": 8192,
+ "description": "SQL predicate applied to the search, in ` IN (...)` form.",
+ "example": "ServiceName IN ('checkout', 'payments')"
+ }
+ }
+ },
+ "SavedSearch": {
+ "type": "object",
+ "required": [
+ "id",
+ "name",
"sourceId"
],
"properties": {
- "sourceId": {
+ "id": {
"type": "string",
- "description": "Source ID to query. Call GET /api/v2/sources to list available sources.\nThe source determines the underlying ClickHouse table (e.g. otel.otel_logs,\notel.otel_traces) and its column schema.\n",
- "example": "69b46cb0d964ce2d0b9506a8"
+ "readOnly": true,
+ "description": "Unique saved search ID. Server-generated.",
+ "example": "507f1f77bcf86cd799439011"
},
- "startTime": {
+ "name": {
"type": "string",
- "format": "date-time",
- "description": "Start of the query window (ISO 8601). Defaults to 15 minutes before endTime. Must be before endTime.\n",
- "example": "2026-05-10T00:00:00Z"
+ "description": "Display name for the saved search.",
+ "example": "Production Errors"
},
- "endTime": {
+ "sourceId": {
"type": "string",
- "format": "date-time",
- "description": "End of the query window (ISO 8601). Defaults to now.",
- "example": "2026-05-10T01:00:00Z"
+ "description": "ID of the source this saved search queries.",
+ "example": "507f1f77bcf86cd799439012"
+ },
+ "select": {
+ "type": "string",
+ "description": "Comma-separated list of column expressions to display. Empty uses the source default.",
+ "example": "Timestamp, ServiceName, Body"
},
"where": {
"type": "string",
- "maxLength": 8192,
- "default": "",
- "description": "Row filter expression. The language is controlled by whereLanguage.\n\nLucene examples (default):\n SeverityText:ERROR\n pipedream.pipeline_name:my-pipeline AND SeverityText:ERROR\n Body:timeout\n\nSQL examples (whereLanguage: \"sql\"):\n SeverityText = 'ERROR'\n `pipedream.pipeline_name` = 'my-pipeline'\n",
+ "description": "Row filter expression. The language is controlled by whereLanguage.",
"example": "SeverityText:ERROR"
},
"whereLanguage": {
@@ -2658,58 +4586,255 @@
"lucene",
"sql"
],
- "example": "lucene",
- "default": "lucene",
- "description": "Language used for the where filter. Default is lucene."
- },
- "select": {
- "type": "string",
- "maxLength": 4096,
- "default": "",
- "description": "Comma-separated list of ClickHouse column expressions to include in\neach result row. When omitted the source's default select expression\nis used.\n\nEach entry is a ClickHouse SQL expression executed under the team's\ndatabase user. Semicolons and subqueries (SELECT keyword) are\nrejected; use column references, map lookups, or function calls only.\n\nHyperDX rewrites known attribute column names to their materialized\nequivalents automatically; you can still pass the logical name.\n",
- "example": "Timestamp,SeverityText,Body,pipedream.pipeline_name"
+ "description": "Language used for the where filter.",
+ "example": "lucene"
},
"orderBy": {
"type": "string",
- "maxLength": 1024,
- "description": "ClickHouse ORDER BY expression. When omitted the source's default\nordering (typically timestamp DESC) is used.\n",
+ "description": "ORDER BY expression. Empty uses the source default.",
"example": "Timestamp DESC"
},
- "maxResults": {
- "type": "integer",
- "minimum": 1,
- "maximum": 2000,
- "default": 100,
- "description": "Maximum number of rows to return. Default is 100, max is 2000."
- },
- "offset": {
- "type": "integer",
- "minimum": 0,
- "maximum": 10000,
- "default": 0,
- "description": "Number of rows to skip (best-effort offset pagination). Default is\n0, max is 10000. Offset pagination is non-deterministic when\nmultiple rows share the same timestamp; for reliable deep paging\nfilter by the last Timestamp value returned in the previous page\ninstead of using a large offset.\n"
- }
- }
- },
- "SearchRow": {
- "type": "object",
- "description": "A single result row. Keys correspond to the requested columns (or the source default columns when columns was omitted). Values are strings or numbers as returned by ClickHouse.\n",
- "additionalProperties": true
- },
- "SearchResponse": {
- "type": "object",
- "properties": {
- "data": {
+ "tags": {
"type": "array",
- "description": "Array of result rows. Each row is an object with keys corresponding to the requested columns.",
+ "maxItems": 50,
"items": {
- "$ref": "#/components/schemas/SearchRow"
- }
+ "type": "string",
+ "maxLength": 32
+ },
+ "description": "Tags used to organize saved searches.",
+ "example": [
+ "production",
+ "errors"
+ ]
},
- "rows": {
- "type": "integer",
- "description": "Number of rows in this response (not total matching rows)."
- }
+ "filters": {
+ "type": "array",
+ "maxItems": 100,
+ "description": "Structured pinned filters applied to the search.",
+ "items": {
+ "$ref": "#/components/schemas/SavedSearchFilter"
+ },
+ "example": [
+ {
+ "type": "sql",
+ "condition": "ServiceName IN ('checkout', 'payments')"
+ }
+ ]
+ },
+ "teamId": {
+ "type": "string",
+ "readOnly": true,
+ "description": "ID of the team that owns the saved search.",
+ "example": "507f1f77bcf86cd799439013"
+ },
+ "createdAt": {
+ "type": "string",
+ "format": "date-time",
+ "readOnly": true,
+ "description": "Creation timestamp.",
+ "example": "2025-01-01T00:00:00.000Z"
+ },
+ "updatedAt": {
+ "type": "string",
+ "format": "date-time",
+ "readOnly": true,
+ "description": "Last update timestamp.",
+ "example": "2025-06-15T10:30:00.000Z"
+ }
+ }
+ },
+ "SavedSearchInput": {
+ "type": "object",
+ "required": [
+ "name",
+ "sourceId"
+ ],
+ "properties": {
+ "name": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "Display name for the saved search.",
+ "example": "Production Errors"
+ },
+ "sourceId": {
+ "type": "string",
+ "description": "ID of the source to query. Must belong to the team.",
+ "example": "507f1f77bcf86cd799439012"
+ },
+ "select": {
+ "type": "string",
+ "maxLength": 4096,
+ "description": "Comma-separated list of column expressions to display. Empty uses the source default.",
+ "example": "Timestamp, ServiceName, Body"
+ },
+ "where": {
+ "type": "string",
+ "maxLength": 8192,
+ "description": "Row filter expression. The language is controlled by whereLanguage.",
+ "example": "SeverityText:ERROR"
+ },
+ "whereLanguage": {
+ "type": "string",
+ "enum": [
+ "lucene",
+ "sql"
+ ],
+ "default": "lucene",
+ "description": "Language used for the where filter.",
+ "example": "lucene"
+ },
+ "orderBy": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "ORDER BY expression. Empty uses the source default.",
+ "example": "Timestamp DESC"
+ },
+ "tags": {
+ "type": "array",
+ "maxItems": 50,
+ "description": "Tags used to organize saved searches.",
+ "items": {
+ "type": "string",
+ "maxLength": 32
+ },
+ "example": [
+ "production",
+ "errors"
+ ]
+ },
+ "filters": {
+ "type": "array",
+ "maxItems": 100,
+ "description": "Structured pinned filters applied to the search.",
+ "items": {
+ "$ref": "#/components/schemas/SavedSearchFilter"
+ },
+ "example": [
+ {
+ "type": "sql",
+ "condition": "ServiceName IN ('checkout', 'payments')"
+ }
+ ]
+ }
+ }
+ },
+ "SavedSearchesListResponse": {
+ "type": "object",
+ "required": [
+ "data",
+ "meta"
+ ],
+ "properties": {
+ "data": {
+ "type": "array",
+ "description": "List of saved search objects.",
+ "items": {
+ "$ref": "#/components/schemas/SavedSearch"
+ }
+ },
+ "meta": {
+ "$ref": "#/components/schemas/PaginationMeta",
+ "description": "Pagination metadata for this result page."
+ }
+ }
+ },
+ "SavedSearchResponseEnvelope": {
+ "type": "object",
+ "properties": {
+ "data": {
+ "$ref": "#/components/schemas/SavedSearch",
+ "description": "The saved search object."
+ }
+ }
+ },
+ "SearchRequest": {
+ "type": "object",
+ "required": [
+ "sourceId"
+ ],
+ "properties": {
+ "sourceId": {
+ "type": "string",
+ "description": "Source ID to query. Call GET /api/v2/sources to list available sources.\nThe source determines the underlying ClickHouse table (e.g. otel.otel_logs,\notel.otel_traces) and its column schema.\n",
+ "example": "69b46cb0d964ce2d0b9506a8"
+ },
+ "startTime": {
+ "type": "string",
+ "format": "date-time",
+ "description": "Start of the query window (ISO 8601). Defaults to 15 minutes before endTime. Must be before endTime.\n",
+ "example": "2026-05-10T00:00:00Z"
+ },
+ "endTime": {
+ "type": "string",
+ "format": "date-time",
+ "description": "End of the query window (ISO 8601). Defaults to now.",
+ "example": "2026-05-10T01:00:00Z"
+ },
+ "where": {
+ "type": "string",
+ "maxLength": 8192,
+ "default": "",
+ "description": "Row filter expression. The language is controlled by whereLanguage.\n\nLucene examples (default):\n SeverityText:ERROR\n pipedream.pipeline_name:my-pipeline AND SeverityText:ERROR\n Body:timeout\n\nSQL examples (whereLanguage: \"sql\"):\n SeverityText = 'ERROR'\n `pipedream.pipeline_name` = 'my-pipeline'\n",
+ "example": "SeverityText:ERROR"
+ },
+ "whereLanguage": {
+ "type": "string",
+ "enum": [
+ "lucene",
+ "sql"
+ ],
+ "example": "lucene",
+ "default": "lucene",
+ "description": "Language used for the where filter. Default is lucene."
+ },
+ "select": {
+ "type": "string",
+ "maxLength": 4096,
+ "default": "",
+ "description": "Comma-separated list of ClickHouse column expressions to include in\neach result row. When omitted the source's default select expression\nis used.\n\nEach entry is a ClickHouse SQL expression executed under the team's\ndatabase user. Semicolons and subqueries (SELECT keyword) are\nrejected; use column references, map lookups, or function calls only.\n\nHyperDX rewrites known attribute column names to their materialized\nequivalents automatically; you can still pass the logical name.\n",
+ "example": "Timestamp,SeverityText,Body,pipedream.pipeline_name"
+ },
+ "orderBy": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "ClickHouse ORDER BY expression. When omitted the source's default\nordering (typically timestamp DESC) is used.\n",
+ "example": "Timestamp DESC"
+ },
+ "maxResults": {
+ "type": "integer",
+ "minimum": 1,
+ "maximum": 2000,
+ "default": 100,
+ "description": "Maximum number of rows to return. Default is 100, max is 2000."
+ },
+ "offset": {
+ "type": "integer",
+ "minimum": 0,
+ "maximum": 10000,
+ "default": 0,
+ "description": "Number of rows to skip (best-effort offset pagination). Default is\n0, max is 10000. Offset pagination is non-deterministic when\nmultiple rows share the same timestamp; for reliable deep paging\nfilter by the last Timestamp value returned in the previous page\ninstead of using a large offset.\n"
+ }
+ }
+ },
+ "SearchRow": {
+ "type": "object",
+ "description": "A single result row. Keys correspond to the requested columns (or the source default columns when columns was omitted). Values are strings or numbers as returned by ClickHouse.\n",
+ "additionalProperties": true
+ },
+ "SearchResponse": {
+ "type": "object",
+ "properties": {
+ "data": {
+ "type": "array",
+ "description": "Array of result rows. Each row is an object with keys corresponding to the requested columns.",
+ "items": {
+ "$ref": "#/components/schemas/SearchRow"
+ }
+ },
+ "rows": {
+ "type": "integer",
+ "description": "Number of rows in this response (not total matching rows)."
+ }
}
},
"QuerySetting": {
@@ -2878,24 +5003,8 @@
},
"minGranularity": {
"type": "string",
- "description": "The granularity of the timestamp column",
- "enum": [
- "1s",
- "15s",
- "30s",
- "1m",
- "5m",
- "15m",
- "30m",
- "1h",
- "2h",
- "6h",
- "12h",
- "1d",
- "2d",
- "7d",
- "30d"
- ],
+ "description": "The granularity of the timestamp column: a positive integer followed by a unit (s, m, h, d). Common values: 1s, 15s, 30s, 1m, 5m, 15m, 30m, 1h, 2h, 6h, 12h, 1d, 2d, 7d, 30d.",
+ "pattern": "^\\d+(s|m|h|d)$",
"example": "5m"
},
"minDate": {
@@ -2922,7 +5031,6 @@
"LogSource": {
"type": "object",
"required": [
- "id",
"name",
"kind",
"connection",
@@ -2933,7 +5041,8 @@
"properties": {
"id": {
"type": "string",
- "description": "Unique source ID.",
+ "readOnly": true,
+ "description": "Unique source ID. Server-generated; ignored if sent in create/update requests.",
"example": "507f1f77bcf86cd799439011"
},
"name": {
@@ -2941,6 +5050,18 @@
"description": "Display name for the source.",
"example": "Logs"
},
+ "section": {
+ "type": "string",
+ "maxLength": 256,
+ "description": "Optional grouping label used to organize sources in the source selector. Sources that share a section value are displayed together.",
+ "example": "Billing"
+ },
+ "disabled": {
+ "type": "boolean",
+ "description": "When true, the source is hidden from source selectors in the UI. Defaults to false.",
+ "nullable": true,
+ "example": false
+ },
"kind": {
"type": "string",
"enum": [
@@ -2982,18 +5103,36 @@
"nullable": true,
"example": "ServiceName"
},
+ "serviceVersionExpression": {
+ "type": "string",
+ "description": "Expression identifying the running release of a service. Defaults to the OpenTelemetry service.version resource attribute when unset. Where services carry the release on different attributes, fall back across them with coalesce(nullIf(a, ''), nullIf(b, '')).",
+ "nullable": true,
+ "example": "ResourceAttributes['service.version']"
+ },
"severityTextExpression": {
"type": "string",
"description": "Expression to extract the severity/log level text.",
"nullable": true,
"example": "SeverityText"
},
+ "severityNumberExpression": {
+ "type": "string",
+ "description": "Expression for the numeric OpenTelemetry SeverityNumber (1-24).",
+ "nullable": true,
+ "example": "SeverityNumber"
+ },
"bodyExpression": {
"type": "string",
"description": "Expression to extract the log message body.",
"nullable": true,
"example": "Body"
},
+ "eventNameExpression": {
+ "type": "string",
+ "description": "Expression for the OpenTelemetry log EventName.",
+ "nullable": true,
+ "example": "EventName"
+ },
"eventAttributesExpression": {
"type": "string",
"description": "Expression to extract event-level attributes.",
@@ -3012,6 +5151,11 @@
"nullable": true,
"example": "TimestampTime"
},
+ "orderByExpression": {
+ "type": "string",
+ "description": "Custom ORDER BY expression that overrides the default ordering. Leave empty to use the auto-detected default.",
+ "example": "Timestamp DESC"
+ },
"metricSourceId": {
"type": "string",
"description": "HyperDX Source for metrics associated with logs. Optional",
@@ -3042,6 +5186,12 @@
"nullable": true,
"example": "Body"
},
+ "knownColumnsListExpression": {
+ "type": "string",
+ "description": "For Distributed table sources whose target tables have non-matching column sets. A list of columns supported across all target tables, used instead of SELECT * when fetching full row data. Leave blank to select all columns.",
+ "nullable": true,
+ "example": "Timestamp, Body, ServiceName"
+ },
"useTextIndexForImplicitColumn": {
"type": "string",
"enum": [
@@ -3104,11 +5254,11 @@
"TraceSource": {
"type": "object",
"required": [
- "id",
"name",
"kind",
"connection",
"from",
+ "defaultTableSelectExpression",
"timestampValueExpression",
"durationExpression",
"durationPrecision",
@@ -3121,7 +5271,8 @@
"properties": {
"id": {
"type": "string",
- "description": "Unique source ID.",
+ "readOnly": true,
+ "description": "Unique source ID. Server-generated; ignored if sent in create/update requests.",
"example": "507f1f77bcf86cd799439021"
},
"name": {
@@ -3129,6 +5280,18 @@
"description": "Display name for the source.",
"example": "Traces"
},
+ "section": {
+ "type": "string",
+ "maxLength": 256,
+ "description": "Optional grouping label used to organize sources in the source selector. Sources that share a section value are displayed together.",
+ "example": "Billing"
+ },
+ "disabled": {
+ "type": "boolean",
+ "description": "When true, the source is hidden from source selectors in the UI. Defaults to false.",
+ "nullable": true,
+ "example": false
+ },
"kind": {
"type": "string",
"enum": [
@@ -3157,7 +5320,6 @@
"defaultTableSelectExpression": {
"type": "string",
"description": "Default columns selected in search results (this can be customized per search later)",
- "nullable": true,
"example": "Timestamp, SpanName, ServiceName, Duration"
},
"timestampValueExpression": {
@@ -3238,6 +5400,17 @@
"nullable": true,
"example": "ServiceName"
},
+ "serviceVersionExpression": {
+ "type": "string",
+ "description": "Expression identifying the running release of a service. Defaults to the OpenTelemetry service.version resource attribute when unset. Where services carry the release on different attributes, fall back across them with coalesce(nullIf(a, ''), nullIf(b, '')).",
+ "nullable": true,
+ "example": "ResourceAttributes['service.version']"
+ },
+ "sampleRateExpression": {
+ "type": "string",
+ "description": "Column or expression for upstream sampling weight (1/N). When set, aggregations (count, avg, sum, quantile) are corrected for sampling. Percentiles use quantileTDigestWeighted, which is an approximation. Leave empty if spans are not sampled.",
+ "example": "SampleRate"
+ },
"resourceAttributesExpression": {
"type": "string",
"description": "Expression to extract resource-level attributes.",
@@ -3256,12 +5429,23 @@
"nullable": true,
"example": "Events"
},
+ "spanLinksValueExpression": {
+ "type": "string",
+ "description": "Expression to extract span links. Used to capture links from a span to spans in other traces. Expected to be Nested ( TraceId String, SpanId String, TraceState String, Attributes Map(LowCardinality(String), String) )",
+ "example": "Links"
+ },
"implicitColumnExpression": {
"type": "string",
"description": "Column used for full text search if no property is specified in a Lucene-based search. Typically the message body of a log.",
"nullable": true,
"example": "SpanName"
},
+ "knownColumnsListExpression": {
+ "type": "string",
+ "description": "For Distributed table sources whose target tables have non-matching column sets. A list of columns supported across all target tables, used instead of SELECT * when fetching full row data. Leave blank to select all columns.",
+ "nullable": true,
+ "example": "Timestamp, Body, ServiceName"
+ },
"useTextIndexForImplicitColumn": {
"type": "string",
"enum": [
@@ -3289,6 +5473,16 @@
},
"nullable": true
},
+ "displayedTimestampValueExpression": {
+ "type": "string",
+ "description": "This DateTime column is used to display and order search results.",
+ "example": "Timestamp"
+ },
+ "orderByExpression": {
+ "type": "string",
+ "description": "Custom ORDER BY expression that overrides the default ordering. Leave empty to use the auto-detected default.",
+ "example": "Timestamp DESC"
+ },
"materializedViews": {
"type": "array",
"description": "Configure materialized views for query optimization. These pre-aggregated views can significantly improve query performance on aggregation queries.",
@@ -3324,7 +5518,6 @@
"MetricSource": {
"type": "object",
"required": [
- "id",
"name",
"kind",
"connection",
@@ -3336,7 +5529,8 @@
"properties": {
"id": {
"type": "string",
- "description": "Unique source ID.",
+ "readOnly": true,
+ "description": "Unique source ID. Server-generated; ignored if sent in create/update requests.",
"example": "507f1f77bcf86cd799439041"
},
"name": {
@@ -3344,10 +5538,22 @@
"description": "Display name for the source.",
"example": "Metrics"
},
- "kind": {
+ "section": {
"type": "string",
- "enum": [
- "metric"
+ "maxLength": 256,
+ "description": "Optional grouping label used to organize sources in the source selector. Sources that share a section value are displayed together.",
+ "example": "Billing"
+ },
+ "disabled": {
+ "type": "boolean",
+ "description": "When true, the source is hidden from source selectors in the UI. Defaults to false.",
+ "nullable": true,
+ "example": false
+ },
+ "kind": {
+ "type": "string",
+ "enum": [
+ "metric"
],
"description": "Source kind discriminator. Must be \"metric\" for metric sources.",
"example": "metric"
@@ -3394,7 +5600,6 @@
"SessionSource": {
"type": "object",
"required": [
- "id",
"name",
"kind",
"connection",
@@ -3404,7 +5609,8 @@
"properties": {
"id": {
"type": "string",
- "description": "Unique source ID.",
+ "readOnly": true,
+ "description": "Unique source ID. Server-generated; ignored if sent in create/update requests.",
"example": "507f1f77bcf86cd799439031"
},
"name": {
@@ -3412,6 +5618,18 @@
"description": "Display name for the source.",
"example": "Sessions"
},
+ "section": {
+ "type": "string",
+ "maxLength": 256,
+ "description": "Optional grouping label used to organize sources in the source selector. Sources that share a section value are displayed together.",
+ "example": "Billing"
+ },
+ "disabled": {
+ "type": "boolean",
+ "description": "When true, the source is hidden from source selectors in the UI. Defaults to false.",
+ "nullable": true,
+ "example": false
+ },
"kind": {
"type": "string",
"enum": [
@@ -3447,6 +5665,77 @@
"type": "string",
"description": "HyperDX Source for traces associated with sessions.",
"example": "507f1f77bcf86cd799439021"
+ },
+ "resourceAttributesExpression": {
+ "type": "string",
+ "description": "Expression to extract resource-level attributes.",
+ "example": "ResourceAttributes"
+ }
+ }
+ },
+ "PromqlSource": {
+ "type": "object",
+ "description": "A source queried with PromQL. It reads either a ClickHouse TimeSeries table (through a ClickHouse connection) or a Prometheus-compatible endpoint (through a connection with isPrometheusEndpoint set to true). PromQL is a preview feature, so creating or updating a PromQL source is rejected unless it is enabled on the deployment (ENABLE_PROMQL).",
+ "required": [
+ "name",
+ "kind",
+ "connection",
+ "from",
+ "timestampValueExpression"
+ ],
+ "properties": {
+ "id": {
+ "type": "string",
+ "readOnly": true,
+ "description": "Unique source ID. Server-generated; ignored if sent in create/update requests.",
+ "example": "507f1f77bcf86cd799439051"
+ },
+ "name": {
+ "type": "string",
+ "description": "Display name for the source.",
+ "example": "Prometheus Metrics"
+ },
+ "section": {
+ "type": "string",
+ "maxLength": 256,
+ "description": "Optional grouping label used to organize sources in the source selector. Sources that share a section value are displayed together.",
+ "example": "Billing"
+ },
+ "disabled": {
+ "type": "boolean",
+ "description": "When true, the source is hidden from source selectors in the UI. Defaults to false.",
+ "nullable": true,
+ "example": false
+ },
+ "kind": {
+ "type": "string",
+ "enum": [
+ "promql"
+ ],
+ "description": "Source kind discriminator. Must be \"promql\" for PromQL sources.",
+ "example": "promql"
+ },
+ "connection": {
+ "type": "string",
+ "description": "ID of the connection used by this source. Either a ClickHouse connection holding the TimeSeries table, or a Prometheus-compatible connection.",
+ "example": "507f1f77bcf86cd799439012"
+ },
+ "from": {
+ "$ref": "#/components/schemas/SourceFrom",
+ "description": "The database and TimeSeries table to query on a ClickHouse connection. Not used for a Prometheus-compatible connection, but still required (empty strings are not accepted — use placeholder values such as \"default\")."
+ },
+ "querySettings": {
+ "type": "array",
+ "description": "Optional ClickHouse query settings applied when querying this source.",
+ "items": {
+ "$ref": "#/components/schemas/QuerySetting"
+ },
+ "nullable": true
+ },
+ "timestampValueExpression": {
+ "type": "string",
+ "description": "Required by the API for all source kinds; not used by PromQL queries.",
+ "example": "timestamp"
}
}
},
@@ -3463,6 +5752,9 @@
},
{
"$ref": "#/components/schemas/SessionSource"
+ },
+ {
+ "$ref": "#/components/schemas/PromqlSource"
}
],
"discriminator": {
@@ -3471,7 +5763,8 @@
"log": "#/components/schemas/LogSource",
"trace": "#/components/schemas/TraceSource",
"metric": "#/components/schemas/MetricSource",
- "session": "#/components/schemas/SessionSource"
+ "session": "#/components/schemas/SessionSource",
+ "promql": "#/components/schemas/PromqlSource"
}
}
},
@@ -3487,6 +5780,15 @@
}
}
},
+ "SourceResponseEnvelope": {
+ "type": "object",
+ "properties": {
+ "data": {
+ "$ref": "#/components/schemas/Source",
+ "description": "The source object."
+ }
+ }
+ },
"SlackWebhook": {
"type": "object",
"required": [
@@ -3523,7 +5825,7 @@
"description": {
"type": "string",
"description": "Webhook description, shown in the UI",
- "example": "Sends critical alerts to the"
+ "example": "Sends critical alerts to the #incidents channel"
},
"updatedAt": {
"type": "string",
@@ -3669,10 +5971,36 @@
}
}
},
+ "PaginationMeta": {
+ "type": "object",
+ "required": [
+ "total",
+ "limit",
+ "offset"
+ ],
+ "properties": {
+ "total": {
+ "type": "integer",
+ "description": "Total number of items matching the query, ignoring pagination.",
+ "example": 142
+ },
+ "limit": {
+ "type": "integer",
+ "description": "Maximum number of items returned in this page.",
+ "example": 50
+ },
+ "offset": {
+ "type": "integer",
+ "description": "Number of items skipped before this page.",
+ "example": 100
+ }
+ }
+ },
"WebhooksListResponse": {
"type": "object",
"required": [
- "data"
+ "data",
+ "meta"
],
"properties": {
"data": {
@@ -3681,6 +6009,86 @@
"items": {
"$ref": "#/components/schemas/Webhook"
}
+ },
+ "meta": {
+ "$ref": "#/components/schemas/PaginationMeta",
+ "description": "Pagination metadata for this result page."
+ }
+ }
+ },
+ "WebhookInput": {
+ "type": "object",
+ "required": [
+ "name",
+ "service",
+ "url"
+ ],
+ "description": "Webhook create/update body. `body`, `headers`, and `queryParams`\nonly take effect for services that issue a templated HTTP request\n(`generic`, `incidentio`). The `slack` service posts a fixed payload\nto its incoming-webhook URL and ignores them, so supplying any of\nthese fields on a `slack` webhook is rejected. `headers` and\n`queryParams` are write-only — they are accepted here but never\nreturned by any read endpoint, so secrets such as auth tokens do\nnot leak. On\nupdate (PUT), omitted readable fields (`description`, `body`) are\ncleared, while omitted `headers`/`queryParams` are preserved —\nsend an explicit `{}` to clear them. Exception: if the destination\n(`url` or `service`) changes, omitted `headers`/`queryParams` are\ncleared rather than preserved, so stored secrets are never forwarded\nto a new destination; re-supply them for the new destination.\n",
+ "properties": {
+ "name": {
+ "type": "string",
+ "maxLength": 1024,
+ "description": "Webhook name. Must be unique per service within the team.",
+ "example": "Production Alerts"
+ },
+ "service": {
+ "type": "string",
+ "enum": [
+ "slack",
+ "incidentio",
+ "generic"
+ ],
+ "description": "Webhook service type.",
+ "example": "slack"
+ },
+ "url": {
+ "type": "string",
+ "format": "uri",
+ "maxLength": 2048,
+ "description": "Webhook destination URL.",
+ "example": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
+ },
+ "description": {
+ "type": "string",
+ "maxLength": 2048,
+ "description": "Webhook description, shown in the UI.",
+ "example": "Sends critical alerts to the #incidents channel"
+ },
+ "body": {
+ "type": "string",
+ "maxLength": 16384,
+ "description": "Optional Handlebars request body template. Only for generic/incidentio; rejected for slack. Rejected with a 400 if it does not compile or calls a helper other than `eq` and the Handlebars built-ins.",
+ "example": "{\"alert\": \"{{title}}\", \"severity\": \"{{level}}\"}"
+ },
+ "headers": {
+ "type": "object",
+ "maxProperties": 100,
+ "description": "Write-only. Custom HTTP headers sent with the webhook request. Never returned on read. Only for generic/incidentio; rejected for slack. Each value is capped at 4096 characters.",
+ "additionalProperties": {
+ "type": "string",
+ "maxLength": 4096
+ },
+ "example": {
+ "Authorization": "Bearer secret-token"
+ }
+ },
+ "queryParams": {
+ "type": "object",
+ "maxProperties": 100,
+ "description": "Write-only. Query parameters appended to the webhook URL. Never returned on read. Only for generic/incidentio; rejected for slack. Each value is capped at 4096 characters.",
+ "additionalProperties": {
+ "type": "string",
+ "maxLength": 4096
+ }
+ }
+ }
+ },
+ "WebhookResponseEnvelope": {
+ "type": "object",
+ "properties": {
+ "data": {
+ "$ref": "#/components/schemas/Webhook",
+ "description": "The webhook object."
}
}
}
@@ -3738,6 +6146,12 @@
"teamId": "65f5e4a3b9e77c001a345678",
"tileId": "65f5e4a3b9e77c001a901234",
"dashboardId": "65f5e4a3b9e77c001a567890",
+ "displayName": "Checkout error spike",
+ "tags": [
+ "checkout",
+ "p1"
+ ],
+ "numConsecutiveWindows": 3,
"createdAt": "2023-03-15T10:20:30.000Z",
"updatedAt": "2023-03-15T14:25:10.000Z"
}
@@ -3757,6 +6171,16 @@
}
}
},
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
"404": {
"description": "Alert not found",
"content": {
@@ -3810,7 +6234,12 @@
"webhookId": "65f5e4a3b9e77c001a789012"
},
"name": "Updated Alert Name",
- "message": "Updated threshold and interval"
+ "message": "Updated threshold and interval",
+ "displayName": "Checkout error spike",
+ "tags": [
+ "checkout",
+ "p1"
+ ]
}
}
}
@@ -3838,6 +6267,16 @@
}
}
},
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
"404": {
"description": "Alert not found",
"content": {
@@ -3901,6 +6340,16 @@
}
}
},
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
"404": {
"description": "Alert not found",
"content": {
@@ -3917,11 +6366,36 @@
"/api/v2/alerts": {
"get": {
"summary": "List Alerts",
- "description": "Retrieves a list of all alerts for the authenticated team",
+ "description": "Retrieves alerts for the authenticated team (paginated). Results are capped at `limit` (default and maximum 1000). When more records exist than are returned, `meta.total` exceeds `data.length`; clients with large collections must page with `limit`/`offset` to retrieve them all.",
"operationId": "listAlerts",
"tags": [
"Alerts"
],
+ "parameters": [
+ {
+ "name": "limit",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "integer",
+ "minimum": 1,
+ "maximum": 1000,
+ "default": 1000
+ },
+ "description": "Maximum number of alerts to return."
+ },
+ {
+ "name": "offset",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "integer",
+ "minimum": 0,
+ "default": 0
+ },
+ "description": "Number of alerts to skip before returning results."
+ }
+ ],
"responses": {
"200": {
"description": "Successfully retrieved alerts",
@@ -3949,10 +6423,20 @@
"teamId": "65f5e4a3b9e77c001a345678",
"tileId": "65f5e4a3b9e77c001a901234",
"dashboardId": "65f5e4a3b9e77c001a567890",
+ "displayName": "Checkout error spike",
+ "tags": [
+ "checkout",
+ "p1"
+ ],
"createdAt": "2023-01-01T00:00:00.000Z",
"updatedAt": "2023-01-01T00:00:00.000Z"
}
- ]
+ ],
+ "meta": {
+ "total": 1,
+ "limit": 1000,
+ "offset": 0
+ }
}
}
}
@@ -3971,6 +6455,16 @@
}
}
}
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
}
}
},
@@ -4003,7 +6497,59 @@
"webhookId": "65f5e4a3b9e77c001a789012"
},
"name": "Error Spike Alert",
- "message": "Error rate has exceeded 100 in the last hour"
+ "message": "Error rate has exceeded 100 in the last hour",
+ "displayName": "Checkout error spike",
+ "tags": [
+ "checkout",
+ "p1"
+ ],
+ "numConsecutiveWindows": 3
+ }
+ },
+ "multiChannelAlert": {
+ "summary": "Create an alert that notifies several webhooks",
+ "value": {
+ "savedSearchId": "65f5e4a3b9e77c001a345678",
+ "threshold": 10,
+ "interval": "5m",
+ "source": "saved_search",
+ "thresholdType": "above",
+ "channels": [
+ {
+ "type": "webhook",
+ "webhookId": "65f5e4a3b9e77c001a789012"
+ },
+ {
+ "type": "webhook",
+ "webhookId": "65f5e4a3b9e77c001a789013"
+ }
+ ],
+ "name": "Error Spike Alert"
+ }
+ },
+ "inlineAlert": {
+ "summary": "Create an inline alert that carries its own chart config",
+ "value": {
+ "source": "inline",
+ "chartConfig": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a123456",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "level:error",
+ "whereLanguage": "lucene"
+ }
+ ]
+ },
+ "threshold": 100,
+ "interval": "5m",
+ "thresholdType": "above",
+ "channel": {
+ "type": "webhook",
+ "webhookId": "65f5e4a3b9e77c001a789012"
+ },
+ "name": "Error log spike"
}
}
}
@@ -4031,6 +6577,16 @@
}
}
},
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
"500": {
"description": "Server error or validation failure",
"content": {
@@ -4293,279 +6849,74 @@
}
}
},
- "/api/v2/dashboards": {
+ "/api/v2/connections": {
"get": {
- "summary": "List Dashboards",
- "description": "Retrieves a list of all dashboards for the authenticated team",
- "operationId": "listDashboards",
+ "summary": "List Connections",
+ "description": "Retrieves a list of all ClickHouse connections for the authenticated team. Passwords are never returned.",
+ "operationId": "listConnections",
"tags": [
- "Dashboards"
+ "Connections"
],
"responses": {
"200": {
- "description": "Successfully retrieved dashboards",
+ "description": "Successfully retrieved connections",
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/DashboardsListResponse"
+ "$ref": "#/components/schemas/ConnectionsListResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
},
- "examples": {
- "dashboards": {
- "summary": "Dashboards list response",
- "value": {
- "data": [
- {
- "id": "65f5e4a3b9e77c001a567890",
- "name": "Infrastructure Monitoring",
- "tiles": [
- {
- "id": "65f5e4a3b9e77c001a901234",
- "name": "Server CPU",
- "x": 0,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "line",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "avg",
- "valueExpression": "cpu.usage",
- "where": "host:server-01"
- }
- ]
- }
- }
- ],
- "tags": [
- "infrastructure",
- "monitoring"
- ],
- "filters": [
- {
- "id": "65f5e4a3b9e77c001a301001",
- "type": "QUERY_EXPRESSION",
- "name": "Environment",
- "expression": "environment",
- "sourceId": "65f5e4a3b9e77c001a111111"
- }
- ]
- },
- {
- "id": "65f5e4a3b9e77c001a567891",
- "name": "API Monitoring",
- "tiles": [
- {
- "id": "65f5e4a3b9e77c001a901235",
- "name": "API Errors",
- "x": 0,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "table",
- "sourceId": "65f5e4a3b9e77c001a111112",
- "select": [
- {
- "aggFn": "count",
- "where": "level:error"
- }
- ],
- "groupBy": "service",
- "orderBy": "count DESC"
- }
- }
- ],
- "tags": [
- "api",
- "monitoring"
- ],
- "filters": [
- {
- "id": "65f5e4a3b9e77c001a301002",
- "type": "QUERY_EXPRESSION",
- "name": "Service",
- "expression": "service_name",
- "sourceId": "65f5e4a3b9e77c001a111112"
- }
- ]
- }
- ]
- }
- }
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
}
}
}
},
- "401": {
- "description": "Unauthorized"
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
}
}
},
"post": {
- "summary": "Create Dashboard",
- "description": "Creates a new dashboard",
- "operationId": "createDashboard",
+ "summary": "Create Connection",
+ "description": "Creates a new ClickHouse connection",
+ "operationId": "createConnection",
"tags": [
- "Dashboards"
+ "Connections"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/CreateDashboardRequest"
- },
- "examples": {
- "simpleTimeSeriesDashboard": {
- "summary": "Dashboard with a line chart",
- "value": {
- "name": "API Monitoring Dashboard",
- "tiles": [
- {
- "name": "API Request Volume",
- "x": 0,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "line",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "count",
- "where": "service:api"
- }
- ]
- }
- }
- ],
- "tags": [
- "api",
- "monitoring"
- ],
- "filters": [
- {
- "type": "QUERY_EXPRESSION",
- "name": "Environment",
- "expression": "environment",
- "sourceId": "65f5e4a3b9e77c001a111111"
- }
- ]
- }
- },
- "complexDashboard": {
- "summary": "Dashboard with multiple chart types",
- "value": {
- "name": "Service Health Overview",
- "tiles": [
- {
- "name": "Request Count",
- "x": 0,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "line",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "count",
- "where": "service:backend"
- }
- ]
- }
- },
- {
- "name": "Error Distribution",
- "x": 6,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "table",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "count",
- "where": "level:error"
- }
- ],
- "groupBy": "errorType",
- "orderBy": "count DESC"
- }
- }
- ],
- "tags": [
- "service-health",
- "production"
- ],
- "filters": [
- {
- "type": "QUERY_EXPRESSION",
- "name": "Service",
- "expression": "service_name",
- "sourceId": "65f5e4a3b9e77c001a111111"
- }
- ]
- }
- }
+ "$ref": "#/components/schemas/CreateConnectionRequest"
}
}
}
},
"responses": {
"200": {
- "description": "Successfully created dashboard",
+ "description": "Successfully created connection",
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/DashboardResponseEnvelope"
- },
- "examples": {
- "createdDashboard": {
- "summary": "Created dashboard response",
- "value": {
- "data": {
- "id": "65f5e4a3b9e77c001a567890",
- "name": "API Monitoring Dashboard",
- "tiles": [
- {
- "id": "65f5e4a3b9e77c001a901234",
- "name": "API Request Volume",
- "x": 0,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "line",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "count",
- "where": "service:api"
- }
- ]
- }
- }
- ],
- "tags": [
- "api",
- "monitoring"
- ],
- "filters": [
- {
- "id": "65f5e4a3b9e77c001a301004",
- "type": "QUERY_EXPRESSION",
- "name": "Environment",
- "expression": "environment",
- "sourceId": "65f5e4a3b9e77c001a111111"
- }
- ]
- }
- }
- }
+ "$ref": "#/components/schemas/ConnectionResponseEnvelope"
}
}
}
@@ -4578,7 +6929,7 @@
"$ref": "#/components/schemas/Error"
},
"example": {
- "message": "Could not find the following source IDs: 68fa86308aa879b977aa6af6"
+ "message": "Body validation failed: name: Required"
}
}
}
@@ -4595,114 +6946,2025 @@
}
}
}
- },
- "500": {
- "description": "Server error or validation failure",
- "content": {
- "application/json": {
- "schema": {
- "$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Dashboard validation failed: name is required"
- }
- }
- }
}
}
}
},
- "/api/v2/dashboards/{id}": {
+ "/api/v2/connections/{id}": {
+ "get": {
+ "summary": "Get Connection",
+ "description": "Retrieves a specific ClickHouse connection by ID. Passwords are never returned.",
+ "operationId": "getConnection",
+ "tags": [
+ "Connections"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Connection ID",
+ "example": "507f1f77bcf86cd799439012"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved connection",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ConnectionResponseEnvelope"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Connection not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Connection not found"
+ }
+ }
+ }
+ }
+ }
+ },
+ "put": {
+ "summary": "Update Connection",
+ "description": "Updates an existing ClickHouse connection.\n\nField semantics: if `password` is omitted or empty the existing\npassword is kept. `hyperdxSettingPrefix` is cleared when set to null\nor an empty string, and `prometheusEndpoint` is cleared when set to\nnull; both are kept unchanged when omitted.\n",
+ "operationId": "updateConnection",
+ "tags": [
+ "Connections"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Connection ID",
+ "example": "507f1f77bcf86cd799439012"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/UpdateConnectionRequest"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successfully updated connection",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/ConnectionResponseEnvelope"
+ }
+ }
+ }
+ },
+ "400": {
+ "description": "Bad request",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Body validation failed: host: Required"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Connection not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Connection not found"
+ }
+ }
+ }
+ }
+ }
+ },
+ "delete": {
+ "summary": "Delete Connection",
+ "description": "Deletes a ClickHouse connection",
+ "operationId": "deleteConnection",
+ "tags": [
+ "Connections"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Connection ID",
+ "example": "507f1f77bcf86cd799439012"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully deleted connection",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/EmptyResponse"
+ },
+ "example": {}
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Connection not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Connection not found"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/dashboards": {
+ "get": {
+ "summary": "List Dashboards",
+ "description": "Retrieves a list of all dashboards for the authenticated team",
+ "operationId": "listDashboards",
+ "tags": [
+ "Dashboards"
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved dashboards",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/DashboardsListResponse"
+ },
+ "examples": {
+ "dashboards": {
+ "summary": "Dashboards list response",
+ "value": {
+ "data": [
+ {
+ "id": "65f5e4a3b9e77c001a567890",
+ "name": "Infrastructure Monitoring",
+ "tiles": [
+ {
+ "id": "65f5e4a3b9e77c001a901234",
+ "name": "Server CPU",
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "avg",
+ "valueExpression": "cpu.usage",
+ "where": "host:server-01"
+ }
+ ]
+ }
+ }
+ ],
+ "tags": [
+ "infrastructure",
+ "monitoring"
+ ],
+ "filters": [
+ {
+ "id": "65f5e4a3b9e77c001a301001",
+ "type": "QUERY_EXPRESSION",
+ "name": "Environment",
+ "expression": "environment",
+ "sourceId": "65f5e4a3b9e77c001a111111"
+ }
+ ]
+ },
+ {
+ "id": "65f5e4a3b9e77c001a567891",
+ "name": "API Monitoring",
+ "tiles": [
+ {
+ "id": "65f5e4a3b9e77c001a901235",
+ "name": "API Errors",
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "table",
+ "sourceId": "65f5e4a3b9e77c001a111112",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "level:error"
+ }
+ ],
+ "groupBy": "service",
+ "orderBy": "count DESC"
+ }
+ }
+ ],
+ "tags": [
+ "api",
+ "monitoring"
+ ],
+ "filters": [
+ {
+ "id": "65f5e4a3b9e77c001a301002",
+ "type": "QUERY_EXPRESSION",
+ "name": "Service",
+ "expression": "service_name",
+ "sourceId": "65f5e4a3b9e77c001a111112"
+ }
+ ]
+ }
+ ]
+ }
+ }
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized"
+ }
+ }
+ },
+ "post": {
+ "summary": "Create Dashboard",
+ "description": "Creates a new dashboard",
+ "operationId": "createDashboard",
+ "tags": [
+ "Dashboards"
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/CreateDashboardRequest"
+ },
+ "examples": {
+ "simpleTimeSeriesDashboard": {
+ "summary": "Dashboard with a line chart",
+ "value": {
+ "name": "API Monitoring Dashboard",
+ "tiles": [
+ {
+ "name": "API Request Volume",
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "service:api"
+ }
+ ]
+ }
+ }
+ ],
+ "tags": [
+ "api",
+ "monitoring"
+ ],
+ "filters": [
+ {
+ "type": "QUERY_EXPRESSION",
+ "name": "Environment",
+ "expression": "environment",
+ "sourceId": "65f5e4a3b9e77c001a111111"
+ }
+ ]
+ }
+ },
+ "complexDashboard": {
+ "summary": "Dashboard with multiple chart types",
+ "value": {
+ "name": "Service Health Overview",
+ "tiles": [
+ {
+ "name": "Request Count",
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "service:backend"
+ }
+ ]
+ }
+ },
+ {
+ "name": "Error Distribution",
+ "x": 6,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "table",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "level:error"
+ }
+ ],
+ "groupBy": "errorType",
+ "orderBy": "count DESC"
+ }
+ }
+ ],
+ "tags": [
+ "service-health",
+ "production"
+ ],
+ "filters": [
+ {
+ "type": "QUERY_EXPRESSION",
+ "name": "Service",
+ "expression": "service_name",
+ "sourceId": "65f5e4a3b9e77c001a111111"
+ },
+ {
+ "type": "STATIC_LIST",
+ "name": "Environment",
+ "options": [
+ "prod",
+ "staging",
+ "dev"
+ ],
+ "isBroadcastEnabled": false,
+ "isVariableEnabled": true,
+ "variableName": "env"
+ }
+ ]
+ }
+ }
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successfully created dashboard",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/DashboardResponseEnvelope"
+ },
+ "examples": {
+ "createdDashboard": {
+ "summary": "Created dashboard response",
+ "value": {
+ "data": {
+ "id": "65f5e4a3b9e77c001a567890",
+ "name": "API Monitoring Dashboard",
+ "tiles": [
+ {
+ "id": "65f5e4a3b9e77c001a901234",
+ "name": "API Request Volume",
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "service:api"
+ }
+ ]
+ }
+ }
+ ],
+ "tags": [
+ "api",
+ "monitoring"
+ ],
+ "filters": [
+ {
+ "id": "65f5e4a3b9e77c001a301004",
+ "type": "QUERY_EXPRESSION",
+ "name": "Environment",
+ "expression": "environment",
+ "sourceId": "65f5e4a3b9e77c001a111111"
+ }
+ ]
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "400": {
+ "description": "Bad request",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Could not find the following source IDs: 68fa86308aa879b977aa6af6"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "500": {
+ "description": "Server error or validation failure",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Dashboard validation failed: name is required"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/dashboards/{id}": {
+ "get": {
+ "summary": "Get Dashboard",
+ "description": "Retrieves a specific dashboard by ID",
+ "operationId": "getDashboard",
+ "tags": [
+ "Dashboards"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Dashboard ID",
+ "example": "65f5e4a3b9e77c001a567890"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved dashboard",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/DashboardResponseEnvelope"
+ },
+ "examples": {
+ "dashboard": {
+ "summary": "Single dashboard response",
+ "value": {
+ "data": {
+ "id": "65f5e4a3b9e77c001a567890",
+ "name": "Infrastructure Monitoring",
+ "tiles": [
+ {
+ "id": "65f5e4a3b9e77c001a901234",
+ "name": "Server CPU",
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "avg",
+ "valueExpression": "cpu.usage",
+ "where": "host:server-01"
+ }
+ ]
+ }
+ },
+ {
+ "id": "65f5e4a3b9e77c001a901235",
+ "name": "Memory Usage",
+ "x": 6,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "avg",
+ "valueExpression": "memory.usage",
+ "where": "host:server-01"
+ }
+ ]
+ }
+ }
+ ],
+ "tags": [
+ "infrastructure",
+ "monitoring"
+ ],
+ "filters": [
+ {
+ "id": "65f5e4a3b9e77c001a301003",
+ "type": "QUERY_EXPRESSION",
+ "name": "Environment",
+ "expression": "environment",
+ "sourceId": "65f5e4a3b9e77c001a111111"
+ }
+ ]
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Dashboard not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Dashboard not found"
+ }
+ }
+ }
+ }
+ }
+ },
+ "put": {
+ "summary": "Update Dashboard",
+ "description": "Updates an existing dashboard.\n\n**Concurrency:** This endpoint does not support optimistic\nconcurrency control. Concurrent PUT requests for the same\ndashboard may silently overwrite each other, which can leave\norphan tile-to-container references on layout-shape edits.\nClients should serialize edits to a given dashboard.\n",
+ "operationId": "updateDashboard",
+ "tags": [
+ "Dashboards"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Dashboard ID",
+ "example": "65f5e4a3b9e77c001a567890"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/UpdateDashboardRequest"
+ },
+ "examples": {
+ "updateDashboard": {
+ "summary": "Update dashboard properties and tiles",
+ "value": {
+ "name": "Updated Dashboard Name",
+ "tiles": [
+ {
+ "id": "65f5e4a3b9e77c001a901234",
+ "name": "Updated Line Chart",
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "level:error"
+ }
+ ]
+ }
+ },
+ {
+ "id": "new-tile-123",
+ "name": "New Number Chart",
+ "x": 6,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "number",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "level:info"
+ }
+ ]
+ }
+ }
+ ],
+ "tags": [
+ "production",
+ "updated"
+ ],
+ "filters": [
+ {
+ "id": "65f5e4a3b9e77c001a301005",
+ "type": "QUERY_EXPRESSION",
+ "name": "Environment",
+ "expression": "environment",
+ "sourceId": "65f5e4a3b9e77c001a111111"
+ }
+ ]
+ }
+ }
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successfully updated dashboard",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/DashboardResponseEnvelope"
+ },
+ "examples": {
+ "updatedDashboard": {
+ "summary": "Updated dashboard response",
+ "value": {
+ "data": {
+ "id": "65f5e4a3b9e77c001a567890",
+ "name": "Updated Dashboard Name",
+ "tiles": [
+ {
+ "id": "65f5e4a3b9e77c001a901234",
+ "name": "Updated Line Chart",
+ "x": 0,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "line",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "level:error"
+ }
+ ]
+ }
+ },
+ {
+ "id": "new-tile-123",
+ "name": "New Number Chart",
+ "x": 6,
+ "y": 0,
+ "w": 6,
+ "h": 3,
+ "config": {
+ "displayType": "number",
+ "sourceId": "65f5e4a3b9e77c001a111111",
+ "select": [
+ {
+ "aggFn": "count",
+ "where": "level:info"
+ }
+ ]
+ }
+ }
+ ],
+ "tags": [
+ "production",
+ "updated"
+ ],
+ "filters": [
+ {
+ "id": "65f5e4a3b9e77c001a301005",
+ "type": "QUERY_EXPRESSION",
+ "name": "Environment",
+ "expression": "environment",
+ "sourceId": "65f5e4a3b9e77c001a111111"
+ }
+ ]
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "400": {
+ "description": "Bad request",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Could not find the following source IDs: 68fa86308aa879b977aa6af6"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Dashboard not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Dashboard not found"
+ }
+ }
+ }
+ },
+ "500": {
+ "description": "Server error or validation failure",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Invalid dashboard configuration"
+ }
+ }
+ }
+ }
+ }
+ },
+ "delete": {
+ "summary": "Delete Dashboard",
+ "description": "Deletes a dashboard",
+ "operationId": "deleteDashboard",
+ "tags": [
+ "Dashboards"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Dashboard ID",
+ "example": "65f5e4a3b9e77c001a567890"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully deleted dashboard",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/EmptyResponse"
+ },
+ "example": {}
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Dashboard not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Dashboard not found"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/dashboards/validate": {
+ "post": {
+ "summary": "Validate Dashboard",
+ "description": "Validates a dashboard body against the same schema and tile rules used\nby POST /api/v2/dashboards. The dashboard is **never persisted**. Use\nthis endpoint at plan time (e.g. from a Terraform provider) to check\nthat a dashboard configuration is valid before applying it.\n",
+ "operationId": "validateDashboard",
+ "tags": [
+ "Dashboards"
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/CreateDashboardRequest"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Validation result. HTTP 200 is always returned for valid **and**\ninvalid bodies — a non-200 response means the request itself\nfailed (auth, server error, etc.).\n",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "required": [
+ "valid",
+ "errors",
+ "normalized"
+ ],
+ "properties": {
+ "valid": {
+ "type": "boolean",
+ "description": "True when the body passes all validation rules."
+ },
+ "errors": {
+ "type": "array",
+ "description": "Validation errors. Empty when valid is true.",
+ "items": {
+ "type": "object",
+ "required": [
+ "path",
+ "message"
+ ],
+ "properties": {
+ "path": {
+ "type": "string",
+ "description": "Dot-separated field path, or empty string for top-level errors."
+ },
+ "message": {
+ "type": "string",
+ "description": "Human-readable error description."
+ }
+ }
+ }
+ },
+ "normalized": {
+ "type": "object",
+ "nullable": true,
+ "description": "The parsed dashboard body with defaults applied (no\npersistence, so no server-assigned tile IDs). Populated\nwhen valid is true, null when valid is false.\n"
+ }
+ }
+ },
+ "examples": {
+ "valid": {
+ "summary": "Valid dashboard body",
+ "value": {
+ "valid": true,
+ "errors": [],
+ "normalized": {
+ "name": "My Dashboard",
+ "tiles": []
+ }
+ }
+ },
+ "invalid": {
+ "summary": "Invalid dashboard body",
+ "value": {
+ "valid": false,
+ "errors": [
+ {
+ "path": "name",
+ "message": "Required"
+ }
+ ],
+ "normalized": null
+ }
+ }
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/saved-searches": {
+ "get": {
+ "summary": "List Saved Searches",
+ "description": "Retrieves saved searches for the authenticated team (paginated). Results are capped at `limit` (default and maximum 1000). When more records exist than are returned, `meta.total` exceeds `data.length`; clients with large collections must page with `limit`/`offset` to retrieve them all.",
+ "operationId": "listSavedSearches",
+ "tags": [
+ "Saved Searches"
+ ],
+ "parameters": [
+ {
+ "name": "limit",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "integer",
+ "minimum": 1,
+ "maximum": 1000,
+ "default": 1000
+ },
+ "description": "Maximum number of saved searches to return."
+ },
+ {
+ "name": "offset",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "integer",
+ "minimum": 0,
+ "default": 0
+ },
+ "description": "Number of saved searches to skip before returning results."
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved saved searches",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SavedSearchesListResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ }
+ }
+ },
+ "post": {
+ "summary": "Create Saved Search",
+ "description": "Creates a new saved search.",
+ "operationId": "createSavedSearch",
+ "tags": [
+ "Saved Searches"
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SavedSearchInput"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successfully created saved search",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SavedSearchResponseEnvelope"
+ }
+ }
+ }
+ },
+ "400": {
+ "description": "Bad request",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/saved-searches/{id}": {
+ "get": {
+ "summary": "Get Saved Search",
+ "description": "Retrieves a specific saved search by ID.",
+ "operationId": "getSavedSearch",
+ "tags": [
+ "Saved Searches"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Saved search ID",
+ "example": "507f1f77bcf86cd799439011"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved saved search",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SavedSearchResponseEnvelope"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Saved search not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ }
+ }
+ },
+ "put": {
+ "summary": "Update Saved Search",
+ "description": "Updates an existing saved search. This is a full replace: send the\nfull object. Every optional field (`select`, `where`, `whereLanguage`,\n`orderBy`, `tags`, `filters`) is always written and falls back to its\ndefault when omitted, so omitting a field resets it rather than\npreserving the stored value.\n",
+ "operationId": "updateSavedSearch",
+ "tags": [
+ "Saved Searches"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Saved search ID",
+ "example": "507f1f77bcf86cd799439011"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SavedSearchInput"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successfully updated saved search",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SavedSearchResponseEnvelope"
+ }
+ }
+ }
+ },
+ "400": {
+ "description": "Bad request",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Saved search not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ }
+ }
+ },
+ "delete": {
+ "summary": "Delete Saved Search",
+ "description": "Deletes a saved search and any alerts attached to it.",
+ "operationId": "deleteSavedSearch",
+ "tags": [
+ "Saved Searches"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Saved search ID",
+ "example": "507f1f77bcf86cd799439011"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully deleted saved search",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/EmptyResponse"
+ },
+ "example": {}
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Saved search not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/search": {
+ "post": {
+ "summary": "Search Raw Logs and Traces",
+ "description": "Fetch individual log or trace rows from a HyperDX source.\n\nThis endpoint mirrors the \"search\" panel mode in the HyperDX UI.\nHyperDX applies the same query optimizations used in the UI:\n - Named attribute columns (e.g. \"pipedream.pipeline_name\") are\n rewritten to their indexed materialized equivalents when the\n source schema exposes them, avoiding slow Map lookups.\n - Rows are ordered by timestamp descending (most recent first).\n - The source's built-in PREWHERE / partition pruning is applied.\n\nAuthentication: Bearer token (personal API key from Team Settings).\n",
+ "operationId": "searchEvents",
+ "tags": [
+ "Search"
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SearchRequest"
+ },
+ "examples": {
+ "recentErrors": {
+ "summary": "Recent errors for a service",
+ "value": {
+ "sourceId": "69b46cb0d964ce2d0b9506a8",
+ "startTime": "2026-05-10T00:00:00Z",
+ "endTime": "2026-05-10T01:00:00Z",
+ "where": "SeverityText:ERROR",
+ "select": "Timestamp,SeverityText,Body,ServiceName",
+ "maxResults": 50
+ }
+ },
+ "pipedreamTaskLogs": {
+ "summary": "Pipedream task logs with materialized column filter",
+ "value": {
+ "sourceId": "69b46cb0d964ce2d0b9506a8",
+ "startTime": "2026-05-10T00:00:00Z",
+ "endTime": "2026-05-10T01:00:00Z",
+ "where": "pipedream.pipeline_name:my-pipeline",
+ "select": "Timestamp,SeverityText,Body,pipedream.pipeline_name,pipedream.stage_name,pipedream.task_index",
+ "maxResults": 200
+ }
+ },
+ "traceSearch": {
+ "summary": "Slow spans for a service",
+ "value": {
+ "sourceId": "69b46cb0d964ce2d0b9508b2",
+ "startTime": "2026-05-10T00:00:00Z",
+ "endTime": "2026-05-10T01:00:00Z",
+ "where": "ServiceName:my-service AND Duration:>1000000000",
+ "select": "Timestamp,TraceId,SpanId,SpanName,ServiceName,Duration,StatusCode",
+ "maxResults": 100
+ }
+ },
+ "paginated": {
+ "summary": "Paginated fetch",
+ "value": {
+ "sourceId": "69b46cb0d964ce2d0b9506a8",
+ "startTime": "2026-05-10T00:00:00Z",
+ "endTime": "2026-05-10T01:00:00Z",
+ "maxResults": 100,
+ "offset": 100
+ }
+ }
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Matching rows returned successfully",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SearchResponse"
+ },
+ "example": {
+ "data": [
+ {
+ "Timestamp": "2026-05-10T00:01:23.456789000Z",
+ "SeverityText": "ERROR",
+ "Body": "connection refused: redis:6379",
+ "ServiceName": "api-service"
+ }
+ ],
+ "rows": 1
+ }
+ }
+ }
+ },
+ "400": {
+ "description": "Invalid request parameters or query error",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "properties": {
+ "message": {
+ "type": "string"
+ }
+ }
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Missing or invalid API key"
+ },
+ "404": {
+ "description": "Source or connection not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "properties": {
+ "message": {
+ "type": "string"
+ }
+ }
+ }
+ }
+ }
+ },
+ "500": {
+ "description": "Server error",
+ "content": {
+ "application/json": {
+ "schema": {
+ "type": "object",
+ "properties": {
+ "message": {
+ "type": "string"
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/sources": {
+ "get": {
+ "summary": "List Sources",
+ "description": "Retrieves a list of all sources for the authenticated team",
+ "operationId": "listSources",
+ "tags": [
+ "Sources"
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved sources",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SourcesListResponse"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ }
+ }
+ },
+ "post": {
+ "summary": "Create Source",
+ "description": "Creates a new source.\n\nThe request body is a source object without the `id` field. If an\n`id` is sent anyway it is silently ignored (stripped before\nvalidation — the request is never rejected because of it).\nGranularity fields\n(`materializedViews[].minGranularity` and\n`metadataMaterializedViews.granularity`) accept the same short format\nthe API returns (e.g. `5m`, `15s`, `1h`, `1d`).\n",
+ "operationId": "createSource",
+ "tags": [
+ "Sources"
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Source"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successfully created source",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SourceResponseEnvelope"
+ }
+ }
+ }
+ },
+ "400": {
+ "description": "Bad request",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Body validation failed: name: Required"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/sources/{id}": {
+ "get": {
+ "summary": "Get Source",
+ "description": "Retrieves a specific source by ID",
+ "operationId": "getSource",
+ "tags": [
+ "Sources"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Source ID",
+ "example": "507f1f77bcf86cd799439011"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved source",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SourceResponseEnvelope"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Source not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Source not found"
+ }
+ }
+ }
+ }
+ }
+ },
+ "put": {
+ "summary": "Update Source",
+ "description": "Updates an existing source. The full source object must be provided;\nthis is a replace, not a patch.\n\nThe request body is a source object without the `id` field. If an\n`id` is sent anyway it is silently ignored (stripped before\nvalidation — never a 400); the path parameter alone identifies the\nsource. Granularity fields (`materializedViews[].minGranularity` and\n`metadataMaterializedViews.granularity`) accept the same short format\nthe API returns (e.g. `5m`, `15s`, `1h`, `1d`).\n",
+ "operationId": "updateSource",
+ "tags": [
+ "Sources"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Source ID",
+ "example": "507f1f77bcf86cd799439011"
+ }
+ ],
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Source"
+ }
+ }
+ }
+ },
+ "responses": {
+ "200": {
+ "description": "Successfully updated source",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/SourceResponseEnvelope"
+ }
+ }
+ }
+ },
+ "400": {
+ "description": "Bad request",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Body validation failed: name: Required"
+ }
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Source not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Source not found"
+ }
+ }
+ }
+ }
+ }
+ },
+ "delete": {
+ "summary": "Delete Source",
+ "description": "Deletes a source",
+ "operationId": "deleteSource",
+ "tags": [
+ "Sources"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Source ID",
+ "example": "507f1f77bcf86cd799439011"
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully deleted source",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/EmptyResponse"
+ },
+ "example": {}
+ }
+ }
+ },
+ "401": {
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Unauthorized access. API key is missing or invalid."
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "404": {
+ "description": "Source not found",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ },
+ "example": {
+ "message": "Source not found"
+ }
+ }
+ }
+ }
+ }
+ }
+ },
+ "/api/v2/team": {
+ "get": {
+ "summary": "Get Team",
+ "description": "Retrieves the authenticated team's basic settings.",
+ "operationId": "getTeam",
+ "tags": [
+ "Team"
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved team"
+ },
+ "403": {
+ "description": "Forbidden"
+ }
+ }
+ }
+ },
+ "/api/v2/team/members": {
+ "get": {
+ "summary": "List Team Members",
+ "description": "Retrieves the team's members.",
+ "operationId": "listTeamMembers",
+ "tags": [
+ "Team"
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved members"
+ }
+ }
+ }
+ },
+ "/api/v2/team/invitation": {
+ "post": {
+ "summary": "Invite Team Member",
+ "description": "Creates a pending invitation for a new email address.",
+ "operationId": "inviteTeamMember",
+ "tags": [
+ "Team"
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully created the invitation"
+ },
+ "400": {
+ "description": "User already exists"
+ }
+ }
+ }
+ },
+ "/api/v2/team/invitations": {
+ "get": {
+ "summary": "List Team Invitations",
+ "description": "Retrieves the team's pending invitations.",
+ "operationId": "listTeamInvitations",
+ "tags": [
+ "Team"
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully retrieved invitations"
+ }
+ }
+ }
+ },
+ "/api/v2/team/invitation/{id}": {
+ "delete": {
+ "summary": "Delete Team Invitation",
+ "description": "Deletes a pending team invitation.",
+ "operationId": "deleteTeamInvitation",
+ "tags": [
+ "Team"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully deleted the invitation"
+ },
+ "404": {
+ "description": "Invitation not found"
+ }
+ }
+ }
+ },
+ "/api/v2/team/member/{id}": {
+ "delete": {
+ "summary": "Remove Team Member",
+ "description": "Removes a member from the team.",
+ "operationId": "removeTeamMember",
+ "tags": [
+ "Team"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successfully removed the member"
+ },
+ "400": {
+ "description": "Cannot remove yourself from the team"
+ },
+ "404": {
+ "description": "Member not found"
+ }
+ }
+ }
+ },
+ "/api/v2/webhooks": {
"get": {
- "summary": "Get Dashboard",
- "description": "Retrieves a specific dashboard by ID",
- "operationId": "getDashboard",
+ "summary": "List Webhooks",
+ "description": "Retrieves webhooks for the authenticated team (paginated). Results are capped at `limit` (default and maximum 1000). When more records exist than are returned, `meta.total` exceeds `data.length`; clients with large collections must page with `limit`/`offset` to retrieve them all.",
+ "operationId": "listWebhooks",
"tags": [
- "Dashboards"
+ "Webhooks"
],
"parameters": [
{
- "name": "id",
- "in": "path",
- "required": true,
+ "name": "limit",
+ "in": "query",
+ "required": false,
"schema": {
- "type": "string"
+ "type": "integer",
+ "minimum": 1,
+ "maximum": 1000,
+ "default": 1000
},
- "description": "Dashboard ID",
- "example": "65f5e4a3b9e77c001a567890"
+ "description": "Maximum number of webhooks to return."
+ },
+ {
+ "name": "offset",
+ "in": "query",
+ "required": false,
+ "schema": {
+ "type": "integer",
+ "minimum": 0,
+ "default": 0
+ },
+ "description": "Number of webhooks to skip before returning results."
}
],
"responses": {
"200": {
- "description": "Successfully retrieved dashboard",
+ "description": "Successfully retrieved webhooks",
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/DashboardResponseEnvelope"
- },
- "examples": {
- "dashboard": {
- "summary": "Single dashboard response",
- "value": {
- "data": {
- "id": "65f5e4a3b9e77c001a567890",
- "name": "Infrastructure Monitoring",
- "tiles": [
- {
- "id": "65f5e4a3b9e77c001a901234",
- "name": "Server CPU",
- "x": 0,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "line",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "avg",
- "valueExpression": "cpu.usage",
- "where": "host:server-01"
- }
- ]
- }
- },
- {
- "id": "65f5e4a3b9e77c001a901235",
- "name": "Memory Usage",
- "x": 6,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "line",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "avg",
- "valueExpression": "memory.usage",
- "where": "host:server-01"
- }
- ]
- }
- }
- ],
- "tags": [
- "infrastructure",
- "monitoring"
- ],
- "filters": [
- {
- "id": "65f5e4a3b9e77c001a301003",
- "type": "QUERY_EXPRESSION",
- "name": "Environment",
- "expression": "environment",
- "sourceId": "65f5e4a3b9e77c001a111111"
- }
- ]
- }
- }
- }
+ "$ref": "#/components/schemas/WebhooksListResponse"
}
}
}
@@ -4720,191 +8982,52 @@
}
}
},
- "404": {
- "description": "Dashboard not found",
+ "403": {
+ "description": "Forbidden",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Dashboard not found"
}
}
}
}
}
},
- "put": {
- "summary": "Update Dashboard",
- "description": "Updates an existing dashboard.\n\n**Concurrency:** This endpoint does not support optimistic\nconcurrency control. Concurrent PUT requests for the same\ndashboard may silently overwrite each other, which can leave\norphan tile-to-container references on layout-shape edits.\nClients should serialize edits to a given dashboard.\n",
- "operationId": "updateDashboard",
+ "post": {
+ "summary": "Create Webhook",
+ "description": "Creates a new webhook for the authenticated team.",
+ "operationId": "createWebhook",
"tags": [
- "Dashboards"
- ],
- "parameters": [
- {
- "name": "id",
- "in": "path",
- "required": true,
- "schema": {
- "type": "string"
- },
- "description": "Dashboard ID",
- "example": "65f5e4a3b9e77c001a567890"
- }
+ "Webhooks"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/UpdateDashboardRequest"
- },
- "examples": {
- "updateDashboard": {
- "summary": "Update dashboard properties and tiles",
- "value": {
- "name": "Updated Dashboard Name",
- "tiles": [
- {
- "id": "65f5e4a3b9e77c001a901234",
- "name": "Updated Line Chart",
- "x": 0,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "line",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "count",
- "where": "level:error"
- }
- ]
- }
- },
- {
- "id": "new-tile-123",
- "name": "New Number Chart",
- "x": 6,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "number",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "count",
- "where": "level:info"
- }
- ]
- }
- }
- ],
- "tags": [
- "production",
- "updated"
- ],
- "filters": [
- {
- "id": "65f5e4a3b9e77c001a301005",
- "type": "QUERY_EXPRESSION",
- "name": "Environment",
- "expression": "environment",
- "sourceId": "65f5e4a3b9e77c001a111111"
- }
- ]
- }
- }
+ "$ref": "#/components/schemas/WebhookInput"
}
}
}
},
"responses": {
"200": {
- "description": "Successfully updated dashboard",
+ "description": "Successfully created webhook",
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/DashboardResponseEnvelope"
- },
- "examples": {
- "updatedDashboard": {
- "summary": "Updated dashboard response",
- "value": {
- "data": {
- "id": "65f5e4a3b9e77c001a567890",
- "name": "Updated Dashboard Name",
- "tiles": [
- {
- "id": "65f5e4a3b9e77c001a901234",
- "name": "Updated Line Chart",
- "x": 0,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "line",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "count",
- "where": "level:error"
- }
- ]
- }
- },
- {
- "id": "new-tile-123",
- "name": "New Number Chart",
- "x": 6,
- "y": 0,
- "w": 6,
- "h": 3,
- "config": {
- "displayType": "number",
- "sourceId": "65f5e4a3b9e77c001a111111",
- "select": [
- {
- "aggFn": "count",
- "where": "level:info"
- }
- ]
- }
- }
- ],
- "tags": [
- "production",
- "updated"
- ],
- "filters": [
- {
- "id": "65f5e4a3b9e77c001a301005",
- "type": "QUERY_EXPRESSION",
- "name": "Environment",
- "expression": "environment",
- "sourceId": "65f5e4a3b9e77c001a111111"
- }
- ]
- }
- }
- }
+ "$ref": "#/components/schemas/WebhookResponseEnvelope"
}
}
}
},
"400": {
- "description": "Bad request",
+ "description": "Bad request or duplicate webhook",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Could not find the following source IDs: 68fa86308aa879b977aa6af6"
}
}
}
@@ -4915,69 +9038,114 @@
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Unauthorized access. API key is missing or invalid."
- }
- }
- }
- },
- "404": {
- "description": "Dashboard not found",
- "content": {
- "application/json": {
- "schema": {
- "$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Dashboard not found"
}
}
}
},
- "500": {
- "description": "Server error or validation failure",
+ "403": {
+ "description": "Forbidden",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Invalid dashboard configuration"
}
}
}
}
}
- },
- "delete": {
- "summary": "Delete Dashboard",
- "description": "Deletes a dashboard",
- "operationId": "deleteDashboard",
+ }
+ },
+ "/api/v2/webhooks/validate": {
+ "post": {
+ "summary": "Validate Webhook",
+ "description": "Validates a webhook body against the same schema, URL rules, and body\ntemplate check used by POST /api/v2/webhooks. The webhook is **never\npersisted**. Use this endpoint at plan time (e.g. from a Terraform\nprovider) to check that a webhook configuration is valid before\napplying it. Name uniqueness is not checked.\n",
+ "operationId": "validateWebhook",
"tags": [
- "Dashboards"
+ "Webhooks"
],
- "parameters": [
- {
- "name": "id",
- "in": "path",
- "required": true,
- "schema": {
- "type": "string"
- },
- "description": "Dashboard ID",
- "example": "65f5e4a3b9e77c001a567890"
+ "requestBody": {
+ "required": true,
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/WebhookInput"
+ }
+ }
}
- ],
+ },
"responses": {
"200": {
- "description": "Successfully deleted dashboard",
+ "description": "Validation result. HTTP 200 is always returned for valid **and**\ninvalid bodies — a non-200 response means the request itself\nfailed (auth, server error, etc.).\n",
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/EmptyResponse"
+ "type": "object",
+ "required": [
+ "valid",
+ "errors",
+ "normalized"
+ ],
+ "properties": {
+ "valid": {
+ "type": "boolean",
+ "description": "True when the body passes all validation rules."
+ },
+ "errors": {
+ "type": "array",
+ "description": "Validation errors. Empty when valid is true.",
+ "items": {
+ "type": "object",
+ "required": [
+ "path",
+ "message"
+ ],
+ "properties": {
+ "path": {
+ "type": "string",
+ "description": "Dot-separated field path, or empty string for top-level errors."
+ },
+ "message": {
+ "type": "string",
+ "description": "Human-readable error description."
+ }
+ }
+ }
+ },
+ "normalized": {
+ "type": "object",
+ "nullable": true,
+ "description": "The parsed webhook body, without the write-only\n`headers` and `queryParams`. Populated when valid is\ntrue, null when valid is false.\n"
+ }
+ }
},
- "example": {}
+ "examples": {
+ "valid": {
+ "summary": "Valid webhook body",
+ "value": {
+ "valid": true,
+ "errors": [],
+ "normalized": {
+ "name": "Alerts",
+ "service": "generic",
+ "url": "https://example.com/webhooks/alerts",
+ "body": "{\"alert\": \"{{title}}\"}"
+ }
+ }
+ },
+ "invalid": {
+ "summary": "Body template that does not compile",
+ "value": {
+ "valid": false,
+ "errors": [
+ {
+ "path": "body",
+ "message": "Invalid webhook body template (line 1, column 12): Parse error on line 1: ..."
+ }
+ ],
+ "normalized": null
+ }
+ }
+ }
}
}
},
@@ -4994,15 +9162,12 @@
}
}
},
- "404": {
- "description": "Dashboard not found",
+ "403": {
+ "description": "Forbidden",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Dashboard not found"
}
}
}
@@ -5010,158 +9175,127 @@
}
}
},
- "/api/v2/search": {
- "post": {
- "summary": "Search Raw Logs and Traces",
- "description": "Fetch individual log or trace rows from a HyperDX source.\n\nThis endpoint mirrors the \"search\" panel mode in the HyperDX UI.\nHyperDX applies the same query optimizations used in the UI:\n - Named attribute columns (e.g. \"pipedream.pipeline_name\") are\n rewritten to their indexed materialized equivalents when the\n source schema exposes them, avoiding slow Map lookups.\n - Rows are ordered by timestamp descending (most recent first).\n - The source's built-in PREWHERE / partition pruning is applied.\n\nAuthentication: Bearer token (personal API key from Team Settings).\n",
- "operationId": "searchEvents",
+ "/api/v2/webhooks/{id}": {
+ "put": {
+ "summary": "Update Webhook",
+ "description": "Replaces an existing webhook. Readable optional fields\n(`description`, `body`) are a full replace: omitting them clears\nthem. The write-only fields `headers` and `queryParams` are never\nreturned on read, so omitting them preserves the stored values;\nsend an explicit empty object (`{}`) to clear them. Exception: if the\ndestination (`url` or `service`) changes, omitted `headers`/\n`queryParams` are cleared rather than preserved so stored secrets are\nnever forwarded to a new destination.\n",
+ "operationId": "updateWebhook",
"tags": [
- "Search"
+ "Webhooks"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Webhook ID",
+ "example": "507f1f77bcf86cd799439011"
+ }
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/SearchRequest"
- },
- "examples": {
- "recentErrors": {
- "summary": "Recent errors for a service",
- "value": {
- "sourceId": "69b46cb0d964ce2d0b9506a8",
- "startTime": "2026-05-10T00:00:00Z",
- "endTime": "2026-05-10T01:00:00Z",
- "where": "SeverityText:ERROR",
- "select": "Timestamp,SeverityText,Body,ServiceName",
- "maxResults": 50
- }
- },
- "pipedreamTaskLogs": {
- "summary": "Pipedream task logs with materialized column filter",
- "value": {
- "sourceId": "69b46cb0d964ce2d0b9506a8",
- "startTime": "2026-05-10T00:00:00Z",
- "endTime": "2026-05-10T01:00:00Z",
- "where": "pipedream.pipeline_name:my-pipeline",
- "select": "Timestamp,SeverityText,Body,pipedream.pipeline_name,pipedream.stage_name,pipedream.task_index",
- "maxResults": 200
- }
- },
- "traceSearch": {
- "summary": "Slow spans for a service",
- "value": {
- "sourceId": "69b46cb0d964ce2d0b9508b2",
- "startTime": "2026-05-10T00:00:00Z",
- "endTime": "2026-05-10T01:00:00Z",
- "where": "ServiceName:my-service AND Duration:>1000000000",
- "select": "Timestamp,TraceId,SpanId,SpanName,ServiceName,Duration,StatusCode",
- "maxResults": 100
- }
- },
- "paginated": {
- "summary": "Paginated fetch",
- "value": {
- "sourceId": "69b46cb0d964ce2d0b9506a8",
- "startTime": "2026-05-10T00:00:00Z",
- "endTime": "2026-05-10T01:00:00Z",
- "maxResults": 100,
- "offset": 100
- }
- }
+ "$ref": "#/components/schemas/WebhookInput"
}
}
}
},
"responses": {
"200": {
- "description": "Matching rows returned successfully",
+ "description": "Successfully updated webhook",
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/SearchResponse"
- },
- "example": {
- "data": [
- {
- "Timestamp": "2026-05-10T00:01:23.456789000Z",
- "SeverityText": "ERROR",
- "Body": "connection refused: redis:6379",
- "ServiceName": "api-service"
- }
- ],
- "rows": 1
+ "$ref": "#/components/schemas/WebhookResponseEnvelope"
}
}
}
},
"400": {
- "description": "Invalid request parameters or query error",
+ "description": "Bad request or duplicate webhook",
"content": {
"application/json": {
"schema": {
- "type": "object",
- "properties": {
- "message": {
- "type": "string"
- }
- }
+ "$ref": "#/components/schemas/Error"
}
}
}
},
"401": {
- "description": "Missing or invalid API key"
+ "description": "Unauthorized",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
+ },
+ "403": {
+ "description": "Forbidden",
+ "content": {
+ "application/json": {
+ "schema": {
+ "$ref": "#/components/schemas/Error"
+ }
+ }
+ }
},
"404": {
- "description": "Source or connection not found",
+ "description": "Webhook not found",
"content": {
"application/json": {
"schema": {
- "type": "object",
- "properties": {
- "message": {
- "type": "string"
- }
- }
+ "$ref": "#/components/schemas/Error"
}
}
}
},
- "500": {
- "description": "Server error",
+ "409": {
+ "description": "Webhook was modified concurrently; retry with current state",
"content": {
"application/json": {
"schema": {
- "type": "object",
- "properties": {
- "message": {
- "type": "string"
- }
- }
+ "$ref": "#/components/schemas/Error"
}
}
}
}
}
- }
- },
- "/api/v2/sources": {
- "get": {
- "summary": "List Sources",
- "description": "Retrieves a list of all sources for the authenticated team",
- "operationId": "listSources",
+ },
+ "delete": {
+ "summary": "Delete Webhook",
+ "description": "Deletes a webhook. Blocked with a 409 while any alert still references it — reassign or remove those alerts first — so deletion never leaves an alert pointing at a missing webhook (which would silently drop notifications). Mirrors the internal webhook delete guard.",
+ "operationId": "deleteWebhook",
"tags": [
- "Sources"
+ "Webhooks"
+ ],
+ "parameters": [
+ {
+ "name": "id",
+ "in": "path",
+ "required": true,
+ "schema": {
+ "type": "string"
+ },
+ "description": "Webhook ID",
+ "example": "507f1f77bcf86cd799439011"
+ }
],
"responses": {
"200": {
- "description": "Successfully retrieved sources",
+ "description": "Successfully deleted webhook",
"content": {
"application/json": {
"schema": {
- "$ref": "#/components/schemas/SourcesListResponse"
- }
+ "$ref": "#/components/schemas/EmptyResponse"
+ },
+ "example": {}
}
}
},
@@ -5171,9 +9305,6 @@
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Unauthorized access. API key is missing or invalid."
}
}
}
@@ -5187,44 +9318,19 @@
}
}
}
- }
- }
- }
- },
- "/api/v2/webhooks": {
- "get": {
- "summary": "List Webhooks",
- "description": "Retrieves a list of all webhooks for the authenticated team",
- "operationId": "listWebhooks",
- "tags": [
- "Webhooks"
- ],
- "responses": {
- "200": {
- "description": "Successfully retrieved webhooks",
- "content": {
- "application/json": {
- "schema": {
- "$ref": "#/components/schemas/WebhooksListResponse"
- }
- }
- }
},
- "401": {
- "description": "Unauthorized",
+ "404": {
+ "description": "Webhook not found",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Error"
- },
- "example": {
- "message": "Unauthorized access. API key is missing or invalid."
}
}
}
},
- "403": {
- "description": "Forbidden",
+ "409": {
+ "description": "Webhook is still referenced by one or more alerts",
"content": {
"application/json": {
"schema": {
diff --git a/packages/api/package.json b/packages/api/package.json
index 9874759096..e1ca44871e 100644
--- a/packages/api/package.json
+++ b/packages/api/package.json
@@ -1,17 +1,28 @@
{
"name": "@hyperdx/api",
- "version": "2.28.0",
+ "version": "2.42.0",
"license": "MIT",
"private": true,
"engines": {
"node": ">=22.16.0"
},
+ "nx": {
+ "targets": {
+ "ci:lint": {
+ "outputs": [
+ "{projectRoot}/openapi.json"
+ ]
+ }
+ }
+ },
"dependencies": {
- "@ai-sdk/anthropic": "^3.0.58",
- "@ai-sdk/openai": "^3.0.47",
+ "@ai-sdk/anthropic": "^4.0.65",
+ "@ai-sdk/openai": "^4.0.78",
+ "@ai-sdk/otel": "^1.0.116",
"@braintree/sanitize-url": "^7.1.1",
+ "@clickhouse/client-common": "1.23.0-head.fae5998.1",
"@esm2cjs/p-queue": "^7.3.0",
- "@hyperdx/common-utils": "^0.20.0",
+ "@hyperdx/common-utils": "^0.32.0",
"@hyperdx/node-opentelemetry": "^0.9.0",
"@hyperdx/passport-local-mongoose": "^9.0.1",
"@modelcontextprotocol/sdk": "^1.27.1",
@@ -20,10 +31,10 @@
"@opentelemetry/sdk-metrics": "^1.30.1",
"@slack/webhook": "^7.0.0",
"@types/node": "^22.15.18",
- "ai": "^6.0.116",
+ "ai": "^7.0.116",
"aws4": "^1.13.2",
"chrono-node": "^2.9.0",
- "compression": "^1.7.4",
+ "compression": "^1.8.2",
"concurrently": "^9.1.2",
"connect-mongo": "^4.6.0",
"cors": "^2.8.5",
@@ -32,14 +43,14 @@
"express": "^4.19.2",
"express-rate-limit": "^6.7.1",
"express-session": "^1.17.3",
- "handlebars": "^4.7.9",
+ "handlebars": "^4.7.10",
"http-graceful-shutdown": "^3.1.13",
- "http-proxy-middleware": "^3.0.5",
- "jsonwebtoken": "^9.0.0",
+ "http-proxy-middleware": "^4.0.0",
+ "ip-address": "^10.7.1",
"lodash": "^4.18.1",
"minimist": "^1.2.7",
"mongodb": "^6.15.0",
- "mongoose": "^6.13.9",
+ "mongoose": "^6.13.10",
"ms": "^2.1.3",
"on-headers": "^1.1.0",
"passport": "^0.6.0",
@@ -47,13 +58,15 @@
"pino": "^10.0.0",
"pino-http": "^11.0.0",
"promised-handlebars": "^2.0.1",
- "protobufjs": "^7.5.8",
+ "protobufjs": "^7.6.5",
"serialize-error": "^8.1.0",
+ "sqlstring": "^2.3.3",
"uuid": "^8.3.2",
"zod": "3.25",
"zod-express-middleware": "^1.4.0"
},
"devDependencies": {
+ "@opentelemetry/sdk-trace-base": "^1.30.1",
"@stoplight/spectral-cli": "^6.15.0",
"@types/compression": "^1.7.3",
"@types/cors": "^2.8.14",
@@ -64,9 +77,11 @@
"@types/minimist": "^1.2.2",
"@types/ms": "^0.7.31",
"@types/passport-local": "^1.0.34",
+ "@types/sqlstring": "^2.3.0",
"@types/supertest": "^2.0.12",
"@types/swagger-jsdoc": "^6",
"@types/uuid": "^8.3.4",
+ "ajv": "^8.20.0",
"jest": "^30.2.0",
"migrate-mongo": "^11.0.0",
"nodemon": "^2.0.20",
@@ -75,29 +90,32 @@
"supertest": "^6.3.1",
"swagger-jsdoc": "^6.2.8",
"swagger-ui-express": "^5.0.1",
- "ts-jest": "^29.4.5",
+ "ts-jest": "^29.4.11",
"ts-node": "^10.8.1",
"tsc-alias": "^1.8.8",
"tsconfig-paths": "^4.2.0",
- "typescript": "^5.9.3"
+ "typescript": "^6.0.3"
},
"scripts": {
"start": "node ./build/index.js",
"dev": "DOTENV_CONFIG_PATH=.env.development nodemon --exec 'ts-node' --transpile-only -r tsconfig-paths/register -r dotenv-expand/config -r '@hyperdx/node-opentelemetry/build/src/tracing' ./src/index.ts",
"dev:mcp": "npx @modelcontextprotocol/inspector",
"dev-task": "DOTENV_CONFIG_PATH=.env.development nodemon --exec 'ts-node' --transpile-only -r tsconfig-paths/register -r dotenv-expand/config -r '@hyperdx/node-opentelemetry/build/src/tracing' ./src/tasks/index.ts",
- "build": "rimraf ./build && tsc && tsc-alias && cp -r ./src/opamp/proto ./build/opamp/",
+ "build": "rimraf ./build && tsc -p tsconfig.build.json && tsc-alias -p tsconfig.build.json && cp -r ./src/opamp/proto ./build/opamp/",
"build:vercel": "rimraf ./build && tsc -p tsconfig.vercel.json && tsc-alias -p tsconfig.vercel.json && cp -r ./src/opamp/proto ./build/opamp/",
- "lint": "npx eslint --quiet . --ext .ts",
+ "lint": "npx eslint . --ext .ts --max-warnings 302",
"lint:fix": "npx eslint . --ext .ts --fix",
"ci:lint": "yarn lint && yarn tsc --noEmit && yarn lint:openapi",
- "ci:int": "DOTENV_CONFIG_PATH=.env.test DOTENV_CONFIG_OVERRIDE=true jest --runInBand --ci --forceExit",
- "dev:int": "DOTENV_CONFIG_PATH=.env.test DOTENV_CONFIG_OVERRIDE=true jest --runInBand --coverage --forceExit",
+ "ci:unit": "jest --ci --coverage",
+ "dev:unit": "jest --watch",
+ "ci:int": "DOTENV_CONFIG_PATH=.env.test DOTENV_CONFIG_OVERRIDE=true jest --config jest.int.config.js --runInBand --ci --forceExit --passWithNoTests",
+ "dev:int": "DOTENV_CONFIG_PATH=.env.test DOTENV_CONFIG_OVERRIDE=true jest --config jest.int.config.js --runInBand --coverage --forceExit",
"dev:migrate-db-create": "ts-node node_modules/.bin/migrate-mongo create -f migrate-mongo-config.ts",
"dev:migrate-db": "ts-node node_modules/.bin/migrate-mongo up -f migrate-mongo-config.ts",
"dev:migrate-ch-create": "migrate create -ext sql -dir ./migrations/ch -seq",
"dev:migrate-ch": "migrate -database 'clickhouse://localhost:9000?database=default&x-multi-statement=true' -path ./migrations/ch up",
"docgen": "ts-node scripts/generate-api-docs.ts",
+ "seed:alerts": "../../scripts/with-slots.sh ts-node --transpile-only -r tsconfig-paths/register scripts/seed-alerts.ts",
"lint:openapi": "yarn docgen && spectral lint openapi.json"
}
}
diff --git a/packages/api/scripts/generate-api-docs.ts b/packages/api/scripts/generate-api-docs.ts
index 0d3978e5d9..facac274c0 100644
--- a/packages/api/scripts/generate-api-docs.ts
+++ b/packages/api/scripts/generate-api-docs.ts
@@ -2,6 +2,7 @@ import fs from 'fs';
import path from 'path';
import swaggerJsdoc from 'swagger-jsdoc';
+// eslint-disable-next-line no-restricted-imports -- run via ts-node without tsconfig-paths, so @/ does not resolve here
import { swaggerOptions } from '../src/utils/swagger';
const specs = swaggerJsdoc(swaggerOptions);
diff --git a/packages/api/scripts/seed-alerts.ts b/packages/api/scripts/seed-alerts.ts
new file mode 100644
index 0000000000..28220ab4ea
--- /dev/null
+++ b/packages/api/scripts/seed-alerts.ts
@@ -0,0 +1,473 @@
+/**
+ * Seeds this worktree's dev-slot Mongo with N alerts, for exercising the alerts
+ * page at scale.
+ *
+ * Half the alerts reference saved searches (up to 2 per search), half reference
+ * dashboard tiles (up to 4 tiles per dashboard, at most one alert per tile).
+ * Every seeded dashboard and saved search carries a tag (default `seeded`) so
+ * the whole batch can be removed again with `--purge`.
+ *
+ * yarn seed:alerts --count 2000
+ * yarn seed:alerts --purge
+ *
+ * The Mongo URI defaults to the dev slot of the current git worktree, from
+ * scripts/slots.sh. Override with --mongo-uri, MONGO_URI, or HDX_DEV_SLOT.
+ * The dev stack must be up (`yarn dev`) and registered once, since
+ * the script attaches everything to the existing team and log source.
+ */
+import { formatTileAlertDisplayName } from '@hyperdx/common-utils/dist/alerts';
+import { DisplayType } from '@hyperdx/common-utils/dist/types';
+import mongoose from 'mongoose';
+
+import Alert, { AlertSource, AlertState } from '@/models/alert';
+import Dashboard from '@/models/dashboard';
+import { SavedSearch } from '@/models/savedSearch';
+import Team from '@/models/team';
+import Webhook, { WebhookService } from '@/models/webhook';
+
+// Several schemas rely on empty strings satisfying `required` (an empty
+// `where` is a valid search). src/models/index.ts installs this for the API;
+// this script deliberately does not import it, to stay clear of config and
+// telemetry bootstrapping.
+mongoose.Schema.Types.String.checkRequired(v => v != null);
+mongoose.set('strictQuery', false);
+
+const DEFAULT_COUNT = 1000;
+const DEFAULT_TAG = 'seeded';
+const MAX_ALERTS_PER_SAVED_SEARCH = 2;
+const MAX_TILES_PER_DASHBOARD = 4;
+const INSERT_CHUNK_SIZE = 500;
+
+/** Roughly what a real team looks like: mostly quiet, a few firing. */
+const STATE_WEIGHTS: [AlertState, number][] = [
+ [AlertState.OK, 79],
+ [AlertState.ALERT, 12],
+ [AlertState.PENDING, 9],
+];
+
+const INTERVALS = ['1m', '5m', '15m', '30m', '1h', '6h', '12h', '1d'] as const;
+
+/** Extra tags beyond the seed tag, so the tag filter has something to filter. */
+const FLAVOR_TAGS = ['production', 'staging', 'team-alpha', 'team-beta'];
+
+const NOTE_TEXT = [
+ 'Runbook: check the upstream queue depth first.',
+ 'Known noisy during deploys — see #incidents.',
+ 'Owned by the platform team. Escalate after 15m.',
+].map(text => `${text}\n\n- [Runbook](https://example.com/runbook)`);
+
+type Args = {
+ count: number;
+ tag: string;
+ mongoUri: string | null;
+ purge: boolean;
+ help: boolean;
+};
+
+const USAGE = [
+ 'Usage: yarn seed:alerts [options]',
+ '',
+ ' -n, --count N number of alerts to create (default 1000)',
+ ' --tag TAG tag applied to seeded dashboards and saved searches',
+ ' (default "seeded")',
+ ' --mongo-uri URI override the dev-slot Mongo URI',
+ ' --purge delete everything carrying --tag instead of seeding',
+].join('\n');
+
+function parseArgs(argv: string[]): Args {
+ const args: Args = {
+ count: DEFAULT_COUNT,
+ tag: DEFAULT_TAG,
+ mongoUri: null,
+ purge: false,
+ help: false,
+ };
+
+ const rest = [...argv];
+ for (let arg = rest.shift(); arg != null; arg = rest.shift()) {
+ const next = () => {
+ const value = rest.shift();
+ if (value == null) throw new Error(`${arg} needs a value`);
+ return value;
+ };
+ switch (arg) {
+ case '-n':
+ case '--count':
+ args.count = Number(next());
+ if (!Number.isInteger(args.count) || args.count < 1) {
+ throw new Error('--count must be a positive integer');
+ }
+ break;
+ case '--tag':
+ args.tag = next();
+ break;
+ case '--mongo-uri':
+ args.mongoUri = next();
+ break;
+ case '--purge':
+ args.purge = true;
+ break;
+ case '-h':
+ case '--help':
+ args.help = true;
+ break;
+ default:
+ throw new Error(`Unknown argument: ${arg}`);
+ }
+ }
+
+ return args;
+}
+
+/**
+ * Mongo URI of this worktree's dev stack. `yarn seed:alerts` runs through
+ * scripts/with-slots.sh, which sets HDX_DEV_MONGO_PORT.
+ */
+function devSlotMongoUri(): string {
+ const port = process.env.HDX_DEV_MONGO_PORT;
+ if (!port) {
+ throw new Error(
+ 'HDX_DEV_MONGO_PORT is not set. Run this with `yarn seed:alerts`, or pass --mongo-uri.',
+ );
+ }
+ return `mongodb://localhost:${port}/hyperdx`;
+}
+
+function randomInt(minInclusive: number, maxInclusive: number): number {
+ return (
+ minInclusive + Math.floor(Math.random() * (maxInclusive - minInclusive + 1))
+ );
+}
+
+function pick(items: readonly T[]): T {
+ return items[randomInt(0, items.length - 1)];
+}
+
+const weightedStates: AlertState[] = STATE_WEIGHTS.flatMap(([state, weight]) =>
+ Array.from({ length: weight }, () => state),
+);
+
+/** Pairs each plan with the id of the document it was inserted as. */
+function* zip(as: A[], bs: B[]): Generator<[A, B]> {
+ const bIterator = bs[Symbol.iterator]();
+ for (const a of as) {
+ const b = bIterator.next();
+ if (b.done === true) return;
+ yield [a, b.value];
+ }
+}
+
+/** Returns the inserted ids as strings; Mongoose casts them back on write. */
+async function insertInChunks(
+ insertMany: (docs: Record[]) => Promise,
+ docs: Record[],
+): Promise {
+ const ids: string[] = [];
+ for (let i = 0; i < docs.length; i += INSERT_CHUNK_SIZE) {
+ const inserted = await insertMany(docs.slice(i, i + INSERT_CHUNK_SIZE));
+ ids.push(...inserted.map(doc => String(doc._id)));
+ }
+ return ids;
+}
+
+function makeAlertDoc({
+ teamId,
+ channel,
+ index,
+ displayName,
+ tags,
+ savedSearchId,
+ dashboardId,
+ tileId,
+}: {
+ teamId: unknown;
+ channel: { type: 'webhook'; webhookId: string };
+ index: number;
+ displayName: string;
+ tags: string[];
+ savedSearchId?: string;
+ dashboardId?: string;
+ tileId?: string;
+}) {
+ return {
+ team: teamId,
+ source: savedSearchId ? AlertSource.SAVED_SEARCH : AlertSource.TILE,
+ // Stored on the alert rather than left to derive from the referenced
+ // saved search or dashboard tile, matching what writers persist now.
+ displayName,
+ tags,
+ savedSearch: savedSearchId ?? null,
+ groupBy: null,
+ dashboard: dashboardId ?? null,
+ tileId: tileId ?? null,
+ interval: pick(INTERVALS),
+ threshold: randomInt(1, 500),
+ thresholdType: 'above',
+ channel,
+ channels: [channel],
+ state: pick(weightedStates),
+ // Notes make rows taller and collapsible, which is what the virtualized
+ // list has to measure rather than assume.
+ note: index % 5 === 0 ? pick(NOTE_TEXT) : null,
+ };
+}
+
+function makeTile(dashboardIndex: number, tileIndex: number, sourceId: string) {
+ return {
+ id: `seeded-tile-${dashboardIndex}-${tileIndex}`,
+ x: (tileIndex % 2) * 6,
+ y: Math.floor(tileIndex / 2) * 3,
+ w: 6,
+ h: 3,
+ config: {
+ name: `Seeded tile ${tileIndex + 1}`,
+ source: sourceId,
+ displayType: DisplayType.Line,
+ select: [
+ {
+ aggFn: 'count',
+ aggCondition: '',
+ aggConditionLanguage: 'lucene',
+ valueExpression: '',
+ },
+ ],
+ where: '',
+ whereLanguage: 'lucene',
+ granularity: 'auto',
+ implicitColumnExpression: 'Body',
+ numberFormat: { output: 'number' },
+ filters: [],
+ },
+ };
+}
+
+function seededWebhookName(tag: string): string {
+ return `Seeded webhook (${tag})`;
+}
+
+async function purge(tag: string) {
+ const [dashboards, savedSearches] = await Promise.all([
+ Dashboard.find({ tags: tag }, { _id: 1 }).lean(),
+ SavedSearch.find({ tags: tag }, { _id: 1 }).lean(),
+ ]);
+ const dashboardIds = dashboards.map(d => d._id);
+ const savedSearchIds = savedSearches.map(s => s._id);
+
+ // Alerts first: an alert whose dashboard or saved search is gone renders as
+ // a nameless row rather than disappearing.
+ const { deletedCount: alertsDeleted } = await Alert.deleteMany({
+ $or: [
+ { dashboard: { $in: dashboardIds } },
+ { savedSearch: { $in: savedSearchIds } },
+ ],
+ });
+ await Dashboard.deleteMany({ _id: { $in: dashboardIds } });
+ await SavedSearch.deleteMany({ _id: { $in: savedSearchIds } });
+ const { deletedCount: webhooksDeleted } = await Webhook.deleteMany({
+ service: WebhookService.Generic,
+ name: seededWebhookName(tag),
+ });
+
+ console.log(
+ `Purged ${alertsDeleted} alerts, ${dashboardIds.length} dashboards, ` +
+ `${savedSearchIds.length} saved searches and ${webhooksDeleted} ` +
+ `webhooks tagged "${tag}".`,
+ );
+}
+
+async function seed(count: number, tag: string) {
+ const team = await Team.findOne({}).lean();
+ if (team == null) {
+ throw new Error(
+ 'No team found — register an account in the dev app first, then re-run.',
+ );
+ }
+ const teamId = team._id;
+
+ const source = await mongoose.connection
+ .collection('sources')
+ .findOne({ team: teamId, kind: 'log' });
+ if (source == null) {
+ throw new Error(`No log source found for team ${String(teamId)}.`);
+ }
+ const sourceId = source._id as mongoose.Types.ObjectId;
+
+ const webhookName = seededWebhookName(tag);
+ const webhook = await Webhook.findOneAndUpdate(
+ { team: teamId, service: WebhookService.Generic, name: webhookName },
+ {
+ $setOnInsert: {
+ team: teamId,
+ service: WebhookService.Generic,
+ name: webhookName,
+ url: 'https://example.com/seeded-webhook',
+ },
+ },
+ { new: true, upsert: true },
+ );
+ const channel = {
+ type: 'webhook' as const,
+ webhookId: String(webhook._id),
+ };
+
+ const savedSearchAlertTotal = Math.ceil(count / 2);
+ const tileAlertTotal = count - savedSearchAlertTotal;
+ const stamp = Date.now();
+ const tagsFor = (index: number) => [
+ tag,
+ FLAVOR_TAGS[index % FLAVOR_TAGS.length],
+ ];
+
+ // --- Saved searches: up to MAX_ALERTS_PER_SAVED_SEARCH alerts each --------
+ const savedSearchPlans: {
+ doc: Record;
+ alerts: number;
+ name: string;
+ tags: string[];
+ }[] = [];
+ for (let remaining = savedSearchAlertTotal; remaining > 0; ) {
+ const alerts = Math.min(
+ remaining,
+ randomInt(1, MAX_ALERTS_PER_SAVED_SEARCH),
+ );
+ const index = savedSearchPlans.length;
+ const name = `Seeded search ${stamp}-${index}`;
+ const tags = tagsFor(index);
+ savedSearchPlans.push({
+ alerts,
+ name,
+ tags,
+ doc: {
+ team: teamId,
+ name,
+ select: '',
+ where: '',
+ whereLanguage: 'lucene',
+ source: sourceId,
+ tags,
+ },
+ });
+ remaining -= alerts;
+ }
+ const savedSearchIds = await insertInChunks(
+ chunk => SavedSearch.insertMany(chunk),
+ savedSearchPlans.map(plan => plan.doc),
+ );
+
+ const alertDocs: Record[] = [];
+ for (const [plan, savedSearchId] of zip(savedSearchPlans, savedSearchIds)) {
+ for (let i = 0; i < plan.alerts; i++) {
+ alertDocs.push(
+ makeAlertDoc({
+ teamId,
+ channel,
+ index: alertDocs.length,
+ displayName: plan.name,
+ tags: plan.tags,
+ savedSearchId,
+ }),
+ );
+ }
+ }
+
+ // --- Dashboards: up to MAX_TILES_PER_DASHBOARD tiles, <=1 alert per tile --
+ const dashboardPlans: {
+ doc: Record;
+ alertedTiles: { id: string; displayName: string }[];
+ tags: string[];
+ }[] = [];
+ for (let remaining = tileAlertTotal; remaining > 0; ) {
+ const index = dashboardPlans.length;
+ const tileCount = randomInt(1, MAX_TILES_PER_DASHBOARD);
+ const tiles = Array.from({ length: tileCount }, (_, tileIndex) =>
+ makeTile(index, tileIndex, String(sourceId)),
+ );
+ // Some tiles are left un-alerted, which is the normal case on a dashboard.
+ const alertedCount = Math.min(remaining, randomInt(1, tileCount));
+ const name = `Seeded dashboard ${stamp}-${index}`;
+ const tags = tagsFor(index);
+ dashboardPlans.push({
+ alertedTiles: tiles.slice(0, alertedCount).map(tile => ({
+ id: tile.id,
+ displayName: formatTileAlertDisplayName(name, tile.config.name),
+ })),
+ tags,
+ doc: {
+ team: teamId,
+ name,
+ tiles,
+ tags,
+ filters: [],
+ },
+ });
+ remaining -= alertedCount;
+ }
+ const dashboardIds = await insertInChunks(
+ chunk => Dashboard.insertMany(chunk),
+ dashboardPlans.map(plan => plan.doc),
+ );
+
+ for (const [plan, dashboardId] of zip(dashboardPlans, dashboardIds)) {
+ for (const tile of plan.alertedTiles) {
+ alertDocs.push(
+ makeAlertDoc({
+ teamId,
+ channel,
+ index: alertDocs.length,
+ displayName: tile.displayName,
+ tags: plan.tags,
+ dashboardId,
+ tileId: tile.id,
+ }),
+ );
+ }
+ }
+
+ await insertInChunks(chunk => Alert.insertMany(chunk), alertDocs);
+
+ const byState = new Map();
+ for (const alert of alertDocs) {
+ const state = String(alert.state);
+ byState.set(state, (byState.get(state) ?? 0) + 1);
+ }
+
+ console.log(
+ [
+ `Seeded ${alertDocs.length} alerts for team ${String(teamId)}:`,
+ ` ${savedSearchAlertTotal} on ${savedSearchIds.length} saved searches ` +
+ `(<=${MAX_ALERTS_PER_SAVED_SEARCH} each)`,
+ ` ${tileAlertTotal} on tiles across ${dashboardIds.length} dashboards ` +
+ `(<=${MAX_TILES_PER_DASHBOARD} tiles each, <=1 alert per tile)`,
+ ` states: ${[...byState]
+ .map(([state, n]) => `${state}=${n}`)
+ .join(', ')}`,
+ `Tagged "${tag}" — remove with: yarn seed:alerts --purge --tag ${tag}`,
+ ].join('\n'),
+ );
+}
+
+async function main() {
+ const args = parseArgs(process.argv.slice(2));
+ if (args.help) {
+ console.log(USAGE);
+ return;
+ }
+ const mongoUri = args.mongoUri ?? process.env.MONGO_URI ?? devSlotMongoUri();
+
+ console.log(`Connecting to ${mongoUri}`);
+ await mongoose.connect(mongoUri, { serverSelectionTimeoutMS: 5000 });
+ try {
+ if (args.purge) {
+ await purge(args.tag);
+ } else {
+ await seed(args.count, args.tag);
+ }
+ } finally {
+ await mongoose.disconnect();
+ }
+}
+
+main().catch(err => {
+ console.error(err instanceof Error ? err.message : err);
+ process.exitCode = 1;
+});
diff --git a/packages/api/src/__tests__/config.test.ts b/packages/api/src/__tests__/config.test.ts
index fedf486e0d..c006077783 100644
--- a/packages/api/src/__tests__/config.test.ts
+++ b/packages/api/src/__tests__/config.test.ts
@@ -59,4 +59,188 @@ describe('config', () => {
});
});
});
+
+ describe('EXTERNAL_API_RATE_LIMIT_MAX', () => {
+ const ORIGINAL = process.env.EXTERNAL_API_RATE_LIMIT_MAX;
+
+ afterEach(() => {
+ if (ORIGINAL === undefined) {
+ delete process.env.EXTERNAL_API_RATE_LIMIT_MAX;
+ } else {
+ process.env.EXTERNAL_API_RATE_LIMIT_MAX = ORIGINAL;
+ }
+ jest.resetModules();
+ });
+
+ it('defaults to 100 when unset', () => {
+ delete process.env.EXTERNAL_API_RATE_LIMIT_MAX;
+
+ jest.isolateModules(() => {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports, n/no-missing-require
+ const config = require('@/config');
+ expect(config.EXTERNAL_API_RATE_LIMIT_MAX).toBe(100);
+ });
+ });
+
+ it('accepts a valid positive integer', () => {
+ process.env.EXTERNAL_API_RATE_LIMIT_MAX = '1000';
+
+ jest.isolateModules(() => {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports, n/no-missing-require
+ const config = require('@/config');
+ expect(config.EXTERNAL_API_RATE_LIMIT_MAX).toBe(1000);
+ });
+ });
+
+ it('accepts exponential notation', () => {
+ process.env.EXTERNAL_API_RATE_LIMIT_MAX = '1e3';
+
+ jest.isolateModules(() => {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports, n/no-missing-require
+ const config = require('@/config');
+ expect(config.EXTERNAL_API_RATE_LIMIT_MAX).toBe(1000);
+ });
+ });
+
+ it.each([
+ ['garbage string (NaN)', 'abc'],
+ ['zero (express-rate-limit v6 treats 0 as unlimited)', '0'],
+ ['negative', '-5'],
+ ['trailing garbage', '100/min'],
+ ['empty string', ''],
+ ['fractional (would block every request at max=0.5)', '0.5'],
+ ['Infinity (would disable the limit entirely)', 'Infinity'],
+ ])(
+ 'falls back to the default of 100 and does not silently disable the limit: %s (%p)',
+ (_desc, raw) => {
+ process.env.EXTERNAL_API_RATE_LIMIT_MAX = raw;
+
+ jest.isolateModules(() => {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports, n/no-missing-require
+ const config = require('@/config');
+ expect(config.EXTERNAL_API_RATE_LIMIT_MAX).toBe(100);
+ });
+ },
+ );
+ });
+
+ describe('EXTERNAL_API_SEARCH_MAX_RESULTS', () => {
+ const ORIGINAL = process.env.EXTERNAL_API_SEARCH_MAX_RESULTS;
+
+ afterEach(() => {
+ if (ORIGINAL === undefined) {
+ delete process.env.EXTERNAL_API_SEARCH_MAX_RESULTS;
+ } else {
+ process.env.EXTERNAL_API_SEARCH_MAX_RESULTS = ORIGINAL;
+ }
+ jest.resetModules();
+ });
+
+ const loadMaxResults = () => {
+ let loaded!: typeof import('@/config');
+ jest.isolateModules(() => {
+ loaded = jest.requireActual('@/config');
+ });
+ return loaded.EXTERNAL_API_SEARCH_MAX_RESULTS;
+ };
+
+ it('defaults to 2000 when unset', () => {
+ delete process.env.EXTERNAL_API_SEARCH_MAX_RESULTS;
+ expect(loadMaxResults()).toBe(2000);
+ });
+
+ it('accepts a valid positive integer', () => {
+ process.env.EXTERNAL_API_SEARCH_MAX_RESULTS = '10000';
+ expect(loadMaxResults()).toBe(10_000);
+ });
+
+ it.each(['abc', '0', '50', '-5', '100rows', '', '0.5', 'Infinity'])(
+ 'falls back to 2000 for %p',
+ raw => {
+ process.env.EXTERNAL_API_SEARCH_MAX_RESULTS = raw;
+ expect(loadMaxResults()).toBe(2000);
+ },
+ );
+ });
+
+ describe('EXPRESS_SESSION_SECRET', () => {
+ const ORIGINAL_SECRET = process.env.EXPRESS_SESSION_SECRET;
+ const ORIGINAL_LOCAL_APP_MODE = process.env.IS_LOCAL_APP_MODE;
+
+ beforeEach(() => {
+ delete process.env.IS_LOCAL_APP_MODE;
+ });
+
+ afterEach(() => {
+ if (ORIGINAL_SECRET === undefined) {
+ delete process.env.EXPRESS_SESSION_SECRET;
+ } else {
+ process.env.EXPRESS_SESSION_SECRET = ORIGINAL_SECRET;
+ }
+ if (ORIGINAL_LOCAL_APP_MODE === undefined) {
+ delete process.env.IS_LOCAL_APP_MODE;
+ } else {
+ process.env.IS_LOCAL_APP_MODE = ORIGINAL_LOCAL_APP_MODE;
+ }
+ jest.resetModules();
+ });
+
+ // The `jest` object's isolateModules returns `jest` itself rather than the
+ // callback's value, so the module has to be captured from inside it.
+ const loadConfig = () => {
+ let loaded!: typeof import('@/config');
+ jest.isolateModules(() => {
+ loaded = jest.requireActual('@/config');
+ });
+ return loaded;
+ };
+
+ it('uses a configured secret verbatim', () => {
+ process.env.EXPRESS_SESSION_SECRET = 'a-deployment-specific-secret';
+
+ const config = loadConfig();
+ expect(config.EXPRESS_SESSION_SECRET).toBe(
+ 'a-deployment-specific-secret',
+ );
+ expect(config.IS_EXPRESS_SESSION_SECRET_GENERATED).toBe(false);
+ });
+
+ it('preserves a configured secret verbatim', () => {
+ process.env.EXPRESS_SESSION_SECRET = ' verbatim secret 👋 ';
+ const config = loadConfig();
+ expect(config.EXPRESS_SESSION_SECRET).toBe(' verbatim secret 👋 ');
+ expect(config.IS_EXPRESS_SESSION_SECRET_GENERATED).toBe(false);
+ });
+
+ it('generates a secret when unset, empty, or publicly known', () => {
+ for (const value of [undefined, '', 'hyperdx is cool 👋']) {
+ if (value === undefined) {
+ delete process.env.EXPRESS_SESSION_SECRET;
+ } else {
+ process.env.EXPRESS_SESSION_SECRET = value;
+ }
+
+ const config = loadConfig();
+ expect(config.EXPRESS_SESSION_SECRET).toMatch(/^[0-9a-f]{64}$/);
+ expect(config.IS_EXPRESS_SESSION_SECRET_GENERATED).toBe(true);
+ }
+ });
+
+ it('keeps the public demo secret only without authentication', () => {
+ process.env.IS_LOCAL_APP_MODE = 'DANGEROUSLY_is_local_app_mode💀';
+ process.env.EXPRESS_SESSION_SECRET = 'hyperdx is cool 👋';
+
+ const config = loadConfig();
+ expect(config.EXPRESS_SESSION_SECRET).toBe('hyperdx is cool 👋');
+ expect(config.IS_EXPRESS_SESSION_SECRET_GENERATED).toBe(false);
+ });
+
+ it('generates a different secret on each load', () => {
+ delete process.env.EXPRESS_SESSION_SECRET;
+
+ expect(loadConfig().EXPRESS_SESSION_SECRET).not.toBe(
+ loadConfig().EXPRESS_SESSION_SECRET,
+ );
+ });
+ });
});
diff --git a/packages/api/src/__tests__/migrations.int.test.ts b/packages/api/src/__tests__/migrations.int.test.ts
new file mode 100644
index 0000000000..52a22dbfb3
--- /dev/null
+++ b/packages/api/src/__tests__/migrations.int.test.ts
@@ -0,0 +1,227 @@
+import mongoose from 'mongoose';
+
+import { createTeam } from '@/controllers/team';
+import { clearDBCollections, closeDB, connectDB } from '@/fixtures';
+import { backfillAlertDisplayFields } from '@/migrations';
+import Alert, { AlertSource, AlertThresholdType } from '@/models/alert';
+import Dashboard from '@/models/dashboard';
+import { SavedSearch } from '@/models/savedSearch';
+
+const baseAlert = {
+ threshold: 1,
+ thresholdType: AlertThresholdType.ABOVE,
+ interval: '5m',
+ channel: { type: null },
+} as const;
+
+const makeSavedSearch = (
+ team: mongoose.Types.ObjectId,
+ fields: { name: string; tags?: string[] },
+) =>
+ new SavedSearch({
+ team,
+ source: new mongoose.Types.ObjectId(),
+ select: '',
+ where: '',
+ whereLanguage: 'lucene',
+ orderBy: '',
+ ...fields,
+ }).save();
+
+describe('backfillAlertDisplayFields', () => {
+ beforeAll(async () => {
+ await connectDB();
+ });
+
+ afterEach(async () => {
+ await clearDBCollections();
+ });
+
+ afterAll(async () => {
+ await closeDB();
+ });
+
+ it('backfills missing alert names and tags from the referenced documents', async () => {
+ const team = await createTeam({ name: 'Test team' });
+ const savedSearch = await makeSavedSearch(team._id, {
+ name: 'Error spikes',
+ tags: ['errors', 'prod'],
+ });
+ const untaggedSearch = await makeSavedSearch(team._id, {
+ name: 'Untagged search',
+ });
+ const dashboard = await new Dashboard({
+ team: team._id,
+ name: 'Service health',
+ tags: ['infra'],
+ tiles: [{ id: 'tile-1', config: { name: 'P95 latency' } }],
+ }).save();
+
+ const [
+ searchAlert,
+ untaggedSearchAlert,
+ tileAlert,
+ missingTileAlert,
+ namedAlert,
+ taggedAlert,
+ clearedTagsAlert,
+ danglingAlert,
+ inlineAlert,
+ ] = await Alert.create([
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.SAVED_SEARCH,
+ savedSearch: savedSearch._id,
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.SAVED_SEARCH,
+ savedSearch: untaggedSearch._id,
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.TILE,
+ dashboard: dashboard._id,
+ tileId: 'tile-1',
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.TILE,
+ dashboard: dashboard._id,
+ tileId: 'deleted-tile',
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.SAVED_SEARCH,
+ savedSearch: savedSearch._id,
+ displayName: 'Custom name',
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.SAVED_SEARCH,
+ savedSearch: savedSearch._id,
+ tags: ['keep-me'],
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.SAVED_SEARCH,
+ savedSearch: savedSearch._id,
+ tags: [],
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.SAVED_SEARCH,
+ savedSearch: new mongoose.Types.ObjectId(),
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.INLINE,
+ chartConfig: { name: 'CPU usage', displayType: 'line' },
+ },
+ ]);
+
+ const { insertedId: legacyAlertId } = await Alert.collection.insertOne({
+ ...baseAlert,
+ team: team._id,
+ savedSearch: savedSearch._id,
+ });
+
+ await backfillAlertDisplayFields();
+
+ const byId = async (id: mongoose.Types.ObjectId | string) =>
+ Alert.findById(id).lean();
+
+ expect(await byId(searchAlert._id)).toMatchObject({
+ displayName: 'Error spikes',
+ tags: ['errors', 'prod'],
+ });
+ expect(await byId(tileAlert._id)).toMatchObject({
+ displayName: 'Service health - P95 latency',
+ tags: ['infra'],
+ });
+ expect((await byId(missingTileAlert._id))?.displayName).toBe(
+ 'Service health - Tile',
+ );
+ expect(await byId(namedAlert._id)).toMatchObject({
+ displayName: 'Custom name',
+ tags: ['errors', 'prod'],
+ });
+ expect(await byId(taggedAlert._id)).toMatchObject({
+ displayName: 'Error spikes',
+ tags: ['keep-me'],
+ });
+ expect(await byId(clearedTagsAlert._id)).toMatchObject({
+ displayName: 'Error spikes',
+ tags: [],
+ });
+ expect(await byId(inlineAlert._id)).toMatchObject({
+ displayName: 'CPU usage',
+ tags: [],
+ });
+ expect(await byId(legacyAlertId)).toMatchObject({
+ displayName: 'Error spikes',
+ });
+
+ const untagged = await byId(untaggedSearchAlert._id);
+ expect(untagged?.displayName).toBe('Untagged search');
+ expect(untagged?.tags).toEqual([]);
+
+ expect((await byId(searchAlert._id))?.updatedAt).toEqual(
+ searchAlert.updatedAt,
+ );
+
+ const dangling = await byId(danglingAlert._id);
+ expect(dangling?.displayName).toBeUndefined();
+ expect(dangling?.tags).toBeUndefined();
+ });
+
+ it('is idempotent: re-running fills newly missing fields and never touches populated ones', async () => {
+ const team = await createTeam({ name: 'Test team' });
+ const savedSearch = await makeSavedSearch(team._id, { name: 'First name' });
+ const [alert, otherAlert] = await Alert.create([
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.SAVED_SEARCH,
+ savedSearch: savedSearch._id,
+ },
+ {
+ ...baseAlert,
+ team: team._id,
+ source: AlertSource.SAVED_SEARCH,
+ savedSearch: savedSearch._id,
+ },
+ ]);
+
+ await backfillAlertDisplayFields();
+ expect((await Alert.findById(alert._id).lean())?.displayName).toBe(
+ 'First name',
+ );
+
+ await SavedSearch.updateOne(
+ { _id: savedSearch._id },
+ { name: 'Second name' },
+ );
+ await Alert.updateOne(
+ { _id: otherAlert._id },
+ { $unset: { displayName: '' } },
+ );
+
+ await backfillAlertDisplayFields();
+ expect((await Alert.findById(alert._id).lean())?.displayName).toBe(
+ 'First name',
+ );
+ expect((await Alert.findById(otherAlert._id).lean())?.displayName).toBe(
+ 'Second name',
+ );
+ });
+});
diff --git a/packages/api/src/__tests__/proxyAuthConfig.test.ts b/packages/api/src/__tests__/proxyAuthConfig.test.ts
new file mode 100644
index 0000000000..be703221cd
--- /dev/null
+++ b/packages/api/src/__tests__/proxyAuthConfig.test.ts
@@ -0,0 +1,186 @@
+// gascity fork: startup-validation tests for the proxy-auth config
+// (config.ts). Kept in a fork-owned file so upstream's config.test.ts stays
+// untouched (it conflicted on every rebase when we appended to it).
+/* eslint-disable security/detect-object-injection -- env names are test constants */
+describe('proxy auth shared-secret rotation', () => {
+ type ProxyAuthConfig = {
+ PROXY_AUTH_SHARED_SECRET: string;
+ PROXY_AUTH_SHARED_SECRET_PREVIOUS: string;
+ };
+
+ const ORIGINAL_PROXY_AUTH_ENABLED = process.env.PROXY_AUTH_ENABLED;
+ const ORIGINAL_SHARED_SECRET = process.env.PROXY_AUTH_SHARED_SECRET;
+ const ORIGINAL_PREVIOUS_SHARED_SECRET =
+ process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS;
+
+ function restoreEnv(name: string, value: string | undefined) {
+ if (value === undefined) {
+ delete process.env[name];
+ } else {
+ process.env[name] = value;
+ }
+ }
+
+ function loadConfig(): ProxyAuthConfig {
+ let config: ProxyAuthConfig | undefined;
+ jest.isolateModules(() => {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports, n/no-missing-require
+ config = require('@/config');
+ });
+ if (!config) throw new Error('config module did not load');
+ return config;
+ }
+
+ afterEach(() => {
+ restoreEnv('PROXY_AUTH_ENABLED', ORIGINAL_PROXY_AUTH_ENABLED);
+ restoreEnv('PROXY_AUTH_SHARED_SECRET', ORIGINAL_SHARED_SECRET);
+ restoreEnv(
+ 'PROXY_AUTH_SHARED_SECRET_PREVIOUS',
+ ORIGINAL_PREVIOUS_SHARED_SECRET,
+ );
+ jest.resetModules();
+ });
+
+ it('keeps proxy-disabled startup behavior when neither secret is configured', () => {
+ process.env.PROXY_AUTH_ENABLED = 'false';
+ delete process.env.PROXY_AUTH_SHARED_SECRET;
+ delete process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS;
+
+ expect(() => loadConfig()).not.toThrow();
+ });
+
+ it('leaves rotation-value validation inert when proxy auth is disabled', () => {
+ process.env.PROXY_AUTH_ENABLED = 'false';
+ process.env.PROXY_AUTH_SHARED_SECRET = 'same-disabled-value';
+ process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS = 'same-disabled-value';
+
+ expect(() => loadConfig()).not.toThrow();
+ });
+
+ it('requires the current secret even when a previous secret is configured', () => {
+ const previousSecret = 'previous-value-must-not-leak';
+ process.env.PROXY_AUTH_ENABLED = 'true';
+ delete process.env.PROXY_AUTH_SHARED_SECRET;
+ process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS = previousSecret;
+
+ let error: unknown;
+ try {
+ loadConfig();
+ } catch (caught) {
+ error = caught;
+ }
+
+ expect(error).toBeInstanceOf(Error);
+ expect(String(error)).not.toContain(previousSecret);
+ });
+
+ it('starts with distinct current and previous secrets', () => {
+ process.env.PROXY_AUTH_ENABLED = 'true';
+ process.env.PROXY_AUTH_SHARED_SECRET = 'current-s3cret-value';
+ process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS = 'previous-s3cret-value';
+
+ const config = loadConfig();
+
+ expect(config.PROXY_AUTH_SHARED_SECRET).toBe('current-s3cret-value');
+ expect(config.PROXY_AUTH_SHARED_SECRET_PREVIOUS).toBe(
+ 'previous-s3cret-value',
+ );
+ });
+
+ it('starts with an empty previous secret when a current secret is configured', () => {
+ process.env.PROXY_AUTH_ENABLED = 'true';
+ process.env.PROXY_AUTH_SHARED_SECRET = 'current-s3cret-value';
+ process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS = '';
+
+ const config = loadConfig();
+
+ expect(config.PROXY_AUTH_SHARED_SECRET).toBe('current-s3cret-value');
+ expect(config.PROXY_AUTH_SHARED_SECRET_PREVIOUS).toBe('');
+ });
+
+ it('rejects equal current and previous secrets without leaking the value', () => {
+ const sharedSecret = 'same-value-must-not-leak';
+ process.env.PROXY_AUTH_ENABLED = 'true';
+ process.env.PROXY_AUTH_SHARED_SECRET = sharedSecret;
+ process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS = sharedSecret;
+
+ let error: unknown;
+ try {
+ loadConfig();
+ } catch (caught) {
+ error = caught;
+ }
+
+ expect(error).toBeInstanceOf(Error);
+ expect(String(error)).not.toContain(sharedSecret);
+ });
+});
+
+// The bundled OpenTelemetry SDK's advanced network capture records every
+// incoming request header on the server span as the request arrives, before
+// consumeProxyAuthSecret runs, and exports it. Proxy auth must not start with
+// it on. The accepted spellings mirror the SDK's own boolean parsing.
+describe('proxy auth with advanced network capture', () => {
+ const ORIGINAL = {
+ PROXY_AUTH_ENABLED: process.env.PROXY_AUTH_ENABLED,
+ PROXY_AUTH_SHARED_SECRET: process.env.PROXY_AUTH_SHARED_SECRET,
+ PROXY_AUTH_SHARED_SECRET_PREVIOUS:
+ process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS,
+ HDX_NODE_ADVANCED_NETWORK_CAPTURE:
+ process.env.HDX_NODE_ADVANCED_NETWORK_CAPTURE,
+ };
+ const SECRET = 'capture-guard-secret-must-not-leak';
+
+ function loadWithCapture(enabled: string, capture: string | undefined) {
+ process.env.PROXY_AUTH_ENABLED = enabled;
+ process.env.PROXY_AUTH_SHARED_SECRET = SECRET;
+ delete process.env.PROXY_AUTH_SHARED_SECRET_PREVIOUS;
+ if (capture === undefined) {
+ delete process.env.HDX_NODE_ADVANCED_NETWORK_CAPTURE;
+ } else {
+ process.env.HDX_NODE_ADVANCED_NETWORK_CAPTURE = capture;
+ }
+ jest.isolateModules(() => {
+ // eslint-disable-next-line @typescript-eslint/no-require-imports, n/no-missing-require
+ require('@/config');
+ });
+ }
+
+ afterEach(() => {
+ for (const [name, value] of Object.entries(ORIGINAL)) {
+ if (value === undefined) {
+ delete process.env[name];
+ } else {
+ process.env[name] = value;
+ }
+ }
+ jest.resetModules();
+ });
+
+ it.each(['true', '1', 'YES', ' True '])(
+ 'refuses to start when capture is %p, without leaking the secret',
+ capture => {
+ let error: unknown;
+ try {
+ loadWithCapture('true', capture);
+ } catch (caught) {
+ error = caught;
+ }
+
+ expect(error).toBeInstanceOf(Error);
+ expect(String(error)).toContain('HDX_NODE_ADVANCED_NETWORK_CAPTURE');
+ expect(String(error)).not.toContain(SECRET);
+ },
+ );
+
+ it.each([undefined, '', 'false', '0', 'no'])(
+ 'starts when capture is %p',
+ capture => {
+ expect(() => loadWithCapture('true', capture)).not.toThrow();
+ },
+ );
+
+ it('leaves the capture check inert when proxy auth is disabled', () => {
+ expect(() => loadWithCapture('false', 'true')).not.toThrow();
+ });
+});
diff --git a/packages/api/src/__tests__/sessionSecretStartup.test.ts b/packages/api/src/__tests__/sessionSecretStartup.test.ts
new file mode 100644
index 0000000000..9f17c1864e
--- /dev/null
+++ b/packages/api/src/__tests__/sessionSecretStartup.test.ts
@@ -0,0 +1,92 @@
+import { IncomingMessage, ServerResponse } from 'http';
+import { Socket } from 'net';
+
+jest.mock('@/api-app', () => ({ __esModule: true, default: jest.fn() }));
+jest.mock('@/controllers/team', () => ({
+ LOCAL_APP_TEAM: { _id: { toString: () => 'local-team' } },
+}));
+jest.mock('@/migrations', () => ({ runStartupMigrations: jest.fn() }));
+jest.mock('@/models', () => ({
+ connectDB: jest.fn().mockResolvedValue(undefined),
+ connectDBWithRetry: jest.fn().mockResolvedValue(undefined),
+}));
+jest.mock('@/opamp/app', () => ({ __esModule: true, default: jest.fn() }));
+jest.mock('@/setupDefaults', () => ({ setupTeamDefaults: jest.fn() }));
+jest.mock('@/utils/tokenEncryption', () => ({
+ verifyTokenEncryption: jest.fn(),
+}));
+jest.mock('@/utils/logger', () => ({
+ __esModule: true,
+ default: { error: jest.fn(), info: jest.fn() },
+}));
+jest.mock('@/utils/instrumentation', () => ({
+ getCounter: jest.fn(() => ({ add: jest.fn() })),
+}));
+jest.mock('http-graceful-shutdown', () => jest.fn());
+jest.mock('http', () => ({
+ ...jest.requireActual('http'),
+ createServer: jest.fn(() => ({ listen: jest.fn() })),
+}));
+
+describe.each(['server', 'serverless'])(
+ 'session secret startup diagnostics (%s first)',
+ first => {
+ const originalEnv = { ...process.env };
+
+ afterEach(() => {
+ process.env = { ...originalEnv };
+ jest.resetModules();
+ });
+
+ it.each([
+ [false, undefined, 1],
+ [false, '', 1],
+ [false, 'hyperdx is cool 👋', 1],
+ [false, 'startup-secret-sentinel', 0],
+ [true, undefined, 0],
+ [true, 'hyperdx is cool 👋', 0],
+ [true, 'startup-secret-sentinel', 0],
+ ])(
+ 'local=%s, secret=%s logs %s times across both entrypoints',
+ async (local, secret, count) => {
+ process.env.APP_TYPE = 'api';
+ if (local) {
+ process.env.IS_LOCAL_APP_MODE = 'DANGEROUSLY_is_local_app_mode💀';
+ } else {
+ delete process.env.IS_LOCAL_APP_MODE;
+ }
+ if (typeof secret === 'undefined')
+ delete process.env.EXPRESS_SESSION_SECRET;
+ else process.env.EXPRESS_SESSION_SECRET = secret;
+
+ const { default: logger } =
+ jest.requireMock('@/utils/logger');
+ const { getCounter } = jest.requireMock<
+ typeof import('@/utils/instrumentation')
+ >('@/utils/instrumentation');
+ const { default: Server } =
+ jest.requireActual('@/server');
+ const { serverlessHandler } =
+ jest.requireActual('@/serverless');
+ const config =
+ jest.requireActual('@/config');
+ expect(logger.error).not.toHaveBeenCalled();
+
+ const req = new IncomingMessage(new Socket());
+ req.url = '/api/health';
+ const res = new ServerResponse(req);
+ if (first === 'server') await new Server().start();
+ await serverlessHandler(req, res);
+ await new Server().start();
+ await serverlessHandler(req, res);
+
+ expect(logger.error).toHaveBeenCalledTimes(count);
+ const counter = jest.mocked(getCounter).mock.results[0].value;
+ expect(counter.add).toHaveBeenCalledTimes(count);
+ expect(
+ JSON.stringify(jest.mocked(logger.error).mock.calls),
+ ).not.toContain(config.EXPRESS_SESSION_SECRET);
+ },
+ );
+ },
+);
diff --git a/packages/api/src/__tests__/setupDefaults.test.ts b/packages/api/src/__tests__/setupDefaults.test.ts
new file mode 100644
index 0000000000..f7b42bc64f
--- /dev/null
+++ b/packages/api/src/__tests__/setupDefaults.test.ts
@@ -0,0 +1,87 @@
+import { SourceKind } from '@hyperdx/common-utils/dist/types';
+
+const configState = { IS_PROMQL_ENABLED: false };
+
+jest.mock('@/config', () => ({
+ DEFAULT_CONNECTIONS: undefined,
+ DEFAULT_SOURCES: JSON.stringify([
+ {
+ kind: 'log',
+ name: 'Logs',
+ connection: 'Local ClickHouse',
+ from: { databaseName: 'default', tableName: 'otel_logs' },
+ timestampValueExpression: 'Timestamp',
+ defaultTableSelectExpression: 'Timestamp, Body',
+ },
+ {
+ kind: 'promql',
+ name: 'PromQL',
+ connection: 'Local ClickHouse',
+ from: { databaseName: 'default', tableName: 'metrics_ts' },
+ timestampValueExpression: 'timestamp',
+ },
+ ]),
+ get IS_PROMQL_ENABLED() {
+ return configState.IS_PROMQL_ENABLED;
+ },
+}));
+
+jest.mock('@/controllers/team', () => ({
+ getTeam: jest.fn().mockResolvedValue({ _id: 'team-1' }),
+}));
+jest.mock('@/controllers/connection', () => ({
+ createConnection: jest.fn(),
+ getConnections: jest
+ .fn()
+ .mockResolvedValue([
+ { _id: 'conn-1', team: 'team-1', name: 'Local ClickHouse' },
+ ]),
+}));
+jest.mock('@/controllers/sources', () => ({
+ getSources: jest.fn().mockResolvedValue([]),
+ createSource: jest.fn(async (_teamId: string, source: { name: string }) => ({
+ _id: `${source.name}-id`,
+ toObject: () => source,
+ })),
+ updateSource: jest.fn(),
+}));
+jest.mock('@/utils/logger', () => ({
+ __esModule: true,
+ default: { info: jest.fn(), warn: jest.fn(), error: jest.fn() },
+}));
+
+import { createSource } from '@/controllers/sources';
+import { setupTeamDefaults } from '@/setupDefaults';
+
+const createdKinds = () =>
+ jest.mocked(createSource).mock.calls.map(([, source]) => source.kind);
+
+describe('setupTeamDefaults', () => {
+ beforeEach(() => {
+ jest.mocked(createSource).mockClear();
+ });
+
+ it('creates a default PromQL source when PromQL is enabled', async () => {
+ configState.IS_PROMQL_ENABLED = true;
+
+ await setupTeamDefaults('team-1');
+
+ expect(createdKinds()).toEqual([SourceKind.Log, SourceKind.Promql]);
+ expect(createSource).toHaveBeenLastCalledWith(
+ 'team-1',
+ expect.objectContaining({
+ kind: SourceKind.Promql,
+ connection: 'conn-1',
+ from: { databaseName: 'default', tableName: 'metrics_ts' },
+ }),
+ );
+ });
+
+ it('skips a default PromQL source when PromQL is disabled', async () => {
+ configState.IS_PROMQL_ENABLED = false;
+
+ await setupTeamDefaults('team-1');
+
+ expect(createdKinds()).toEqual([SourceKind.Log]);
+ });
+});
diff --git a/packages/api/src/api-app.ts b/packages/api/src/api-app.ts
index 5f4c3181d4..501b2f9af9 100644
--- a/packages/api/src/api-app.ts
+++ b/packages/api/src/api-app.ts
@@ -9,17 +9,23 @@ import mcpRouter from './mcp/app';
import { isUserAuthenticated } from './middleware/auth';
import defaultCors from './middleware/cors';
import { appErrorHandler } from './middleware/error';
+import { consumeProxyAuthSecret } from './middleware/proxyHeaderAuth';
import routers from './routers/api';
import clickhouseProxyRouter from './routers/api/clickhouseProxy';
import connectionsRouter from './routers/api/connections';
import favoritesRouter from './routers/api/favorites';
+import iacRouter from './routers/api/iac';
import pinnedFiltersRouter from './routers/api/pinnedFilters';
import savedSearchRouter from './routers/api/savedSearch';
import sourcesRouter from './routers/api/sources';
import externalRoutersV2 from './routers/external-api/v2';
import usageStats from './tasks/usageStats';
+import { registerAITelemetry } from './utils/aiTelemetry';
import logger, { expressLogger } from './utils/logger';
import passport from './utils/passport';
+import { captureProxyBody } from './utils/proxyBody';
+
+registerAITelemetry();
const app: express.Application = express();
@@ -51,10 +57,19 @@ if (!config.IS_CI && config.FRONTEND_URL) {
}
app.disable('x-powered-by');
+// gascity fork: take the proxy-auth shared secret off the request before any
+// middleware can log or forward it (see middleware/proxyHeaderAuth.ts).
+app.use(consumeProxyAuthSecret);
app.use(compression());
-app.use(express.json({ limit: '32mb' }));
-app.use(express.text({ limit: '32mb' }));
-app.use(express.urlencoded({ extended: false, limit: '32mb' }));
+app.use(express.json({ limit: '32mb', verify: captureProxyBody }));
+app.use(express.text({ limit: '32mb', verify: captureProxyBody }));
+app.use(
+ express.urlencoded({
+ extended: false,
+ limit: '32mb',
+ verify: captureProxyBody,
+ }),
+);
app.use(session(sess));
if (!config.IS_LOCAL_APP_MODE) {
@@ -108,6 +123,7 @@ app.use('/saved-search', isUserAuthenticated, savedSearchRouter);
app.use('/favorites', isUserAuthenticated, favoritesRouter);
app.use('/pinned-filters', isUserAuthenticated, pinnedFiltersRouter);
app.use('/clickhouse-proxy', isUserAuthenticated, clickhouseProxyRouter);
+app.use('/iac', isUserAuthenticated, iacRouter);
if (config.IS_PROMQL_ENABLED) {
app.use('/v1/prometheus', isUserAuthenticated, routers.prometheusRouter);
}
diff --git a/packages/api/src/clickhouse.ts b/packages/api/src/clickhouse.ts
new file mode 100644
index 0000000000..f9bb00910c
--- /dev/null
+++ b/packages/api/src/clickhouse.ts
@@ -0,0 +1,26 @@
+import type {
+ ClickhouseClientOptions,
+ Logger as ClickhouseLogger,
+} from '@hyperdx/common-utils/dist/clickhouse';
+import { ClickhouseClient as BaseClickhouseClient } from '@hyperdx/common-utils/dist/clickhouse/node';
+
+import logger from '@/utils/logger';
+
+const customLogger: ClickhouseLogger = {
+ trace: ({ module, message, args }) =>
+ logger.trace({ ...args, module }, message),
+ debug: ({ module, message, args }) =>
+ logger.debug({ ...args, module }, message),
+ info: ({ module, message, args }) =>
+ logger.info({ ...args, module }, message),
+ warn: ({ module, message, args, err }) =>
+ logger.warn({ ...args, module, err }, message),
+ error: ({ module, message, args, err }) =>
+ logger.error({ ...args, module, err }, message),
+};
+
+export class ClickhouseClient extends BaseClickhouseClient {
+ constructor(options: ClickhouseClientOptions) {
+ super({ ...options, customLogger: options.customLogger ?? customLogger });
+ }
+}
diff --git a/packages/api/src/clickhouse/__tests__/__snapshots__/renderChartConfig.test.ts.snap b/packages/api/src/clickhouse/__tests__/__snapshots__/renderChartConfig.int.test.ts.snap
similarity index 98%
rename from packages/api/src/clickhouse/__tests__/__snapshots__/renderChartConfig.test.ts.snap
rename to packages/api/src/clickhouse/__tests__/__snapshots__/renderChartConfig.int.test.ts.snap
index e40f178c51..5cb786df24 100644
--- a/packages/api/src/clickhouse/__tests__/__snapshots__/renderChartConfig.test.ts.snap
+++ b/packages/api/src/clickhouse/__tests__/__snapshots__/renderChartConfig.int.test.ts.snap
@@ -426,6 +426,15 @@ exports[`renderChartConfig Query Metrics - Sum calculates min_rate/max_rate corr
]
`;
+exports[`renderChartConfig Query Metrics - Sum cross-scope same-key attributes split into two series under variadic AttributesHash (HDX-4466) 1`] = `
+[
+ {
+ "Value": 0,
+ "__hdx_time_bucket": "2022-01-05T00:00:00Z",
+ },
+]
+`;
+
exports[`renderChartConfig Query Metrics - Sum handles counter resets correctly for sum metrics 1`] = `
[
{
diff --git a/packages/api/src/clickhouse/__tests__/renderChartConfig.int.test.ts b/packages/api/src/clickhouse/__tests__/renderChartConfig.int.test.ts
new file mode 100644
index 0000000000..4f93e19282
--- /dev/null
+++ b/packages/api/src/clickhouse/__tests__/renderChartConfig.int.test.ts
@@ -0,0 +1,4857 @@
+// TODO: we might want to move this test file to common-utils package
+
+import { ChSql, chSql } from '@hyperdx/common-utils/dist/clickhouse';
+import { ClickhouseClient } from '@hyperdx/common-utils/dist/clickhouse/node';
+import { getMetadata } from '@hyperdx/common-utils/dist/core/metadata';
+import { renderChartConfig } from '@hyperdx/common-utils/dist/core/renderChartConfig';
+import {
+ DisplayType,
+ MetricsDataType,
+ QuerySettings,
+} from '@hyperdx/common-utils/dist/types';
+import _ from 'lodash';
+import ms from 'ms';
+
+import * as config from '@/config';
+import { createTeam } from '@/controllers/team';
+import {
+ bucketExponentialHistogramObservations,
+ bulkInsertLogs,
+ bulkInsertMetricsGauge,
+ bulkInsertMetricsHistogram,
+ bulkInsertMetricsSum,
+ DEFAULT_DATABASE,
+ DEFAULT_LOGS_TABLE,
+ DEFAULT_METRICS_TABLE,
+ executeSqlCommand,
+ getServer,
+ seedExponentialHistogramMetric,
+} from '@/fixtures';
+import Connection from '@/models/connection';
+import { Source } from '@/models/source';
+
+const TEST_METRIC_TABLES = {
+ sum: DEFAULT_METRICS_TABLE.SUM,
+ gauge: DEFAULT_METRICS_TABLE.GAUGE,
+ histogram: DEFAULT_METRICS_TABLE.HISTOGRAM,
+ summary: DEFAULT_METRICS_TABLE.SUMMARY,
+ 'exponential histogram': DEFAULT_METRICS_TABLE.EXPONENTIAL_HISTOGRAM,
+};
+
+const querySettings: QuerySettings = [
+ { setting: 'optimize_read_in_order', value: '0' },
+ { setting: 'cast_keep_nullable', value: '1' },
+ { setting: 'count_distinct_implementation', value: 'uniqCombined64' },
+ { setting: 'async_insert_busy_timeout_min_ms', value: '20000' },
+];
+
+describe('renderChartConfig', () => {
+ const server = getServer();
+
+ const now = new Date('2022-01-05').getTime();
+ let team, connection, logSource, metricSource, metadata;
+ let clickhouseClient: ClickhouseClient;
+
+ const nowPlus = time_val => new Date(now + ms(time_val));
+ const toClickHouseISOString = (date: Date) =>
+ date.toISOString().replace('.000Z', 'Z');
+
+ const queryData = async (chsql: ChSql) => {
+ try {
+ const res = await clickhouseClient.query<'JSON'>({
+ query: chsql.sql,
+ query_params: chsql.params,
+ format: 'JSON',
+ });
+ const json = await res.json();
+ return json.data;
+ } catch (err) {
+ console.error('[ClickhouseClient] Error:', err);
+ throw err;
+ }
+ };
+
+ const expectNonTimeseriesResults = (
+ results: unknown[],
+ expected: object[],
+ ) => {
+ expect(results).toHaveLength(expected.length);
+ expect(results).toEqual(expect.arrayContaining(expected));
+ expect(
+ results.every(
+ result =>
+ typeof result === 'object' &&
+ result !== null &&
+ !('__hdx_time_bucket' in result),
+ ),
+ ).toBe(true);
+ };
+
+ beforeAll(async () => {
+ await server.start();
+ });
+
+ afterAll(async () => {
+ await server.stop();
+ });
+
+ beforeEach(async () => {
+ team = await createTeam({ name: 'My Team' });
+ connection = await Connection.create({
+ team: team._id,
+ name: 'Default',
+ host: config.CLICKHOUSE_HOST,
+ username: config.CLICKHOUSE_USER,
+ password: config.CLICKHOUSE_PASSWORD,
+ });
+ logSource = await Source.create({
+ kind: 'log',
+ team: team._id,
+ from: {
+ databaseName: DEFAULT_DATABASE,
+ tableName: DEFAULT_LOGS_TABLE,
+ },
+ timestampValueExpression: 'Timestamp',
+ connection: connection.id,
+ name: 'Logs',
+ });
+ metricSource = await Source.create({
+ kind: 'metric',
+ team: team._id,
+ from: {
+ databaseName: DEFAULT_DATABASE,
+ tableName: '',
+ },
+ timestampValueExpression: 'TimeUnix',
+ connection: connection.id,
+ name: 'OTLPMetrics',
+ });
+ clickhouseClient = new ClickhouseClient({
+ host: connection.host,
+ username: connection.username,
+ password: connection.password,
+ });
+ metadata = getMetadata(clickhouseClient);
+ });
+
+ afterEach(async () => {
+ await server.clearDBs();
+ jest.clearAllMocks();
+ });
+
+ describe('aggFn', () => {
+ afterAll(async () => {
+ await executeSqlCommand('DROP TABLE IF EXISTS agg_fn_str_test');
+ await executeSqlCommand('DROP TABLE IF EXISTS agg_fn_default_test');
+ });
+
+ it('numeric agg functions should handle numeric values as strings', async () => {
+ await executeSqlCommand(`
+ CREATE TABLE agg_fn_str_test(
+ ts UInt64,
+ strVal String
+ ) ENGINE = MergeTree
+ ORDER BY ts
+ `);
+ await executeSqlCommand(`
+ INSERT INTO agg_fn_str_test(ts, strVal) VALUES
+ (fromUnixTimestamp64Milli(1519211811570), '3'),
+ (fromUnixTimestamp64Milli(1519211811770), '-1'),
+ (fromUnixTimestamp64Milli(1519211811870), '1.1'),
+ (fromUnixTimestamp64Milli(1519211811970), '-1.1'),
+ `);
+
+ const query = await renderChartConfig(
+ {
+ select: [
+ { aggFn: 'avg', valueExpression: 'strVal' },
+ { aggFn: 'max', valueExpression: 'strVal' },
+ { aggFn: 'min', valueExpression: 'strVal' },
+ { aggFn: 'quantile', level: 0.5, valueExpression: 'strVal' },
+ { aggFn: 'sum', valueExpression: 'strVal' },
+ ],
+ from: {
+ databaseName: DEFAULT_DATABASE,
+ tableName: `agg_fn_str_test`,
+ },
+ where: '',
+ connection: connection.id,
+ timestampValueExpression: 'ts',
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+
+ it('numeric agg functions should use default values for other types', async () => {
+ await executeSqlCommand(`
+ CREATE TABLE agg_fn_default_test(
+ ts UInt64,
+ strVal String,
+ boolVal Bool,
+ enumVal Enum('a' = 1, 'b' = 2),
+ nullVal Nullable(String),
+ ) ENGINE = MergeTree
+ ORDER BY ts
+ `);
+ await executeSqlCommand(`
+ INSERT INTO agg_fn_default_test(ts, strVal, boolVal, enumVal, nullVal) VALUES
+ (fromUnixTimestamp64Milli(1519211811570), 'a', false, 'b', NULL)
+ `);
+ const query = await renderChartConfig(
+ {
+ select: [
+ { aggFn: 'avg', valueExpression: 'strVal' },
+ { aggFn: 'max', valueExpression: 'strVal' },
+ { aggFn: 'min', valueExpression: 'strVal' },
+ { aggFn: 'quantile', level: 0.5, valueExpression: 'strVal' },
+ { aggFn: 'sum', valueExpression: 'strVal' },
+ ],
+ from: {
+ databaseName: DEFAULT_DATABASE,
+ tableName: `agg_fn_default_test`,
+ },
+ where: '',
+ connection: connection.id,
+ timestampValueExpression: 'ts',
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+ });
+
+ describe('Query Events - Logs', () => {
+ it('simple select + where query logs', async () => {
+ const now = new Date('2023-11-16T22:12:00.000Z');
+ await bulkInsertLogs([
+ {
+ ServiceName: 'api',
+ Timestamp: now,
+ SeverityText: 'error',
+ Body: 'Oh no! Something went wrong!',
+ },
+ {
+ ServiceName: 'api',
+ Timestamp: now,
+ SeverityText: 'info',
+ Body: 'This is a test message.',
+ },
+ ]);
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ valueExpression: 'Body',
+ },
+ ],
+ from: logSource.from,
+ where: `SeverityText = 'error'`,
+ timestampValueExpression: 'Timestamp',
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ it('simple select + group by query logs', async () => {
+ const now = new Date('2023-11-16T22:12:00.000Z');
+ await bulkInsertLogs([
+ {
+ ServiceName: 'api',
+ Timestamp: now,
+ SeverityText: 'error',
+ Body: 'Oh no! Something went wrong!',
+ },
+ {
+ ServiceName: 'app',
+ Timestamp: now,
+ SeverityText: 'info',
+ Body: 'This is a test message.',
+ },
+ ]);
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ valueExpression: 'ServiceName',
+ },
+ {
+ valueExpression: 'count()',
+ alias: 'count',
+ },
+ ],
+ from: logSource.from,
+ where: '',
+ timestampValueExpression: 'Timestamp',
+ connection: connection.id,
+ groupBy: 'ServiceName',
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ // TODO: add more tests (including events chart, using filters, etc)
+ });
+
+ describe('Query Metrics - Gauge', () => {
+ beforeEach(async () => {
+ const gaugePointsA = [
+ { value: 50, timestamp: now },
+ { value: 25, timestamp: now + ms('1m') },
+ { value: 12.5, timestamp: now + ms('2m') },
+ { value: 6.25, timestamp: now + ms('3m') },
+ { value: 100, timestamp: now + ms('6m') },
+ { value: 75, timestamp: now + ms('7m') },
+ { value: 10, timestamp: now + ms('8m') },
+ { value: 80, timestamp: now + ms('9m') },
+ ].map(point => ({
+ MetricName: 'test.cpu',
+ ServiceName: 'db',
+ ResourceAttributes: {
+ host: 'host1',
+ ip: '127.0.0.1',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ }));
+ const gaugePointsB = [
+ { value: 1, timestamp: now },
+ { value: 2, timestamp: now + ms('1m') },
+ { value: 3, timestamp: now + ms('2m') },
+ { value: 4, timestamp: now + ms('3m') },
+ { value: 5, timestamp: now + ms('6m') },
+ { value: 6, timestamp: now + ms('7m') },
+ { value: 5, timestamp: now + ms('8m') },
+ { value: 4, timestamp: now + ms('9m') },
+ ].map(point => ({
+ MetricName: 'test.cpu',
+ ServiceName: 'db',
+ ResourceAttributes: {
+ host: 'host2',
+ ip: '127.0.2',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ }));
+ await bulkInsertMetricsGauge([...gaugePointsA, ...gaugePointsB]);
+ });
+
+ it('single max/avg/sum gauge', async () => {
+ const avgQuery = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'avg',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(avgQuery)).toMatchSnapshot();
+ const maxQuery = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'max',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(maxQuery)).toMatchSnapshot();
+ const sumQuery = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'sum',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(sumQuery)).toMatchSnapshot();
+ });
+
+ it('single avg gauge with where', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'avg',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: 'ServiceName:"db" AND ResourceAttributes.host:"host1"',
+ whereLanguage: 'lucene',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ it('single avg gauge with group-by', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'avg',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ groupBy: `ResourceAttributes['host']`,
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ it('single max gauge with delta', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'max',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ isDelta: true,
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ it('single max gauge with delta and group by', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'max',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ isDelta: true,
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ groupBy: `ResourceAttributes['host']`,
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ describe('with a dotted materialized column', () => {
+ const table = `${DEFAULT_DATABASE}.${DEFAULT_METRICS_TABLE.GAUGE}`;
+
+ beforeEach(async () => {
+ await executeSqlCommand(
+ `ALTER TABLE ${table} ADD COLUMN IF NOT EXISTS \`host.name\` String MATERIALIZED ResourceAttributes['host']`,
+ );
+ await executeSqlCommand(
+ `ALTER TABLE ${table} ADD COLUMN IF NOT EXISTS value_band String ALIAS if(Value > 20, 'high', 'low')`,
+ );
+ });
+
+ afterEach(async () => {
+ await executeSqlCommand(
+ `ALTER TABLE ${table} DROP COLUMN IF EXISTS \`host.name\``,
+ );
+ await executeSqlCommand(
+ `ALTER TABLE ${table} DROP COLUMN IF EXISTS value_band`,
+ );
+ });
+
+ it('filters and groups on the column', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'max',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: 'host.name:"host1"',
+ whereLanguage: 'lucene',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ groupBy: '`host.name`',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toEqual(
+ Array(2).fill(expect.objectContaining({ 'host.name': 'host1' })),
+ );
+ });
+
+ it('keeps a group per value of a column that varies within a series', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'max',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: 'host.name:"host1"',
+ whereLanguage: 'lucene',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ groupBy: 'value_band',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const rows = await queryData>(query);
+ expect(rows.map(r => r.value_band).sort()).toEqual([
+ 'high',
+ 'high',
+ 'low',
+ 'low',
+ ]);
+ });
+
+ it('does not split series by a computed column the chart does not group by', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'max',
+ metricName: 'test.cpu',
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: 'host.name:"host1"',
+ whereLanguage: 'lucene',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(
+ (await queryData>(query)).map(
+ r => r['max(toFloat64OrDefault(toString(LastValue)))'],
+ ),
+ ).toEqual([6.25, 80]);
+ });
+ });
+ });
+
+ describe('Query Metrics - Sum', () => {
+ beforeEach(async () => {
+ // Rate: 8, 1, 8, 25
+ const sumPointsA = [
+ { value: 0, timestamp: now - ms('1m') }, // 0
+ { value: 1, timestamp: now },
+ { value: 8, timestamp: now + ms('4m') }, // 8
+ { value: 8, timestamp: now + ms('6m') },
+ { value: 9, timestamp: now + ms('9m') }, // 9
+ { value: 15, timestamp: now + ms('11m') },
+ { value: 17, timestamp: now + ms('14m') }, // 17
+ { value: 32, timestamp: now + ms('16m') },
+ { value: 42, timestamp: now + ms('19m') }, // 42
+ ].map(point => ({
+ MetricName: 'test.users',
+ ServiceName: 'api',
+ ResourceAttributes: {
+ host: 'host1',
+ ip: '127.0.0.1',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ }));
+ // Rate: 11, 78, 5805, 78729
+ // Sum: 12, 79, 5813, 78754
+ const sumPointsB = [
+ { value: 3, timestamp: now - ms('1m') }, // 3
+ { value: 3, timestamp: now },
+ { value: 14, timestamp: now + ms('4m') }, // 14
+ { value: 15, timestamp: now + ms('6m') },
+ { value: 92, timestamp: now + ms('9m') }, // 92
+ { value: 653, timestamp: now + ms('11m') },
+ { value: 5897, timestamp: now + ms('14m') }, // 5897
+ { value: 9323, timestamp: now + ms('16m') },
+ { value: 84626, timestamp: now + ms('19m') }, // 84626
+ ].map(point => ({
+ MetricName: 'test.users',
+ ServiceName: 'api',
+ ResourceAttributes: {
+ host: 'host2',
+ ip: '127.0.0.1',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ }));
+ const sumPointsC = [
+ { value: 0, timestamp: now - ms('1m') }, // 0
+ { value: 1, timestamp: now },
+ { value: 8, timestamp: now + ms('1m') }, // 8
+ { value: 0, timestamp: now + ms('2m') },
+ { value: 7, timestamp: now + ms('2m') },
+ { value: 7, timestamp: now + ms('10m') }, // 9
+ { value: 15, timestamp: now + ms('12m') },
+ { value: 17, timestamp: now + ms('14m') }, // 17
+ { value: 0, timestamp: now + ms('16m') },
+ { value: 42, timestamp: now + ms('19m') }, // 42
+ ].map(point => ({
+ MetricName: 'counter.reset',
+ ServiceName: 'api',
+ ResourceAttributes: {
+ host: 'host3',
+ ip: '127.0.1',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ }));
+ const sumPointsD = [
+ { value: 0, timestamp: now - ms('1m') }, // 0
+ { value: 1, timestamp: now },
+ { value: 8, timestamp: now + ms('1m') }, // 8
+ { value: 0, timestamp: now + ms('2m') },
+ { value: 7, timestamp: now + ms('2m') },
+ { value: 7, timestamp: now + ms('10m') }, // 9
+ { value: 15, timestamp: now + ms('12m') },
+ { value: 17, timestamp: now + ms('14m') }, // 17
+ { value: 0, timestamp: now + ms('16m') },
+ { value: 42, timestamp: now + ms('19m') }, // 42
+ ].map(point => ({
+ MetricName: 'counter.min_reset',
+ ServiceName: 'api',
+ ResourceAttributes: {
+ host: 'MIN_VARIANT_0',
+ ip: '127.0.1',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ }));
+ const sumPointsE = [
+ { value: 0, timestamp: now - ms('1m') },
+ { value: 2, timestamp: now },
+ { value: 9, timestamp: now + ms('1m') },
+ { value: 0, timestamp: now + ms('2m') },
+ { value: 15, timestamp: now + ms('2m') },
+ { value: 25, timestamp: now + ms('10m') },
+ { value: 35, timestamp: now + ms('12m') },
+ { value: 57, timestamp: now + ms('14m') },
+ { value: 0, timestamp: now + ms('16m') },
+ { value: 92, timestamp: now + ms('19m') },
+ ].map(point => ({
+ MetricName: 'counter.min_reset',
+ ServiceName: 'api',
+ ResourceAttributes: {
+ host: 'MAX_VARIANT_1',
+ ip: '127.0.1',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ }));
+ const podAgePoints = [
+ { Value: 518400, TimeUnix: new Date(now - ms('1m')) },
+ { Value: 604800, TimeUnix: new Date(now) },
+ { Value: 691200, TimeUnix: new Date(now + ms('1m')) },
+ { Value: 777600, TimeUnix: new Date(now + ms('2m')) },
+ { Value: 864000, TimeUnix: new Date(now + ms('3m')) },
+ { Value: 950400, TimeUnix: new Date(now + ms('4m')) },
+ { Value: 1641600, TimeUnix: new Date(now + ms('12m')) },
+ ].map(point => ({
+ MetricName: 'k8s.pod.uptime',
+ ServiceName: 'api',
+ ResourceAttributes: {
+ host: 'cluster-node-1',
+ ip: '127.0.0.1',
+ },
+ IsMonotonic: true,
+ AggregationTemporality: 2,
+ ...point,
+ }));
+ await bulkInsertMetricsSum([
+ ...sumPointsA,
+ ...sumPointsB,
+ ...sumPointsC,
+ ...sumPointsD,
+ ...sumPointsE,
+ ...podAgePoints,
+ ]);
+ });
+
+ it('single sum rate', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'sum',
+ metricName: 'test.users',
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ describe('with a dotted materialized column', () => {
+ const table = `${DEFAULT_DATABASE}.${DEFAULT_METRICS_TABLE.SUM}`;
+
+ beforeEach(async () => {
+ await executeSqlCommand(
+ `ALTER TABLE ${table} ADD COLUMN IF NOT EXISTS \`host.name\` String MATERIALIZED ResourceAttributes['host']`,
+ );
+ await executeSqlCommand(
+ `ALTER TABLE ${table} ADD COLUMN IF NOT EXISTS value_band String ALIAS if(Value > 8, 'high', 'low')`,
+ );
+ });
+
+ afterEach(async () => {
+ await executeSqlCommand(
+ `ALTER TABLE ${table} DROP COLUMN IF EXISTS \`host.name\``,
+ );
+ await executeSqlCommand(
+ `ALTER TABLE ${table} DROP COLUMN IF EXISTS value_band`,
+ );
+ });
+
+ it('keeps a group per value of a column that varies within a series', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'sum',
+ metricName: 'test.users',
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: 'host.name:"host1"',
+ whereLanguage: 'lucene',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '5 minute',
+ groupBy: 'value_band',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const rows = await queryData<{
+ __hdx_time_bucket: string;
+ value_band: string;
+ Value: number;
+ }>(query);
+ const secondBucket = rows.filter(
+ r => new Date(r.__hdx_time_bucket).getTime() === now + ms('5m'),
+ );
+ // The 8 -> 9 step lands in `high`; the rate window must not reset
+ // when the label changes.
+ expect(secondBucket.map(r => [r.value_band, r.Value]).sort()).toEqual([
+ ['high', 1],
+ ['low', 0],
+ ]);
+ });
+
+ it('filters and groups on the column', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'sum',
+ metricName: 'test.users',
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: 'host.name:"host1"',
+ whereLanguage: 'lucene',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '5 minute',
+ groupBy: '`host.name`',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toEqual(
+ Array(4).fill(expect.objectContaining({ 'host.name': 'host1' })),
+ );
+ });
+
+ it('ranks increase groups by the column', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'increase',
+ metricName: 'test.users',
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '5 minute',
+ groupBy: '`host.name`',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toEqual(
+ expect.arrayContaining([
+ expect.objectContaining({ 'host.name': 'host1' }),
+ expect.objectContaining({ 'host.name': 'host2' }),
+ ]),
+ );
+ });
+ });
+
+ it('sum values as without rate computation', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ metricName: 'k8s.pod.uptime',
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: 'ServiceName:api',
+ whereLanguage: 'lucene',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '5 minutes',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ it('handles counter resets correctly for sum metrics', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'sum',
+ metricName: 'counter.reset',
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '10 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(query)).toMatchSnapshot();
+ });
+
+ it('calculates min_rate/max_rate correctly for sum metrics', async () => {
+ // Raw Data is
+ // MIN_VARIANT_0: [0, 1, 8, 0, 7, 7, 15, 17, 0, 42]
+ // MIN_VARIANT_1: [0, 2, 9, 0, 15, 25 35, 57, 0, 92]
+ //
+ // Based on the data inserted in the fixture, the expected stream of values
+ // for each series after adjusting for the zero reset should be:
+ // MIN_VARIANT_0: [0, 1, 8, 8, 15, 15, 23, 25, 25, 67]
+ // MIN_VARIANT_1: [0, 2, 9, 9, 24, 34, 44, 66, 66, 158]
+ //
+ // At the 10 minute buckets, should result in three buckets for each where
+ // the first bucket is outside the query window.
+ // MIN_VARIANT_0: [0], [1, 8, 8, 15], [15, 23, 25, 25, 67]]
+ // MIN_VARIANT_1: [0], [2, 9, 9, 24], [34, 44, 66, 66, 158]]
+ //
+ // When comparing the value at the end of the buckets over the filtered
+ // time frame it should result in the following counts added per bucket as:
+ // MIN_VARIANT_0: [15, 52]
+ // MIN_VARIANT_1: [24, 134]
+ //
+ // These values are what we apply the aggregation functions to.
+ const minQuery = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'min',
+ metricName: 'counter.min_reset',
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '10 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(minQuery)).toMatchSnapshot('minSum');
+
+ const maxQuery = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'max',
+ metricName: 'counter.min_reset',
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '10 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ expect(await queryData(maxQuery)).toMatchSnapshot('maxSum');
+ });
+
+ it('new series appearing mid-window: v2 suppresses first-row spike', async () => {
+ // Scenario: An existing series (customerA) has data from the start,
+ // and a new series (customerB) appears mid-window with a high
+ // cumulative counter value (e.g. 10000). A naive inter-bucket diff
+ // approach would leak the full cumulative value as a spike on the
+ // first bucket of customerB. The v2 lagInFrame-based Rate yields 0
+ // for the first row of each series, preventing the spike.
+
+ // Use a unique metric name to isolate from the beforeEach data.
+ const metricName = 'customer.events.total';
+
+ // customerA: present from the start, steady increase
+ const customerAPoints = [
+ { value: 0, timestamp: now - ms('1m') },
+ { value: 10, timestamp: now + ms('2m') },
+ { value: 20, timestamp: now + ms('7m') },
+ { value: 30, timestamp: now + ms('12m') },
+ { value: 40, timestamp: now + ms('17m') },
+ ].map(point => ({
+ MetricName: metricName,
+ ServiceName: 'api',
+ ResourceAttributes: { customer: 'customerA' },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ }));
+
+ // customerB: appears at +10m with a high cumulative value (10000),
+ // then increases to 10050 by +17m. The real per-bucket increase is
+ // only 50. A naive approach would show 10000 in the bucket where it
+ // first appears.
+ const customerBPoints = [
+ { value: 10000, timestamp: now + ms('10m') },
+ { value: 10020, timestamp: now + ms('12m') },
+ { value: 10050, timestamp: now + ms('17m') },
+ ].map(point => ({
+ MetricName: metricName,
+ ServiceName: 'api',
+ ResourceAttributes: { customer: 'customerB' },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ }));
+
+ await bulkInsertMetricsSum([...customerAPoints, ...customerBPoints]);
+
+ // --- v2 query (current code with lagInFrame) ---
+ const v2Query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'sum',
+ metricName: metricName,
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('20m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const v2Results = await queryData(v2Query);
+
+ // v2 should NOT have a spike — the max bucket value should be much
+ // smaller than customerB's initial cumulative value (10000). The real
+ // per-bucket increases across both series are at most ~50.
+ const v2Values = v2Results.map((r: any) => Number(r.Value));
+ const v2MaxValue = Math.max(...v2Values);
+ expect(v2MaxValue).toBeLessThan(100);
+
+ // Verify we actually got data back (not an empty result set).
+ expect(v2Results.length).toBeGreaterThan(0);
+ });
+
+ it('cross-scope same-key attributes split into two series under variadic AttributesHash (HDX-4466)', async () => {
+ // HDX-4466 pinning test. Two cumulative Sum rows that carry the same
+ // logical attribute set ({service.name: api, host: h1}) but distribute
+ // the host key across different attribute scopes:
+ //
+ // Row A: ResourceAttributes={service.name: api, host: h1}, Attributes={}
+ // Row B: ResourceAttributes={service.name: api}, Attributes={host: h1}
+ //
+ // AttributesHash is now computed as the variadic
+ // cityHash64(ScopeAttributes, ResourceAttributes, Attributes)
+ // for all metric schemas (Map and JSON), so the two rows hash
+ // distinctly: each lands in its own series. Both rows are then the
+ // first row of their respective series, the lagInFrame Rate is NULL
+ // for both, and the bucketed Sum aggregation collapses to 0 for the
+ // bucket holding them.
+ //
+ // Pre-HDX-4466 (Map schema only) this was different: the Map path
+ // wrapped the three maps in mapConcat() before hashing, which
+ // collapsed both rows into a single series. Row B's lagInFrame
+ // captured the cumulative increase (110 - 100 = 10) and the bucket
+ // reported Value=10. The pre-refactor commit pinned that behaviour;
+ // this assertion records the variadic outcome that replaced it.
+ const metricName = 'hdx4466.cross_scope.events.total';
+
+ await bulkInsertMetricsSum([
+ {
+ MetricName: metricName,
+ ServiceName: 'api',
+ ResourceAttributes: { 'service.name': 'api', host: 'h1' },
+ Attributes: {},
+ Value: 100,
+ TimeUnix: new Date(now + ms('1m')),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ },
+ {
+ MetricName: metricName,
+ ServiceName: 'api',
+ ResourceAttributes: { 'service.name': 'api' },
+ Attributes: { host: 'h1' },
+ Value: 110,
+ TimeUnix: new Date(now + ms('2m')),
+ IsMonotonic: true,
+ AggregationTemporality: 2, // Cumulative
+ },
+ ]);
+
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'sum',
+ metricName,
+ metricType: MetricsDataType.Sum,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('5m'))],
+ granularity: '5 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ const results = await queryData(query);
+
+ // Under variadic AttributesHash, both rows are first-in-series, so
+ // no rate is captured and the bucketed Sum collapses to 0. A
+ // regression that reintroduced mapConcat-style collapsing would
+ // produce Value=10 in the populated bucket, failing this check.
+ const totalValue = results.reduce(
+ (acc: number, r: any) => acc + (r.Value == null ? 0 : Number(r.Value)),
+ 0,
+ );
+ expect(totalValue).toBe(0);
+
+ // Snapshot pins the bucketed output for the two-series-split case.
+ expect(results).toMatchSnapshot();
+ });
+ });
+
+ describe('Query Metrics - Histogram', () => {
+ beforeEach(async () => {
+ const histPointsA = [
+ {
+ BucketCounts: [0, 0, 0],
+ Count: 0,
+ TimeUnix: new Date(now),
+ },
+ {
+ BucketCounts: [10, 10, 10],
+ Count: 30,
+ TimeUnix: new Date(now + ms('1m')),
+ },
+ ].map(point => ({
+ MetricName: 'test.two_timestamps_lower_bound',
+ ResourceAttributes: {
+ host: 'test2',
+ ip: '127.0.0.1',
+ },
+ AggregationTemporality: 2, // Cumulative
+ ExplicitBounds: [10, 30],
+ ...point,
+ }));
+ const histPointsB = [
+ {
+ BucketCounts: [0, 0, 0],
+ TimeUnix: new Date(now),
+ },
+ {
+ BucketCounts: [10, 0, 0],
+ TimeUnix: new Date(now + ms('1m')),
+ },
+ ].map(point => ({
+ MetricName: 'test.two_timestamps_lower_bound_inf',
+ ResourceAttributes: {
+ host: 'test2',
+ ip: '127.0.0.1',
+ },
+ AggregationTemporality: 2, // Cumulative
+ ExplicitBounds: [1, 30],
+ ...point,
+ }));
+ const histPointsC = [
+ {
+ BucketCounts: [0, 0, 0],
+ TimeUnix: new Date(now),
+ },
+ {
+ BucketCounts: [0, 0, 10],
+ TimeUnix: new Date(now + ms('1m')),
+ },
+ ].map(point => ({
+ MetricName: 'test.two_timestamps_upper_bound_inf',
+ ResourceAttributes: {
+ host: 'test2',
+ ip: '127.0.0.1',
+ },
+ AggregationTemporality: 2, // Cumulative
+ ExplicitBounds: [0, 30],
+ ...point,
+ }));
+ const histPointsD = [
+ {
+ BucketCounts: [5, 5, 5],
+ TimeUnix: new Date(now),
+ },
+ {
+ BucketCounts: [0, 0, 0],
+ TimeUnix: new Date(now + ms('1m')),
+ },
+ {
+ BucketCounts: [10, 10, 10],
+ TimeUnix: new Date(now + ms('2m')),
+ },
+ ].map(point => ({
+ MetricName: 'test.three_timestamps_bounded',
+ ResourceAttributes: {
+ host: 'test3',
+ ip: '127.0.0.1',
+ },
+ AggregationTemporality: 2, // Cumulative
+ ExplicitBounds: [10, 30],
+ ...point,
+ }));
+ const histPointsE = [
+ {
+ BucketCounts: [1, 1, 1, 1, 1, 1],
+ TimeUnix: new Date(now),
+ ResourceAttributes: { host: 'test-a' },
+ },
+ {
+ BucketCounts: [2, 2, 2, 2, 2, 2],
+ TimeUnix: new Date(now + ms('5s')),
+ ResourceAttributes: { host: 'test-b' },
+ },
+ {
+ BucketCounts: [2, 1, 2, 1, 2, 1],
+ TimeUnix: new Date(now + ms('1m')),
+ ResourceAttributes: { host: 'test-a' },
+ },
+ {
+ BucketCounts: [3, 3, 2, 2, 3, 3],
+ TimeUnix: new Date(now + ms('65s')),
+ ResourceAttributes: { host: 'test-b' },
+ },
+ ].map(point => ({
+ MetricName: 'test.multiple_series',
+ AggregationTemporality: 2, // Cumulative
+ ExplicitBounds: [1, 2, 5, 8, 13],
+ ...point,
+ }));
+ const histPointsF = [
+ ...['host-a', 'host-b'].flatMap(host =>
+ ['service-1', 'service-2', 'service-3'].map(service => ({
+ TimeUnix: nowPlus('-1m'),
+ ResourceAttributes: { host, service },
+ BucketCounts: [0, 0, 0, 0, 0, 0],
+ Count: 0,
+ Sum: 0,
+ })),
+ ),
+ {
+ TimeUnix: new Date(now),
+ ResourceAttributes: { host: 'host-a', service: 'service-1' },
+ BucketCounts: [5, 2, 0, 0, 0, 0],
+ Count: 7,
+ Sum: 18,
+ },
+ {
+ TimeUnix: new Date(now),
+ ResourceAttributes: { host: 'host-a', service: 'service-2' },
+ BucketCounts: [11, 0, 0, 0, 0, 0],
+ Count: 11,
+ Sum: 22,
+ },
+ {
+ TimeUnix: new Date(now),
+ ResourceAttributes: { host: 'host-a', service: 'service-3' },
+ BucketCounts: [0, 5, 0, 0, 0, 0],
+ Count: 5,
+ Sum: 31,
+ },
+ {
+ TimeUnix: new Date(now),
+ ResourceAttributes: { host: 'host-b', service: 'service-1' },
+ BucketCounts: [6, 3, 0, 0, 0, 0],
+ Count: 9,
+ Sum: 37,
+ },
+ {
+ TimeUnix: new Date(now),
+ ResourceAttributes: { host: 'host-b', service: 'service-2' },
+ BucketCounts: [1, 0, 0, 0, 0, 0],
+ Count: 1,
+ Sum: 3,
+ },
+ {
+ TimeUnix: new Date(now),
+ ResourceAttributes: { host: 'host-b', service: 'service-3' },
+ BucketCounts: [4, 2, 0, 0, 0, 0],
+ Count: 6,
+ Sum: 25,
+ },
+ //----------
+ {
+ TimeUnix: nowPlus('30s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-1' },
+ BucketCounts: [9, 3, 1, 0, 0, 0],
+ Count: 13,
+ Sum: 54,
+ },
+ {
+ TimeUnix: nowPlus('34s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-2' },
+ BucketCounts: [17, 2, 0, 0, 0, 0],
+ Count: 19,
+ Sum: 52,
+ },
+ {
+ TimeUnix: nowPlus('38s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-3' },
+ BucketCounts: [1, 7, 0, 0, 0, 0],
+ Count: 8,
+ Sum: 51,
+ },
+ {
+ TimeUnix: nowPlus('29s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-1' },
+ BucketCounts: [12, 3, 0, 0, 0, 0],
+ Count: 15,
+ Sum: 52,
+ },
+ {
+ TimeUnix: nowPlus('31s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-2' },
+ BucketCounts: [9, 1, 0, 0, 0, 0],
+ Count: 10,
+ Sum: 52,
+ },
+ {
+ TimeUnix: nowPlus('45s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-3' },
+ BucketCounts: [4, 6, 0, 0, 0, 0],
+ Count: 10,
+ Sum: 61,
+ },
+ //----------
+ {
+ TimeUnix: nowPlus('90s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-1' },
+ BucketCounts: [9, 6, 6, 4, 2, 0],
+ Count: 27,
+ Sum: 1015,
+ },
+ {
+ TimeUnix: nowPlus('94s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-2' },
+ BucketCounts: [17, 4, 5, 7, 0, 0],
+ Count: 33,
+ Sum: 655,
+ },
+ {
+ TimeUnix: nowPlus('98s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-3' },
+ BucketCounts: [1, 7, 0, 3, 2, 4],
+ Count: 17,
+ Sum: 3741,
+ },
+ {
+ TimeUnix: nowPlus('89s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-1' },
+ BucketCounts: [19, 5, 0, 0, 0, 0],
+ Count: 24,
+ Sum: 85,
+ },
+ {
+ TimeUnix: nowPlus('91s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-2' },
+ BucketCounts: [12, 1, 1, 0, 0, 0],
+ Count: 14,
+ Sum: 109,
+ },
+ {
+ TimeUnix: nowPlus('105s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-3' },
+ BucketCounts: [4, 8, 0, 0, 0, 0],
+ Count: 12,
+ Sum: 79,
+ },
+ //----------
+ {
+ TimeUnix: nowPlus('150s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-1' },
+ BucketCounts: [9, 9, 11, 8, 4, 0],
+ Count: 41,
+ Sum: 1955,
+ },
+ {
+ TimeUnix: nowPlus('154s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-2' },
+ BucketCounts: [17, 6, 10, 14, 0, 0],
+ Count: 47,
+ Sum: 1292,
+ },
+ {
+ TimeUnix: nowPlus('158s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-3' },
+ BucketCounts: [1, 7, 0, 3, 2, 18],
+ Count: 31,
+ Sum: 22459,
+ },
+ {
+ TimeUnix: nowPlus('149s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-1' },
+ BucketCounts: [26, 7, 0, 0, 0, 0],
+ Count: 33,
+ Sum: 120,
+ },
+ {
+ TimeUnix: nowPlus('151s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-2' },
+ BucketCounts: [15, 1, 2, 0, 0, 0],
+ Count: 18,
+ Sum: 155,
+ },
+ {
+ TimeUnix: nowPlus('165s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-3' },
+ BucketCounts: [4, 10, 0, 0, 0, 0],
+ Count: 14,
+ Sum: 95,
+ },
+ //----------
+ {
+ TimeUnix: nowPlus('210s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-1' },
+ BucketCounts: [19, 12, 11, 8, 4, 0],
+ Count: 54,
+ Sum: 2003,
+ },
+ {
+ TimeUnix: nowPlus('214s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-2' },
+ BucketCounts: [37, 10, 10, 14, 0, 0],
+ Count: 71,
+ Sum: 1337,
+ },
+ {
+ TimeUnix: nowPlus('218s'),
+ ResourceAttributes: { host: 'host-a', service: 'service-3' },
+ BucketCounts: [20, 0, 0, 0, 0, 0],
+ Count: 20,
+ Sum: 29,
+ },
+ {
+ TimeUnix: nowPlus('209s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-1' },
+ BucketCounts: [30, 9, 0, 0, 0, 0],
+ Count: 39,
+ Sum: 139,
+ },
+ {
+ TimeUnix: nowPlus('211s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-2' },
+ BucketCounts: [21, 4, 3, 0, 0, 0],
+ Count: 28,
+ Sum: 223,
+ },
+ {
+ TimeUnix: nowPlus('225s'),
+ ResourceAttributes: { host: 'host-b', service: 'service-3' },
+ BucketCounts: [14, 11, 0, 0, 0, 0],
+ Count: 25,
+ Sum: 121,
+ },
+ ].map(point => ({
+ MetricName: 'test.send_latency',
+ AggregationTemporality: 2,
+ ExplicitBounds: [5, 10, 50, 100, 500],
+ ...point,
+ }));
+
+ await bulkInsertMetricsHistogram([
+ ...histPointsA,
+ ...histPointsB,
+ ...histPointsC,
+ ...histPointsD,
+ ...histPointsE,
+ ...histPointsF,
+ ]);
+ });
+
+ it('two_timestamps_bounded histogram (p50)', async () => {
+ /*
+ This test starts with 2 data points with bounds of [10, 30]:
+ t0: [0, 0, 0]
+ t1: [10, 10, 10]
+
+ Since the AggregationTemporality is 2(cumulative), we need to calculate the delta between the two points:
+ delta: [10, 10, 10] - [0, 0, 0] = [10, 10, 10]
+
+ We need to interpolate between the lower and upper bounds of the second bucket:
+ cum sum = [10, 20, 30]
+ rank = 0.5 * 30 = 15 (between bounds 10 - 30)
+ interpolate: 10 + (30 - 10) * ((15 - 10) / (20 - 10)) = 20
+ */
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ level: 0.5,
+ metricName: 'test.two_timestamps_lower_bound',
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('2m'))],
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+
+ it('two_timestamps_bounded histogram (p90)', async () => {
+ /*
+ This test starts with 2 data points with bounds of [10, 30]:
+ t0: [0, 0, 0]
+ t1: [10, 10, 10]
+
+ Since the AggregationTemporality is 2(cumulative), we need to calculate the delta between the two points:
+ delta: [10, 10, 10] - [0, 0, 0] = [10, 10, 10]
+
+ Total observations: 10 + 10 + 10 = 30
+ Cumulative counts: [10, 20, 30]
+ p90 point:
+ Rank = 0.9 * 30 = 27
+ This falls in the third bucket (since 20 < 27 ≤ 30)
+
+ We need to interpolate between the lower and upper bounds of the third bucket:
+ Lower bound: 30
+ Upper bound: Infinity (but we use the upper bound of the previous bucket for the last bucket)
+ Position in bucket: (27 - 20) / (30 - 20) = 0.7
+ Interpolated value: 30 (since it's in the last bucket, we return the upper bound of the previous bucket)
+
+ Thus the first point value would be 0 since it's at the start of the bounds.
+ The second point value would be 30 since that is the 90th percentile point value delta from the first point.
+ */
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ level: 0.9,
+ metricName: 'test.two_timestamps_lower_bound',
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('2m'))],
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+
+ it('two_timestamps_bounded histogram (p25)', async () => {
+ /*
+ This test starts with 2 data points with bounds of [10, 30]:
+ t0: [0, 0, 0]
+ t1: [10, 10, 10]
+
+ Since the AggregationTemporality is 2(cumulative), we need to calculate the delta between the two points:
+ delta: [10, 10, 10] - [0, 0, 0] = [10, 10, 10]
+
+ Total observations: 10 + 10 + 10 = 30
+ Cumulative counts: [10, 20, 30]
+ p25 point:
+ Rank = 0.25 * 30 = 7.5
+ This falls in the first bucket (since 0 < 7.5 ≤ 10)
+
+ We need to interpolate between the lower and upper bounds of the first bucket:
+ Lower bound: 0 (implicit lower bound for first bucket)
+ Upper bound: 10 (first explicit bound)
+ Position in bucket: 7.5 / 10 = 0.75
+ Interpolated value: 0 + 0.75 * (10 - 0) = 7.5
+
+ Since all observations are in the first bucket which has an upper bound of 1:
+ For the first bucket (≤ 0), the algorithm would interpolate, but since all values are in this bucket and it's the lowest bucket, it would return 10
+ Thus the value columns in res should be [0, 10]
+ */
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ level: 0.25,
+ metricName: 'test.two_timestamps_lower_bound',
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('2m'))],
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+
+ it('two_timestamps_lower_bound_inf histogram (p50)', async () => {
+ /*
+ This test starts with 2 data points with bounds of [1, 30]:
+ t0: [0, 0, 0]
+ t1: [10, 0, 0]
+
+ Since the AggregationTemporality is 2(cumulative), we need to calculate the delta between the two points:
+ delta: [10, 0, 0] - [0, 0, 0] = [10, 0, 0]
+
+ Total observations: 10 + 0 + 0 = 10
+ Cumulative counts: [10, 10, 10]
+ p50 point:
+ Rank = 0.5 * 10 = 5
+ This falls in the first bucket (since 5 < 10)
+
+ Since all observations are in the first bucket which has an upper bound of 1:
+ For the first bucket (≤ 0), the algorithm would interpolate, but since all values are in this bucket and it's the lowest bucket, it would return 1
+ Thus the value columns in res should be [0, 1]
+ */
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ level: 0.5,
+ metricName: 'test.two_timestamps_lower_bound_inf',
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('2m'))],
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+
+ it('should bucket correctly when no grouping is defined', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ level: 0.5,
+ metricName: 'test.send_latency',
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('5m'))],
+ granularity: '2 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+
+ it('should bucket correctly when grouping by a single attribute', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ level: 0.5,
+ metricName: 'test.send_latency',
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('5m'))],
+ groupBy: `ResourceAttributes['host']`,
+ granularity: '2 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+
+ it('should bucket correctly when grouping by multiple attributes', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ level: 0.5,
+ metricName: 'test.send_latency',
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('5m'))],
+ groupBy: `ResourceAttributes['host'], ResourceAttributes['service']`,
+ granularity: '2 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+ const res = await queryData(query);
+ expect(res).toMatchSnapshot();
+ });
+
+ describe.each([
+ {
+ description: 'without group by',
+ groupBy: undefined,
+ metricSuffix: 'ungrouped',
+ },
+ {
+ description: 'grouped by ServiceName',
+ groupBy: 'ServiceName',
+ metricSuffix: 'service-name',
+ },
+ {
+ description: 'grouped by an attribute',
+ groupBy: "Attributes['service']",
+ metricSuffix: 'attribute',
+ },
+ ])('$description', ({ groupBy, metricSuffix }) => {
+ const metricName = `test.histogram.${metricSuffix}`;
+
+ beforeEach(async () => {
+ await bulkInsertMetricsHistogram([
+ {
+ MetricName: metricName,
+ ServiceName: 'api',
+ ResourceAttributes: {},
+ Attributes: { service: 'api' },
+ AggregationTemporality: 1,
+ ExplicitBounds: [10],
+ BucketCounts: [2, 0],
+ Count: 2,
+ TimeUnix: new Date(now),
+ },
+ {
+ MetricName: metricName,
+ ServiceName: 'worker',
+ ResourceAttributes: {},
+ Attributes: { service: 'worker' },
+ AggregationTemporality: 1,
+ ExplicitBounds: [10],
+ BucketCounts: [0, 1],
+ Count: 1,
+ TimeUnix: new Date(now),
+ },
+ ]);
+ });
+
+ const queryHistogram = async (
+ select: { aggFn: 'count' } | { aggFn: 'quantile'; level: number },
+ ) => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ ...select,
+ metricName,
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), nowPlus('1m')],
+ groupBy,
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ return queryData(query);
+ };
+
+ it('counts observations', async () => {
+ const results = await queryHistogram({ aggFn: 'count' });
+
+ expect(results).toHaveLength(groupBy ? 2 : 1);
+ expect(results).toEqual(
+ groupBy
+ ? expect.arrayContaining([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ group: ['api'],
+ Value: '2',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ group: ['worker'],
+ Value: '1',
+ },
+ ])
+ : [
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: '3',
+ },
+ ],
+ );
+ });
+
+ it('calculates quantiles', async () => {
+ const results = await queryHistogram({
+ aggFn: 'quantile',
+ level: 0.95,
+ });
+
+ expect(results).toHaveLength(groupBy ? 2 : 1);
+ expect(results).toEqual(
+ groupBy
+ ? expect.arrayContaining([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ group: ['api'],
+ Value: 9.5,
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ group: ['worker'],
+ Value: 10,
+ },
+ ])
+ : [
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: 10,
+ },
+ ],
+ );
+ });
+ });
+
+ const queryNonTimeseriesHistogram = async (
+ metricName: string,
+ select: { aggFn: 'count' } | { aggFn: 'quantile'; level: number },
+ ) => {
+ const query = await renderChartConfig(
+ {
+ displayType: DisplayType.Table,
+ select: [
+ {
+ ...select,
+ metricName,
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), nowPlus('4m')],
+ groupBy: `Attributes['route']`,
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ return queryData(query);
+ };
+
+ it('aggregates histogram count and quantile across every timestamp in each non-timeseries group', async () => {
+ const metricName = 'test.non.timeseries.histogram.grouped';
+ await bulkInsertMetricsHistogram([
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/ten' },
+ AggregationTemporality: 1,
+ ExplicitBounds: [10, 30],
+ BucketCounts: [10, 0, 0],
+ Count: 10,
+ TimeUnix: new Date(now),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/ten' },
+ AggregationTemporality: 1,
+ ExplicitBounds: [10, 30],
+ BucketCounts: [0, 10, 0],
+ Count: 10,
+ TimeUnix: nowPlus('1m'),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/hundred' },
+ AggregationTemporality: 1,
+ ExplicitBounds: [100, 300],
+ BucketCounts: [6, 0, 0],
+ Count: 6,
+ TimeUnix: new Date(now),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/hundred' },
+ AggregationTemporality: 1,
+ ExplicitBounds: [100, 300],
+ BucketCounts: [0, 6, 0],
+ Count: 6,
+ TimeUnix: nowPlus('2m'),
+ },
+ ]);
+
+ // /ten sums 10 + 10 = 20 observations; its standalone p50s are 5 and
+ // 20, while the combined [10, 10, 0] distribution has p50 10.
+ // /hundred sums 6 + 6 = 12; its standalone p50s are 50 and 200, while
+ // the combined [6, 6, 0] distribution has p50 100. The combined
+ // quantiles cannot match if only one timestamp is calculated.
+ expectNonTimeseriesResults(
+ await queryNonTimeseriesHistogram(metricName, { aggFn: 'count' }),
+ [
+ { group: ['/ten'], Value: '20' },
+ { group: ['/hundred'], Value: '12' },
+ ],
+ );
+ expectNonTimeseriesResults(
+ await queryNonTimeseriesHistogram(metricName, {
+ aggFn: 'quantile',
+ level: 0.5,
+ }),
+ [
+ { group: ['/ten'], Value: 10 },
+ { group: ['/hundred'], Value: 100 },
+ ],
+ );
+ });
+
+ it('uses the first visible cumulative histogram point as the baseline', async () => {
+ const metricName = 'test.non.timeseries.histogram.boundaries';
+ await bulkInsertMetricsHistogram([
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ AggregationTemporality: 2,
+ ExplicitBounds: [10],
+ BucketCounts: [2, 0],
+ Count: 2,
+ TimeUnix: nowPlus('-1m'),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ AggregationTemporality: 2,
+ ExplicitBounds: [10],
+ BucketCounts: [2, 2],
+ Count: 4,
+ TimeUnix: new Date(now),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ AggregationTemporality: 2,
+ ExplicitBounds: [10],
+ BucketCounts: [2, 4],
+ Count: 6,
+ TimeUnix: nowPlus('1m'),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ AggregationTemporality: 2,
+ ExplicitBounds: [10],
+ BucketCounts: [102, 104],
+ Count: 206,
+ TimeUnix: nowPlus('3m'),
+ },
+ ]);
+
+ const renderQuery = (select: {
+ aggFn: 'count' | 'quantile';
+ level?: number;
+ }) =>
+ renderChartConfig(
+ {
+ displayType: DisplayType.Number,
+ select: [
+ {
+ ...select,
+ metricName,
+ metricType: MetricsDataType.Histogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), nowPlus('2m')],
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ // Only [0m, 2m] is scanned: the 0m cumulative count 4 is the zero-delta
+ // baseline, and the 1m point contributes 6 - 4 = 2. Its bucket delta is
+ // [2, 4] - [2, 2] = [0, 2], whose p50 is the finite lower boundary 10
+ // of the overflow bucket. The -1m and 3m points contribute nothing.
+ expect(await queryData(await renderQuery({ aggFn: 'count' }))).toEqual([
+ { Value: '2' },
+ ]);
+ expect(
+ await queryData(await renderQuery({ aggFn: 'quantile', level: 0.5 })),
+ ).toEqual([{ Value: 10 }]);
+ });
+
+ it('aggregates reset-adjusted histogram count and quantile across each non-timeseries group', async () => {
+ const metricName = 'test.non.timeseries.histogram.reset.grouped';
+ await bulkInsertMetricsHistogram([
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/reset' },
+ AggregationTemporality: 2,
+ ExplicitBounds: [10, 30],
+ BucketCounts: [2, 0, 0],
+ Count: 2,
+ TimeUnix: new Date(now),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/reset' },
+ AggregationTemporality: 2,
+ ExplicitBounds: [10, 30],
+ BucketCounts: [6, 0, 0],
+ Count: 6,
+ TimeUnix: nowPlus('1m'),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/reset' },
+ AggregationTemporality: 2,
+ ExplicitBounds: [10, 30],
+ BucketCounts: [0, 4, 0],
+ Count: 4,
+ TimeUnix: nowPlus('2m'),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/normal' },
+ AggregationTemporality: 2,
+ ExplicitBounds: [10, 30],
+ BucketCounts: [0, 2, 0],
+ Count: 2,
+ TimeUnix: new Date(now),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/normal' },
+ AggregationTemporality: 2,
+ ExplicitBounds: [10, 30],
+ BucketCounts: [4, 2, 0],
+ Count: 6,
+ TimeUnix: nowPlus('1m'),
+ },
+ {
+ MetricName: metricName,
+ ResourceAttributes: {},
+ Attributes: { route: '/normal' },
+ AggregationTemporality: 2,
+ ExplicitBounds: [10, 30],
+ BucketCounts: [4, 6, 0],
+ Count: 10,
+ TimeUnix: nowPlus('3m'),
+ },
+ ]);
+
+ // /reset uses 0 for its first point, [6, 0, 0] - [2, 0, 0] =
+ // [4, 0, 0] for its second, then detects the decrease and uses the
+ // current [0, 4, 0] after reset. /normal produces those same two
+ // interval distributions by subtraction. Both sum to count 8 and
+ // buckets [4, 4, 0], whose p50 is the shared boundary 10.
+ expectNonTimeseriesResults(
+ await queryNonTimeseriesHistogram(metricName, { aggFn: 'count' }),
+ [
+ { group: ['/reset'], Value: '8' },
+ { group: ['/normal'], Value: '8' },
+ ],
+ );
+ expectNonTimeseriesResults(
+ await queryNonTimeseriesHistogram(metricName, {
+ aggFn: 'quantile',
+ level: 0.5,
+ }),
+ [
+ { group: ['/reset'], Value: 10 },
+ { group: ['/normal'], Value: 10 },
+ ],
+ );
+ });
+ });
+
+ describe('Query Metrics - Exponential Histogram', () => {
+ type CountResult = {
+ __hdx_time_bucket?: string;
+ Value: string;
+ group?: string[];
+ };
+
+ type QuantileResult = {
+ __hdx_time_bucket?: string;
+ Value: number;
+ group?: string[];
+ };
+
+ const isQuantileResult = (result: unknown): result is QuantileResult =>
+ typeof result === 'object' &&
+ result !== null &&
+ (!('__hdx_time_bucket' in result) ||
+ typeof result.__hdx_time_bucket === 'string') &&
+ 'Value' in result &&
+ typeof result.Value === 'number' &&
+ (!('group' in result) ||
+ (Array.isArray(result.group) &&
+ result.group.every(value => typeof value === 'string')));
+
+ const isCountResult = (result: unknown): result is CountResult =>
+ typeof result === 'object' &&
+ result !== null &&
+ (!('__hdx_time_bucket' in result) ||
+ typeof result.__hdx_time_bucket === 'string') &&
+ 'Value' in result &&
+ typeof result.Value === 'string' &&
+ (!('group' in result) ||
+ (Array.isArray(result.group) &&
+ result.group.every(value => typeof value === 'string')));
+
+ const queryCount = async (
+ metricName: string,
+ {
+ dateRange = [new Date(now), nowPlus('3m')],
+ granularity = '1 minute',
+ groupBy,
+ displayType,
+ }: {
+ dateRange?: [Date, Date];
+ granularity?: '1 minute' | '2 minute' | null;
+ groupBy?: string;
+ displayType?: DisplayType;
+ } = {},
+ ) => {
+ const query = await renderChartConfig(
+ {
+ displayType:
+ displayType ??
+ (granularity === null ? DisplayType.Number : DisplayType.Line),
+ select: [
+ {
+ aggFn: 'count',
+ metricName,
+ metricType: MetricsDataType.ExponentialHistogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange,
+ groupBy,
+ granularity: granularity ?? undefined,
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ const results = await queryData(query);
+ if (!results.every(isCountResult)) {
+ throw new Error('unexpected exponential histogram count query result');
+ }
+ return results.sort(
+ (left, right) =>
+ (left.__hdx_time_bucket ?? '').localeCompare(
+ right.__hdx_time_bucket ?? '',
+ ) ||
+ JSON.stringify(left.group ?? []).localeCompare(
+ JSON.stringify(right.group ?? []),
+ ),
+ );
+ };
+
+ const queryQuantile = async (
+ level: number,
+ metricName: string,
+ {
+ dateRange = [new Date(now), nowPlus('2m')],
+ granularity = '1 minute',
+ groupBy,
+ where = '',
+ displayType,
+ }: {
+ dateRange?: [Date, Date];
+ granularity?: '1 minute' | null;
+ groupBy?: string;
+ where?: string;
+ displayType?: DisplayType;
+ } = {},
+ ) => {
+ const query = await renderChartConfig(
+ {
+ displayType:
+ displayType ??
+ (granularity === null ? DisplayType.Number : DisplayType.Line),
+ select: [
+ {
+ aggFn: 'quantile',
+ level,
+ metricName,
+ metricType: MetricsDataType.ExponentialHistogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where,
+ metricTables: TEST_METRIC_TABLES,
+ dateRange,
+ groupBy,
+ granularity: granularity ?? undefined,
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ const results = await queryData(query);
+ if (!results.every(isQuantileResult)) {
+ throw new Error('unexpected exponential histogram query result');
+ }
+ return results;
+ };
+
+ const seedGroupedCumulativeSeries = async ({
+ attributes,
+ metricName,
+ observations,
+ }: {
+ attributes: Record;
+ metricName: string;
+ observations: number[];
+ }) =>
+ seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: attributes,
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: attributes,
+ ...bucketExponentialHistogramObservations(observations),
+ },
+ ],
+ });
+
+ it('aggregates exponential-histogram count and quantile across every timestamp in each non-timeseries group', async () => {
+ const metricName = 'test.non.timeseries.exponential.histogram.grouped';
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { route: '/two' },
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/two' },
+ ...bucketExponentialHistogramObservations([8, 8]),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { route: '/four' },
+ ...bucketExponentialHistogramObservations([4, 4]),
+ },
+ {
+ TimeUnix: nowPlus('2m'),
+ Attributes: { route: '/four' },
+ ...bucketExponentialHistogramObservations([16, 16]),
+ },
+ ],
+ });
+
+ // Each group sums two observations at each timestamp, so both counts
+ // are 2 + 2 = 4. The standalone p50s are sqrt(2) and sqrt(32) for
+ // /two, and sqrt(8) and sqrt(128) for /four. Their whole-range p50s
+ // (2 and 4) therefore prove that both timestamps were combined.
+ expectNonTimeseriesResults(
+ await queryCount(metricName, {
+ granularity: null,
+ groupBy: `Attributes['route']`,
+ displayType: DisplayType.Table,
+ }),
+ [
+ { group: ['/two'], Value: '4' },
+ { group: ['/four'], Value: '4' },
+ ],
+ );
+ expectNonTimeseriesResults(
+ await queryQuantile(0.5, metricName, {
+ granularity: null,
+ groupBy: `Attributes['route']`,
+ displayType: DisplayType.Table,
+ }),
+ [
+ { group: ['/two'], Value: 2 },
+ { group: ['/four'], Value: 4 },
+ ],
+ );
+ });
+
+ it('uses the first visible cumulative exponential-histogram point as the baseline', async () => {
+ const metricName = 'test.non.timeseries.exponential.histogram.boundaries';
+ const baseline = Array.from({ length: 10 }, () => 1024);
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations(baseline),
+ },
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([...baseline, 4]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([...baseline, 4, 4]),
+ },
+ {
+ TimeUnix: nowPlus('3m'),
+ ...bucketExponentialHistogramObservations([
+ ...baseline,
+ 4,
+ 4,
+ ...Array.from({ length: 20 }, () => 2048),
+ ]),
+ },
+ ],
+ });
+
+ // Only [0m, 2m] is scanned. The 0m cumulative count 11 is the
+ // zero-delta baseline; the 1m point contributes 12 - 11 = 1. That one
+ // new observation is 4, whose exponential bucket midpoint is sqrt(8).
+ // The 10-count -1m point and the large 3m point are outside the range.
+ expect(
+ await queryCount(metricName, {
+ dateRange: [new Date(now), nowPlus('2m')],
+ granularity: null,
+ }),
+ ).toEqual([{ Value: '1' }]);
+ expect(
+ await queryQuantile(0.5, metricName, {
+ dateRange: [new Date(now), nowPlus('2m')],
+ granularity: null,
+ }),
+ ).toEqual([{ Value: Math.sqrt(8) }]);
+ });
+
+ it('aggregates every exponential-histogram reset mode across each non-timeseries group', async () => {
+ const metricName = 'test.non.timeseries.exponential.reset.grouped';
+ const originalStartTime = nowPlus('-1m');
+ const unknownResetTime = nowPlus('1m');
+
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: originalStartTime,
+ Attributes: { route: '/known' },
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: originalStartTime,
+ Attributes: { route: '/known' },
+ ...bucketExponentialHistogramObservations([2, 2, 4, 4]),
+ },
+ {
+ TimeUnix: nowPlus('2m'),
+ StartTimeUnix: nowPlus('90s'),
+ Attributes: { route: '/known' },
+ ...bucketExponentialHistogramObservations([8, 8]),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: originalStartTime,
+ Attributes: { route: '/decrease' },
+ ...bucketExponentialHistogramObservations([2, 2, 4, 4]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: originalStartTime,
+ Attributes: { route: '/decrease' },
+ ...bucketExponentialHistogramObservations([2, 2, 4, 4, 8, 8]),
+ },
+ {
+ TimeUnix: nowPlus('2m'),
+ StartTimeUnix: originalStartTime,
+ Attributes: { route: '/decrease' },
+ ...bucketExponentialHistogramObservations([16, 16]),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: originalStartTime,
+ Attributes: { route: '/unknown' },
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: unknownResetTime,
+ StartTimeUnix: unknownResetTime,
+ Attributes: { route: '/unknown' },
+ ...bucketExponentialHistogramObservations([4, 4]),
+ },
+ {
+ TimeUnix: nowPlus('2m'),
+ StartTimeUnix: unknownResetTime,
+ Attributes: { route: '/unknown' },
+ ...bucketExponentialHistogramObservations([4, 4, 8, 8]),
+ },
+ ],
+ });
+
+ const queryOptions: {
+ granularity: null;
+ groupBy: string;
+ displayType: DisplayType;
+ } = {
+ granularity: null,
+ groupBy: `Attributes['route']`,
+ displayType: DisplayType.Table,
+ };
+ // /known contributes two 4s before its changed start time makes the two
+ // current 8s a reset delta: count 4, combined p50 4.
+ // /decrease contributes two 8s, then its count decrease makes the two
+ // current 16s a reset delta: count 4, combined p50 8.
+ // /unknown contributes zero at its StartTimeUnix == TimeUnix marker;
+ // the next point subtracts that marker and contributes two 8s:
+ // count 2 and p50 sqrt(32).
+ expectNonTimeseriesResults(await queryCount(metricName, queryOptions), [
+ { group: ['/known'], Value: '4' },
+ { group: ['/decrease'], Value: '4' },
+ { group: ['/unknown'], Value: '2' },
+ ]);
+ expectNonTimeseriesResults(
+ await queryQuantile(0.5, metricName, queryOptions),
+ [
+ { group: ['/known'], Value: 4 },
+ { group: ['/decrease'], Value: 8 },
+ { group: ['/unknown'], Value: Math.sqrt(32) },
+ ],
+ );
+ });
+
+ it('counts delta-temporality observations directly', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.count.delta',
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([-4, 0, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ ],
+ });
+
+ expect(await queryCount('test.count.delta')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: '3',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: '2',
+ },
+ ]);
+ });
+
+ it('subtracts cumulative counts using a warm-up point outside the requested range', async () => {
+ const startTime = nowPlus('-2m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.count.cumulative.warmup',
+ points: [
+ {
+ TimeUnix: nowPlus('-1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 4, 8]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 4, 8, 16, 32]),
+ },
+ ],
+ });
+
+ expect(await queryCount('test.count.cumulative.warmup')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: '1',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: '2',
+ },
+ ]);
+ });
+
+ it('does not attribute a predecessor-less cumulative count to its first visible bucket', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.count.cumulative.missing.predecessor',
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('-10m'),
+ ...bucketExponentialHistogramObservations([2, 4, 8]),
+ },
+ ],
+ });
+
+ expect(
+ await queryCount('test.count.cumulative.missing.predecessor'),
+ ).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: '0',
+ },
+ ]);
+ });
+
+ it('suppresses an unknown-start reset and uses it as the next baseline', async () => {
+ const resetTime = nowPlus('1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.count.cumulative.unknown.reset',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ {
+ TimeUnix: resetTime,
+ StartTimeUnix: resetTime,
+ ...bucketExponentialHistogramObservations([-4, 0, 2]),
+ },
+ {
+ TimeUnix: nowPlus('2m'),
+ StartTimeUnix: resetTime,
+ ...bucketExponentialHistogramObservations([-4, -2, 0, 2, 4]),
+ },
+ ],
+ });
+
+ expect(await queryCount('test.count.cumulative.unknown.reset')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: '0',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(resetTime),
+ Value: '0',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('2m')),
+ Value: '2',
+ },
+ ]);
+ });
+
+ it('uses the current count after a known cumulative reset', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.count.cumulative.known.reset',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations([2, 4, 8]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('30s'),
+ ...bucketExponentialHistogramObservations([-4, 0]),
+ },
+ ],
+ });
+
+ expect(await queryCount('test.count.cumulative.known.reset')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: '0',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: '2',
+ },
+ ]);
+ });
+
+ it('uses the current count after a decrease-inferred cumulative reset', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.count.cumulative.decrease',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 4, 8]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([16]),
+ },
+ ],
+ });
+
+ expect(await queryCount('test.count.cumulative.decrease')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: '0',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: '1',
+ },
+ ]);
+ });
+
+ it('keeps count continuity across scale and bucket-layout changes', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.count.scale.change',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([-16, 0, 2, 4], -1),
+ },
+ ],
+ });
+
+ expect(await queryCount('test.count.scale.change')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: '0',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: '3',
+ },
+ ]);
+ });
+
+ it('sums series by the requested group and time bucket', async () => {
+ await Promise.all([
+ seedGroupedCumulativeSeries({
+ attributes: { host: 'host-a', service: 'api' },
+ metricName: 'test.count.grouped',
+ observations: [2, 4],
+ }),
+ seedGroupedCumulativeSeries({
+ attributes: { host: 'host-a', service: 'worker' },
+ metricName: 'test.count.grouped',
+ observations: [8],
+ }),
+ seedGroupedCumulativeSeries({
+ attributes: { host: 'host-b', service: 'api' },
+ metricName: 'test.count.grouped',
+ observations: [16, 32, 64],
+ }),
+ ]);
+
+ expect(
+ await queryCount('test.count.grouped', {
+ groupBy: `Attributes['host']`,
+ }),
+ ).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ group: ['host-a'],
+ Value: '0',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ group: ['host-b'],
+ Value: '0',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['host-a'],
+ Value: '3',
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['host-b'],
+ Value: '3',
+ },
+ ]);
+ });
+
+ it('calculates a quantile for a single series', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.request.duration',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ ],
+ });
+
+ expect(await queryQuantile(0.5, 'test.request.duration')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: 2,
+ },
+ ]);
+ });
+
+ it('returns no rows when no data points match the query', async () => {
+ expect(await queryQuantile(0.5, 'test.metric.never.seeded')).toEqual([]);
+ });
+
+ it('uses logarithmic interpolation within a positive exponential bucket', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.positive.interpolation',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([4, 4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, 'test.positive.interpolation');
+
+ // At scale 0, both observations occupy (2, 4]. Halfway through an
+ // exponential bucket is its geometric midpoint: 2 * sqrt(2), not 3.
+ expect(result.__hdx_time_bucket).toBe(
+ toClickHouseISOString(nowPlus('1m')),
+ );
+ expect(result.Value).toBeCloseTo(Math.sqrt(8));
+ });
+
+ it('orders negative buckets before zero and interpolates them toward zero', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.negative.interpolation',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([-4, -4, 0, 2]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.25, 'test.negative.interpolation');
+
+ // The first two observations are in [-4, -2). Rank 1 is halfway through
+ // that bucket, whose logarithmic midpoint is -2 * sqrt(2).
+ expect(result.Value).toBeCloseTo(-Math.sqrt(8));
+ });
+
+ it('orders multiple populated negative buckets from the largest magnitude toward zero', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.negative.multiple.bucket.order',
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([-8, -8, -4, -2]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.375,
+ 'test.negative.multiple.bucket.order',
+ );
+
+ // Negative buckets are traversed as [-8, -4), [-4, -2), [-2, -1).
+ // Rank 1.5 is 75% of the way through the first two-count bucket, so
+ // logarithmic interpolation toward zero gives -2^2.25.
+ expect(result.Value).toBeCloseTo(-(2 ** 2.25));
+ });
+
+ it('returns zero when the requested rank falls in the zero bucket', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.zero.bucket',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([-2, 0, 0, 4]),
+ },
+ ],
+ });
+
+ // One negative observation precedes two zero observations, so rank 2
+ // falls in the zero bucket and the median is exactly zero.
+ expect(await queryQuantile(0.5, 'test.zero.bucket')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: 0,
+ },
+ ]);
+ });
+
+ it('returns a negative bucket upper boundary when the rank ends at the zero bucket border', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.negative.zero.border',
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([-4, -4, 0, 0]),
+ },
+ ],
+ });
+
+ // Rank 2 ends exactly at the top of negative bucket [-4, -2), just
+ // before the zero bucket begins. The sign-flip boundary must resolve to
+ // the bucket's upper boundary -2, not to zero.
+ expect(await queryQuantile(0.5, 'test.negative.zero.border')).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: -2,
+ },
+ ]);
+ });
+
+ it('returns the outer bucket boundaries for quantile levels zero and one', async () => {
+ const metricName = 'test.quantile.endpoints';
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([-8, -4, 2, 4]),
+ },
+ ],
+ });
+
+ // Level zero starts at the lower numeric boundary of the first negative
+ // bucket. Level one ends at the upper boundary of the last positive one.
+ expect((await queryQuantile(0, metricName))[0]?.Value).toBe(-8);
+ expect((await queryQuantile(1, metricName))[0]?.Value).toBe(4);
+ });
+
+ it('returns single-sided outer boundaries for quantile levels zero and one', async () => {
+ const negativeOnly = 'test.quantile.endpoints.negative.only';
+ const positiveOnly = 'test.quantile.endpoints.positive.only';
+ await seedExponentialHistogramMetric({
+ metricName: negativeOnly,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([-4, -2]),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName: positiveOnly,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ ],
+ });
+
+ // With only negative buckets, level one must end at the upper (least
+ // negative) boundary of [-2, -1) and stay negative. With only positive
+ // buckets, level zero starts at the lower boundary of (1, 2].
+ expect((await queryQuantile(1, negativeOnly))[0]?.Value).toBe(-1);
+ expect((await queryQuantile(0, positiveOnly))[0]?.Value).toBe(1);
+ });
+
+ it('returns zero for quantile levels zero and one when only the zero bucket is populated', async () => {
+ const metricName = 'test.quantile.endpoints.zero.only';
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([0, 0]),
+ },
+ ],
+ });
+
+ // With empty negative and positive bucket arrays, both endpoint levels
+ // resolve within the zero bucket, which represents exactly zero.
+ expect((await queryQuantile(0, metricName))[0]?.Value).toBe(0);
+ expect((await queryQuantile(1, metricName))[0]?.Value).toBe(0);
+ });
+
+ it('subtracts cumulative bucket counts instead of quantiling lifetime counts', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.delta',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 2, 4, 4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, 'test.cumulative.delta');
+
+ // Only the two newly recorded 4s belong to the second interval. Including
+ // the lifetime 2s would place the median at 2 instead of 2 * sqrt(2).
+ expect(result.__hdx_time_bucket).toBe(
+ toClickHouseISOString(nowPlus('1m')),
+ );
+ expect(result.Value).toBeCloseTo(Math.sqrt(8));
+ });
+
+ it('uses a cumulative warm-up point outside the requested range as the subtraction baseline', async () => {
+ const startTime = nowPlus('-2m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.warmup',
+ points: [
+ {
+ TimeUnix: nowPlus('-1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 2, 4, 4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, 'test.cumulative.warmup');
+
+ // The -1m point is fetched only to establish the baseline. It must not be
+ // returned, and the first visible bucket contains only the two new 4s.
+ expect(result.__hdx_time_bucket).toBe(
+ toClickHouseISOString(new Date(now)),
+ );
+ expect(result.Value).toBeCloseTo(Math.sqrt(8));
+ });
+
+ it('emits no value for a predecessor-less cumulative point whose start predates the requested range', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.missing.predecessor',
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('-10m'),
+ ...bucketExponentialHistogramObservations([2, 2, 4, 4]),
+ },
+ ],
+ });
+
+ // With no predecessor, the portion of the lifetime counts belonging to
+ // this interval is unknowable. Returning no quantile avoids a false spike.
+ expect(
+ await queryQuantile(0.5, 'test.cumulative.missing.predecessor'),
+ ).toEqual([]);
+ });
+
+ it('emits no value for a predecessor-less cumulative point whose start falls inside the requested range', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.midrange.start',
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('30s'),
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ ],
+ });
+
+ // Even though this new series' entire lifetime lies inside the requested
+ // range, a predecessor-less cumulative point is suppressed rather than
+ // attributed to the chart bucket it first appears in.
+ expect(
+ await queryQuantile(0.5, 'test.cumulative.midrange.start'),
+ ).toEqual([]);
+ });
+
+ it('emits no value when the cumulative baseline precedes the one-interval warm-up window', async () => {
+ const startTime = nowPlus('-3m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.stale.baseline',
+ points: [
+ {
+ TimeUnix: nowPlus('-2m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 2, 4, 4]),
+ },
+ ],
+ });
+
+ // The warm-up fetch reaches back exactly one granularity interval, so a
+ // baseline two minutes before the range is never read. The in-range
+ // point is then predecessor-less and contributes nothing.
+ expect(
+ await queryQuantile(0.5, 'test.cumulative.stale.baseline'),
+ ).toEqual([]);
+ });
+
+ it('treats an unknown-start reset point as zero contribution and uses it as the next baseline', async () => {
+ const originalStartTime = nowPlus('-1m');
+ const resetTime = nowPlus('1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.unknown.reset',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: originalStartTime,
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: resetTime,
+ StartTimeUnix: resetTime,
+ ...bucketExponentialHistogramObservations([4, 4]),
+ },
+ {
+ TimeUnix: nowPlus('2m'),
+ StartTimeUnix: resetTime,
+ ...bucketExponentialHistogramObservations([2, 4, 4]),
+ },
+ ],
+ });
+
+ // StartTimeUnix == TimeUnix marks an unknown-start reset and contributes
+ // nothing at 1m. The 2m point subtracts that marker, leaving only one 2.
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.cumulative.unknown.reset',
+ );
+ expect(result.__hdx_time_bucket).toBe(
+ toClickHouseISOString(nowPlus('2m')),
+ );
+ expect(result.Value).toBeCloseTo(Math.sqrt(2));
+ });
+
+ it('handles an unknown-start cumulative reset spanning negative, zero, and positive buckets', async () => {
+ const resetTime = nowPlus('1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.unknown.reset.all.buckets',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: resetTime,
+ StartTimeUnix: resetTime,
+ ...bucketExponentialHistogramObservations([-4, 0, 2]),
+ },
+ {
+ TimeUnix: nowPlus('2m'),
+ StartTimeUnix: resetTime,
+ ...bucketExponentialHistogramObservations([-4, -2, 0, 2]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.cumulative.unknown.reset.all.buckets',
+ );
+
+ // The reset marker's negative, zero, and positive counts contribute
+ // nothing. The next point subtracts all three sides of that baseline,
+ // leaving only the newly recorded -2 in negative bucket [-2, -1).
+ expect(result.__hdx_time_bucket).toBe(
+ toClickHouseISOString(nowPlus('2m')),
+ );
+ expect(result.Value).toBeCloseTo(-Math.sqrt(2));
+ });
+
+ it('uses all current counts after a known cumulative reset', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.known.reset',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('30s'),
+ ...bucketExponentialHistogramObservations([4, 4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, 'test.cumulative.known.reset');
+
+ // The changed start time is earlier than the point time, so this is a
+ // known reset with an implicit zero baseline. Both new-sequence 4s count.
+ expect(result.Value).toBeCloseTo(Math.sqrt(8));
+ });
+
+ it('uses the complete negative, zero, and positive distribution after a known cumulative reset', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.known.reset.all.buckets',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations([-4, -4, 0, 0, 2, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('30s'),
+ ...bucketExponentialHistogramObservations([-2, 0, 4]),
+ },
+ ],
+ });
+
+ // The changed start time begins a true reset sequence, so none of the old
+ // negative, zero, or positive counts are subtracted. In the new three-count
+ // distribution, rank 1.5 falls in the zero bucket.
+ expect(
+ await queryQuantile(0.5, 'test.cumulative.known.reset.all.buckets'),
+ ).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: 0,
+ },
+ ]);
+ });
+
+ it('detects a cumulative reset from a decreased zero count with an unchanged start time', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.reset.zero.decrease',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([0, 0, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([0, 2, 4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.cumulative.reset.zero.decrease',
+ );
+
+ // ZeroCount falls from two to one while every existing positive bucket
+ // is nondecreasing. The decrease alone identifies a reset, so the entire
+ // current [0, 2, 4] distribution is used and p50 is sqrt(2).
+ expect(result.Value).toBeCloseTo(Math.sqrt(2));
+ });
+
+ it('detects a cumulative reset from a decreased positive bucket with an unchanged start time', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.reset.positive.decrease',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([0, 2, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([0, 2, 4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.cumulative.reset.positive.decrease',
+ );
+
+ // The count in positive bucket (1, 2] falls from two to one while the
+ // zero count is unchanged. Treating the point as a new sequence yields
+ // the current [0, 2, 4] distribution and p50 = sqrt(2).
+ expect(result.Value).toBeCloseTo(Math.sqrt(2));
+ });
+
+ it('detects a cumulative reset from a decreased negative bucket with an unchanged start time', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.reset.negative.decrease',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([0, -2, -2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([0, -2, -4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.cumulative.reset.negative.decrease',
+ );
+
+ // The [-2, -1) bucket falls from two to one while ZeroCount is stable.
+ // The current [-4, -2, 0] distribution is therefore used directly;
+ // rank 1.5 lies halfway through [-2, -1), giving -sqrt(2).
+ expect(result.Value).toBeCloseTo(-Math.sqrt(2));
+ });
+
+ it('detects a cumulative reset when a previous bucket disappears outside the current range', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.reset.disappearing.bucket',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([4, 4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.cumulative.reset.disappearing.bucket',
+ );
+
+ // PositiveOffset moves from bucket index 0 to 1. Looking up the missing
+ // old bucket as zero detects its decrease and treats both current 4s as
+ // the new sequence, whose p50 is the bucket midpoint sqrt(8).
+ expect(result.Value).toBeCloseTo(Math.sqrt(8));
+ });
+
+ it('detects a cumulative reset when the current range shrinks below a previous upper bucket', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.reset.shrinking.range',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 8]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 2]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.cumulative.reset.shrinking.range',
+ );
+
+ // The current array covers only bucket index 0, while the previous
+ // point held a positive count in bucket index 2 above that range. The
+ // vanished count is a decrease, so both current 2s form a new sequence
+ // whose p50 is the bucket midpoint sqrt(2).
+ expect(result.Value).toBeCloseTo(Math.sqrt(2));
+ });
+
+ it('detects a cumulative reset when the current negative range shrinks below a previous upper bucket', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.cumulative.reset.shrinking.negative.range',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([-2, -8]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([-2, -2]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.cumulative.reset.shrinking.negative.range',
+ );
+
+ // Mirror of the positive shrinking-range case: the previous -8 count
+ // sits above the current negative bucket range, so its disappearance
+ // marks a reset and both current -2s count, giving p50 -sqrt(2).
+ expect(result.Value).toBeCloseTo(-Math.sqrt(2));
+ });
+
+ it('uses delta-temporality bucket counts directly without subtracting the previous point', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.delta.temporality',
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ ],
+ });
+
+ const results = await queryQuantile(0.5, 'test.delta.temporality');
+
+ // The second point is already an interval distribution. Its median rank
+ // ends at the top of the (1, 2] bucket, so the result is exactly 2. If it
+ // were treated as cumulative, only the newly appearing 4 bucket remains.
+ expect(results).toHaveLength(2);
+ const firstInterval = results.find(
+ result =>
+ result.__hdx_time_bucket === toClickHouseISOString(new Date(now)),
+ );
+ const secondInterval = results.find(
+ result =>
+ result.__hdx_time_bucket === toClickHouseISOString(nowPlus('1m')),
+ );
+ expect(firstInterval?.Value).toBeCloseTo(Math.sqrt(2));
+ expect(secondInterval?.Value).toBe(2);
+ });
+
+ it('ignores unspecified-temporality exponential histograms', async () => {
+ const metricName = 'test.unspecified.temporality';
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 0,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([2]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ ],
+ });
+
+ expect(await queryQuantile(0.5, metricName)).toEqual([]);
+ });
+
+ it('sums multiple cumulative interval deltas from one series within a single chart bucket', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.multiple.points.same.chart.bucket',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('20s'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2]),
+ },
+ {
+ TimeUnix: nowPlus('40s'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ ],
+ });
+
+ // The two sub-minute increases are [2] and [4]. They must be summed into
+ // one minute bucket before selecting a quantile; [2, 4] has p50 = 2.
+ expect(
+ await queryQuantile(0.5, 'test.multiple.points.same.chart.bucket'),
+ ).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: 2,
+ },
+ ]);
+ });
+
+ it('sums multiple delta points from one series within a single chart bucket', async () => {
+ const metricName = 'test.multiple.delta.points.same.chart.bucket';
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('20s'),
+ ...bucketExponentialHistogramObservations([2]),
+ },
+ {
+ TimeUnix: nowPlus('40s'),
+ ...bucketExponentialHistogramObservations([4]),
+ },
+ ],
+ });
+
+ // Both delta points fall inside the first minute bucket, so their
+ // interval counts sum to [2, 4], whose median rank reaches the upper
+ // boundary of the (1, 2] bucket.
+ expect(await queryQuantile(0.5, metricName)).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(new Date(now)),
+ Value: 2,
+ },
+ ]);
+ });
+
+ it('combines cumulative and delta temporality series within the same output group', async () => {
+ const metricName = 'test.mixed.temporality.group';
+ const groupAttributes = { route: '/mixed' };
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ ResourceAttributes: { producer: 'cumulative' },
+ Attributes: { ...groupAttributes, instance: 'cumulative-a' },
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('-1m'),
+ ResourceAttributes: { producer: 'cumulative' },
+ Attributes: { ...groupAttributes, instance: 'cumulative-a' },
+ ...bucketExponentialHistogramObservations([2]),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ResourceAttributes: { producer: 'delta' },
+ Attributes: { ...groupAttributes, instance: 'delta-a' },
+ ...bucketExponentialHistogramObservations([4]),
+ },
+ ],
+ });
+
+ // The cumulative series contributes its [2] increase and the delta series
+ // contributes [4] directly. After spatial aggregation, rank 1 reaches the
+ // upper boundary of the first positive bucket, producing p50 = 2.
+ expect(
+ await queryQuantile(0.5, metricName, {
+ groupBy: "Attributes['route']",
+ }),
+ ).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['/mixed'],
+ Value: 2,
+ },
+ ]);
+ });
+
+ it('combines mixed temporalities and mixed scales within the same output group', async () => {
+ const metricName = 'test.mixed.temporality.and.scale.group';
+ const groupAttributes = { route: '/mixed-scale-temporality' };
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ Attributes: { ...groupAttributes, instance: 'cumulative-fine' },
+ ...bucketExponentialHistogramObservations([], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('-1m'),
+ Attributes: { ...groupAttributes, instance: 'cumulative-fine' },
+ ...bucketExponentialHistogramObservations([2], 1),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { ...groupAttributes, instance: 'delta-coarse' },
+ ...bucketExponentialHistogramObservations([16], -1),
+ },
+ ],
+ });
+
+ // The cumulative scale-1 observation downscales into scale -1 bucket 0;
+ // the delta observation is already in bucket 1. Their combined p50 ends
+ // at bucket 0's upper boundary, which is 4.
+ expect(
+ await queryQuantile(0.5, metricName, {
+ groupBy: "Attributes['route']",
+ }),
+ ).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['/mixed-scale-temporality'],
+ Value: 4,
+ },
+ ]);
+ });
+
+ it('normalizes different scales across independent series before combining their buckets', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.scale.across.series',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], -1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([4], -1),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName: 'test.scale.across.series',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([4], 1),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, 'test.scale.across.series');
+
+ // Both observations normalize into scale -1 bucket 0, whose bounds are
+ // (1, 4]. Their median is the bucket's geometric midpoint, 2.
+ expect(result.Value).toBeCloseTo(2);
+ });
+
+ it('normalizes an unaligned multi-bucket range across multiple coarser bucket boundaries', async () => {
+ const metricName = 'test.scale.unaligned.multi.bucket.range';
+ const fineObservations = [
+ { index: 1, count: 1 },
+ { index: 2, count: 2 },
+ { index: 3, count: 3 },
+ { index: 4, count: 4 },
+ { index: 5, count: 5 },
+ { index: 6, count: 6 },
+ { index: 7, count: 7 },
+ { index: 8, count: 8 },
+ { index: 9, count: 9 },
+ ].flatMap(({ index, count }) =>
+ Array.from({ length: count }, () => 2 ** ((index + 0.5) / 2)),
+ );
+
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], -1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], -1),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations(fineObservations, 1),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, metricName);
+
+ // Scale 1 indexes 1..9 have counts 1..9. Normalizing to scale -1 uses
+ // a divisor of 4, and the unaligned offset 1 produces coarse counts
+ // [1+2+3, 4+5+6+7, 8+9] = [6, 22, 17]. Rank 22.5 is therefore 75%
+ // through coarse bucket index 1, whose bounds are (4, 16]. Log-linear
+ // interpolation gives 4 * 4^0.75 = sqrt(128).
+ expect(result.Value).toBeCloseTo(Math.sqrt(128));
+ });
+
+ it('normalizes an unaligned negative multi-bucket range across multiple coarser bucket boundaries', async () => {
+ const metricName = 'test.scale.unaligned.negative.multi.bucket.range';
+ const fineObservations = [
+ { index: 1, count: 1 },
+ { index: 2, count: 2 },
+ { index: 3, count: 3 },
+ { index: 4, count: 4 },
+ { index: 5, count: 5 },
+ { index: 6, count: 6 },
+ { index: 7, count: 7 },
+ { index: 8, count: 8 },
+ { index: 9, count: 9 },
+ ].flatMap(({ index, count }) =>
+ Array.from({ length: count }, () => -(2 ** ((index + 0.5) / 2))),
+ );
+
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], -1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], -1),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations(fineObservations, 1),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, metricName);
+
+ // Magnitude indexes 1..9 normalize to counts [6, 22, 17] at scale -1.
+ // Negative ordering reverses those to [17, 22, 6]. Rank 22.5 is 25%
+ // through magnitude bucket index 1, interpolating from -16 toward -4
+ // to produce -sqrt(128).
+ expect(result.Value).toBeCloseTo(-Math.sqrt(128));
+ });
+
+ it('preserves internal empty buckets while normalizing a sparse fine-scale range', async () => {
+ const metricName = 'test.scale.sparse.internal.empty.buckets';
+ const fineObservations = [1, 9].map(index => 2 ** ((index + 0.5) / 2));
+
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], -1),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations(fineObservations, 1),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.75, metricName);
+
+ // Fine indexes 1 and 9 create a dense source array with seven internal
+ // zeroes. At scale -1 they become coarse counts [1, 0, 1]. Rank 1.5
+ // skips the empty middle bucket and lands halfway through (16, 64].
+ expect(result.Value).toBeCloseTo(32);
+ });
+
+ it('normalizes a scale change within one cumulative series before calculating its delta', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.scale.within.series',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 4], 0),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, 'test.scale.within.series');
+
+ // After the scale-1 baseline is downscaled to scale 0, its 2 is removed
+ // from the cumulative point and only the newly recorded 4 remains.
+ expect(result.Value).toBeCloseTo(Math.sqrt(8));
+ });
+
+ it('normalizes mixed scales and an in-series scale change before aggregating a group', async () => {
+ const metricName = 'test.mixed.scale.group';
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ Attributes: { route: '/mixed-scale', instance: 'changing' },
+ ...bucketExponentialHistogramObservations([2], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ Attributes: { route: '/mixed-scale', instance: 'changing' },
+ ...bucketExponentialHistogramObservations([2, 4], 0),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { route: '/mixed-scale', instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], -1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/mixed-scale', instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([4], -1),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, metricName, {
+ groupBy: "Attributes['route']",
+ });
+
+ // The changing series first normalizes its scale-1 baseline and scale-0
+ // cumulative point to scale -1, leaving one new 4. The coarse series adds
+ // another 4. Both land in scale -1 bucket (1, 4], whose midpoint is 2.
+ expect(result).toMatchObject({
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['/mixed-scale'],
+ });
+ expect(result.Value).toBeCloseTo(2);
+ });
+
+ it('floors negative bucket indexes when normalizing to a coarser scale', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.negative.bucket.index',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], 0),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([0.5], 0),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName: 'test.negative.bucket.index',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([0.5], 1),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, 'test.negative.bucket.index');
+
+ // Fine-scale index -3 must floor to coarse-scale index -2, not truncate
+ // to -1. Scale-0 bucket -2 is (0.25, 0.5], with midpoint sqrt(0.125).
+ expect(result.Value).toBeCloseTo(Math.sqrt(0.125));
+ });
+
+ it('floors negative bucket indexes for negative observations when normalizing to a coarser scale', async () => {
+ const metricName = 'test.negative.side.bucket.index';
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], 0),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([-0.5], 0),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([-0.5], 1),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, metricName);
+
+ // The negative buckets downscale separately from the positive ones, so
+ // fine-scale magnitude index -3 must also floor to coarse index -2.
+ // Scale-0 negative bucket -2 spans [-0.5, -0.25), with logarithmic
+ // midpoint -sqrt(0.125).
+ expect(result.Value).toBeCloseTo(-Math.sqrt(0.125));
+ });
+
+ it('interpolates at the finest valid scale without downscaling', async () => {
+ const scale = 20;
+ await seedExponentialHistogramMetric({
+ metricName: 'test.scale.maximum.fine.interpolation',
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([2, 2], scale),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.scale.maximum.fine.interpolation',
+ );
+
+ // At scale 20, value 2 occupies index 2^20 - 1. Its midpoint is only
+ // half a scale-20 logarithmic step below 2.
+ const expected = 2 ** (1 - 0.5 / 2 ** scale);
+ expect(result.Value).toBeCloseTo(expected);
+ });
+
+ it('normalizes across the full valid scale range without overflowing the divisor', async () => {
+ const metricName = 'test.scale.full.range.normalization';
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'coarse' },
+ ...bucketExponentialHistogramObservations([], -10),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { instance: 'fine' },
+ ...bucketExponentialHistogramObservations([2, 2], 20),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, metricName);
+
+ // The scale difference is 30, so the divisor is 2^30. Fine index
+ // 2^20 - 1 maps to scale -10 bucket 0, whose logarithmic midpoint is
+ // 2^512. Compare relatively because the expected value is enormous.
+ expect(result.Value / 2 ** 512).toBeCloseTo(1);
+ });
+
+ it('handles a cumulative bucket offset expanding toward smaller values', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.offset.expansion',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([4]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, 'test.offset.expansion');
+
+ // Adding 2 expands PositiveOffset from 1 to 0. Absolute bucket indexes
+ // align the old 4 correctly, leaving the new 2 as the only delta count.
+ expect(result.Value).toBeCloseTo(Math.sqrt(2));
+ });
+
+ it('handles a cumulative negative-bucket offset expanding toward smaller magnitudes', async () => {
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName: 'test.negative.offset.expansion',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([-4]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ...bucketExponentialHistogramObservations([-2, -4]),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.negative.offset.expansion',
+ );
+
+ // Adding -2 expands NegativeOffset from magnitude index 1 to 0. The old
+ // -4 count remains aligned by absolute index, leaving -2 as the delta.
+ expect(result.Value).toBeCloseTo(-Math.sqrt(2));
+ });
+
+ it('normalizes a scale change before using all counts from a known reset', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.scale.change.with.known.reset',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations([2, 2], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('30s'),
+ ...bucketExponentialHistogramObservations([4, 4], 0),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(
+ 0.5,
+ 'test.scale.change.with.known.reset',
+ );
+
+ // Both points normalize to scale 0 before reset detection. The changed
+ // start time then selects the complete current [4, 4] distribution.
+ expect(result.Value).toBeCloseTo(Math.sqrt(8));
+ });
+
+ it('ignores other metrics, filtered attribute series, and points outside the requested range', async () => {
+ await seedExponentialHistogramMetric({
+ metricName: 'test.isolated.metric',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { route: '/target' },
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/target' },
+ ...bucketExponentialHistogramObservations([2, 4]),
+ },
+ {
+ TimeUnix: nowPlus('3m'),
+ Attributes: { route: '/target' },
+ ...bucketExponentialHistogramObservations([2, 4, 1024, 1024]),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName: 'test.isolated.metric',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { route: '/noise' },
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/noise' },
+ ...bucketExponentialHistogramObservations([1024, 1024]),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName: 'test.unrelated.metric',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([2048, 2048]),
+ },
+ ],
+ });
+
+ // Only /target's in-range [2, 4] distribution is selected. The 3m point
+ // may be read by the forward warm-up interval but must be filtered from
+ // the final chart, while the other route and metric must not contribute.
+ expect(
+ await queryQuantile(0.5, 'test.isolated.metric', {
+ where: "Attributes['route'] = '/target'",
+ }),
+ ).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: 2,
+ },
+ ]);
+ });
+
+ it('excludes lower-scale filtered series and unrelated metrics before choosing the normalized scale', async () => {
+ const metricName = 'test.filtered.scale.isolation';
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { route: '/target' },
+ ...bucketExponentialHistogramObservations([], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/target' },
+ ...bucketExponentialHistogramObservations([2, 2], 1),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/noise' },
+ ...bucketExponentialHistogramObservations([2 ** 512], -10),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName: 'test.filtered.scale.unrelated.metric',
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/target' },
+ ...bucketExponentialHistogramObservations([2 ** 512], -10),
+ },
+ ],
+ });
+
+ const [result] = await queryQuantile(0.5, metricName, {
+ where: "Attributes['route'] = '/target'",
+ });
+
+ // Filtering leaves only the scale-1 target before min(Scale) is taken.
+ // Its two 2s remain in scale-1 bucket (sqrt(2), 2], with midpoint 2^0.75.
+ expect(result.Value).toBeCloseTo(2 ** 0.75);
+ });
+
+ it('calculates quantiles without time bucketing when granularity is omitted', async () => {
+ const metricName = 'test.no.granularity';
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: nowPlus('-1m'),
+ ...bucketExponentialHistogramObservations([4, 4]),
+ },
+ ],
+ });
+
+ expect(
+ await queryQuantile(0.5, metricName, { granularity: null }),
+ ).toEqual([
+ {
+ Value: Math.sqrt(8),
+ },
+ ]);
+ });
+
+ it('returns an exponential-histogram quantile under a custom value alias', async () => {
+ const metricName = 'test.custom.quantile.alias';
+ await seedExponentialHistogramMetric({
+ metricName,
+ aggregationTemporality: 1,
+ points: [
+ {
+ TimeUnix: nowPlus('1m'),
+ ...bucketExponentialHistogramObservations([4, 4]),
+ },
+ ],
+ });
+
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ alias: 'P50',
+ level: 0.5,
+ metricName,
+ metricType: MetricsDataType.ExponentialHistogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), nowPlus('2m')],
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ expect(await queryData(query)).toEqual([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ P50: Math.sqrt(8),
+ },
+ ]);
+ });
+
+ it('uses one normalized scale across output groups while calculating each group independently', async () => {
+ const metricName = 'test.scale.across.output.groups';
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { route: '/fine' },
+ ...bucketExponentialHistogramObservations([], 1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/fine' },
+ ...bucketExponentialHistogramObservations([2, 2], 1),
+ },
+ ],
+ });
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: { route: '/coarse' },
+ ...bucketExponentialHistogramObservations([], -1),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: { route: '/coarse' },
+ ...bucketExponentialHistogramObservations([16, 16], -1),
+ },
+ ],
+ });
+
+ const results = await queryQuantile(0.5, metricName, {
+ groupBy: "Attributes['route']",
+ });
+ const fine = results.find(result => _.isEqual(result.group, ['/fine']));
+ const coarse = results.find(result =>
+ _.isEqual(result.group, ['/coarse']),
+ );
+
+ // The global minimum scale is -1. The fine group's 2s therefore land in
+ // (1, 4] and produce midpoint 2, while the coarse group's 16s remain in
+ // (4, 16] and produce midpoint 8. Counts never cross group boundaries.
+ expect(results).toHaveLength(2);
+ expect(fine?.Value).toBeCloseTo(2);
+ expect(coarse?.Value).toBeCloseTo(8);
+ });
+
+ it('calculates each single-attribute group quantile across multiple series in that group', async () => {
+ const metricName = 'test.groupby.route';
+ await seedGroupedCumulativeSeries({
+ metricName,
+ attributes: { route: '/checkout', instance: 'checkout-a' },
+ observations: [2],
+ });
+ await seedGroupedCumulativeSeries({
+ metricName,
+ attributes: { route: '/checkout', instance: 'checkout-b' },
+ observations: [4],
+ });
+ await seedGroupedCumulativeSeries({
+ metricName,
+ attributes: { route: '/search', instance: 'search-a' },
+ observations: [4, 4],
+ });
+ await seedGroupedCumulativeSeries({
+ metricName,
+ attributes: { route: '/search', instance: 'search-b' },
+ observations: [4, 4],
+ });
+
+ const results = await queryQuantile(0.5, metricName, {
+ groupBy: "Attributes['route']",
+ });
+ const checkout = results.find(result =>
+ _.isEqual(result.group, ['/checkout']),
+ );
+ const search = results.find(result =>
+ _.isEqual(result.group, ['/search']),
+ );
+
+ expect(results).toHaveLength(2);
+ // /checkout combines one 2 and one 4 from distinct instances. Rank 1
+ // reaches the upper boundary of the first bucket, so its p50 is 2.
+ expect(checkout).toEqual({
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['/checkout'],
+ Value: 2,
+ });
+ // /search combines four 4s across its two instances. Rank 2 is halfway
+ // through scale-0 bucket (2, 4], producing its geometric midpoint.
+ expect(search?.Value).toBeCloseTo(Math.sqrt(8));
+ expect(search?.__hdx_time_bucket).toBe(
+ toClickHouseISOString(nowPlus('1m')),
+ );
+ });
+
+ it('keeps cumulative predecessor chains separate for groups outside metric attributes', async () => {
+ const metricName = 'test.groupby.service-name';
+ const startTime = nowPlus('-1m');
+ await seedExponentialHistogramMetric({
+ metricName,
+ points: [
+ {
+ TimeUnix: new Date(now),
+ StartTimeUnix: startTime,
+ ServiceName: 'service-a',
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ StartTimeUnix: startTime,
+ ServiceName: 'service-a',
+ ...bucketExponentialHistogramObservations([2]),
+ },
+ {
+ TimeUnix: nowPlus('20s'),
+ StartTimeUnix: startTime,
+ ServiceName: 'service-b',
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('80s'),
+ StartTimeUnix: startTime,
+ ServiceName: 'service-b',
+ ...bucketExponentialHistogramObservations([2]),
+ },
+ ],
+ });
+
+ const results = await queryQuantile(0.5, metricName, {
+ groupBy: 'ServiceName',
+ });
+
+ expect(results).toHaveLength(2);
+ expect(results).toEqual(
+ expect.arrayContaining([
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['service-a'],
+ Value: Math.sqrt(2),
+ },
+ {
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['service-b'],
+ Value: Math.sqrt(2),
+ },
+ ]),
+ );
+ });
+
+ it('calculates each composite group quantile across multiple series in that group', async () => {
+ const metricName = 'test.groupby.route.method';
+ await seedGroupedCumulativeSeries({
+ metricName,
+ attributes: { route: '/orders', method: 'GET', instance: 'get-a' },
+ observations: [0],
+ });
+ await seedGroupedCumulativeSeries({
+ metricName,
+ attributes: { route: '/orders', method: 'GET', instance: 'get-b' },
+ observations: [0, 2, 2],
+ });
+ await seedGroupedCumulativeSeries({
+ metricName,
+ attributes: { route: '/orders', method: 'POST', instance: 'post-a' },
+ observations: [-4, -4],
+ });
+ await seedGroupedCumulativeSeries({
+ metricName,
+ attributes: { route: '/orders', method: 'POST', instance: 'post-b' },
+ observations: [-4, -4],
+ });
+
+ const results = await queryQuantile(0.5, metricName, {
+ groupBy: "Attributes['route'], Attributes['method']",
+ });
+ const get = results.find(result =>
+ _.isEqual(result.group, ['/orders', 'GET']),
+ );
+ const post = results.find(result =>
+ _.isEqual(result.group, ['/orders', 'POST']),
+ );
+
+ expect(results).toHaveLength(2);
+ // GET combines two zero counts and two positive counts across its two
+ // instances. Rank 2 lands at the end of the zero bucket, so p50 is zero.
+ expect(get).toEqual({
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ group: ['/orders', 'GET'],
+ Value: 0,
+ });
+ // POST combines four -4 observations across two instances. Halfway
+ // through negative bucket [-4, -2) is the logarithmic value -sqrt(8).
+ expect(post?.Value).toBeCloseTo(-Math.sqrt(8));
+ expect(post?.__hdx_time_bucket).toBe(
+ toClickHouseISOString(nowPlus('1m')),
+ );
+ });
+ });
+
+ describe('K8s Semantic Convention Migrations with metricNameSql', () => {
+ beforeEach(async () => {
+ // Insert gauge metrics with old semantic convention (ScopeVersion < 0.125.0)
+ const oldVersionGaugePoints = [
+ { value: 45, timestamp: now, ScopeVersion: '0.124.0' },
+ { value: 50, timestamp: now + ms('1m'), ScopeVersion: '0.124.0' },
+ { value: 55, timestamp: now + ms('2m'), ScopeVersion: '0.124.0' },
+ ].map(point => ({
+ MetricName: 'k8s.pod.cpu.utilization',
+ ServiceName: 'k8s-monitor',
+ ResourceAttributes: {
+ 'k8s.pod.name': 'test-pod',
+ 'k8s.namespace.name': 'default',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ ScopeVersion: point.ScopeVersion,
+ }));
+
+ // Insert gauge metrics with new semantic convention (ScopeVersion >= 0.125.0)
+ const newVersionGaugePoints = [
+ { value: 60, timestamp: now + ms('3m'), ScopeVersion: '0.125.0' },
+ { value: 65, timestamp: now + ms('4m'), ScopeVersion: '0.125.0' },
+ { value: 70, timestamp: now + ms('5m'), ScopeVersion: '0.126.0' },
+ ].map(point => ({
+ MetricName: 'k8s.pod.cpu.usage',
+ ServiceName: 'k8s-monitor',
+ ResourceAttributes: {
+ 'k8s.pod.name': 'test-pod',
+ 'k8s.namespace.name': 'default',
+ },
+ Value: point.value,
+ TimeUnix: new Date(point.timestamp),
+ ScopeVersion: point.ScopeVersion,
+ }));
+
+ await bulkInsertMetricsGauge([
+ ...oldVersionGaugePoints,
+ ...newVersionGaugePoints,
+ ]);
+ });
+
+ it('should query k8s.pod.cpu.utilization gauge metric using metricNameSql to handle both old and new conventions', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'avg',
+ metricName: 'k8s.pod.cpu.utilization',
+ metricNameSql:
+ "MetricName IN ('k8s.pod.cpu.utilization', 'k8s.pod.cpu.usage')",
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ const res = await queryData(query);
+ // Should return data from both old (k8s.pod.cpu.utilization) and new (k8s.pod.cpu.usage) metric names
+ expect(res.length).toBeGreaterThan(0);
+ expect(res).toMatchSnapshot();
+
+ // Verify the SQL contains the IN-based metric name condition
+ expect(query.sql).toContain('k8s.pod.cpu.usage');
+ expect(query.sql).toContain('k8s.pod.cpu.utilization');
+ expect(query.sql).toMatch(/MetricName IN /);
+ });
+
+ it('should handle gauge metric with metricNameSql and groupBy', async () => {
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'avg',
+ metricName: 'k8s.pod.cpu.utilization',
+ metricNameSql:
+ "MetricName IN ('k8s.pod.cpu.utilization', 'k8s.pod.cpu.usage')",
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '1 minute',
+ groupBy: `ResourceAttributes['k8s.pod.name']`,
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ const res = await queryData(query);
+ expect(res.length).toBeGreaterThan(0);
+ expect(res).toMatchSnapshot();
+ });
+
+ it('should merge exponential histogram buckets across metricNameSql aliases after calculating per-name deltas', async () => {
+ const attributes = { service: 'shared-service' };
+ await Promise.all([
+ seedExponentialHistogramMetric({
+ metricName: 'container.cpu.utilization',
+ points: [
+ {
+ TimeUnix: new Date(now),
+ Attributes: attributes,
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('1m'),
+ Attributes: attributes,
+ ...bucketExponentialHistogramObservations([2, 2, 2, 8]),
+ },
+ ],
+ }),
+ seedExponentialHistogramMetric({
+ metricName: 'container.cpu.usage',
+ points: [
+ {
+ TimeUnix: nowPlus('10s'),
+ Attributes: attributes,
+ ...bucketExponentialHistogramObservations([]),
+ },
+ {
+ TimeUnix: nowPlus('70s'),
+ Attributes: attributes,
+ ...bucketExponentialHistogramObservations([
+ 2, 2, 2, 2, 2, 2, 2, 8, 8, 8,
+ ]),
+ },
+ ],
+ }),
+ ]);
+
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'quantile',
+ level: 0.75,
+ metricName: 'container.cpu.utilization',
+ metricNameSql:
+ "MetricName IN ('container.cpu.utilization', 'container.cpu.usage')",
+ metricType: MetricsDataType.ExponentialHistogram,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), nowPlus('2m')],
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ const res = await queryData(query);
+ expect(res).toHaveLength(1);
+ expect(res[0]).toEqual({
+ __hdx_time_bucket: toClickHouseISOString(nowPlus('1m')),
+ Value: expect.closeTo(2 ** (17 / 8)),
+ });
+ });
+
+ it('should handle metrics without metricNameSql (backward compatibility)', async () => {
+ // Test querying the old metric name directly without migration SQL
+ const query = await renderChartConfig(
+ {
+ select: [
+ {
+ aggFn: 'avg',
+ metricName: 'k8s.pod.cpu.utilization',
+ // No metricNameSql provided - should query old name only
+ metricType: MetricsDataType.Gauge,
+ valueExpression: 'Value',
+ },
+ ],
+ from: metricSource.from,
+ where: '',
+ metricTables: TEST_METRIC_TABLES,
+ dateRange: [new Date(now), new Date(now + ms('10m'))],
+ granularity: '1 minute',
+ timestampValueExpression: metricSource.timestampValueExpression,
+ connection: connection.id,
+ },
+ metadata,
+ querySettings,
+ );
+
+ const res = await queryData(query);
+ // Should only return data from old metric name (k8s.pod.cpu.utilization)
+ expect(res).toMatchSnapshot();
+
+ // Verify the SQL uses simple string comparison (not IN-based)
+ expect(query.sql).toContain("MetricName = 'k8s.pod.cpu.utilization'");
+ expect(query.sql).not.toMatch(/MetricName IN /);
+ });
+ });
+
+ describe('Query settings', () => {
+ it('handles the the query settings', async () => {
+ const now = new Date('2023-11-16T22:12:00.000Z');
+
+ await bulkInsertLogs([
+ {
+ ServiceName: 'api',
+ Timestamp: now,
+ SeverityText: 'error',
+ Body: 'Oh no! Something went wrong!',
+ },
+ {
+ ServiceName: 'api',
+ Timestamp: now,
+ SeverityText: 'info',
+ Body: 'This is a test message.',
+ },
+ ]);
+
+ const query = await renderChartConfig(
+ {
+ select: [{ valueExpression: 'Body' }],
+ from: logSource.from,
+ where: '',
+ timestampValueExpression: 'Timestamp',
+ connection: connection.id,
+ settings: chSql`max_result_rows = 1`,
+ },
+ metadata,
+ [...querySettings, { setting: 'result_overflow_mode', value: 'break' }],
+ );
+
+ const res = await queryData(query);
+ // ensures `result_overflow_mode = break` is applied, otherwise query would error.
+ expect(res).toHaveLength(2);
+ expect(res).toMatchSnapshot();
+ });
+ });
+});
diff --git a/packages/api/src/clickhouse/__tests__/renderChartConfig.test.ts b/packages/api/src/clickhouse/__tests__/renderChartConfig.test.ts
deleted file mode 100644
index 7ea8a36442..0000000000
--- a/packages/api/src/clickhouse/__tests__/renderChartConfig.test.ts
+++ /dev/null
@@ -1,1670 +0,0 @@
-// TODO: we might want to move this test file to common-utils package
-
-import { ChSql, chSql } from '@hyperdx/common-utils/dist/clickhouse';
-import { ClickhouseClient } from '@hyperdx/common-utils/dist/clickhouse/node';
-import { getMetadata } from '@hyperdx/common-utils/dist/core/metadata';
-import { renderChartConfig } from '@hyperdx/common-utils/dist/core/renderChartConfig';
-import {
- MetricsDataType,
- QuerySettings,
-} from '@hyperdx/common-utils/dist/types';
-import _ from 'lodash';
-import ms from 'ms';
-
-import * as config from '@/config';
-import { createTeam } from '@/controllers/team';
-import {
- bulkInsertLogs,
- bulkInsertMetricsGauge,
- bulkInsertMetricsHistogram,
- bulkInsertMetricsSum,
- DEFAULT_DATABASE,
- DEFAULT_LOGS_TABLE,
- DEFAULT_METRICS_TABLE,
- executeSqlCommand,
- getServer,
-} from '@/fixtures';
-import Connection from '@/models/connection';
-import { Source } from '@/models/source';
-
-const TEST_METRIC_TABLES = {
- sum: DEFAULT_METRICS_TABLE.SUM,
- gauge: DEFAULT_METRICS_TABLE.GAUGE,
- histogram: DEFAULT_METRICS_TABLE.HISTOGRAM,
- summary: DEFAULT_METRICS_TABLE.SUMMARY,
- 'exponential histogram': DEFAULT_METRICS_TABLE.EXPONENTIAL_HISTOGRAM,
-};
-
-const querySettings: QuerySettings = [
- { setting: 'optimize_read_in_order', value: '0' },
- { setting: 'cast_keep_nullable', value: '1' },
- { setting: 'count_distinct_implementation', value: 'uniqCombined64' },
- { setting: 'async_insert_busy_timeout_min_ms', value: '20000' },
-];
-
-describe('renderChartConfig', () => {
- const server = getServer();
-
- const now = new Date('2022-01-05').getTime();
- let team, connection, logSource, metricSource, metadata;
- let clickhouseClient: ClickhouseClient;
-
- const nowPlus = time_val => new Date(now + ms(time_val));
-
- const queryData = async (chsql: ChSql) => {
- try {
- const res = await clickhouseClient.query<'JSON'>({
- query: chsql.sql,
- query_params: chsql.params,
- format: 'JSON',
- });
- const json = await res.json();
- return json.data;
- } catch (err) {
- console.error('[ClickhouseClient] Error:', err);
- throw err;
- }
- };
-
- beforeAll(async () => {
- await server.start();
- });
-
- afterAll(async () => {
- await server.stop();
- });
-
- beforeEach(async () => {
- team = await createTeam({ name: 'My Team' });
- connection = await Connection.create({
- team: team._id,
- name: 'Default',
- host: config.CLICKHOUSE_HOST,
- username: config.CLICKHOUSE_USER,
- password: config.CLICKHOUSE_PASSWORD,
- });
- logSource = await Source.create({
- kind: 'log',
- team: team._id,
- from: {
- databaseName: DEFAULT_DATABASE,
- tableName: DEFAULT_LOGS_TABLE,
- },
- timestampValueExpression: 'Timestamp',
- connection: connection.id,
- name: 'Logs',
- });
- metricSource = await Source.create({
- kind: 'metric',
- team: team._id,
- from: {
- databaseName: DEFAULT_DATABASE,
- tableName: '',
- },
- timestampValueExpression: 'TimeUnix',
- connection: connection.id,
- name: 'OTLPMetrics',
- });
- clickhouseClient = new ClickhouseClient({
- host: connection.host,
- username: connection.username,
- password: connection.password,
- });
- metadata = getMetadata(clickhouseClient);
- });
-
- afterEach(async () => {
- await server.clearDBs();
- jest.clearAllMocks();
- });
-
- describe('aggFn', () => {
- afterAll(async () => {
- await executeSqlCommand('DROP TABLE IF EXISTS agg_fn_str_test');
- await executeSqlCommand('DROP TABLE IF EXISTS agg_fn_default_test');
- });
-
- it('numeric agg functions should handle numeric values as strings', async () => {
- await executeSqlCommand(`
- CREATE TABLE agg_fn_str_test(
- ts UInt64,
- strVal String
- ) ENGINE = MergeTree
- ORDER BY ts
- `);
- await executeSqlCommand(`
- INSERT INTO agg_fn_str_test(ts, strVal) VALUES
- (fromUnixTimestamp64Milli(1519211811570), '3'),
- (fromUnixTimestamp64Milli(1519211811770), '-1'),
- (fromUnixTimestamp64Milli(1519211811870), '1.1'),
- (fromUnixTimestamp64Milli(1519211811970), '-1.1'),
- `);
-
- const query = await renderChartConfig(
- {
- select: [
- { aggFn: 'avg', valueExpression: 'strVal' },
- { aggFn: 'max', valueExpression: 'strVal' },
- { aggFn: 'min', valueExpression: 'strVal' },
- { aggFn: 'quantile', level: 0.5, valueExpression: 'strVal' },
- { aggFn: 'sum', valueExpression: 'strVal' },
- ],
- from: {
- databaseName: DEFAULT_DATABASE,
- tableName: `agg_fn_str_test`,
- },
- where: '',
- connection: connection.id,
- timestampValueExpression: 'ts',
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
-
- it('numeric agg functions should use default values for other types', async () => {
- await executeSqlCommand(`
- CREATE TABLE agg_fn_default_test(
- ts UInt64,
- strVal String,
- boolVal Bool,
- enumVal Enum('a' = 1, 'b' = 2),
- nullVal Nullable(String),
- ) ENGINE = MergeTree
- ORDER BY ts
- `);
- await executeSqlCommand(`
- INSERT INTO agg_fn_default_test(ts, strVal, boolVal, enumVal, nullVal) VALUES
- (fromUnixTimestamp64Milli(1519211811570), 'a', false, 'b', NULL)
- `);
- const query = await renderChartConfig(
- {
- select: [
- { aggFn: 'avg', valueExpression: 'strVal' },
- { aggFn: 'max', valueExpression: 'strVal' },
- { aggFn: 'min', valueExpression: 'strVal' },
- { aggFn: 'quantile', level: 0.5, valueExpression: 'strVal' },
- { aggFn: 'sum', valueExpression: 'strVal' },
- ],
- from: {
- databaseName: DEFAULT_DATABASE,
- tableName: `agg_fn_default_test`,
- },
- where: '',
- connection: connection.id,
- timestampValueExpression: 'ts',
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
- });
-
- describe('Query Events - Logs', () => {
- it('simple select + where query logs', async () => {
- const now = new Date('2023-11-16T22:12:00.000Z');
- await bulkInsertLogs([
- {
- ServiceName: 'api',
- Timestamp: now,
- SeverityText: 'error',
- Body: 'Oh no! Something went wrong!',
- },
- {
- ServiceName: 'api',
- Timestamp: now,
- SeverityText: 'info',
- Body: 'This is a test message.',
- },
- ]);
- const query = await renderChartConfig(
- {
- select: [
- {
- valueExpression: 'Body',
- },
- ],
- from: logSource.from,
- where: `SeverityText = 'error'`,
- timestampValueExpression: 'Timestamp',
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
-
- expect(await queryData(query)).toMatchSnapshot();
- });
-
- it('simple select + group by query logs', async () => {
- const now = new Date('2023-11-16T22:12:00.000Z');
- await bulkInsertLogs([
- {
- ServiceName: 'api',
- Timestamp: now,
- SeverityText: 'error',
- Body: 'Oh no! Something went wrong!',
- },
- {
- ServiceName: 'app',
- Timestamp: now,
- SeverityText: 'info',
- Body: 'This is a test message.',
- },
- ]);
- const query = await renderChartConfig(
- {
- select: [
- {
- valueExpression: 'ServiceName',
- },
- {
- valueExpression: 'count()',
- alias: 'count',
- },
- ],
- from: logSource.from,
- where: '',
- timestampValueExpression: 'Timestamp',
- connection: connection.id,
- groupBy: 'ServiceName',
- },
- metadata,
- querySettings,
- );
- expect(await queryData(query)).toMatchSnapshot();
- });
-
- // TODO: add more tests (including events chart, using filters, etc)
- });
-
- describe('Query Metrics - Gauge', () => {
- beforeEach(async () => {
- const gaugePointsA = [
- { value: 50, timestamp: now },
- { value: 25, timestamp: now + ms('1m') },
- { value: 12.5, timestamp: now + ms('2m') },
- { value: 6.25, timestamp: now + ms('3m') },
- { value: 100, timestamp: now + ms('6m') },
- { value: 75, timestamp: now + ms('7m') },
- { value: 10, timestamp: now + ms('8m') },
- { value: 80, timestamp: now + ms('9m') },
- ].map(point => ({
- MetricName: 'test.cpu',
- ServiceName: 'db',
- ResourceAttributes: {
- host: 'host1',
- ip: '127.0.0.1',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- }));
- const gaugePointsB = [
- { value: 1, timestamp: now },
- { value: 2, timestamp: now + ms('1m') },
- { value: 3, timestamp: now + ms('2m') },
- { value: 4, timestamp: now + ms('3m') },
- { value: 5, timestamp: now + ms('6m') },
- { value: 6, timestamp: now + ms('7m') },
- { value: 5, timestamp: now + ms('8m') },
- { value: 4, timestamp: now + ms('9m') },
- ].map(point => ({
- MetricName: 'test.cpu',
- ServiceName: 'db',
- ResourceAttributes: {
- host: 'host2',
- ip: '127.0.2',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- }));
- await bulkInsertMetricsGauge([...gaugePointsA, ...gaugePointsB]);
- });
-
- it('single max/avg/sum gauge', async () => {
- const avgQuery = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'avg',
- metricName: 'test.cpu',
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '5 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(avgQuery)).toMatchSnapshot();
- const maxQuery = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'max',
- metricName: 'test.cpu',
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '5 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(maxQuery)).toMatchSnapshot();
- const sumQuery = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'sum',
- metricName: 'test.cpu',
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '5 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(sumQuery)).toMatchSnapshot();
- });
-
- it('single avg gauge with where', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'avg',
- metricName: 'test.cpu',
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: 'ServiceName:"db" AND ResourceAttributes.host:"host1"',
- whereLanguage: 'lucene',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '5 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(query)).toMatchSnapshot();
- });
-
- it('single avg gauge with group-by', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'avg',
- metricName: 'test.cpu',
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '5 minute',
- groupBy: `ResourceAttributes['host']`,
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(query)).toMatchSnapshot();
- });
-
- it('single max gauge with delta', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'max',
- metricName: 'test.cpu',
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- isDelta: true,
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '5 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(query)).toMatchSnapshot();
- });
-
- it('single max gauge with delta and group by', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'max',
- metricName: 'test.cpu',
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- isDelta: true,
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '5 minute',
- groupBy: `ResourceAttributes['host']`,
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(query)).toMatchSnapshot();
- });
- });
-
- describe('Query Metrics - Sum', () => {
- beforeEach(async () => {
- // Rate: 8, 1, 8, 25
- const sumPointsA = [
- { value: 0, timestamp: now - ms('1m') }, // 0
- { value: 1, timestamp: now },
- { value: 8, timestamp: now + ms('4m') }, // 8
- { value: 8, timestamp: now + ms('6m') },
- { value: 9, timestamp: now + ms('9m') }, // 9
- { value: 15, timestamp: now + ms('11m') },
- { value: 17, timestamp: now + ms('14m') }, // 17
- { value: 32, timestamp: now + ms('16m') },
- { value: 42, timestamp: now + ms('19m') }, // 42
- ].map(point => ({
- MetricName: 'test.users',
- ServiceName: 'api',
- ResourceAttributes: {
- host: 'host1',
- ip: '127.0.0.1',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- IsMonotonic: true,
- AggregationTemporality: 2, // Cumulative
- }));
- // Rate: 11, 78, 5805, 78729
- // Sum: 12, 79, 5813, 78754
- const sumPointsB = [
- { value: 3, timestamp: now - ms('1m') }, // 3
- { value: 3, timestamp: now },
- { value: 14, timestamp: now + ms('4m') }, // 14
- { value: 15, timestamp: now + ms('6m') },
- { value: 92, timestamp: now + ms('9m') }, // 92
- { value: 653, timestamp: now + ms('11m') },
- { value: 5897, timestamp: now + ms('14m') }, // 5897
- { value: 9323, timestamp: now + ms('16m') },
- { value: 84626, timestamp: now + ms('19m') }, // 84626
- ].map(point => ({
- MetricName: 'test.users',
- ServiceName: 'api',
- ResourceAttributes: {
- host: 'host2',
- ip: '127.0.0.1',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- IsMonotonic: true,
- AggregationTemporality: 2, // Cumulative
- }));
- const sumPointsC = [
- { value: 0, timestamp: now - ms('1m') }, // 0
- { value: 1, timestamp: now },
- { value: 8, timestamp: now + ms('1m') }, // 8
- { value: 0, timestamp: now + ms('2m') },
- { value: 7, timestamp: now + ms('2m') },
- { value: 7, timestamp: now + ms('10m') }, // 9
- { value: 15, timestamp: now + ms('12m') },
- { value: 17, timestamp: now + ms('14m') }, // 17
- { value: 0, timestamp: now + ms('16m') },
- { value: 42, timestamp: now + ms('19m') }, // 42
- ].map(point => ({
- MetricName: 'counter.reset',
- ServiceName: 'api',
- ResourceAttributes: {
- host: 'host3',
- ip: '127.0.1',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- IsMonotonic: true,
- AggregationTemporality: 2, // Cumulative
- }));
- const sumPointsD = [
- { value: 0, timestamp: now - ms('1m') }, // 0
- { value: 1, timestamp: now },
- { value: 8, timestamp: now + ms('1m') }, // 8
- { value: 0, timestamp: now + ms('2m') },
- { value: 7, timestamp: now + ms('2m') },
- { value: 7, timestamp: now + ms('10m') }, // 9
- { value: 15, timestamp: now + ms('12m') },
- { value: 17, timestamp: now + ms('14m') }, // 17
- { value: 0, timestamp: now + ms('16m') },
- { value: 42, timestamp: now + ms('19m') }, // 42
- ].map(point => ({
- MetricName: 'counter.min_reset',
- ServiceName: 'api',
- ResourceAttributes: {
- host: 'MIN_VARIANT_0',
- ip: '127.0.1',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- IsMonotonic: true,
- AggregationTemporality: 2, // Cumulative
- }));
- const sumPointsE = [
- { value: 0, timestamp: now - ms('1m') },
- { value: 2, timestamp: now },
- { value: 9, timestamp: now + ms('1m') },
- { value: 0, timestamp: now + ms('2m') },
- { value: 15, timestamp: now + ms('2m') },
- { value: 25, timestamp: now + ms('10m') },
- { value: 35, timestamp: now + ms('12m') },
- { value: 57, timestamp: now + ms('14m') },
- { value: 0, timestamp: now + ms('16m') },
- { value: 92, timestamp: now + ms('19m') },
- ].map(point => ({
- MetricName: 'counter.min_reset',
- ServiceName: 'api',
- ResourceAttributes: {
- host: 'MAX_VARIANT_1',
- ip: '127.0.1',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- IsMonotonic: true,
- AggregationTemporality: 2, // Cumulative
- }));
- const podAgePoints = [
- { Value: 518400, TimeUnix: new Date(now - ms('1m')) },
- { Value: 604800, TimeUnix: new Date(now) },
- { Value: 691200, TimeUnix: new Date(now + ms('1m')) },
- { Value: 777600, TimeUnix: new Date(now + ms('2m')) },
- { Value: 864000, TimeUnix: new Date(now + ms('3m')) },
- { Value: 950400, TimeUnix: new Date(now + ms('4m')) },
- { Value: 1641600, TimeUnix: new Date(now + ms('12m')) },
- ].map(point => ({
- MetricName: 'k8s.pod.uptime',
- ServiceName: 'api',
- ResourceAttributes: {
- host: 'cluster-node-1',
- ip: '127.0.0.1',
- },
- IsMonotonic: true,
- AggregationTemporality: 2,
- ...point,
- }));
- await bulkInsertMetricsSum([
- ...sumPointsA,
- ...sumPointsB,
- ...sumPointsC,
- ...sumPointsD,
- ...sumPointsE,
- ...podAgePoints,
- ]);
- });
-
- it('single sum rate', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'sum',
- metricName: 'test.users',
- metricType: MetricsDataType.Sum,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('20m'))],
- granularity: '5 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(query)).toMatchSnapshot();
- });
-
- it('sum values as without rate computation', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- metricName: 'k8s.pod.uptime',
- metricType: MetricsDataType.Sum,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: 'ServiceName:api',
- whereLanguage: 'lucene',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('20m'))],
- granularity: '5 minutes',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(query)).toMatchSnapshot();
- });
-
- it('handles counter resets correctly for sum metrics', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'sum',
- metricName: 'counter.reset',
- metricType: MetricsDataType.Sum,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('20m'))],
- granularity: '10 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(query)).toMatchSnapshot();
- });
-
- it('calculates min_rate/max_rate correctly for sum metrics', async () => {
- // Raw Data is
- // MIN_VARIANT_0: [0, 1, 8, 0, 7, 7, 15, 17, 0, 42]
- // MIN_VARIANT_1: [0, 2, 9, 0, 15, 25 35, 57, 0, 92]
- //
- // Based on the data inserted in the fixture, the expected stream of values
- // for each series after adjusting for the zero reset should be:
- // MIN_VARIANT_0: [0, 1, 8, 8, 15, 15, 23, 25, 25, 67]
- // MIN_VARIANT_1: [0, 2, 9, 9, 24, 34, 44, 66, 66, 158]
- //
- // At the 10 minute buckets, should result in three buckets for each where
- // the first bucket is outside the query window.
- // MIN_VARIANT_0: [0], [1, 8, 8, 15], [15, 23, 25, 25, 67]]
- // MIN_VARIANT_1: [0], [2, 9, 9, 24], [34, 44, 66, 66, 158]]
- //
- // When comparing the value at the end of the buckets over the filtered
- // time frame it should result in the following counts added per bucket as:
- // MIN_VARIANT_0: [15, 52]
- // MIN_VARIANT_1: [24, 134]
- //
- // These values are what we apply the aggregation functions to.
- const minQuery = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'min',
- metricName: 'counter.min_reset',
- metricType: MetricsDataType.Sum,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('20m'))],
- granularity: '10 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(minQuery)).toMatchSnapshot('minSum');
-
- const maxQuery = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'max',
- metricName: 'counter.min_reset',
- metricType: MetricsDataType.Sum,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('20m'))],
- granularity: '10 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- expect(await queryData(maxQuery)).toMatchSnapshot('maxSum');
- });
-
- it('new series appearing mid-window: v2 suppresses first-row spike', async () => {
- // Scenario: An existing series (customerA) has data from the start,
- // and a new series (customerB) appears mid-window with a high
- // cumulative counter value (e.g. 10000). A naive inter-bucket diff
- // approach would leak the full cumulative value as a spike on the
- // first bucket of customerB. The v2 lagInFrame-based Rate yields 0
- // for the first row of each series, preventing the spike.
-
- // Use a unique metric name to isolate from the beforeEach data.
- const metricName = 'customer.events.total';
-
- // customerA: present from the start, steady increase
- const customerAPoints = [
- { value: 0, timestamp: now - ms('1m') },
- { value: 10, timestamp: now + ms('2m') },
- { value: 20, timestamp: now + ms('7m') },
- { value: 30, timestamp: now + ms('12m') },
- { value: 40, timestamp: now + ms('17m') },
- ].map(point => ({
- MetricName: metricName,
- ServiceName: 'api',
- ResourceAttributes: { customer: 'customerA' },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- IsMonotonic: true,
- AggregationTemporality: 2, // Cumulative
- }));
-
- // customerB: appears at +10m with a high cumulative value (10000),
- // then increases to 10050 by +17m. The real per-bucket increase is
- // only 50. A naive approach would show 10000 in the bucket where it
- // first appears.
- const customerBPoints = [
- { value: 10000, timestamp: now + ms('10m') },
- { value: 10020, timestamp: now + ms('12m') },
- { value: 10050, timestamp: now + ms('17m') },
- ].map(point => ({
- MetricName: metricName,
- ServiceName: 'api',
- ResourceAttributes: { customer: 'customerB' },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- IsMonotonic: true,
- AggregationTemporality: 2, // Cumulative
- }));
-
- await bulkInsertMetricsSum([...customerAPoints, ...customerBPoints]);
-
- // --- v2 query (current code with lagInFrame) ---
- const v2Query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'sum',
- metricName: metricName,
- metricType: MetricsDataType.Sum,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('20m'))],
- granularity: '5 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- const v2Results = await queryData(v2Query);
-
- // v2 should NOT have a spike — the max bucket value should be much
- // smaller than customerB's initial cumulative value (10000). The real
- // per-bucket increases across both series are at most ~50.
- const v2Values = v2Results.map((r: any) => Number(r.Value));
- const v2MaxValue = Math.max(...v2Values);
- expect(v2MaxValue).toBeLessThan(100);
-
- // Verify we actually got data back (not an empty result set).
- expect(v2Results.length).toBeGreaterThan(0);
- });
- });
-
- describe('Query Metrics - Histogram', () => {
- beforeEach(async () => {
- const histPointsA = [
- {
- BucketCounts: [0, 0, 0],
- TimeUnix: new Date(now),
- },
- {
- BucketCounts: [10, 10, 10],
- TimeUnix: new Date(now + ms('1m')),
- },
- ].map(point => ({
- MetricName: 'test.two_timestamps_lower_bound',
- ResourceAttributes: {
- host: 'test2',
- ip: '127.0.0.1',
- },
- AggregationTemporality: 2, // Cumulative
- ExplicitBounds: [10, 30],
- ...point,
- }));
- const histPointsB = [
- {
- BucketCounts: [0, 0, 0],
- TimeUnix: new Date(now),
- },
- {
- BucketCounts: [10, 0, 0],
- TimeUnix: new Date(now + ms('1m')),
- },
- ].map(point => ({
- MetricName: 'test.two_timestamps_lower_bound_inf',
- ResourceAttributes: {
- host: 'test2',
- ip: '127.0.0.1',
- },
- AggregationTemporality: 2, // Cumulative
- ExplicitBounds: [1, 30],
- ...point,
- }));
- const histPointsC = [
- {
- BucketCounts: [0, 0, 0],
- TimeUnix: new Date(now),
- },
- {
- BucketCounts: [0, 0, 10],
- TimeUnix: new Date(now + ms('1m')),
- },
- ].map(point => ({
- MetricName: 'test.two_timestamps_upper_bound_inf',
- ResourceAttributes: {
- host: 'test2',
- ip: '127.0.0.1',
- },
- AggregationTemporality: 2, // Cumulative
- ExplicitBounds: [0, 30],
- ...point,
- }));
- const histPointsD = [
- {
- BucketCounts: [5, 5, 5],
- TimeUnix: new Date(now),
- },
- {
- BucketCounts: [0, 0, 0],
- TimeUnix: new Date(now + ms('1m')),
- },
- {
- BucketCounts: [10, 10, 10],
- TimeUnix: new Date(now + ms('2m')),
- },
- ].map(point => ({
- MetricName: 'test.three_timestamps_bounded',
- ResourceAttributes: {
- host: 'test3',
- ip: '127.0.0.1',
- },
- AggregationTemporality: 2, // Cumulative
- ExplicitBounds: [10, 30],
- ...point,
- }));
- const histPointsE = [
- {
- BucketCounts: [1, 1, 1, 1, 1, 1],
- TimeUnix: new Date(now),
- ResourceAttributes: { host: 'test-a' },
- },
- {
- BucketCounts: [2, 2, 2, 2, 2, 2],
- TimeUnix: new Date(now + ms('5s')),
- ResourceAttributes: { host: 'test-b' },
- },
- {
- BucketCounts: [2, 1, 2, 1, 2, 1],
- TimeUnix: new Date(now + ms('1m')),
- ResourceAttributes: { host: 'test-a' },
- },
- {
- BucketCounts: [3, 3, 2, 2, 3, 3],
- TimeUnix: new Date(now + ms('65s')),
- ResourceAttributes: { host: 'test-b' },
- },
- ].map(point => ({
- MetricName: 'test.multiple_series',
- AggregationTemporality: 2, // Cumulative
- ExplicitBounds: [1, 2, 5, 8, 13],
- ...point,
- }));
- const histPointsF = [
- {
- TimeUnix: new Date(now),
- ResourceAttributes: { host: 'host-a', service: 'service-1' },
- BucketCounts: [5, 2, 0, 0, 0, 0],
- Count: 7,
- Sum: 18,
- },
- {
- TimeUnix: new Date(now),
- ResourceAttributes: { host: 'host-a', service: 'service-2' },
- BucketCounts: [11, 0, 0, 0, 0, 0],
- Count: 11,
- Sum: 22,
- },
- {
- TimeUnix: new Date(now),
- ResourceAttributes: { host: 'host-a', service: 'service-3' },
- BucketCounts: [0, 5, 0, 0, 0, 0],
- Count: 5,
- Sum: 31,
- },
- {
- TimeUnix: new Date(now),
- ResourceAttributes: { host: 'host-b', service: 'service-1' },
- BucketCounts: [6, 3, 0, 0, 0, 0],
- Count: 9,
- Sum: 37,
- },
- {
- TimeUnix: new Date(now),
- ResourceAttributes: { host: 'host-b', service: 'service-2' },
- BucketCounts: [1, 0, 0, 0, 0, 0],
- Count: 1,
- Sum: 3,
- },
- {
- TimeUnix: new Date(now),
- ResourceAttributes: { host: 'host-b', service: 'service-3' },
- BucketCounts: [4, 2, 0, 0, 0, 0],
- Count: 6,
- Sum: 25,
- },
- //----------
- {
- TimeUnix: nowPlus('30s'),
- ResourceAttributes: { host: 'host-a', service: 'service-1' },
- BucketCounts: [9, 3, 1, 0, 0, 0],
- Count: 13,
- Sum: 54,
- },
- {
- TimeUnix: nowPlus('34s'),
- ResourceAttributes: { host: 'host-a', service: 'service-2' },
- BucketCounts: [17, 2, 0, 0, 0, 0],
- Count: 19,
- Sum: 52,
- },
- {
- TimeUnix: nowPlus('38s'),
- ResourceAttributes: { host: 'host-a', service: 'service-3' },
- BucketCounts: [1, 7, 0, 0, 0, 0],
- Count: 8,
- Sum: 51,
- },
- {
- TimeUnix: nowPlus('29s'),
- ResourceAttributes: { host: 'host-b', service: 'service-1' },
- BucketCounts: [12, 3, 0, 0, 0, 0],
- Count: 15,
- Sum: 52,
- },
- {
- TimeUnix: nowPlus('31s'),
- ResourceAttributes: { host: 'host-b', service: 'service-2' },
- BucketCounts: [9, 1, 0, 0, 0, 0],
- Count: 10,
- Sum: 52,
- },
- {
- TimeUnix: nowPlus('45s'),
- ResourceAttributes: { host: 'host-b', service: 'service-3' },
- BucketCounts: [4, 6, 0, 0, 0, 0],
- Count: 10,
- Sum: 61,
- },
- //----------
- {
- TimeUnix: nowPlus('90s'),
- ResourceAttributes: { host: 'host-a', service: 'service-1' },
- BucketCounts: [9, 6, 6, 4, 2, 0],
- Count: 27,
- Sum: 1015,
- },
- {
- TimeUnix: nowPlus('94s'),
- ResourceAttributes: { host: 'host-a', service: 'service-2' },
- BucketCounts: [17, 4, 5, 7, 0, 0],
- Count: 33,
- Sum: 655,
- },
- {
- TimeUnix: nowPlus('98s'),
- ResourceAttributes: { host: 'host-a', service: 'service-3' },
- BucketCounts: [1, 7, 0, 3, 2, 4],
- Count: 17,
- Sum: 3741,
- },
- {
- TimeUnix: nowPlus('89s'),
- ResourceAttributes: { host: 'host-b', service: 'service-1' },
- BucketCounts: [19, 5, 0, 0, 0, 0],
- Count: 24,
- Sum: 85,
- },
- {
- TimeUnix: nowPlus('91s'),
- ResourceAttributes: { host: 'host-b', service: 'service-2' },
- BucketCounts: [12, 1, 1, 0, 0, 0],
- Count: 14,
- Sum: 109,
- },
- {
- TimeUnix: nowPlus('105s'),
- ResourceAttributes: { host: 'host-b', service: 'service-3' },
- BucketCounts: [4, 8, 0, 0, 0, 0],
- Count: 12,
- Sum: 79,
- },
- //----------
- {
- TimeUnix: nowPlus('150s'),
- ResourceAttributes: { host: 'host-a', service: 'service-1' },
- BucketCounts: [9, 9, 11, 8, 4, 0],
- Count: 41,
- Sum: 1955,
- },
- {
- TimeUnix: nowPlus('154s'),
- ResourceAttributes: { host: 'host-a', service: 'service-2' },
- BucketCounts: [17, 6, 10, 14, 0, 0],
- Count: 47,
- Sum: 1292,
- },
- {
- TimeUnix: nowPlus('158s'),
- ResourceAttributes: { host: 'host-a', service: 'service-3' },
- BucketCounts: [1, 7, 0, 3, 2, 18],
- Count: 31,
- Sum: 22459,
- },
- {
- TimeUnix: nowPlus('149s'),
- ResourceAttributes: { host: 'host-b', service: 'service-1' },
- BucketCounts: [26, 7, 0, 0, 0, 0],
- Count: 33,
- Sum: 120,
- },
- {
- TimeUnix: nowPlus('151s'),
- ResourceAttributes: { host: 'host-b', service: 'service-2' },
- BucketCounts: [15, 1, 2, 0, 0, 0],
- Count: 18,
- Sum: 155,
- },
- {
- TimeUnix: nowPlus('165s'),
- ResourceAttributes: { host: 'host-b', service: 'service-3' },
- BucketCounts: [4, 10, 0, 0, 0, 0],
- Count: 14,
- Sum: 95,
- },
- //----------
- {
- TimeUnix: nowPlus('210s'),
- ResourceAttributes: { host: 'host-a', service: 'service-1' },
- BucketCounts: [19, 12, 11, 8, 4, 0],
- Count: 54,
- Sum: 2003,
- },
- {
- TimeUnix: nowPlus('214s'),
- ResourceAttributes: { host: 'host-a', service: 'service-2' },
- BucketCounts: [37, 10, 10, 14, 0, 0],
- Count: 71,
- Sum: 1337,
- },
- {
- TimeUnix: nowPlus('218s'),
- ResourceAttributes: { host: 'host-a', service: 'service-3' },
- BucketCounts: [20, 0, 0, 0, 0, 0],
- Count: 20,
- Sum: 29,
- },
- {
- TimeUnix: nowPlus('209s'),
- ResourceAttributes: { host: 'host-b', service: 'service-1' },
- BucketCounts: [30, 9, 0, 0, 0, 0],
- Count: 39,
- Sum: 139,
- },
- {
- TimeUnix: nowPlus('211s'),
- ResourceAttributes: { host: 'host-b', service: 'service-2' },
- BucketCounts: [21, 4, 3, 0, 0, 0],
- Count: 28,
- Sum: 223,
- },
- {
- TimeUnix: nowPlus('225s'),
- ResourceAttributes: { host: 'host-b', service: 'service-3' },
- BucketCounts: [14, 11, 0, 0, 0, 0],
- Count: 25,
- Sum: 121,
- },
- ].map(point => ({
- MetricName: 'test.send_latency',
- AggregationTemporality: 2,
- ExplicitBounds: [5, 10, 50, 100, 500],
- ...point,
- }));
-
- await bulkInsertMetricsHistogram([
- ...histPointsA,
- ...histPointsB,
- ...histPointsC,
- ...histPointsD,
- ...histPointsE,
- ...histPointsF,
- ]);
- });
-
- it('two_timestamps_bounded histogram (p50)', async () => {
- /*
- This test starts with 2 data points with bounds of [10, 30]:
- t0: [0, 0, 0]
- t1: [10, 10, 10]
-
- Since the AggregationTemporality is 2(cumulative), we need to calculate the delta between the two points:
- delta: [10, 10, 10] - [0, 0, 0] = [10, 10, 10]
-
- We need to interpolate between the lower and upper bounds of the second bucket:
- cum sum = [10, 20, 30]
- rank = 0.5 * 30 = 15 (between bounds 10 - 30)
- interpolate: 10 + (30 - 10) * ((15 - 10) / (20 - 10)) = 20
- */
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'quantile',
- level: 0.5,
- metricName: 'test.two_timestamps_lower_bound',
- metricType: MetricsDataType.Histogram,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('2m'))],
- granularity: '1 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
-
- it('two_timestamps_bounded histogram (p90)', async () => {
- /*
- This test starts with 2 data points with bounds of [10, 30]:
- t0: [0, 0, 0]
- t1: [10, 10, 10]
-
- Since the AggregationTemporality is 2(cumulative), we need to calculate the delta between the two points:
- delta: [10, 10, 10] - [0, 0, 0] = [10, 10, 10]
-
- Total observations: 10 + 10 + 10 = 30
- Cumulative counts: [10, 20, 30]
- p90 point:
- Rank = 0.9 * 30 = 27
- This falls in the third bucket (since 20 < 27 ≤ 30)
-
- We need to interpolate between the lower and upper bounds of the third bucket:
- Lower bound: 30
- Upper bound: Infinity (but we use the upper bound of the previous bucket for the last bucket)
- Position in bucket: (27 - 20) / (30 - 20) = 0.7
- Interpolated value: 30 (since it's in the last bucket, we return the upper bound of the previous bucket)
-
- Thus the first point value would be 0 since it's at the start of the bounds.
- The second point value would be 30 since that is the 90th percentile point value delta from the first point.
- */
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'quantile',
- level: 0.9,
- metricName: 'test.two_timestamps_lower_bound',
- metricType: MetricsDataType.Histogram,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('2m'))],
- granularity: '1 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
-
- it('two_timestamps_bounded histogram (p25)', async () => {
- /*
- This test starts with 2 data points with bounds of [10, 30]:
- t0: [0, 0, 0]
- t1: [10, 10, 10]
-
- Since the AggregationTemporality is 2(cumulative), we need to calculate the delta between the two points:
- delta: [10, 10, 10] - [0, 0, 0] = [10, 10, 10]
-
- Total observations: 10 + 10 + 10 = 30
- Cumulative counts: [10, 20, 30]
- p25 point:
- Rank = 0.25 * 30 = 7.5
- This falls in the first bucket (since 0 < 7.5 ≤ 10)
-
- We need to interpolate between the lower and upper bounds of the first bucket:
- Lower bound: 0 (implicit lower bound for first bucket)
- Upper bound: 10 (first explicit bound)
- Position in bucket: 7.5 / 10 = 0.75
- Interpolated value: 0 + 0.75 * (10 - 0) = 7.5
-
- Since all observations are in the first bucket which has an upper bound of 1:
- For the first bucket (≤ 0), the algorithm would interpolate, but since all values are in this bucket and it's the lowest bucket, it would return 10
- Thus the value columns in res should be [0, 10]
- */
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'quantile',
- level: 0.25,
- metricName: 'test.two_timestamps_lower_bound',
- metricType: MetricsDataType.Histogram,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('2m'))],
- granularity: '1 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
-
- it('two_timestamps_lower_bound_inf histogram (p50)', async () => {
- /*
- This test starts with 2 data points with bounds of [1, 30]:
- t0: [0, 0, 0]
- t1: [10, 0, 0]
-
- Since the AggregationTemporality is 2(cumulative), we need to calculate the delta between the two points:
- delta: [10, 0, 0] - [0, 0, 0] = [10, 0, 0]
-
- Total observations: 10 + 0 + 0 = 10
- Cumulative counts: [10, 10, 10]
- p50 point:
- Rank = 0.5 * 10 = 5
- This falls in the first bucket (since 5 < 10)
-
- Since all observations are in the first bucket which has an upper bound of 1:
- For the first bucket (≤ 0), the algorithm would interpolate, but since all values are in this bucket and it's the lowest bucket, it would return 1
- Thus the value columns in res should be [0, 1]
- */
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'quantile',
- level: 0.5,
- metricName: 'test.two_timestamps_lower_bound_inf',
- metricType: MetricsDataType.Histogram,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('2m'))],
- granularity: '1 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
-
- it('should bucket correctly when no grouping is defined', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'quantile',
- level: 0.5,
- metricName: 'test.send_latency',
- metricType: MetricsDataType.Histogram,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('5m'))],
- granularity: '2 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
-
- it('should bucket correctly when grouping by a single attribute', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'quantile',
- level: 0.5,
- metricName: 'test.send_latency',
- metricType: MetricsDataType.Histogram,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('5m'))],
- groupBy: `ResourceAttributes['host']`,
- granularity: '2 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
-
- it('should bucket correctly when grouping by multiple attributes', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'quantile',
- level: 0.5,
- metricName: 'test.send_latency',
- metricType: MetricsDataType.Histogram,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('5m'))],
- groupBy: `ResourceAttributes['host'], ResourceAttributes['service']`,
- granularity: '2 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
- const res = await queryData(query);
- expect(res).toMatchSnapshot();
- });
- });
-
- describe('K8s Semantic Convention Migrations with metricNameSql', () => {
- beforeEach(async () => {
- // Insert gauge metrics with old semantic convention (ScopeVersion < 0.125.0)
- const oldVersionGaugePoints = [
- { value: 45, timestamp: now, ScopeVersion: '0.124.0' },
- { value: 50, timestamp: now + ms('1m'), ScopeVersion: '0.124.0' },
- { value: 55, timestamp: now + ms('2m'), ScopeVersion: '0.124.0' },
- ].map(point => ({
- MetricName: 'k8s.pod.cpu.utilization',
- ServiceName: 'k8s-monitor',
- ResourceAttributes: {
- 'k8s.pod.name': 'test-pod',
- 'k8s.namespace.name': 'default',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- ScopeVersion: point.ScopeVersion,
- }));
-
- // Insert gauge metrics with new semantic convention (ScopeVersion >= 0.125.0)
- const newVersionGaugePoints = [
- { value: 60, timestamp: now + ms('3m'), ScopeVersion: '0.125.0' },
- { value: 65, timestamp: now + ms('4m'), ScopeVersion: '0.125.0' },
- { value: 70, timestamp: now + ms('5m'), ScopeVersion: '0.126.0' },
- ].map(point => ({
- MetricName: 'k8s.pod.cpu.usage',
- ServiceName: 'k8s-monitor',
- ResourceAttributes: {
- 'k8s.pod.name': 'test-pod',
- 'k8s.namespace.name': 'default',
- },
- Value: point.value,
- TimeUnix: new Date(point.timestamp),
- ScopeVersion: point.ScopeVersion,
- }));
-
- await bulkInsertMetricsGauge([
- ...oldVersionGaugePoints,
- ...newVersionGaugePoints,
- ]);
- });
-
- it('should query k8s.pod.cpu.utilization gauge metric using metricNameSql to handle both old and new conventions', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'avg',
- metricName: 'k8s.pod.cpu.utilization',
- metricNameSql:
- "MetricName IN ('k8s.pod.cpu.utilization', 'k8s.pod.cpu.usage')",
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '1 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
-
- const res = await queryData(query);
- // Should return data from both old (k8s.pod.cpu.utilization) and new (k8s.pod.cpu.usage) metric names
- expect(res.length).toBeGreaterThan(0);
- expect(res).toMatchSnapshot();
-
- // Verify the SQL contains the IN-based metric name condition
- expect(query.sql).toContain('k8s.pod.cpu.usage');
- expect(query.sql).toContain('k8s.pod.cpu.utilization');
- expect(query.sql).toMatch(/MetricName IN /);
- });
-
- it('should handle gauge metric with metricNameSql and groupBy', async () => {
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'avg',
- metricName: 'k8s.pod.cpu.utilization',
- metricNameSql:
- "MetricName IN ('k8s.pod.cpu.utilization', 'k8s.pod.cpu.usage')",
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '1 minute',
- groupBy: `ResourceAttributes['k8s.pod.name']`,
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
-
- const res = await queryData(query);
- expect(res.length).toBeGreaterThan(0);
- expect(res).toMatchSnapshot();
- });
-
- it('should handle metrics without metricNameSql (backward compatibility)', async () => {
- // Test querying the old metric name directly without migration SQL
- const query = await renderChartConfig(
- {
- select: [
- {
- aggFn: 'avg',
- metricName: 'k8s.pod.cpu.utilization',
- // No metricNameSql provided - should query old name only
- metricType: MetricsDataType.Gauge,
- valueExpression: 'Value',
- },
- ],
- from: metricSource.from,
- where: '',
- metricTables: TEST_METRIC_TABLES,
- dateRange: [new Date(now), new Date(now + ms('10m'))],
- granularity: '1 minute',
- timestampValueExpression: metricSource.timestampValueExpression,
- connection: connection.id,
- },
- metadata,
- querySettings,
- );
-
- const res = await queryData(query);
- // Should only return data from old metric name (k8s.pod.cpu.utilization)
- expect(res).toMatchSnapshot();
-
- // Verify the SQL uses simple string comparison (not IN-based)
- expect(query.sql).toContain("MetricName = 'k8s.pod.cpu.utilization'");
- expect(query.sql).not.toMatch(/MetricName IN /);
- });
- });
-
- describe('Query settings', () => {
- it('handles the the query settings', async () => {
- const now = new Date('2023-11-16T22:12:00.000Z');
-
- await bulkInsertLogs([
- {
- ServiceName: 'api',
- Timestamp: now,
- SeverityText: 'error',
- Body: 'Oh no! Something went wrong!',
- },
- {
- ServiceName: 'api',
- Timestamp: now,
- SeverityText: 'info',
- Body: 'This is a test message.',
- },
- ]);
-
- const query = await renderChartConfig(
- {
- select: [{ valueExpression: 'Body' }],
- from: logSource.from,
- where: '',
- timestampValueExpression: 'Timestamp',
- connection: connection.id,
- settings: chSql`max_result_rows = 1`,
- },
- metadata,
- [...querySettings, { setting: 'result_overflow_mode', value: 'break' }],
- );
-
- const res = await queryData(query);
- // ensures `result_overflow_mode = break` is applied, otherwise query would error.
- expect(res).toHaveLength(2);
- expect(res).toMatchSnapshot();
- });
- });
-});
diff --git a/packages/api/src/config.ts b/packages/api/src/config.ts
index 63dcb36520..bd8813ba89 100644
--- a/packages/api/src/config.ts
+++ b/packages/api/src/config.ts
@@ -1,8 +1,9 @@
+import crypto from 'crypto';
+
const env = process.env;
// DEFAULTS
const DEFAULT_APP_TYPE = 'api';
-const DEFAULT_EXPRESS_SESSION = 'hyperdx is cool 👋';
const DEFAULT_FRONTEND_URL = env.HYPERDX_APP_PORT
? `http://localhost:${env.HYPERDX_APP_PORT}`
: '';
@@ -13,8 +14,20 @@ export const APP_TYPE = (env.APP_TYPE || DEFAULT_APP_TYPE) as
| 'api'
| 'scheduled-task';
export const CODE_VERSION = env.CODE_VERSION ?? '';
-export const EXPRESS_SESSION_SECRET = (env.EXPRESS_SESSION_SECRET ||
- DEFAULT_EXPRESS_SESSION) as string;
+// Only for single container local deployments, disable authentication
+export const IS_LOCAL_APP_MODE =
+ env.IS_LOCAL_APP_MODE === 'DANGEROUSLY_is_local_app_mode💀';
+const LEGACY_PUBLIC_SESSION_SECRET = 'hyperdx is cool 👋';
+// Authenticated deployments never use the public demo secret. Without a safe
+// configured value, generate one per process (sessions end on restart).
+export const IS_EXPRESS_SESSION_SECRET_GENERATED =
+ !env.EXPRESS_SESSION_SECRET ||
+ (!IS_LOCAL_APP_MODE &&
+ env.EXPRESS_SESSION_SECRET === LEGACY_PUBLIC_SESSION_SECRET);
+export const EXPRESS_SESSION_SECRET =
+ !IS_EXPRESS_SESSION_SECRET_GENERATED && env.EXPRESS_SESSION_SECRET
+ ? env.EXPRESS_SESSION_SECRET
+ : crypto.randomBytes(32).toString('hex');
export const FRONTEND_URL = (env.FRONTEND_URL ||
DEFAULT_FRONTEND_URL) as string;
const HYPERDX_IMAGE = env.HYPERDX_IMAGE;
@@ -28,6 +41,11 @@ export const IS_LOCAL_IMAGE = HYPERDX_IMAGE === 'all-in-one-noauth';
export const IS_INLINE_API = env.HDX_PREVIEW_INLINE_API === 'true';
export const FRONTEND_REDIRECT_BASE = IS_INLINE_API ? '' : FRONTEND_URL;
export const INGESTION_API_KEY = env.INGESTION_API_KEY ?? '';
+// Opt-in: emit the contrib `datadogreceiver` on the collector so a
+// Datadog Agent can ship APM traces (DD trace API -> OTLP -> ClickHouse).
+// Off by default because the receiver has no per-team bearer-token auth like
+// `otlp/hyperdx`, so enabling it opens an unauthenticated ingest port (:8126).
+export const ENABLE_DATADOG_RECEIVER = env.ENABLE_DATADOG_RECEIVER === 'true';
export const HYPERDX_API_KEY = env.HYPERDX_API_KEY as string;
export const HYPERDX_LOG_LEVEL = env.HYPERDX_LOG_LEVEL as string;
export const IS_CI = NODE_ENV === 'test';
@@ -38,12 +56,14 @@ export const OTEL_SERVICE_NAME = env.OTEL_SERVICE_NAME as string;
export const PORT = Number.parseInt(env.PORT as string);
export const OPAMP_PORT = Number.parseInt(env.OPAMP_PORT as string);
export const USAGE_STATS_ENABLED = env.USAGE_STATS_ENABLED !== 'false';
+export const WEBHOOK_HOSTNAME_ALLOWLIST = env.WEBHOOK_HOSTNAME_ALLOWLIST ?? '';
export const RUN_SCHEDULED_TASKS_EXTERNALLY =
env.RUN_SCHEDULED_TASKS_EXTERNALLY === 'true';
-// Only for single container local deployments, disable authentication
-export const IS_LOCAL_APP_MODE =
- env.IS_LOCAL_APP_MODE === 'DANGEROUSLY_is_local_app_mode💀';
+// 32-byte key (base64 or hex). Setting it is what turns on encryption of
+// stored third-party tokens; unset means they are stored in plain text. See
+// utils/tokenEncryption.ts.
+export const TOKEN_ENCRYPTION_KEY = env.TOKEN_ENCRYPTION_KEY;
// gascity fork: reverse-proxy (forward-auth) SSO. See GASCITY-FORK.md.
// Trust an identity header set by a trusted upstream gate (oauth2-proxy in front
@@ -53,7 +73,8 @@ export const IS_LOCAL_APP_MODE =
// client-supplied PROXY_AUTH_HEADER / PROXY_AUTH_SECRET_HEADER on inbound and
// re-set them only after a successful SSO, AND a NetworkPolicy MUST ensure only
// the gate can reach this port. PROXY_AUTH_SHARED_SECRET is a strong in-app
-// backstop (a value the browser never sees); PROXY_AUTH_ALLOWED_EMAIL_DOMAINS is
+// backstop (values the browser never sees); one optional previous value permits
+// an attended rotation window. PROXY_AUTH_ALLOWED_EMAIL_DOMAINS is
// defense-in-depth, NOT the primary control. With PROXY_AUTH_ENABLED unset, all
// of this is inert and stock auth behavior is unchanged.
export const IS_PROXY_AUTH_ENABLED = env.PROXY_AUTH_ENABLED === 'true';
@@ -67,6 +88,8 @@ export const PROXY_AUTH_ALLOWED_EMAIL_DOMAINS = (
.map(d => d.trim().toLowerCase())
.filter(Boolean);
export const PROXY_AUTH_SHARED_SECRET = env.PROXY_AUTH_SHARED_SECRET || '';
+export const PROXY_AUTH_SHARED_SECRET_PREVIOUS =
+ env.PROXY_AUTH_SHARED_SECRET_PREVIOUS || '';
export const PROXY_AUTH_SECRET_HEADER = (
env.PROXY_AUTH_SECRET_HEADER || 'x-hdx-proxy-auth-secret'
).toLowerCase();
@@ -87,6 +110,32 @@ if (IS_PROXY_AUTH_ENABLED && !PROXY_AUTH_SHARED_SECRET) {
'Refusing to start in header-only-trust mode.',
);
}
+if (
+ IS_PROXY_AUTH_ENABLED &&
+ PROXY_AUTH_SHARED_SECRET_PREVIOUS &&
+ PROXY_AUTH_SHARED_SECRET_PREVIOUS === PROXY_AUTH_SHARED_SECRET
+) {
+ throw new Error(
+ 'PROXY_AUTH_SHARED_SECRET_PREVIOUS must differ from ' +
+ 'PROXY_AUTH_SHARED_SECRET when proxy auth is enabled.',
+ );
+}
+// The bundled OpenTelemetry SDK's advanced network capture records every
+// incoming request header on the server span as the request arrives, before
+// consumeProxyAuthSecret can take the shared secret off it, and exports it
+// along with the session and gate cookies. Same truthy spellings as the SDK.
+if (
+ IS_PROXY_AUTH_ENABLED &&
+ ['true', 'yes', '1'].includes(
+ (env.HDX_NODE_ADVANCED_NETWORK_CAPTURE ?? '').toLowerCase().trim(),
+ )
+) {
+ throw new Error(
+ 'PROXY_AUTH_ENABLED=true is incompatible with ' +
+ 'HDX_NODE_ADVANCED_NETWORK_CAPTURE: it exports request headers, ' +
+ 'including the proxy-auth secret, before the app can remove them.',
+ );
+}
// Only used to bootstrap empty instances
export const DEFAULT_CONNECTIONS = env.DEFAULT_CONNECTIONS;
@@ -94,6 +143,22 @@ export const DEFAULT_SOURCES = env.DEFAULT_SOURCES;
export const IS_PROMQL_ENABLED = env.ENABLE_PROMQL === 'true';
+export const EXTERNAL_API_RATE_LIMIT_MAX = (() => {
+ const parsed = Number(env.EXTERNAL_API_RATE_LIMIT_MAX);
+ return Number.isSafeInteger(parsed) && parsed > 0 ? parsed : 100;
+})();
+
+export const EXTERNAL_API_SEARCH_DEFAULT_MAX_RESULTS = 100;
+
+export const EXTERNAL_API_SEARCH_MAX_RESULTS = (() => {
+ const parsed = Number(env.EXTERNAL_API_SEARCH_MAX_RESULTS);
+ // A cap below the default would make requests without maxResults invalid.
+ return Number.isSafeInteger(parsed) &&
+ parsed >= EXTERNAL_API_SEARCH_DEFAULT_MAX_RESULTS
+ ? parsed
+ : 2000;
+})();
+
// FOR CI ONLY
export const CLICKHOUSE_HOST = env.CLICKHOUSE_HOST as string;
export const CLICKHOUSE_USER = env.CLICKHOUSE_USER as string;
diff --git a/packages/api/src/controllers/__tests__/ai.test.ts b/packages/api/src/controllers/__tests__/ai.test.ts
index 2f7d3a048f..34387aa3ba 100644
--- a/packages/api/src/controllers/__tests__/ai.test.ts
+++ b/packages/api/src/controllers/__tests__/ai.test.ts
@@ -48,7 +48,10 @@ function setConfig(overrides: Record) {
Object.assign(mockConfig, overrides);
}
-import { getAIModel } from '@/controllers/ai';
+import { DisplayType, SourceKind } from '@hyperdx/common-utils/dist/types';
+
+import { getAIModel, getChartConfigFromResolvedConfig } from '@/controllers/ai';
+import type { ISource } from '@/models/source';
beforeEach(() => {
setConfig({});
@@ -252,3 +255,44 @@ describe('openai provider', () => {
});
});
});
+
+describe('getChartConfigFromResolvedConfig', () => {
+ // @ts-expect-error mock source only has the fields getChartConfigFromResolvedConfig reads
+ const source: ISource = {
+ id: 'source-id',
+ kind: SourceKind.Trace,
+ connection: 'connection-id',
+ from: { databaseName: 'default', tableName: 'otel_traces' },
+ timestampValueExpression: 'Timestamp',
+ };
+
+ it('maps percentile aggregations to quantile with a level', () => {
+ const result = getChartConfigFromResolvedConfig(
+ {
+ displayType: DisplayType.Table,
+ select: [
+ { aggregationFunction: 'max', property: 'Duration' },
+ {
+ aggregationFunction: 'p95',
+ property: 'Duration',
+ condition: "ServiceName = 'api'",
+ },
+ ],
+ groupBy: 'SpanName',
+ timeRange: 'Past 1h',
+ },
+ source,
+ );
+
+ expect(result.select).toEqual([
+ { aggFn: 'max', valueExpression: 'Duration' },
+ {
+ aggFn: 'quantile',
+ level: 0.95,
+ valueExpression: 'Duration',
+ aggCondition: "ServiceName = 'api'",
+ aggConditionLanguage: 'sql',
+ },
+ ]);
+ });
+});
diff --git a/packages/api/src/controllers/__tests__/alertHistory.int.test.ts b/packages/api/src/controllers/__tests__/alertHistory.int.test.ts
new file mode 100644
index 0000000000..12554ebcd8
--- /dev/null
+++ b/packages/api/src/controllers/__tests__/alertHistory.int.test.ts
@@ -0,0 +1,1456 @@
+import { ObjectId } from 'mongodb';
+import { Types } from 'mongoose';
+
+import {
+ ALERT_EVALUATION_GROUPS_LIMIT,
+ getAlertEvaluations,
+ getAlertTransitionsInRange,
+ getRecentAlertHistories,
+ getRecentAlertHistoriesBatch,
+} from '@/controllers/alertHistory';
+import { clearDBCollections, closeDB, connectDB } from '@/fixtures';
+import Alert, { AlertState } from '@/models/alert';
+import AlertHistory from '@/models/alertHistory';
+import Team from '@/models/team';
+
+describe('alertHistory controller', () => {
+ beforeAll(async () => {
+ await connectDB();
+ });
+
+ afterEach(async () => {
+ await clearDBCollections();
+ });
+
+ afterAll(async () => {
+ await closeDB();
+ });
+
+ describe('getRecentAlertHistories', () => {
+ it('should return empty array when no histories exist', async () => {
+ const alertId = new ObjectId();
+ const histories = await getRecentAlertHistories({
+ alertId,
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toEqual([]);
+ });
+
+ it('should return recent alert histories for a given alert', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const now = new Date(Date.now() - 60000);
+ const earlier = new Date(Date.now() - 120000);
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: now,
+ state: AlertState.ALERT,
+ counts: 5,
+ lastValues: [{ startTime: now, count: 5 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: earlier,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: earlier, count: 0 }],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(2);
+ expect(histories[0].createdAt).toEqual(now);
+ expect(histories[0].state).toBe(AlertState.ALERT);
+ expect(histories[0].counts).toBe(5);
+ expect(histories[1].createdAt).toEqual(earlier);
+ expect(histories[1].state).toBe(AlertState.OK);
+ expect(histories[1].counts).toBe(0);
+ });
+
+ it('should respect the limit parameter', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ // Create 5 histories
+ for (let i = 0; i < 5; i++) {
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: new Date(Date.now() - i * 60000),
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [
+ { startTime: new Date(Date.now() - i * 60000), count: 0 },
+ ],
+ });
+ }
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 3,
+ });
+
+ expect(histories).toHaveLength(3);
+ });
+
+ it('should group histories by createdAt timestamp', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const timestamp = new Date(Date.now() - 60000);
+
+ // Create multiple histories with the same timestamp
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: timestamp, count: 0 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: timestamp, count: 0 }],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(1);
+ expect(histories[0].createdAt).toEqual(timestamp);
+ expect(histories[0].counts).toBe(0); // 0 + 0
+ expect(histories[0].lastValues).toHaveLength(2);
+ });
+
+ it('should set state to ALERT if any grouped history has ALERT state', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const timestamp = new Date(Date.now() - 60000);
+
+ // Create histories with mixed states at the same timestamp
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: timestamp, count: 0 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.ALERT,
+ counts: 3,
+ lastValues: [{ startTime: timestamp, count: 3 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: timestamp, count: 0 }],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(1);
+ expect(histories[0].state).toBe(AlertState.ALERT);
+ expect(histories[0].counts).toBe(3); // 0 + 3 + 0
+ });
+
+ it('should set state to OK when all grouped histories are OK', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const timestamp = new Date(Date.now() - 60000);
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: timestamp, count: 0 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: timestamp, count: 0 }],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(1);
+ expect(histories[0].state).toBe(AlertState.OK);
+ });
+
+ it('should sort histories by createdAt in descending order', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const oldest = new Date(Date.now() - 180000);
+ const middle = new Date(Date.now() - 120000);
+ const newest = new Date(Date.now() - 60000);
+
+ // Create in random order
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: middle,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: middle, count: 0 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: newest,
+ state: AlertState.ALERT,
+ counts: 3,
+ lastValues: [{ startTime: newest, count: 3 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: oldest,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: oldest, count: 0 }],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(3);
+ expect(histories[0].createdAt).toEqual(newest);
+ expect(histories[1].createdAt).toEqual(middle);
+ expect(histories[2].createdAt).toEqual(oldest);
+ });
+
+ it('should sort lastValues by startTime in ascending order', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const timestamp = new Date(Date.now() - 60000);
+ const older = new Date(Date.now() - 120000);
+ const newer = new Date(Date.now() - 30000);
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: older, count: 0 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: newer, count: 0 }],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(1);
+ expect(histories[0].lastValues).toHaveLength(2);
+ expect(histories[0].lastValues[0].startTime).toEqual(older);
+ expect(histories[0].lastValues[1].startTime).toEqual(newer);
+ });
+
+ it('should only return histories for the specified alert', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert1 = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const alert2 = await Alert.create({
+ team: team._id,
+ threshold: 200,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const timestamp = new Date(Date.now() - 60000);
+
+ await AlertHistory.create({
+ alert: alert1._id,
+ createdAt: timestamp,
+ state: AlertState.ALERT,
+ counts: 5,
+ lastValues: [{ startTime: timestamp, count: 5 }],
+ });
+
+ await AlertHistory.create({
+ alert: alert2._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: timestamp, count: 0 }],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert1._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(1);
+ expect(histories[0].state).toBe(AlertState.ALERT);
+ expect(histories[0].counts).toBe(5);
+ });
+
+ it('surfaces ERROR windows with their recorded errors', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const errorWindow = new Date(Date.now() - 60000);
+ const okWindow = new Date(Date.now() - 120000);
+ const errorTimestamp = new Date(Date.now() - 55000);
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: errorWindow,
+ state: AlertState.ERROR,
+ counts: 0,
+ lastValues: [],
+ errors: [
+ {
+ timestamp: errorTimestamp,
+ type: 'QUERY_TIMEOUT',
+ message: 'Alert query did not complete within the 300s timeout',
+ },
+ ],
+ });
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: okWindow,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: okWindow, count: 0 }],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(2);
+ expect(histories[0].state).toBe(AlertState.ERROR);
+ expect(histories[0].errors).toHaveLength(1);
+ expect(histories[0].errors![0].type).toBe('QUERY_TIMEOUT');
+ expect(histories[0].errors![0].message).toContain('300s timeout');
+ expect(histories[1].state).toBe(AlertState.OK);
+ expect(histories[1].errors).toBeUndefined();
+ });
+
+ it('lets ALERT/PENDING outrank ERROR within a grouped window, but ERROR outrank OK', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const alertWindow = new Date(Date.now() - 60000);
+ const okWindow = new Date(Date.now() - 120000);
+ const makeError = () => ({
+ timestamp: new Date(),
+ type: 'WEBHOOK_ERROR',
+ message: 'Failed to send webhook notification.',
+ });
+
+ // Window that fired AND recorded a notification error → shows ALERT
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: alertWindow,
+ state: AlertState.ALERT,
+ counts: 2,
+ lastValues: [{ startTime: alertWindow, count: 2 }],
+ });
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: alertWindow,
+ state: AlertState.ERROR,
+ counts: 0,
+ lastValues: [],
+ errors: [makeError()],
+ });
+
+ // Window that was OK but the resolve notification failed → shows ERROR
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: okWindow,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: okWindow, count: 0 }],
+ });
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: okWindow,
+ state: AlertState.ERROR,
+ counts: 0,
+ lastValues: [],
+ errors: [makeError()],
+ });
+
+ const histories = await getRecentAlertHistories({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ });
+
+ expect(histories).toHaveLength(2);
+ expect(histories[0].state).toBe(AlertState.ALERT);
+ // Errors from the ERROR row are still surfaced on the merged window
+ expect(histories[0].errors).toHaveLength(1);
+ expect(histories[1].state).toBe(AlertState.ERROR);
+ expect(histories[1].errors).toHaveLength(1);
+ });
+ });
+
+ describe('getAlertEvaluations', () => {
+ const createAlert = async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ return Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+ };
+
+ const createOkWindow = (alertId: Types.ObjectId, createdAt: Date) =>
+ AlertHistory.create({
+ alert: alertId,
+ createdAt,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: createdAt, count: 0 }],
+ });
+
+ it('only returns windows within [startTime, endTime]', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const at = (minsAgo: number) => new Date(now - minsAgo * 60_000);
+
+ await createOkWindow(alert._id, at(5)); // after endTime
+ await createOkWindow(alert._id, at(15)); // in range
+ await createOkWindow(alert._id, at(20)); // in range
+ await createOkWindow(alert._id, at(40)); // before startTime
+
+ const page = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ startTime: at(30),
+ endTime: at(10),
+ });
+
+ expect(page.data).toHaveLength(2);
+ expect(page.data[0].createdAt).toEqual(at(15));
+ expect(page.data[1].createdAt).toEqual(at(20));
+ expect(page.hasMore).toBe(false);
+ expect(page.nextBefore).toBeUndefined();
+ });
+
+ it('paginates older windows via the nextBefore cursor when the page fills up', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const windows = [1, 2, 3, 4].map(i => new Date(now - i * 5 * 60_000));
+ for (const createdAt of windows) {
+ await createOkWindow(alert._id, createdAt);
+ }
+
+ const startTime = new Date(now - 60 * 60_000);
+ const endTime = new Date(now);
+
+ const firstPage = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 2,
+ startTime,
+ endTime,
+ });
+ expect(firstPage.data).toHaveLength(2);
+ expect(firstPage.data[0].createdAt).toEqual(windows[0]);
+ expect(firstPage.data[1].createdAt).toEqual(windows[1]);
+ expect(firstPage.hasMore).toBe(true);
+ expect(firstPage.nextBefore).toEqual(windows[1]);
+
+ const secondPage = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 2,
+ startTime,
+ endTime,
+ before: firstPage.nextBefore,
+ });
+ expect(secondPage.data).toHaveLength(2);
+ expect(secondPage.data[0].createdAt).toEqual(windows[2]);
+ expect(secondPage.data[1].createdAt).toEqual(windows[3]);
+ });
+
+ it('keeps paging across gaps: the scan bound truncates but nextBefore advances', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const at = (minsAgo: number) => new Date(now - minsAgo * 60_000);
+
+ // One recent window, then a gap far wider than the per-request scan
+ // bound ((limit + 1) × interval = 3 minutes for limit=2 / 1m interval),
+ // then an old window still inside the requested range.
+ await createOkWindow(alert._id, at(1));
+ await createOkWindow(alert._id, at(20));
+
+ const startTime = at(30);
+ const endTime = at(0);
+
+ // First page: finds the recent window; the scan bound stops long
+ // before startTime, so more may exist.
+ const firstPage = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '1m',
+ limit: 2,
+ startTime,
+ endTime,
+ });
+ expect(firstPage.data).toHaveLength(1);
+ expect(firstPage.data[0].createdAt).toEqual(at(1));
+ expect(firstPage.hasMore).toBe(true);
+ expect(firstPage.nextBefore).toBeDefined();
+
+ // Follow the cursor until the old window is found or the range is
+ // exhausted. Every hop advances by at least one scan bound, so this
+ // terminates.
+ let before = firstPage.nextBefore;
+ let found: Date | undefined;
+ for (let i = 0; i < 20 && before != null; i++) {
+ const page = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '1m',
+ limit: 2,
+ startTime,
+ endTime,
+ before,
+ });
+ if (page.data.length > 0) {
+ found = page.data[0].createdAt;
+ break;
+ }
+ expect(page.hasMore).toBe(true);
+ // Empty pages still advance the cursor
+ expect(page.nextBefore!.getTime()).toBeLessThan(before.getTime());
+ before = page.nextBefore;
+ }
+ expect(found).toEqual(at(20));
+ });
+
+ it('reports hasMore=false once the scan reaches startTime', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const at = (minsAgo: number) => new Date(now - minsAgo * 60_000);
+
+ await createOkWindow(alert._id, at(5));
+
+ const page = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ startTime: at(20),
+ endTime: at(0),
+ });
+
+ expect(page.data).toHaveLength(1);
+ expect(page.hasMore).toBe(false);
+ expect(page.nextBefore).toBeUndefined();
+ // Non-grouped rows carry no per-group breakdown
+ expect(page.data[0].groups).toBeUndefined();
+ expect(page.data[0].groupsTotal).toBeUndefined();
+ });
+
+ it('breaks grouped windows down per group, firing-first', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const windowStart = new Date(now - 5 * 60_000);
+ const bucket = new Date(now - 10 * 60_000);
+
+ const createGroupRow = (
+ group: string,
+ state: AlertState,
+ count: number,
+ fired?: boolean,
+ ) =>
+ AlertHistory.create({
+ alert: alert._id,
+ createdAt: windowStart,
+ state,
+ counts: state === AlertState.OK ? 0 : 1,
+ lastValues: [{ startTime: bucket, count }],
+ group,
+ ...(fired != null && { fired }),
+ });
+
+ await createGroupRow('ServiceName:web', AlertState.OK, 3);
+ await createGroupRow('ServiceName:api', AlertState.ALERT, 14, true);
+ await createGroupRow('ServiceName:worker', AlertState.PENDING, 9);
+ // A notification failure recorded for the window: contributes errors,
+ // never a group entry.
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: windowStart,
+ state: AlertState.ERROR,
+ counts: 0,
+ lastValues: [],
+ errors: [
+ {
+ timestamp: new Date(),
+ type: 'WEBHOOK_ERROR',
+ message: 'Failed to send webhook notification.',
+ },
+ ],
+ });
+
+ const page = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ startTime: new Date(now - 60 * 60_000),
+ endTime: new Date(now),
+ });
+
+ expect(page.data).toHaveLength(1);
+ const window = page.data[0];
+ // Overall window state merges the per-group states
+ expect(window.state).toBe(AlertState.ALERT);
+ expect(window.groupsTotal).toBe(3);
+ expect(window.groups).toHaveLength(3);
+ // Firing-first ordering: ALERT > PENDING > OK
+ expect(window.groups!.map(g => g.group)).toEqual([
+ 'ServiceName:api',
+ 'ServiceName:worker',
+ 'ServiceName:web',
+ ]);
+ expect(window.groups![0]).toMatchObject({
+ group: 'ServiceName:api',
+ state: AlertState.ALERT,
+ counts: 1,
+ fired: true,
+ });
+ expect(window.groups![0].lastValue).toMatchObject({ count: 14 });
+ expect(window.groups![2]).toMatchObject({
+ group: 'ServiceName:web',
+ state: AlertState.OK,
+ counts: 0,
+ });
+ // The ERROR row surfaces as window errors, not as a group
+ expect(window.errors).toHaveLength(1);
+ expect(window.groups!.every(g => g.state !== AlertState.ERROR)).toBe(
+ true,
+ );
+ });
+
+ it('surfaces evaluation analytics, preferring the successful evaluation over a failed attempt', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const windowStart = new Date(now - 5 * 60_000);
+ const bucket = new Date(now - 10 * 60_000);
+
+ // Failed first attempt for this window (its query ran 300s)
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: windowStart,
+ state: AlertState.ERROR,
+ counts: 0,
+ lastValues: [],
+ errors: [
+ {
+ timestamp: new Date(),
+ type: 'QUERY_TIMEOUT',
+ message: 'timed out',
+ },
+ ],
+ analytics: { queryDurationMs: 300_000 },
+ });
+ // Successful retry
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: windowStart,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: bucket, count: 1 }],
+ analytics: {
+ queryDurationMs: 1_200,
+ webhookDurationMs: 340,
+ backfilledBuckets: 0,
+ },
+ });
+
+ const page = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ startTime: new Date(now - 60 * 60_000),
+ endTime: new Date(now),
+ });
+
+ expect(page.data).toHaveLength(1);
+ expect(page.data[0].analytics).toMatchObject({
+ queryDurationMs: 1_200,
+ webhookDurationMs: 340,
+ backfilledBuckets: 0,
+ });
+ });
+
+ it('falls back to the failed attempt analytics when no successful row exists', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const windowStart = new Date(now - 5 * 60_000);
+
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: windowStart,
+ state: AlertState.ERROR,
+ counts: 0,
+ lastValues: [],
+ errors: [
+ {
+ timestamp: new Date(),
+ type: 'QUERY_TIMEOUT',
+ message: 'timed out',
+ },
+ ],
+ analytics: { queryDurationMs: 300_000 },
+ });
+
+ const page = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ startTime: new Date(now - 60 * 60_000),
+ endTime: new Date(now),
+ });
+
+ expect(page.data).toHaveLength(1);
+ expect(page.data[0].analytics).toMatchObject({
+ queryDurationMs: 300_000,
+ // Derived from lastValues (none) for rows lacking the field
+ backfilledBuckets: 0,
+ });
+ });
+
+ it('derives backfilled buckets from lastValues for rows written before analytics existed', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const windowStart = new Date(now - 5 * 60_000);
+ const bucketAt = (minsAgo: number) => new Date(now - minsAgo * 60_000);
+
+ // Legacy backfill row: one evaluation covering three buckets, no
+ // analytics field.
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: windowStart,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [
+ { startTime: bucketAt(20), count: 0 },
+ { startTime: bucketAt(15), count: 0 },
+ { startTime: bucketAt(10), count: 0 },
+ ],
+ });
+
+ const page = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ startTime: new Date(now - 60 * 60_000),
+ endTime: new Date(now),
+ });
+
+ expect(page.data).toHaveLength(1);
+ expect(page.data[0].analytics).toEqual({ backfilledBuckets: 2 });
+ });
+
+ it('caps the per-group breakdown and reports the pre-cap total', async () => {
+ const alert = await createAlert();
+ const now = Date.now();
+ const windowStart = new Date(now - 5 * 60_000);
+ const bucket = new Date(now - 10 * 60_000);
+
+ const total = ALERT_EVALUATION_GROUPS_LIMIT + 5;
+ // One firing group buried among many OK groups: firing-first ordering
+ // must keep it visible after the cap.
+ const rows = Array.from({ length: total }, (_, i) => ({
+ alert: alert._id,
+ createdAt: windowStart,
+ state: i === total - 1 ? AlertState.ALERT : AlertState.OK,
+ counts: i === total - 1 ? 1 : 0,
+ lastValues: [{ startTime: bucket, count: i }],
+ group: `ServiceName:svc-${String(i).padStart(3, '0')}`,
+ }));
+ await AlertHistory.insertMany(rows);
+
+ const page = await getAlertEvaluations({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ limit: 10,
+ startTime: new Date(now - 60 * 60_000),
+ endTime: new Date(now),
+ });
+
+ expect(page.data).toHaveLength(1);
+ const window = page.data[0];
+ expect(window.groupsTotal).toBe(total);
+ expect(window.groups).toHaveLength(ALERT_EVALUATION_GROUPS_LIMIT);
+ expect(window.groups![0].state).toBe(AlertState.ALERT);
+ expect(window.groups![0].group).toBe(
+ `ServiceName:svc-${String(total - 1).padStart(3, '0')}`,
+ );
+ });
+ });
+
+ describe('getRecentAlertHistoriesBatch', () => {
+ it('should return empty map when no alerts are provided', async () => {
+ const result = await getRecentAlertHistoriesBatch([], 20);
+ expect(result.size).toBe(0);
+ });
+
+ it('should return histories for multiple alerts in a single batch call', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert1 = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+ const alert2 = await Alert.create({
+ team: team._id,
+ threshold: 200,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const now = new Date(Date.now() - 60000);
+ const earlier = new Date(Date.now() - 120000);
+
+ await AlertHistory.create({
+ alert: alert1._id,
+ createdAt: now,
+ state: AlertState.ALERT,
+ counts: 5,
+ lastValues: [{ startTime: now, count: 5 }],
+ });
+ await AlertHistory.create({
+ alert: alert1._id,
+ createdAt: earlier,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: earlier, count: 0 }],
+ });
+ await AlertHistory.create({
+ alert: alert2._id,
+ createdAt: now,
+ state: AlertState.OK,
+ counts: 1,
+ lastValues: [{ startTime: now, count: 1 }],
+ });
+
+ const result = await getRecentAlertHistoriesBatch(
+ [
+ { alertId: new ObjectId(alert1._id), interval: '5m' },
+ { alertId: new ObjectId(alert2._id), interval: '5m' },
+ ],
+ 20,
+ );
+
+ expect(result.size).toBe(2);
+
+ const alert1Histories = result.get(alert1._id.toString());
+ expect(alert1Histories).toHaveLength(2);
+ expect(alert1Histories![0].createdAt).toEqual(now);
+ expect(alert1Histories![0].state).toBe(AlertState.ALERT);
+ expect(alert1Histories![1].createdAt).toEqual(earlier);
+ expect(alert1Histories![1].state).toBe(AlertState.OK);
+
+ const alert2Histories = result.get(alert2._id.toString());
+ expect(alert2Histories).toHaveLength(1);
+ expect(alert2Histories![0].state).toBe(AlertState.OK);
+ expect(alert2Histories![0].counts).toBe(1);
+ });
+
+ it('should return empty array for alerts with no history', async () => {
+ const alertId = new ObjectId();
+
+ const result = await getRecentAlertHistoriesBatch(
+ [{ alertId, interval: '5m' }],
+ 20,
+ );
+
+ expect(result.size).toBe(1);
+ expect(result.get(alertId.toString())).toEqual([]);
+ });
+
+ it('should respect the limit parameter per alert', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ // Create 5 histories
+ for (let i = 0; i < 5; i++) {
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: new Date(Date.now() - i * 60000),
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [
+ { startTime: new Date(Date.now() - i * 60000), count: 0 },
+ ],
+ });
+ }
+
+ const result = await getRecentAlertHistoriesBatch(
+ [{ alertId: new ObjectId(alert._id), interval: '5m' }],
+ 3,
+ );
+
+ expect(result.get(alert._id.toString())).toHaveLength(3);
+ });
+
+ it('should detect ALERT state when any grouped history has ALERT state', async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ const alert = await Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+
+ const timestamp = new Date(Date.now() - 60000);
+
+ // Create histories with mixed states at the same timestamp
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.OK,
+ counts: 0,
+ lastValues: [{ startTime: timestamp, count: 0 }],
+ });
+ await AlertHistory.create({
+ alert: alert._id,
+ createdAt: timestamp,
+ state: AlertState.ALERT,
+ counts: 3,
+ lastValues: [{ startTime: timestamp, count: 3 }],
+ });
+
+ const result = await getRecentAlertHistoriesBatch(
+ [{ alertId: new ObjectId(alert._id), interval: '5m' }],
+ 20,
+ );
+
+ const histories = result.get(alert._id.toString());
+ expect(histories).toHaveLength(1);
+ expect(histories![0].state).toBe(AlertState.ALERT);
+ expect(histories![0].counts).toBe(3);
+ });
+ });
+
+ describe('getAlertTransitionsInRange', () => {
+ // Minutes-ago helper so all fixtures stay recent (avoids the 30d TTL).
+ const now = Date.now();
+ const t = (minsAgo: number) => new Date(now - minsAgo * 60_000);
+
+ const createAlert = async () => {
+ const team = await Team.create({ name: 'Test Team' });
+ return Alert.create({
+ team: team._id,
+ threshold: 100,
+ interval: '5m',
+ channel: { type: null },
+ });
+ };
+
+ const createHistory = (
+ alertId: Types.ObjectId,
+ createdAt: Date,
+ state: AlertState,
+ counts: number,
+ ) =>
+ AlertHistory.create({
+ alert: alertId,
+ createdAt,
+ state,
+ counts,
+ lastValues: [{ startTime: createdAt, count: counts }],
+ });
+
+ it('returns empty array when no history exists', async () => {
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(),
+ interval: '5m',
+ startTime: t(60),
+ endTime: t(0),
+ });
+ expect(transitions).toEqual([]);
+ });
+
+ it('emits a firing and a recovery for an ALERT episode', async () => {
+ const alert = await createAlert();
+ await createHistory(alert._id, t(25), AlertState.OK, 0);
+ await createHistory(alert._id, t(20), AlertState.ALERT, 7);
+ await createHistory(alert._id, t(15), AlertState.ALERT, 8);
+ await createHistory(alert._id, t(10), AlertState.OK, 0);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(28),
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(2);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ expect(transitions[0].createdAt).toBe(t(20).toISOString());
+ expect(transitions[1].state).toBe(AlertState.OK);
+ expect(transitions[1].createdAt).toBe(t(10).toISOString());
+ });
+
+ it('detects a firing at the range edge using the preceding window', async () => {
+ const alert = await createAlert();
+ // Window just before the range establishes the prior (non-firing) state.
+ await createHistory(alert._id, t(30), AlertState.OK, 0);
+ await createHistory(alert._id, t(25), AlertState.ALERT, 4);
+ await createHistory(alert._id, t(20), AlertState.ALERT, 5);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(27),
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ expect(transitions[0].createdAt).toBe(t(25).toISOString());
+ });
+
+ it('emits a firing when history begins already in ALERT', async () => {
+ // A freshly-created alert that is firing from its first evaluation: there
+ // is no preceding non-ALERT window, but it should still be marked.
+ const alert = await createAlert();
+ await createHistory(alert._id, t(20), AlertState.ALERT, 1);
+ await createHistory(alert._id, t(15), AlertState.ALERT, 1);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(25),
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ expect(transitions[0].createdAt).toBe(t(20).toISOString());
+ });
+
+ it('pins a carry-in firing marker to the range start when already firing on entry', async () => {
+ // Firing before and throughout the range: a single firing marker should
+ // appear at the range start, not at any interior window.
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.ALERT, 5);
+ await createHistory(alert._id, t(25), AlertState.ALERT, 5);
+ await createHistory(alert._id, t(20), AlertState.ALERT, 5);
+
+ const startTime = t(27);
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime,
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ expect(transitions[0].createdAt).toBe(startTime.toISOString());
+ });
+
+ it('pins a carry-in marker when no window falls inside the range', async () => {
+ // Alert interval wider than the dashboard window: the only evaluation is
+ // before startTime, so nothing lands in-range, but the alert is firing.
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.ALERT, 1);
+
+ const startTime = t(20);
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '1h', // lookback reaches back past the t(30) window
+ startTime,
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ expect(transitions[0].createdAt).toBe(startTime.toISOString());
+ });
+
+ it('pins a carry-in firing then shows the in-range recovery', async () => {
+ // Fired before the range and recovered inside it: without the carry-in
+ // marker the recovery would appear orphaned.
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.ALERT, 5); // before range
+ await createHistory(alert._id, t(25), AlertState.ALERT, 5); // in range, firing
+ await createHistory(alert._id, t(20), AlertState.OK, 0); // recovery in range
+
+ const startTime = t(27);
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime,
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(2);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ expect(transitions[0].createdAt).toBe(startTime.toISOString());
+ expect(transitions[1].state).toBe(AlertState.OK);
+ expect(transitions[1].createdAt).toBe(t(20).toISOString());
+ });
+
+ it('treats PENDING as non-firing for boundary detection', async () => {
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.OK, 0);
+ await createHistory(alert._id, t(25), AlertState.PENDING, 2);
+ await createHistory(alert._id, t(20), AlertState.ALERT, 6);
+ await createHistory(alert._id, t(15), AlertState.PENDING, 1);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(27),
+ endTime: t(5),
+ });
+
+ expect(transitions.map(tr => tr.state)).toEqual([
+ AlertState.ALERT,
+ AlertState.OK,
+ ]);
+ expect(transitions[0].createdAt).toBe(t(20).toISOString());
+ expect(transitions[1].createdAt).toBe(t(15).toISOString());
+ });
+
+ it('ignores ERROR windows so a failed evaluation mid-firing is not a recovery', async () => {
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.OK, 0);
+ await createHistory(alert._id, t(25), AlertState.ALERT, 3);
+ // Evaluation failed mid-episode — must not read as a recovery + refire
+ await createHistory(alert._id, t(20), AlertState.ERROR, 0);
+ await createHistory(alert._id, t(15), AlertState.ALERT, 4);
+ await createHistory(alert._id, t(10), AlertState.OK, 0);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(40),
+ endTime: t(0),
+ });
+
+ expect(transitions.map(tr => tr.state)).toEqual([
+ AlertState.ALERT,
+ AlertState.OK,
+ ]);
+ expect(transitions[0].createdAt).toBe(t(25).toISOString());
+ expect(transitions[1].createdAt).toBe(t(10).toISOString());
+ });
+
+ it('only considers history for the specified alert', async () => {
+ const alert1 = await createAlert();
+ const alert2 = await createAlert();
+ await createHistory(alert1._id, t(30), AlertState.OK, 0);
+ await createHistory(alert1._id, t(25), AlertState.ALERT, 3);
+ // Noise on another alert must not leak in.
+ await createHistory(alert2._id, t(25), AlertState.OK, 0);
+ await createHistory(alert2._id, t(20), AlertState.ALERT, 9);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert1._id),
+ interval: '5m',
+ startTime: t(27),
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ });
+
+ describe('bucketStart', () => {
+ const createHistoryWithBuckets = (
+ alertId: Types.ObjectId,
+ createdAt: Date,
+ state: AlertState,
+ bucketStarts: Date[],
+ group?: string,
+ ) =>
+ AlertHistory.create({
+ alert: alertId,
+ createdAt,
+ state,
+ counts: state === AlertState.ALERT ? 1 : 0,
+ ...(group != null ? { group } : {}),
+ lastValues: bucketStarts.map(startTime => ({
+ startTime,
+ count: 1,
+ })),
+ });
+
+ it('emits the newest evaluated bucket start for each transition', async () => {
+ // Evaluation at t(20) covered 1-minute buckets t(25)..t(21) (anomaly
+ // alerts bucket finer than the interval) — the marker belongs on the
+ // newest bucket, where the chart plots the transitioning value.
+ const alert = await createAlert();
+ await createHistory(alert._id, t(25), AlertState.OK, 0);
+ await createHistoryWithBuckets(alert._id, t(20), AlertState.ALERT, [
+ t(25),
+ t(24),
+ t(23),
+ t(22),
+ t(21),
+ ]);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(22),
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].createdAt).toBe(t(20).toISOString());
+ expect(transitions[0].bucketStart).toBe(t(21).toISOString());
+ });
+
+ it('falls back to createdAt − interval when the window has no lastValues', async () => {
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.OK, 0);
+ await createHistoryWithBuckets(alert._id, t(20), AlertState.ALERT, []);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(28),
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].bucketStart).toBe(t(25).toISOString());
+ });
+
+ it("derives a recovery's bucketStart from the recovering window's lastValues", async () => {
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.OK, 0);
+ await createHistoryWithBuckets(alert._id, t(25), AlertState.ALERT, [
+ t(27),
+ t(26),
+ ]);
+ await createHistoryWithBuckets(alert._id, t(20), AlertState.OK, [
+ t(22),
+ t(21),
+ ]);
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(26),
+ endTime: t(5),
+ });
+
+ expect(transitions.map(tr => tr.state)).toEqual([
+ AlertState.ALERT,
+ AlertState.OK,
+ ]);
+ expect(transitions[1].createdAt).toBe(t(20).toISOString());
+ expect(transitions[1].bucketStart).toBe(t(21).toISOString());
+ });
+
+ it('floors bucketStart at the range start for an edge crossing', async () => {
+ // A crossing whose evaluation lands just inside the range derives a
+ // bucketStart one bucket earlier — before startTime. It is floored at
+ // startTime so the marker never renders left of a carry-in pin.
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.OK, 0);
+ await createHistoryWithBuckets(alert._id, t(25), AlertState.ALERT, []);
+
+ const startTime = t(25); // crossing evaluated exactly at range start
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime,
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ // Empty lastValues → createdAt − interval = t(30), before startTime —
+ // floored to startTime.
+ expect(transitions[0].bucketStart).toBe(startTime.toISOString());
+ });
+
+ it('keeps a recovery after the carry-in pin it follows', async () => {
+ // Firing on entry, recovering on the first in-range tick: the
+ // recovery's derived bucket start precedes startTime, but the carry-in
+ // pin sits at startTime — the floor keeps firing → recovery ordered.
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.ALERT, 5);
+ await createHistoryWithBuckets(alert._id, t(25), AlertState.OK, []);
+
+ const startTime = t(25);
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime,
+ endTime: t(5),
+ });
+
+ expect(transitions.map(tr => tr.state)).toEqual([
+ AlertState.ALERT,
+ AlertState.OK,
+ ]);
+ expect(transitions[0].bucketStart).toBe(startTime.toISOString());
+ expect(
+ new Date(transitions[1].bucketStart!) >=
+ new Date(transitions[0].bucketStart!),
+ ).toBe(true);
+ });
+
+ it('takes the newest bucket across group rows in one window', async () => {
+ // Group-by alerts write one history row per group; the marker should
+ // reflect the newest bucket evaluated by any of them.
+ const alert = await createAlert();
+ await createHistory(alert._id, t(25), AlertState.OK, 0);
+ await createHistoryWithBuckets(
+ alert._id,
+ t(20),
+ AlertState.ALERT,
+ [t(23)],
+ 'group-a',
+ );
+ await createHistoryWithBuckets(
+ alert._id,
+ t(20),
+ AlertState.OK,
+ [t(21)],
+ 'group-b',
+ );
+
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime: t(22),
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].state).toBe(AlertState.ALERT);
+ expect(transitions[0].bucketStart).toBe(t(21).toISOString());
+ });
+
+ it('pins the carry-in marker bucketStart to the range start', async () => {
+ const alert = await createAlert();
+ await createHistory(alert._id, t(30), AlertState.ALERT, 5);
+ await createHistory(alert._id, t(25), AlertState.ALERT, 5);
+
+ const startTime = t(27);
+ const transitions = await getAlertTransitionsInRange({
+ alertId: new ObjectId(alert._id),
+ interval: '5m',
+ startTime,
+ endTime: t(5),
+ });
+
+ expect(transitions).toHaveLength(1);
+ expect(transitions[0].bucketStart).toBe(startTime.toISOString());
+ });
+ });
+ });
+});
diff --git a/packages/api/src/controllers/__tests__/alertHistory.test.ts b/packages/api/src/controllers/__tests__/alertHistory.test.ts
deleted file mode 100644
index ab14bfaa18..0000000000
--- a/packages/api/src/controllers/__tests__/alertHistory.test.ts
+++ /dev/null
@@ -1,521 +0,0 @@
-import { ObjectId } from 'mongodb';
-
-import {
- getRecentAlertHistories,
- getRecentAlertHistoriesBatch,
-} from '@/controllers/alertHistory';
-import { clearDBCollections, closeDB, connectDB } from '@/fixtures';
-import Alert, { AlertState } from '@/models/alert';
-import AlertHistory from '@/models/alertHistory';
-import Team from '@/models/team';
-
-describe('alertHistory controller', () => {
- beforeAll(async () => {
- await connectDB();
- });
-
- afterEach(async () => {
- await clearDBCollections();
- });
-
- afterAll(async () => {
- await closeDB();
- });
-
- describe('getRecentAlertHistories', () => {
- it('should return empty array when no histories exist', async () => {
- const alertId = new ObjectId();
- const histories = await getRecentAlertHistories({
- alertId,
- interval: '5m',
- limit: 10,
- });
-
- expect(histories).toEqual([]);
- });
-
- it('should return recent alert histories for a given alert', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- const now = new Date(Date.now() - 60000);
- const earlier = new Date(Date.now() - 120000);
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: now,
- state: AlertState.ALERT,
- counts: 5,
- lastValues: [{ startTime: now, count: 5 }],
- });
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: earlier,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: earlier, count: 0 }],
- });
-
- const histories = await getRecentAlertHistories({
- alertId: new ObjectId(alert._id),
- interval: '5m',
- limit: 10,
- });
-
- expect(histories).toHaveLength(2);
- expect(histories[0].createdAt).toEqual(now);
- expect(histories[0].state).toBe(AlertState.ALERT);
- expect(histories[0].counts).toBe(5);
- expect(histories[1].createdAt).toEqual(earlier);
- expect(histories[1].state).toBe(AlertState.OK);
- expect(histories[1].counts).toBe(0);
- });
-
- it('should respect the limit parameter', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- // Create 5 histories
- for (let i = 0; i < 5; i++) {
- await AlertHistory.create({
- alert: alert._id,
- createdAt: new Date(Date.now() - i * 60000),
- state: AlertState.OK,
- counts: 0,
- lastValues: [
- { startTime: new Date(Date.now() - i * 60000), count: 0 },
- ],
- });
- }
-
- const histories = await getRecentAlertHistories({
- alertId: new ObjectId(alert._id),
- interval: '5m',
- limit: 3,
- });
-
- expect(histories).toHaveLength(3);
- });
-
- it('should group histories by createdAt timestamp', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- const timestamp = new Date(Date.now() - 60000);
-
- // Create multiple histories with the same timestamp
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: timestamp, count: 0 }],
- });
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: timestamp, count: 0 }],
- });
-
- const histories = await getRecentAlertHistories({
- alertId: new ObjectId(alert._id),
- interval: '5m',
- limit: 10,
- });
-
- expect(histories).toHaveLength(1);
- expect(histories[0].createdAt).toEqual(timestamp);
- expect(histories[0].counts).toBe(0); // 0 + 0
- expect(histories[0].lastValues).toHaveLength(2);
- });
-
- it('should set state to ALERT if any grouped history has ALERT state', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- const timestamp = new Date(Date.now() - 60000);
-
- // Create histories with mixed states at the same timestamp
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: timestamp, count: 0 }],
- });
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.ALERT,
- counts: 3,
- lastValues: [{ startTime: timestamp, count: 3 }],
- });
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: timestamp, count: 0 }],
- });
-
- const histories = await getRecentAlertHistories({
- alertId: new ObjectId(alert._id),
- interval: '5m',
- limit: 10,
- });
-
- expect(histories).toHaveLength(1);
- expect(histories[0].state).toBe(AlertState.ALERT);
- expect(histories[0].counts).toBe(3); // 0 + 3 + 0
- });
-
- it('should set state to OK when all grouped histories are OK', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- const timestamp = new Date(Date.now() - 60000);
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: timestamp, count: 0 }],
- });
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: timestamp, count: 0 }],
- });
-
- const histories = await getRecentAlertHistories({
- alertId: new ObjectId(alert._id),
- interval: '5m',
- limit: 10,
- });
-
- expect(histories).toHaveLength(1);
- expect(histories[0].state).toBe(AlertState.OK);
- });
-
- it('should sort histories by createdAt in descending order', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- const oldest = new Date(Date.now() - 180000);
- const middle = new Date(Date.now() - 120000);
- const newest = new Date(Date.now() - 60000);
-
- // Create in random order
- await AlertHistory.create({
- alert: alert._id,
- createdAt: middle,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: middle, count: 0 }],
- });
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: newest,
- state: AlertState.ALERT,
- counts: 3,
- lastValues: [{ startTime: newest, count: 3 }],
- });
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: oldest,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: oldest, count: 0 }],
- });
-
- const histories = await getRecentAlertHistories({
- alertId: new ObjectId(alert._id),
- interval: '5m',
- limit: 10,
- });
-
- expect(histories).toHaveLength(3);
- expect(histories[0].createdAt).toEqual(newest);
- expect(histories[1].createdAt).toEqual(middle);
- expect(histories[2].createdAt).toEqual(oldest);
- });
-
- it('should sort lastValues by startTime in ascending order', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- const timestamp = new Date(Date.now() - 60000);
- const older = new Date(Date.now() - 120000);
- const newer = new Date(Date.now() - 30000);
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: older, count: 0 }],
- });
-
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: newer, count: 0 }],
- });
-
- const histories = await getRecentAlertHistories({
- alertId: new ObjectId(alert._id),
- interval: '5m',
- limit: 10,
- });
-
- expect(histories).toHaveLength(1);
- expect(histories[0].lastValues).toHaveLength(2);
- expect(histories[0].lastValues[0].startTime).toEqual(older);
- expect(histories[0].lastValues[1].startTime).toEqual(newer);
- });
-
- it('should only return histories for the specified alert', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert1 = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- const alert2 = await Alert.create({
- team: team._id,
- threshold: 200,
- interval: '5m',
- channel: { type: null },
- });
-
- const timestamp = new Date(Date.now() - 60000);
-
- await AlertHistory.create({
- alert: alert1._id,
- createdAt: timestamp,
- state: AlertState.ALERT,
- counts: 5,
- lastValues: [{ startTime: timestamp, count: 5 }],
- });
-
- await AlertHistory.create({
- alert: alert2._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: timestamp, count: 0 }],
- });
-
- const histories = await getRecentAlertHistories({
- alertId: new ObjectId(alert1._id),
- interval: '5m',
- limit: 10,
- });
-
- expect(histories).toHaveLength(1);
- expect(histories[0].state).toBe(AlertState.ALERT);
- expect(histories[0].counts).toBe(5);
- });
- });
-
- describe('getRecentAlertHistoriesBatch', () => {
- it('should return empty map when no alerts are provided', async () => {
- const result = await getRecentAlertHistoriesBatch([], 20);
- expect(result.size).toBe(0);
- });
-
- it('should return histories for multiple alerts in a single batch call', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert1 = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
- const alert2 = await Alert.create({
- team: team._id,
- threshold: 200,
- interval: '5m',
- channel: { type: null },
- });
-
- const now = new Date(Date.now() - 60000);
- const earlier = new Date(Date.now() - 120000);
-
- await AlertHistory.create({
- alert: alert1._id,
- createdAt: now,
- state: AlertState.ALERT,
- counts: 5,
- lastValues: [{ startTime: now, count: 5 }],
- });
- await AlertHistory.create({
- alert: alert1._id,
- createdAt: earlier,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: earlier, count: 0 }],
- });
- await AlertHistory.create({
- alert: alert2._id,
- createdAt: now,
- state: AlertState.OK,
- counts: 1,
- lastValues: [{ startTime: now, count: 1 }],
- });
-
- const result = await getRecentAlertHistoriesBatch(
- [
- { alertId: new ObjectId(alert1._id), interval: '5m' },
- { alertId: new ObjectId(alert2._id), interval: '5m' },
- ],
- 20,
- );
-
- expect(result.size).toBe(2);
-
- const alert1Histories = result.get(alert1._id.toString());
- expect(alert1Histories).toHaveLength(2);
- expect(alert1Histories![0].createdAt).toEqual(now);
- expect(alert1Histories![0].state).toBe(AlertState.ALERT);
- expect(alert1Histories![1].createdAt).toEqual(earlier);
- expect(alert1Histories![1].state).toBe(AlertState.OK);
-
- const alert2Histories = result.get(alert2._id.toString());
- expect(alert2Histories).toHaveLength(1);
- expect(alert2Histories![0].state).toBe(AlertState.OK);
- expect(alert2Histories![0].counts).toBe(1);
- });
-
- it('should return empty array for alerts with no history', async () => {
- const alertId = new ObjectId();
-
- const result = await getRecentAlertHistoriesBatch(
- [{ alertId, interval: '5m' }],
- 20,
- );
-
- expect(result.size).toBe(1);
- expect(result.get(alertId.toString())).toEqual([]);
- });
-
- it('should respect the limit parameter per alert', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- // Create 5 histories
- for (let i = 0; i < 5; i++) {
- await AlertHistory.create({
- alert: alert._id,
- createdAt: new Date(Date.now() - i * 60000),
- state: AlertState.OK,
- counts: 0,
- lastValues: [
- { startTime: new Date(Date.now() - i * 60000), count: 0 },
- ],
- });
- }
-
- const result = await getRecentAlertHistoriesBatch(
- [{ alertId: new ObjectId(alert._id), interval: '5m' }],
- 3,
- );
-
- expect(result.get(alert._id.toString())).toHaveLength(3);
- });
-
- it('should detect ALERT state when any grouped history has ALERT state', async () => {
- const team = await Team.create({ name: 'Test Team' });
- const alert = await Alert.create({
- team: team._id,
- threshold: 100,
- interval: '5m',
- channel: { type: null },
- });
-
- const timestamp = new Date(Date.now() - 60000);
-
- // Create histories with mixed states at the same timestamp
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.OK,
- counts: 0,
- lastValues: [{ startTime: timestamp, count: 0 }],
- });
- await AlertHistory.create({
- alert: alert._id,
- createdAt: timestamp,
- state: AlertState.ALERT,
- counts: 3,
- lastValues: [{ startTime: timestamp, count: 3 }],
- });
-
- const result = await getRecentAlertHistoriesBatch(
- [{ alertId: new ObjectId(alert._id), interval: '5m' }],
- 20,
- );
-
- const histories = result.get(alert._id.toString());
- expect(histories).toHaveLength(1);
- expect(histories![0].state).toBe(AlertState.ALERT);
- expect(histories![0].counts).toBe(3);
- });
- });
-});
diff --git a/packages/api/src/controllers/__tests__/alerts.test.ts b/packages/api/src/controllers/__tests__/alerts.test.ts
new file mode 100644
index 0000000000..a9de723a9c
--- /dev/null
+++ b/packages/api/src/controllers/__tests__/alerts.test.ts
@@ -0,0 +1,138 @@
+import {
+ AlertChartConfig,
+ DisplayType,
+} from '@hyperdx/common-utils/dist/types';
+
+import { makeAlert } from '@/controllers/alerts';
+import { AlertChannel, AlertSource, AlertThresholdType } from '@/models/alert';
+
+// A channel type this repo doesn't define -- see
+// models/__tests__/alert.test.ts for why. `value: any` (rather than an `as`
+// cast) keeps this off the no-unsafe-type-assertion budget while still
+// producing a value typed as AlertChannel for the call below.
+const foreignChannel = (value: any): AlertChannel => value;
+
+describe('makeAlert', () => {
+ // makeAlert mirrors channels[0] into `channel` for readers that predate
+ // multi-channel support. That mirroring must stay opaque: a downstream
+ // fork's channel types must survive verbatim, not get projected onto
+ // webhook-shaped fields.
+ it('mirrors channels[0] into channel verbatim, preserving fields this repo does not define', () => {
+ const exotic = foreignChannel({
+ type: 'email',
+ emailRecipients: ['ops@example.test'],
+ });
+
+ const result = makeAlert({
+ interval: '5m',
+ threshold: 1,
+ thresholdType: AlertThresholdType.ABOVE,
+ channels: [exotic],
+ });
+
+ expect(result.channel).toEqual(exotic);
+ expect(result.channels).toEqual([exotic]);
+ });
+
+ const chartConfig: AlertChartConfig = {
+ name: 'Errors',
+ source: 'source-id',
+ displayType: DisplayType.Line,
+ select: [
+ {
+ aggFn: 'count',
+ aggCondition: '',
+ aggConditionLanguage: 'lucene',
+ valueExpression: '',
+ },
+ ],
+ where: '',
+ whereLanguage: 'lucene',
+ };
+
+ it('persists chartConfig for inline alerts and clears the other source references', () => {
+ const result = makeAlert({
+ interval: '5m',
+ threshold: 1,
+ thresholdType: AlertThresholdType.ABOVE,
+ channels: [{ type: 'webhook', webhookId: 'webhook-id' }],
+ source: AlertSource.INLINE,
+ chartConfig,
+ });
+
+ expect(result.chartConfig).toEqual(chartConfig);
+ expect(result.savedSearch).toBeNull();
+ expect(result.groupBy).toBeNull();
+ expect(result.dashboard).toBeNull();
+ expect(result.tileId).toBeNull();
+ });
+
+ it('clears chartConfig when the alert source is not chart', () => {
+ // Converting an inline alert to another source must not leave the old
+ // config behind (mirrors how savedSearch/dashboard references clear).
+ const result = makeAlert({
+ interval: '5m',
+ threshold: 1,
+ thresholdType: AlertThresholdType.ABOVE,
+ channels: [{ type: 'webhook', webhookId: 'webhook-id' }],
+ source: AlertSource.SAVED_SEARCH,
+ savedSearchId: 'saved-search-id',
+ chartConfig,
+ });
+
+ expect(result.chartConfig).toBeNull();
+ expect(result.savedSearch).toBe('saved-search-id');
+ });
+
+ describe('displayName and tags', () => {
+ const base = {
+ interval: '5m' as const,
+ threshold: 1,
+ thresholdType: AlertThresholdType.ABOVE,
+ channels: [{ type: 'webhook' as const, webhookId: 'webhook-id' }],
+ source: AlertSource.SAVED_SEARCH,
+ savedSearchId: 'saved-search-id',
+ };
+ const refs = {
+ savedSearch: { name: 'Checkout errors', tags: ['checkout'] },
+ };
+
+ it('derives from the referenced entity when omitted', () => {
+ const result = makeAlert(base, undefined, refs);
+
+ expect(result.displayName).toBe('Checkout errors');
+ expect(result.tags).toEqual(['checkout']);
+ });
+
+ it('stores explicit values', () => {
+ const result = makeAlert(
+ { ...base, displayName: 'Custom', tags: [] },
+ undefined,
+ refs,
+ );
+
+ expect(result.displayName).toBe('Custom');
+ expect(result.tags).toEqual([]);
+ });
+
+ // Storing the read-path fallback would freeze "Alert" onto a document
+ // whose saved search is perfectly nameable.
+ it('stores null when the referenced entity was not loaded', () => {
+ const result = makeAlert(base);
+
+ expect(result.displayName).toBeNull();
+ expect(result.tags).toBeNull();
+ });
+
+ it('treats null as "derive"', () => {
+ const result = makeAlert(
+ { ...base, displayName: null, tags: null },
+ undefined,
+ refs,
+ );
+
+ expect(result.displayName).toBe('Checkout errors');
+ expect(result.tags).toEqual(['checkout']);
+ });
+ });
+});
diff --git a/packages/api/src/controllers/__tests__/alertsPage.test.ts b/packages/api/src/controllers/__tests__/alertsPage.test.ts
new file mode 100644
index 0000000000..c3362e3099
--- /dev/null
+++ b/packages/api/src/controllers/__tests__/alertsPage.test.ts
@@ -0,0 +1,226 @@
+import { MAX_TAGS } from '@hyperdx/common-utils/dist/types';
+
+import {
+ alertsPageQuerySchema,
+ buildAlertsPageFilter,
+} from '@/controllers/alertsPage';
+import { AlertSource, AlertState } from '@/models/alert';
+import { encodeCursor } from '@/utils/pagination';
+
+const TEAM_ID = '6500000000000000000000aa';
+const ALERT_ID = '6500000000000000000000bb';
+
+const parse = (query: Record) =>
+ alertsPageQuerySchema.parse(query);
+
+const buildFilter = (query: Record = {}) =>
+ buildAlertsPageFilter(TEAM_ID, parse(query));
+
+describe('alertsPageQuerySchema', () => {
+ it('accepts an empty query', () => {
+ expect(parse({})).toEqual({});
+ });
+
+ it('normalizes a repeatable param whether it arrives once or many times', () => {
+ expect(parse({ tag: 'prod' }).tag).toEqual(['prod']);
+ expect(parse({ tag: ['prod', 'web'] }).tag).toEqual(['prod', 'web']);
+ expect(parse({ tag: '' }).tag).toBeUndefined();
+ });
+
+ it('rejects more tags than an alert can carry', () => {
+ expect(() => parse({ tag: Array(MAX_TAGS + 1).fill('t') })).toThrow();
+ });
+
+ it('coerces limit and rejects out-of-range values', () => {
+ expect(parse({ limit: '25' }).limit).toBe(25);
+ expect(() => parse({ limit: '0' })).toThrow();
+ expect(() => parse({ limit: '501' })).toThrow();
+ expect(() => parse({ limit: '1.5' })).toThrow();
+ });
+
+ it('accepts limit and tag count at their inclusive bounds', () => {
+ expect(parse({ limit: '1' }).limit).toBe(1);
+ expect(parse({ limit: '500' }).limit).toBe(500);
+ expect(parse({ tag: Array(MAX_TAGS).fill('t') }).tag).toHaveLength(
+ MAX_TAGS,
+ );
+ });
+
+ it('trims search and rejects an overlong one', () => {
+ expect(parse({ search: ' errors ' }).search).toBe('errors');
+ expect(() => parse({ search: 'x'.repeat(513) })).toThrow();
+ });
+
+ it('rejects values outside the source and state enums', () => {
+ expect(parse({ source: 'tile' }).source).toEqual([AlertSource.TILE]);
+ expect(() => parse({ source: 'nope' })).toThrow();
+ expect(() => parse({ state: 'nope' })).toThrow();
+ });
+
+ it('rejects a non-ObjectId createdBy', () => {
+ expect(() => parse({ createdBy: 'not-an-id' })).toThrow();
+ });
+
+ it('survives re-parsing its own output', () => {
+ // processRequest writes the parsed query back onto req.query, and the
+ // handler parses that again to recover the coerced types.
+ const once = parse({
+ limit: '2',
+ tag: ['prod', 'web'],
+ source: 'tile',
+ state: 'OK',
+ search: ' errors ',
+ createdBy: ALERT_ID,
+ });
+ expect(alertsPageQuerySchema.parse(once)).toEqual(once);
+ });
+
+ it('rejects a cursor with no limit to continue', () => {
+ const cursor = encodeCursor({ n: 'a', id: ALERT_ID });
+ expect(() => parse({ cursor })).toThrow('cursor requires limit');
+ expect(parse({ cursor, limit: '2' }).cursor).toBe(cursor);
+ });
+
+ it.each([
+ ['garbage', 'not a cursor!'],
+ ['a non-ObjectId id', encodeCursor({ n: 'a', id: 'nope' })],
+ // JSON.stringify drops an undefined value, so a writer that doesn't
+ // normalize a missing displayName to null issues a cursor like this.
+ ['a missing name', encodeCursor({ id: ALERT_ID })],
+ ])('rejects a cursor with %s', (_label, cursor) => {
+ expect(() => parse({ cursor, limit: '2' })).toThrow('invalid cursor');
+ });
+});
+
+describe('buildAlertsPageFilter', () => {
+ it('scopes to the team and nothing else when unfiltered', () => {
+ expect(buildFilter()).toEqual({ team: TEAM_ID });
+ });
+
+ it('matches any requested tag', () => {
+ expect(buildFilter({ tag: ['prod', 'web'] }).tags).toEqual({
+ $in: ['prod', 'web'],
+ });
+ });
+
+ it('matches documents with no source field when saved_search is requested', () => {
+ // `source` postdates the first alerts; those documents are saved searches.
+ expect(
+ buildFilter({ source: [AlertSource.SAVED_SEARCH, AlertSource.TILE] })
+ .source,
+ ).toEqual({ $in: [AlertSource.SAVED_SEARCH, AlertSource.TILE, null] });
+
+ expect(buildFilter({ source: AlertSource.TILE }).source).toEqual({
+ $in: [AlertSource.TILE],
+ });
+ });
+
+ it('matches documents with no state field when ok is requested', () => {
+ // `state` postdates the first alerts; those documents hydrate as OK.
+ expect(
+ buildFilter({ state: [AlertState.OK, AlertState.ALERT] }).state,
+ ).toEqual({ $in: [AlertState.OK, AlertState.ALERT, null] });
+
+ expect(buildFilter({ state: AlertState.ALERT }).state).toEqual({
+ $in: [AlertState.ALERT],
+ });
+ });
+
+ it('filters on state and createdBy by equality', () => {
+ const filter = buildFilter({
+ state: [AlertState.ALERT],
+ createdBy: ALERT_ID,
+ });
+ expect(filter.state).toEqual({ $in: [AlertState.ALERT] });
+ expect(filter.createdBy).toBe(ALERT_ID);
+ });
+
+ it('treats regex metacharacters in search as literals', () => {
+ expect(buildFilter({ search: 'a.b(c' }).displayName).toEqual({
+ $regex: 'a\\.b\\(c',
+ $options: 'i',
+ });
+ });
+
+ it('adds no displayName predicate for a blank search', () => {
+ // A cleared search box sends `search=`, and trimming a whitespace-only one
+ // lands in the same place. Neither may become a match-everything regex.
+ expect(buildFilter({ search: '' }).displayName).toBeUndefined();
+ expect(buildFilter({ search: ' ' }).displayName).toBeUndefined();
+ });
+
+ it('pages past a named alert with a scoped tie-break', () => {
+ const filter = buildFilter({
+ limit: '2',
+ cursor: encodeCursor({ n: 'Checkout', id: ALERT_ID }),
+ });
+
+ // The _id tie-break is scoped to the cursor's own name so neither branch
+ // rescans the run of rows sharing it.
+ expect(filter.displayName).toBeUndefined();
+ expect(filter.$or).toEqual([
+ { displayName: { $gt: 'Checkout' } },
+ { displayName: 'Checkout', _id: { $gt: ALERT_ID } },
+ ]);
+ });
+
+ it('pages out of the unnamed block into the named alerts', () => {
+ const filter = buildFilter({
+ limit: '2',
+ cursor: encodeCursor({ n: null, id: ALERT_ID }),
+ });
+
+ expect(filter.displayName).toBeUndefined();
+ expect(filter.$or).toEqual([
+ { displayName: null, _id: { $gt: ALERT_ID } },
+ { displayName: { $gte: '' } },
+ ]);
+ });
+
+ it('keeps the search predicate alongside a named cursor', () => {
+ const filter = buildFilter({
+ limit: '2',
+ search: 'err',
+ cursor: encodeCursor({ n: 'Checkout', id: ALERT_ID }),
+ });
+
+ expect(filter.displayName).toEqual({ $regex: 'err', $options: 'i' });
+ expect(filter.$or).toEqual([
+ { displayName: { $gt: 'Checkout' } },
+ { displayName: 'Checkout', _id: { $gt: ALERT_ID } },
+ ]);
+ });
+
+ it('keeps the search predicate alongside a null cursor', () => {
+ const filter = buildFilter({
+ limit: '2',
+ search: 'err',
+ cursor: encodeCursor({ n: null, id: ALERT_ID }),
+ });
+
+ expect(filter.displayName).toEqual({ $regex: 'err', $options: 'i' });
+ expect(filter.$or).toEqual([
+ { displayName: null, _id: { $gt: ALERT_ID } },
+ { displayName: { $gte: '' } },
+ ]);
+ });
+
+ it('leaves $or to the cursor when the widened filters are applied', () => {
+ // The missing-field branches spell "documents with no source/state field"
+ // as `$in: [..., null]` rather than a second `$or`, which would overwrite
+ // the cursor's and silently restart the walk.
+ const filter = buildFilter({
+ limit: '2',
+ source: AlertSource.SAVED_SEARCH,
+ state: AlertState.OK,
+ cursor: encodeCursor({ n: null, id: ALERT_ID }),
+ });
+
+ expect(filter.source).toEqual({ $in: [AlertSource.SAVED_SEARCH, null] });
+ expect(filter.state).toEqual({ $in: [AlertState.OK, null] });
+ expect(filter.$or).toEqual([
+ { displayName: null, _id: { $gt: ALERT_ID } },
+ { displayName: { $gte: '' } },
+ ]);
+ });
+});
diff --git a/packages/api/src/controllers/__tests__/metadataCacheLimits.int.test.ts b/packages/api/src/controllers/__tests__/metadataCacheLimits.int.test.ts
new file mode 100644
index 0000000000..bf20db2a40
--- /dev/null
+++ b/packages/api/src/controllers/__tests__/metadataCacheLimits.int.test.ts
@@ -0,0 +1,245 @@
+import http from 'http';
+import request from 'supertest';
+
+import app from '@/api-app';
+import { ClickhouseClient } from '@/clickhouse';
+import * as config from '@/config';
+import * as aiController from '@/controllers/ai';
+import {
+ clearDBCollections,
+ closeTestFixtureClickHouseClient,
+ connectDB,
+ executeSqlCommand,
+} from '@/fixtures';
+import { mongooseConnection } from '@/models';
+import Connection from '@/models/connection';
+import { LogSource } from '@/models/source';
+import User, { type UserDocument } from '@/models/user';
+
+jest.mock('@/config', () => ({
+ ...jest.requireActual('@/config'),
+ AI_PROVIDER: 'openai',
+ AI_API_KEY: 'test-only',
+ AI_MODEL_NAME: 'test-only',
+ AI_BASE_URL: 'http://127.0.0.1:1',
+}));
+
+jest.mock('ai', () => ({
+ ...jest.requireActual('ai'),
+ generateText: jest.fn().mockResolvedValue({
+ output: {
+ displayType: 'table',
+ select: [{ aggregationFunction: 'count', property: 'Body' }],
+ timeRange: 'Past 1h',
+ },
+ }),
+}));
+
+const server = http.createServer(app);
+const agent = request.agent(server);
+const metadataSpy = jest.spyOn(aiController, 'getAIMetadata');
+const querySpy = jest.spyOn(ClickhouseClient.prototype, 'query');
+const tables: string[] = [];
+let user: UserDocument;
+// Stabilize hourly cache keys without giving MongoDB expired test sessions.
+const initialTime = new Date();
+initialTime.setUTCMinutes(15, 0, 0);
+const bucketTime = new Date(initialTime);
+bucketTime.setUTCMinutes(0, 0, 0);
+const fixtureTimestamp = bucketTime
+ .toISOString()
+ .slice(0, 19)
+ .replace('T', ' ');
+
+beforeAll(async () => {
+ jest.useFakeTimers({
+ now: initialTime,
+ doNotFake: [
+ 'hrtime',
+ 'nextTick',
+ 'performance',
+ 'queueMicrotask',
+ 'setImmediate',
+ 'clearImmediate',
+ 'setInterval',
+ 'clearInterval',
+ 'setTimeout',
+ 'clearTimeout',
+ ],
+ });
+ await connectDB();
+ await clearDBCollections();
+ await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
+ const credentials = {
+ email: 'metadata-cache@example.invalid',
+ password: 'Local-fixture-only-7368!',
+ };
+ if (!(await User.findOne({ email: credentials.email }))) {
+ await agent
+ .post('/register/password')
+ .send({ ...credentials, confirmPassword: credentials.password })
+ .expect(200);
+ }
+ await agent.post('/login/password').send(credentials).expect(303);
+ const found = await User.findOne({ email: credentials.email });
+ if (!found) throw new Error('Native registration did not persist user');
+ user = found;
+}, 60000);
+
+afterAll(async () => {
+ jest.useRealTimers();
+ await new Promise((resolve, reject) =>
+ server.close(err => (err ? reject(err) : resolve())),
+ );
+ for (const table of tables)
+ await executeSqlCommand(`DROP TABLE IF EXISTS ${table}`);
+ await clearDBCollections();
+ await closeTestFixtureClickHouseClient();
+ await mongooseConnection.close();
+});
+
+beforeEach(() => {
+ jest.setSystemTime(initialTime);
+ querySpy.mockClear();
+ metadataSpy.mockClear();
+});
+
+async function sourceFixture(name: string, connectionName = name) {
+ const table = `metadata_limits_${name}`;
+ const rollup = `${table}_kv`;
+ tables.push(table, rollup);
+ await executeSqlCommand(
+ `CREATE TABLE IF NOT EXISTS ${table} (Timestamp DateTime64(3), Body String, LogAttributes Map(String, String)) ENGINE=MergeTree ORDER BY Timestamp`,
+ );
+ await executeSqlCommand(
+ `CREATE TABLE IF NOT EXISTS ${rollup} (Timestamp DateTime, ColumnIdentifier String, Key String, Value String, count UInt64) ENGINE=MergeTree ORDER BY (ColumnIdentifier, Timestamp, Key)`,
+ );
+ await executeSqlCommand(`TRUNCATE TABLE ${table}`);
+ await executeSqlCommand(`TRUNCATE TABLE ${rollup}`);
+ await executeSqlCommand(
+ `INSERT INTO ${table} SELECT toDateTime64('${fixtureTimestamp}', 3), 'fixture', mapFromArrays(arrayMap(x -> concat('key', leftPad(toString(x), 3, '0')), range(100)), arrayMap(x -> 'value', range(100)))`,
+ );
+ await executeSqlCommand(
+ `INSERT INTO ${rollup} SELECT toDateTime('${fixtureTimestamp}'), 'LogAttributes', concat('key', leftPad(toString(number), 3, '0')), 'value', 100-number FROM numbers(100)`,
+ );
+ let connection = await Connection.findOne({
+ team: user.team,
+ name: connectionName,
+ });
+ connection ??= await Connection.create({
+ team: user.team,
+ name: connectionName,
+ host: config.CLICKHOUSE_HOST,
+ username: config.CLICKHOUSE_USER,
+ password: config.CLICKHOUSE_PASSWORD,
+ });
+ let source = await LogSource.findOne({ team: user.team, name });
+ source ??= await LogSource.create({
+ name,
+ team: user.team,
+ connection: connection._id,
+ from: { databaseName: 'default', tableName: table },
+ timestampValueExpression: 'Timestamp',
+ bodyExpression: 'Body',
+ eventAttributesExpression: 'LogAttributes',
+ metadataMaterializedViews: { kvRollupTable: rollup, granularity: '1 hour' },
+ });
+ return source;
+}
+
+async function mcp(sourceId: string, accessKey = user.accessKey) {
+ const response = await request(server)
+ .post('/mcp')
+ .set('Authorization', `Bearer ${accessKey}`)
+ .set('Accept', 'application/json, text/event-stream')
+ .send({
+ jsonrpc: '2.0',
+ id: 1,
+ method: 'tools/call',
+ params: { name: 'clickstack_describe_source', arguments: { sourceId } },
+ });
+ if (response.status !== 200)
+ return { status: response.status, count: -1, isError: true };
+ const data = response.text
+ .split('\n')
+ .find((line: string) => line.startsWith('data: '));
+ const envelope = data ? JSON.parse(data.slice(6)) : response.body;
+ const tool = envelope.result;
+ if (!tool) throw new Error(JSON.stringify(envelope));
+ if (tool.isError)
+ return { status: response.status, count: -1, isError: true };
+ const body = JSON.parse(
+ tool.content.find((item: { type: string }) => item.type === 'text').text,
+ );
+ return {
+ status: response.status,
+ count: body.source.mapAttributeKeys.LogAttributes.length,
+ isError: false,
+ sourceId: body.source.id,
+ connectionId: body.source.connectionId,
+ partial: body.source.partial ?? false,
+ };
+}
+
+async function ai(sourceId: string) {
+ const response = await agent
+ .post('/ai/assistant')
+ .send({ text: 'Count logs', sourceId });
+ if (response.status !== 200) return { status: response.status, count: -1 };
+ const invocation = metadataSpy.mock.results.at(-1);
+ if (!invocation || invocation.type !== 'return')
+ throw new Error('Native getAIMetadata was not called');
+ const value: Awaited> =
+ await invocation.value;
+ return {
+ status: response.status,
+ count: value.allFields.filter(
+ field => field.path[0] === 'LogAttributes' && field.path.length > 1,
+ ).length,
+ sourceId: response.body.source,
+ connectionId: response.body.connection,
+ };
+}
+
+function keyQueries() {
+ return querySpy.mock.calls
+ .map(call => call[0])
+ .filter(call => /SELECT Key\s+FROM/.test(call.query))
+ .map(call => ({
+ query: call.query,
+ query_params: call.query_params,
+ connectionId: call.connectionId,
+ clickhouse_settings: call.clickhouse_settings,
+ }));
+}
+
+test('MCP 50 then AI default preserves each requested limit', async () => {
+ const source = await sourceFixture('mcp_ai');
+ const first = await mcp(source.id);
+ const second = await ai(source.id);
+ expect(first).toMatchObject({
+ status: 200,
+ count: 50,
+ isError: false,
+ partial: false,
+ });
+ expect(second).toMatchObject({ status: 200, count: 100 });
+ expect((await mcp(source.id)).count).toBe(50);
+ expect((await ai(source.id)).count).toBe(100);
+ expect(keyQueries()).toHaveLength(2);
+});
+test('AI default then MCP 50 preserves each requested limit', async () => {
+ const source = await sourceFixture('ai_mcp');
+ const first = await ai(source.id);
+ const second = await mcp(source.id);
+ expect(first).toMatchObject({ status: 200, count: 100 });
+ expect(second).toMatchObject({
+ status: 200,
+ count: 50,
+ isError: false,
+ partial: false,
+ });
+ expect((await mcp(source.id)).count).toBe(50);
+ expect((await ai(source.id)).count).toBe(100);
+ expect(keyQueries()).toHaveLength(2);
+});
diff --git a/packages/api/src/controllers/__tests__/sources.int.test.ts b/packages/api/src/controllers/__tests__/sources.int.test.ts
new file mode 100644
index 0000000000..71a7d73466
--- /dev/null
+++ b/packages/api/src/controllers/__tests__/sources.int.test.ts
@@ -0,0 +1,39 @@
+import mongoose from 'mongoose';
+
+import { getSource } from '@/controllers/sources';
+import { clearDBCollections, closeDB, connectDB } from '@/fixtures';
+
+describe('sources controller', () => {
+ beforeAll(async () => {
+ await connectDB();
+ });
+
+ afterEach(async () => {
+ await clearDBCollections();
+ });
+
+ afterAll(async () => {
+ await closeDB();
+ });
+
+ describe('getSource', () => {
+ it('returns null when sourceId is not a valid ObjectId', async () => {
+ // Non-ObjectId strings used to bubble a Mongoose CastError up
+ // through MCP tools as "Cast to ObjectId failed for value ...".
+ // The wrapper now short-circuits before hitting MongoDB so the
+ // caller's not-found branch fires cleanly.
+ const team = new mongoose.Types.ObjectId().toString();
+
+ expect(await getSource(team, 'not-an-objectid')).toBeNull();
+ expect(await getSource(team, '')).toBeNull();
+ expect(await getSource(team, ' ')).toBeNull();
+ });
+
+ it('returns null for a well-formed but missing ObjectId', async () => {
+ const team = new mongoose.Types.ObjectId().toString();
+ const missingSourceId = new mongoose.Types.ObjectId().toString();
+
+ expect(await getSource(team, missingSourceId)).toBeNull();
+ });
+ });
+});
diff --git a/packages/api/src/controllers/__tests__/team.test.ts b/packages/api/src/controllers/__tests__/team.int.test.ts
similarity index 63%
rename from packages/api/src/controllers/__tests__/team.test.ts
rename to packages/api/src/controllers/__tests__/team.int.test.ts
index a97a06989c..1330ae83ba 100644
--- a/packages/api/src/controllers/__tests__/team.test.ts
+++ b/packages/api/src/controllers/__tests__/team.int.test.ts
@@ -1,6 +1,20 @@
-import { createTeam, getTeam, getTeamByApiKey } from '@/controllers/team';
+import * as config from '@/config';
+import {
+ createTeam,
+ getTeam,
+ getTeamByApiKey,
+ getTeamInviteUrl,
+} from '@/controllers/team';
import { clearDBCollections, closeDB, connectDB } from '@/fixtures';
+describe('getTeamInviteUrl', () => {
+ it('builds the join-team URL for a token', () => {
+ expect(getTeamInviteUrl('abc123')).toBe(
+ `${config.FRONTEND_URL}/join-team?token=abc123`,
+ );
+ });
+});
+
describe('team controller', () => {
beforeAll(async () => {
await connectDB();
diff --git a/packages/api/src/controllers/__tests__/timeseriesEngine.int.test.ts b/packages/api/src/controllers/__tests__/timeseriesEngine.int.test.ts
new file mode 100644
index 0000000000..0ea9586610
--- /dev/null
+++ b/packages/api/src/controllers/__tests__/timeseriesEngine.int.test.ts
@@ -0,0 +1,478 @@
+import { ClickhouseClient } from '@/clickhouse';
+import * as config from '@/config';
+import {
+ queryLabelNames,
+ queryLabelValues,
+} from '@/controllers/timeseriesEngine';
+import {
+ closeTestFixtureClickHouseClient,
+ DEFAULT_DATABASE,
+ dropTimeSeriesTable,
+ executeSqlCommand,
+ seedTimeSeriesTagsTable,
+ TimeSeriesFixtureSeries,
+} from '@/fixtures';
+
+const CONNECTION_ID = 'timeseries-engine-int-test';
+const BOUNDED_TABLE = 'test_ts_label_values';
+const UNBOUNDED_TABLE = 'test_ts_label_values_no_bounds';
+
+const HOUR_SEC = 3600;
+const OLD_START_SEC = 1600000000;
+const RECENT_START_SEC = 1700000000;
+const sec = (s: number) => s * 1000;
+
+// Three series: one only in the distant past, two only in the recent window
+// sharing a metric name (so DISTINCT has something to collapse). Only the old
+// series carries `region`, so a window can exclude a label name too.
+const SERIES: TimeSeriesFixtureSeries[] = [
+ {
+ metricName: 'old_metric',
+ tags: { job: 'batch', region: 'eu' },
+ startSec: OLD_START_SEC,
+ endSec: OLD_START_SEC + HOUR_SEC,
+ },
+ {
+ metricName: 'recent_metric',
+ tags: { job: 'api' },
+ startSec: RECENT_START_SEC,
+ endSec: RECENT_START_SEC + HOUR_SEC,
+ },
+ {
+ metricName: 'recent_metric',
+ tags: { job: 'web' },
+ startSec: RECENT_START_SEC,
+ endSec: RECENT_START_SEC + HOUR_SEC,
+ },
+];
+
+describe('timeseriesEngine controller', () => {
+ let client: ClickhouseClient;
+
+ const labelValues = (
+ args: Partial[0]> & {
+ labelName: string;
+ },
+ ) =>
+ queryLabelValues({
+ client,
+ connectionId: CONNECTION_ID,
+ databaseName: DEFAULT_DATABASE,
+ tableName: BOUNDED_TABLE,
+ ...args,
+ });
+
+ beforeAll(async () => {
+ client = new ClickhouseClient({
+ host: config.CLICKHOUSE_HOST,
+ username: config.CLICKHOUSE_USER,
+ password: config.CLICKHOUSE_PASSWORD,
+ });
+ await seedTimeSeriesTagsTable({
+ table: BOUNDED_TABLE,
+ series: SERIES,
+ withSamples: true,
+ });
+ await seedTimeSeriesTagsTable({
+ table: UNBOUNDED_TABLE,
+ series: SERIES,
+ storeTimeBounds: false,
+ });
+ });
+
+ afterAll(async () => {
+ for (const table of [BOUNDED_TABLE, UNBOUNDED_TABLE]) {
+ await dropTimeSeriesTable({ table });
+ }
+ await client.close();
+ await closeTestFixtureClickHouseClient();
+ });
+
+ describe('queryLabelValues', () => {
+ describe('__name__', () => {
+ it('returns every metric name, deduplicated and sorted', async () => {
+ expect(await labelValues({ labelName: '__name__' })).toEqual([
+ 'old_metric',
+ 'recent_metric',
+ ]);
+ });
+
+ it('keeps only metrics whose series overlap the window', async () => {
+ expect(
+ await labelValues({
+ labelName: '__name__',
+ startMs: sec(RECENT_START_SEC),
+ endMs: sec(RECENT_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['recent_metric']);
+
+ expect(
+ await labelValues({
+ labelName: '__name__',
+ startMs: sec(OLD_START_SEC),
+ endMs: sec(OLD_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['old_metric']);
+ });
+
+ // A lone `end` bounds min_time (nothing may start after the window) and a
+ // lone `start` bounds max_time (nothing may end before it). Getting the
+ // pairing backwards still filters, just wrongly, so both directions are
+ // pinned.
+ it('applies a lone end bound against min_time', async () => {
+ expect(
+ await labelValues({
+ labelName: '__name__',
+ endMs: sec(OLD_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['old_metric']);
+ });
+
+ it('applies a lone start bound against max_time', async () => {
+ expect(
+ await labelValues({
+ labelName: '__name__',
+ startMs: sec(RECENT_START_SEC),
+ }),
+ ).toEqual(['recent_metric']);
+ });
+
+ it('returns nothing for a window no series covers', async () => {
+ expect(
+ await labelValues({
+ labelName: '__name__',
+ startMs: sec(OLD_START_SEC + 2 * HOUR_SEC),
+ endMs: sec(OLD_START_SEC + 3 * HOUR_SEC),
+ }),
+ ).toEqual([]);
+ });
+ });
+
+ describe('tag labels', () => {
+ it('returns the label values, deduplicated and sorted', async () => {
+ expect(await labelValues({ labelName: 'job' })).toEqual([
+ 'api',
+ 'batch',
+ 'web',
+ ]);
+ });
+
+ it('scopes label values to the window', async () => {
+ expect(
+ await labelValues({
+ labelName: 'job',
+ startMs: sec(RECENT_START_SEC),
+ endMs: sec(RECENT_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['api', 'web']);
+ });
+
+ it('returns nothing for a label no series carries', async () => {
+ expect(await labelValues({ labelName: 'instance' })).toEqual([]);
+ });
+
+ // The label name reaches both the WHERE clause and the value expression.
+ // Parameterized, this is an ordinary miss; interpolated, it is a syntax
+ // error — so the assertion is "empty", not "throws".
+ it('parameterizes the label name rather than interpolating it', async () => {
+ expect(await labelValues({ labelName: `job') OR 1=1 --` })).toEqual([]);
+ });
+ });
+
+ // A selector matches a series only if it has a sample in the window, which
+ // is why BOUNDED_TABLE is seeded with samples.
+ describe('match[] selectors', () => {
+ it('keeps only the values carried by matching series', async () => {
+ expect(
+ await labelValues({
+ labelName: 'job',
+ match: ['recent_metric{job="api"}'],
+ }),
+ ).toEqual(['api']);
+ });
+
+ // Prometheus unions the selectors rather than intersecting them.
+ it('unions the series matched by each selector', async () => {
+ expect(
+ await labelValues({
+ labelName: 'job',
+ match: ['recent_metric{job="api"}', 'old_metric'],
+ }),
+ ).toEqual(['api', 'batch']);
+ });
+
+ it('scopes metric names to the selector', async () => {
+ expect(
+ await labelValues({
+ labelName: '__name__',
+ match: ['{job="batch"}'],
+ }),
+ ).toEqual(['old_metric']);
+ });
+
+ it('evaluates regex matchers', async () => {
+ expect(
+ await labelValues({
+ labelName: '__name__',
+ match: ['{job=~"a.*|w.*"}'],
+ }),
+ ).toEqual(['recent_metric']);
+ });
+
+ // Both filters have to survive: the selector alone would answer with
+ // every job, the window alone with every recent one.
+ it('narrows the selector by the time window', async () => {
+ expect(
+ await labelValues({
+ labelName: 'job',
+ match: ['{job=~".+"}'],
+ startMs: sec(RECENT_START_SEC),
+ endMs: sec(RECENT_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['api', 'web']);
+ });
+
+ it('returns nothing when no series matches', async () => {
+ expect(
+ await labelValues({ labelName: 'job', match: ['absent_metric'] }),
+ ).toEqual([]);
+ });
+
+ it('treats an empty selector list as no filter', async () => {
+ expect(await labelValues({ labelName: 'job', match: [] })).toEqual([
+ 'api',
+ 'batch',
+ 'web',
+ ]);
+ });
+
+ it('surfaces a malformed selector as an error', async () => {
+ await expect(
+ labelValues({ labelName: 'job', match: ['recent_metric{'] }),
+ ).rejects.toThrow(/while parsing PromQL query/);
+ });
+
+ // A selector is caller-supplied text. Interpolated, this closes the
+ // enclosing string and the query fails; parameterized, it is a plain miss
+ // — so the assertion is "empty", not "throws".
+ it('parameterizes the selector rather than interpolating it', async () => {
+ expect(
+ await labelValues({
+ labelName: 'job',
+ match: [`{job="a') OR 1=1 --"}`],
+ }),
+ ).toEqual([]);
+ });
+
+ it('collapses a repeated identical selector', async () => {
+ expect(
+ await labelValues({
+ labelName: 'job',
+ match: ['old_metric', 'old_metric'],
+ }),
+ ).toEqual(['batch']);
+ });
+
+ it('applies the limit after the selector', async () => {
+ expect(
+ await labelValues({
+ labelName: 'job',
+ match: ['{job=~".+"}'],
+ limit: 1,
+ }),
+ ).toEqual(['api']);
+ });
+
+ // The label predicate and the selector have to AND together rather than
+ // one standing in for the other.
+ it('returns nothing when the matched series lack the label', async () => {
+ expect(
+ await labelValues({ labelName: 'region', match: ['recent_metric'] }),
+ ).toEqual([]);
+ });
+ });
+
+ describe('limit', () => {
+ it('caps the number of values returned, keeping the ordering', async () => {
+ expect(await labelValues({ labelName: '__name__', limit: 1 })).toEqual([
+ 'old_metric',
+ ]);
+ });
+
+ // Prometheus reads limit=0 as unlimited, so it must not become `LIMIT 0`.
+ it('treats a zero limit as unlimited', async () => {
+ expect(await labelValues({ labelName: '__name__', limit: 0 })).toEqual([
+ 'old_metric',
+ 'recent_metric',
+ ]);
+ });
+
+ it('returns everything when the limit exceeds the result set', async () => {
+ expect(
+ await labelValues({ labelName: '__name__', limit: 100 }),
+ ).toEqual(['old_metric', 'recent_metric']);
+ });
+ });
+
+ // store_min_time_and_max_time = 0 leaves the tags table without the columns
+ // the predicate reads, and referencing a missing column is a hard error.
+ // Time bounds are best-effort narrowing, so the request widens instead.
+ describe('table without min_time/max_time', () => {
+ it('ignores the bounds instead of failing', async () => {
+ expect(
+ await queryLabelValues({
+ client,
+ connectionId: CONNECTION_ID,
+ databaseName: DEFAULT_DATABASE,
+ tableName: UNBOUNDED_TABLE,
+ labelName: '__name__',
+ startMs: sec(RECENT_START_SEC),
+ endMs: sec(RECENT_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['old_metric', 'recent_metric']);
+ });
+
+ // A selector needs those columns to evaluate at all. Unlike the bounds it
+ // cannot degrade to "no filter" — that would answer a different question.
+ it('rejects match[] rather than dropping it', async () => {
+ await expect(
+ queryLabelValues({
+ client,
+ connectionId: CONNECTION_ID,
+ databaseName: DEFAULT_DATABASE,
+ tableName: UNBOUNDED_TABLE,
+ labelName: 'job',
+ match: ['recent_metric'],
+ }),
+ ).rejects.toThrow(/store_min_time_and_max_time = 1/);
+ });
+
+ it('still filters on the label itself', async () => {
+ expect(
+ await queryLabelValues({
+ client,
+ connectionId: CONNECTION_ID,
+ databaseName: DEFAULT_DATABASE,
+ tableName: UNBOUNDED_TABLE,
+ labelName: 'job',
+ }),
+ ).toEqual(['api', 'batch', 'web']);
+ });
+ });
+
+ it('rejects when the table is not a TimeSeries table', async () => {
+ await executeSqlCommand(
+ `CREATE OR REPLACE TABLE ${DEFAULT_DATABASE}.test_ts_plain (ts DateTime) ENGINE = MergeTree ORDER BY ts`,
+ );
+ try {
+ await expect(
+ labelValues({
+ labelName: '__name__',
+ tableName: 'test_ts_plain',
+ }),
+ ).rejects.toThrow(/TimeSeries table only/);
+ } finally {
+ await executeSqlCommand(
+ `DROP TABLE IF EXISTS ${DEFAULT_DATABASE}.test_ts_plain`,
+ );
+ }
+ });
+ });
+
+ describe('queryLabelNames', () => {
+ const labelNames = (
+ args: Partial[0]> = {},
+ ) =>
+ queryLabelNames({
+ client,
+ connectionId: CONNECTION_ID,
+ databaseName: DEFAULT_DATABASE,
+ tableName: BOUNDED_TABLE,
+ ...args,
+ });
+
+ // `__name__` is not a key of `tags`, so it only appears if folded back in.
+ it('returns every label name including __name__, deduplicated and sorted', async () => {
+ expect(await labelNames()).toEqual(['__name__', 'job', 'region']);
+ });
+
+ it('keeps only labels carried by series overlapping the window', async () => {
+ expect(
+ await labelNames({
+ startMs: sec(RECENT_START_SEC),
+ endMs: sec(RECENT_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['__name__', 'job']);
+ });
+
+ it('returns nothing for a window no series covers', async () => {
+ expect(
+ await labelNames({
+ startMs: sec(OLD_START_SEC + 2 * HOUR_SEC),
+ endMs: sec(OLD_START_SEC + 3 * HOUR_SEC),
+ }),
+ ).toEqual([]);
+ });
+
+ it('caps the number of names returned, keeping the ordering', async () => {
+ expect(await labelNames({ limit: 2 })).toEqual(['__name__', 'job']);
+ });
+
+ it('treats a zero limit as unlimited', async () => {
+ expect(await labelNames({ limit: 0 })).toEqual([
+ '__name__',
+ 'job',
+ 'region',
+ ]);
+ });
+
+ it('ignores the bounds on a table without min_time/max_time', async () => {
+ expect(
+ await labelNames({
+ tableName: UNBOUNDED_TABLE,
+ startMs: sec(RECENT_START_SEC),
+ endMs: sec(RECENT_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['__name__', 'job', 'region']);
+ });
+
+ describe('match[] selectors', () => {
+ // Only the old series carries `region`, so a selector that excludes it
+ // has to drop a label name from the answer.
+ it('keeps only the names carried by matching series', async () => {
+ expect(await labelNames({ match: ['recent_metric'] })).toEqual([
+ '__name__',
+ 'job',
+ ]);
+ });
+
+ it('unions the series matched by each selector', async () => {
+ expect(
+ await labelNames({ match: ['{job="web"}', 'old_metric'] }),
+ ).toEqual(['__name__', 'job', 'region']);
+ });
+
+ // Both filters have to survive: the selector alone would answer with
+ // `region` too, the window alone with every recent label.
+ it('narrows the selector by the time window', async () => {
+ expect(
+ await labelNames({
+ match: ['{job=~".+"}'],
+ startMs: sec(RECENT_START_SEC),
+ endMs: sec(RECENT_START_SEC + HOUR_SEC),
+ }),
+ ).toEqual(['__name__', 'job']);
+ });
+
+ it('returns nothing when no series matches', async () => {
+ expect(await labelNames({ match: ['absent_metric'] })).toEqual([]);
+ });
+
+ it('rejects match[] on a table without min_time/max_time', async () => {
+ await expect(
+ labelNames({ tableName: UNBOUNDED_TABLE, match: ['recent_metric'] }),
+ ).rejects.toThrow(/store_min_time_and_max_time = 1/);
+ });
+ });
+ });
+});
diff --git a/packages/api/src/controllers/__tests__/timeseriesEngine.test.ts b/packages/api/src/controllers/__tests__/timeseriesEngine.test.ts
new file mode 100644
index 0000000000..f1f0d2b36e
--- /dev/null
+++ b/packages/api/src/controllers/__tests__/timeseriesEngine.test.ts
@@ -0,0 +1,47 @@
+import {
+ formatMatrixResponse,
+ formatVectorResponse,
+} from '@/controllers/timeseriesEngine';
+
+describe('formatMatrixResponse', () => {
+ it('converts prometheusQueryRange rows into Prometheus matrix shape', () => {
+ const rows = [
+ {
+ tags: [
+ ['__name__', 'http_requests_total'],
+ ['method', 'GET'],
+ ] as [string, string][],
+ samples: [
+ [1700000000, 5],
+ ['2023-11-14T22:14:20.000Z', 7],
+ ] as [string | number, number][],
+ },
+ ];
+ expect(formatMatrixResponse(rows as any)).toEqual([
+ {
+ metric: { __name__: 'http_requests_total', method: 'GET' },
+ values: [
+ [1700000000, '5'],
+ [1700000060, '7'],
+ ],
+ },
+ ]);
+ });
+
+ it('returns empty array for empty input', () => {
+ expect(formatMatrixResponse([])).toEqual([]);
+ });
+});
+
+describe('formatVectorResponse', () => {
+ it('converts prometheusQuery rows into Prometheus vector shape', () => {
+ const rows = [
+ { tags: [['service', 'api']], timestamp: 1700000000, value: 42 },
+ { tags: [], timestamp: '2023-11-14T22:13:20.000Z', value: 3 },
+ ];
+ expect(formatVectorResponse(rows as any)).toEqual([
+ { metric: { service: 'api' }, value: [1700000000, '42'] },
+ { metric: {}, value: [1700000000, '3'] },
+ ]);
+ });
+});
diff --git a/packages/api/src/controllers/ai.ts b/packages/api/src/controllers/ai.ts
index 48f14b97da..90b201cea5 100644
--- a/packages/api/src/controllers/ai.ts
+++ b/packages/api/src/controllers/ai.ts
@@ -1,22 +1,23 @@
import { createAnthropic } from '@ai-sdk/anthropic';
import { createOpenAI } from '@ai-sdk/openai';
-import { ClickhouseClient } from '@hyperdx/common-utils/dist/clickhouse/node';
import {
getMetadata,
TableMetadata,
+ tcFromSource,
} from '@hyperdx/common-utils/dist/core/metadata';
+import { resolvePercentileAggFn } from '@hyperdx/common-utils/dist/core/utils';
import {
AILineTableResponse,
AssistantLineTableConfigSchema,
ChartConfigWithDateRange,
pickSampleWeightExpressionProps,
- SourceKind,
} from '@hyperdx/common-utils/dist/types';
import type { LanguageModel } from 'ai';
import * as chrono from 'chrono-node';
import ms from 'ms';
import z from 'zod';
+import { ClickhouseClient } from '@/clickhouse';
import * as config from '@/config';
import { ISource } from '@/models/source';
import { parseJSON } from '@/utils/common';
@@ -107,6 +108,8 @@ export async function getAIMetadata(source: ISource) {
databaseName,
tableName,
connectionId,
+ metadataMVs: tcFromSource(source).metadataMVs,
+ timestampValueExpression: source.timestampValueExpression,
});
// TODO: Dedup with DBSearchPageFilters.tsx logic
@@ -255,16 +258,18 @@ export function getChartConfigFromResolvedConfig(
return {
displayType: resObject.displayType,
- select: resObject.select.map(s => ({
- aggFn: s.aggregationFunction,
- valueExpression: s.property,
- ...(s.condition
- ? {
- aggCondition: s.condition,
- aggConditionLanguage: 'sql',
- }
- : {}),
- })),
+ select: resObject.select.map(s =>
+ resolvePercentileAggFn({
+ aggFn: s.aggregationFunction,
+ valueExpression: s.property,
+ ...(s.condition
+ ? {
+ aggCondition: s.condition,
+ aggConditionLanguage: 'sql',
+ }
+ : {}),
+ }),
+ ),
from: {
tableName: source.from.tableName,
databaseName: source.from.databaseName,
diff --git a/packages/api/src/controllers/alertHistory.ts b/packages/api/src/controllers/alertHistory.ts
index 93ed9c1468..0c120d05cf 100644
--- a/packages/api/src/controllers/alertHistory.ts
+++ b/packages/api/src/controllers/alertHistory.ts
@@ -1,41 +1,143 @@
import PQueue from '@esm2cjs/p-queue';
import {
+ ALERT_EVALUATION_GROUPS_LIMIT,
ALERT_INTERVAL_TO_MINUTES,
AlertInterval,
+ AlertTransition,
} from '@hyperdx/common-utils/dist/types';
import { ObjectId } from 'mongodb';
-import { AlertState } from '@/models/alert';
-import AlertHistory, { IAlertHistory } from '@/models/alertHistory';
+import { AlertState, IAlertError } from '@/models/alert';
+import AlertHistory, {
+ IAlertHistory,
+ IAlertHistoryAnalytics,
+} from '@/models/alertHistory';
+
+// Re-exported for API-side consumers/tests; the app imports it from
+// common-utils to explain the cap in the UI.
+export { ALERT_EVALUATION_GROUPS_LIMIT };
// Max parallel per-alert queries to avoid overwhelming the DB connection pool
export const ALERT_HISTORY_QUERY_CONCURRENCY = 20;
+/** Alert evaluation interval in milliseconds. */
+const intervalToMs = (interval: AlertInterval): number =>
+ ALERT_INTERVAL_TO_MINUTES[interval] * 60 * 1000;
+
type GroupedAlertHistory = {
_id: Date;
states: string[];
counts: number;
lastValues: IAlertHistory['lastValues'][];
+ errors: IAlertError[][];
};
+function groupStateToOverallState(states: string[]): AlertState {
+ if (states.includes(AlertState.ALERT)) {
+ return AlertState.ALERT;
+ }
+
+ if (states.includes(AlertState.PENDING)) {
+ return AlertState.PENDING;
+ }
+
+ // An evaluation window that neither fired nor was pending, but recorded an
+ // error (query failure, or a notification failure on an OK window), is
+ // surfaced as ERROR.
+ if (states.includes(AlertState.ERROR)) {
+ return AlertState.ERROR;
+ }
+
+ return AlertState.OK;
+}
+
+/** Dedupe errors by type+message, keeping the most recent occurrence. */
+function dedupeErrors(errors: IAlertError[]): IAlertError[] {
+ const map = new Map();
+ for (const error of errors) {
+ const key = `${error.type}||${error.message}`;
+ const existing = map.get(key);
+ if (!existing || error.timestamp > existing.timestamp) {
+ map.set(key, error);
+ }
+ }
+ return Array.from(map.values()).sort(
+ (a, b) => b.timestamp.getTime() - a.timestamp.getTime(),
+ );
+}
+
function mapGroupedHistories(
groupedHistories: GroupedAlertHistory[],
): Omit[] {
- return groupedHistories.map(group => ({
- createdAt: group._id,
- state: group.states.includes(AlertState.ALERT)
- ? AlertState.ALERT
- : AlertState.OK,
- counts: group.counts,
- lastValues: group.lastValues
- .flat()
- .sort((a, b) => a.startTime.getTime() - b.startTime.getTime()),
- }));
+ return groupedHistories.map(group => {
+ // $push skips documents where the field is missing, but be defensive
+ // about nulls in case of engine differences (e.g. DocumentDB).
+ const errors = dedupeErrors(
+ (group.errors ?? []).flat().filter((e): e is IAlertError => e != null),
+ );
+ return {
+ createdAt: group._id,
+ state: groupStateToOverallState(group.states),
+ counts: group.counts,
+ lastValues: group.lastValues
+ .flat()
+ .sort((a, b) => a.startTime.getTime() - b.startTime.getTime()),
+ ...(errors.length > 0 && { errors }),
+ };
+ });
+}
+
+/**
+ * Fetch grouped evaluation windows (one entry per createdAt, newest first)
+ * for the given alert within the createdAt bounds, capped at `limit` groups.
+ */
+async function fetchGroupedWindows(
+ alertId: ObjectId,
+ createdAt: Record,
+ limit: number,
+): Promise[]> {
+ const groupedHistories = await AlertHistory.aggregate([
+ {
+ $match: {
+ alert: new ObjectId(alertId),
+ createdAt,
+ },
+ },
+ { $sort: { createdAt: -1 } },
+ {
+ $group: {
+ _id: '$createdAt',
+ states: {
+ $push: '$state',
+ },
+ counts: {
+ $sum: '$counts',
+ },
+ lastValues: {
+ $push: '$lastValues',
+ },
+ errors: {
+ $push: '$errors',
+ },
+ },
+ },
+ {
+ $sort: {
+ _id: -1,
+ },
+ },
+ {
+ $limit: limit,
+ },
+ ]);
+
+ return mapGroupedHistories(groupedHistories);
}
/**
* Gets the most recent alert histories for a given alert ID,
- * limiting to the given number of entries.
+ * limiting to the given number of entries. Results are one entry per
+ * evaluation window (grouped by createdAt), newest first.
*/
export async function getRecentAlertHistories({
alertId,
@@ -46,27 +148,189 @@ export async function getRecentAlertHistories({
interval: AlertInterval;
limit: number;
}): Promise[]> {
- const lookbackMs = limit * ALERT_INTERVAL_TO_MINUTES[interval] * 60 * 1000;
+ // One extra interval of slack so a window sitting exactly `limit` intervals
+ // back (the newest window is up to one interval old) isn't cut off by the
+ // lookback bound.
+ const lookbackMs = (limit + 1) * intervalToMs(interval);
+ return fetchGroupedWindows(
+ alertId,
+ { $gte: new Date(Date.now() - lookbackMs) },
+ limit,
+ );
+}
- const groupedHistories = await AlertHistory.aggregate([
+type AlertEvaluationGroupEntry = {
+ group: string;
+ state: AlertState;
+ counts: number;
+ /** The group's most recent bucket value in this window, if any. */
+ lastValue?: { startTime: Date; count: number };
+ /** True when a notification was actually sent for this group. */
+ fired?: boolean;
+};
+
+type AlertEvaluationEntry = Omit & {
+ /**
+ * Per-group breakdown for group-by alerts, firing-first, capped at
+ * ALERT_EVALUATION_GROUPS_LIMIT. Absent for non-grouped alerts.
+ */
+ groups?: AlertEvaluationGroupEntry[];
+ /** Total number of groups evaluated in this window (before the cap). */
+ groupsTotal?: number;
+};
+
+export type AlertEvaluationsPage = {
+ data: AlertEvaluationEntry[];
+ /** True when older windows may exist within [startTime, ...). */
+ hasMore: boolean;
+ /** Cursor for the next-older page (pass as `before`). Set when hasMore. */
+ nextBefore?: Date;
+};
+
+// One AlertHistory row's fields, as pushed into a per-window sub-array by the
+// evaluations aggregation (unlike the alerts-page pipeline, group identity is
+// preserved so windows can be broken down per group).
+type EvaluationWindowRow = {
+ group?: string;
+ state: AlertState;
+ counts?: number;
+ lastValues?: IAlertHistory['lastValues'];
+ fired?: boolean;
+ errors?: IAlertError[];
+ analytics?: IAlertHistoryAnalytics;
+};
+
+type StructuredWindow = {
+ _id: Date;
+ rows: EvaluationWindowRow[];
+};
+
+// Firing-first ordering for the per-group breakdown.
+const groupStatePriority = (state: AlertState): number => {
+ switch (state) {
+ case AlertState.ALERT:
+ return 0;
+ case AlertState.PENDING:
+ return 1;
+ default:
+ return 2;
+ }
+};
+
+/**
+ * Pick the evaluation analytics for a window. A window's rows all carry the
+ * same evaluation-level analytics, except when it contains rows from two
+ * evaluations (a failed attempt's ERROR row + the successful retry's rows) —
+ * prefer the successful evaluation's. For rows written before analytics
+ * existed, `backfilledBuckets` is derived from the distinct bucket times so
+ * the "Backfilled Buckets" column works on historical data.
+ */
+function resolveWindowAnalytics(
+ rows: EvaluationWindowRow[],
+ lastValues: IAlertHistory['lastValues'],
+): IAlertHistoryAnalytics | undefined {
+ const analytics =
+ rows.find(r => r.state !== AlertState.ERROR && r.analytics != null)
+ ?.analytics ?? rows.find(r => r.analytics != null)?.analytics;
+
+ if (analytics?.backfilledBuckets != null) {
+ return analytics;
+ }
+
+ const distinctBuckets = new Set(lastValues.map(v => v.startTime.getTime()))
+ .size;
+ const derivedBackfilled = Math.max(0, distinctBuckets - 1);
+ if (analytics == null && derivedBackfilled === 0) {
+ return undefined;
+ }
+ return { ...analytics, backfilledBuckets: derivedBackfilled };
+}
+
+function mapStructuredWindow(window: StructuredWindow): AlertEvaluationEntry {
+ const rows = window.rows ?? [];
+ const errors = dedupeErrors(
+ rows.flatMap(r => r.errors ?? []).filter(e => e != null),
+ );
+ const lastValues = rows
+ .flatMap(r => r.lastValues ?? [])
+ .sort((a, b) => a.startTime.getTime() - b.startTime.getTime());
+ const analytics = resolveWindowAnalytics(rows, lastValues);
+
+ // Per-group breakdown: rows carrying a group identity. ERROR rows never
+ // carry one (they record the evaluation failure, not a group result).
+ const groupRows = rows.filter(
+ r => r.group != null && r.group !== '' && r.state !== AlertState.ERROR,
+ );
+ const groups = groupRows
+ .map(r => {
+ const groupLastValues = r.lastValues ?? [];
+ const lastValue =
+ groupLastValues.length > 0
+ ? groupLastValues.reduce((latest, v) =>
+ v.startTime.getTime() > latest.startTime.getTime() ? v : latest,
+ )
+ : undefined;
+ return {
+ group: r.group as string,
+ state: r.state,
+ counts: r.counts ?? 0,
+ ...(lastValue != null && { lastValue }),
+ ...(r.fired != null && { fired: r.fired }),
+ };
+ })
+ .sort(
+ (a, b) =>
+ groupStatePriority(a.state) - groupStatePriority(b.state) ||
+ (b.lastValue?.count ?? Number.NEGATIVE_INFINITY) -
+ (a.lastValue?.count ?? Number.NEGATIVE_INFINITY) ||
+ a.group.localeCompare(b.group),
+ );
+
+ return {
+ createdAt: window._id,
+ state: groupStateToOverallState(rows.map(r => r.state)),
+ counts: rows.reduce((sum, r) => sum + (r.counts ?? 0), 0),
+ lastValues,
+ ...(errors.length > 0 && { errors }),
+ ...(analytics != null && { analytics }),
+ ...(groups.length > 0 && {
+ groups: groups.slice(0, ALERT_EVALUATION_GROUPS_LIMIT),
+ groupsTotal: groups.length,
+ }),
+ };
+}
+
+/**
+ * Fetch evaluation windows (one entry per createdAt, newest first) within the
+ * createdAt bounds, capped at `limit` windows, preserving per-group rows so
+ * grouped alerts can be broken down per group.
+ */
+async function fetchStructuredWindows(
+ alertId: ObjectId,
+ createdAt: Record,
+ limit: number,
+): Promise {
+ const windows = await AlertHistory.aggregate([
{
$match: {
alert: new ObjectId(alertId),
- createdAt: { $gte: new Date(Date.now() - lookbackMs) },
+ createdAt,
},
},
{ $sort: { createdAt: -1 } },
{
$group: {
_id: '$createdAt',
- states: {
- $push: '$state',
- },
- counts: {
- $sum: '$counts',
- },
- lastValues: {
- $push: '$lastValues',
+ rows: {
+ $push: {
+ group: '$group',
+ state: '$state',
+ counts: '$counts',
+ lastValues: '$lastValues',
+ fired: '$fired',
+ errors: '$errors',
+ analytics: '$analytics',
+ },
},
},
},
@@ -80,7 +344,83 @@ export async function getRecentAlertHistories({
},
]);
- return mapGroupedHistories(groupedHistories);
+ return windows.map(mapStructuredWindow);
+}
+
+/**
+ * Paginated evaluation windows for the alert detail page, newest first,
+ * scoped to [startTime, endTime].
+ *
+ * Every request scans a hard-bounded slice of at most ~(limit + 1) intervals
+ * of history (anchored at `before ?? endTime`), so a wide time range can
+ * never force an unbounded scan — group-by alerts can have many rows per
+ * window, and the $group stage processes every matched row.
+ *
+ * Because of that bound, a page may end before reaching `startTime` even if
+ * fewer than `limit` windows were returned (a gap with no evaluations). The
+ * returned `nextBefore` cursor always advances past the scanned slice, so
+ * callers can keep paging across gaps: pass it as `before` on the next call.
+ */
+export async function getAlertEvaluations({
+ alertId,
+ interval,
+ limit,
+ startTime,
+ endTime,
+ before,
+}: {
+ alertId: ObjectId;
+ interval: AlertInterval;
+ limit: number;
+ startTime: Date;
+ endTime: Date;
+ before?: Date;
+}): Promise {
+ const intervalMs = intervalToMs(interval);
+ // One extra interval of slack so a window sitting exactly `limit` intervals
+ // back isn't cut off by the scan bound.
+ const scanMs = (limit + 1) * intervalMs;
+
+ // Upper bound: the page cursor when provided (exclusive), else the range
+ // end (inclusive). A cursor past the range end is ignored.
+ const usableBefore =
+ before != null && before.getTime() <= endTime.getTime()
+ ? before
+ : undefined;
+ const pageEndMs = usableBefore?.getTime() ?? endTime.getTime();
+
+ // Lower bound: the scan bound, clamped to the requested range start.
+ const scanFloorMs = Math.max(startTime.getTime(), pageEndMs - scanMs);
+
+ const createdAt: Record = {
+ $gte: new Date(scanFloorMs),
+ };
+ if (usableBefore != null) {
+ createdAt.$lt = usableBefore;
+ } else {
+ createdAt.$lte = endTime;
+ }
+
+ // Fetch one extra window to detect count truncation.
+ const windows = await fetchStructuredWindows(alertId, createdAt, limit + 1);
+
+ const truncatedByCount = windows.length > limit;
+ const data = truncatedByCount ? windows.slice(0, limit) : windows;
+ // More windows may exist when the page filled up, or when the scan bound
+ // stopped before reaching the range start.
+ const truncatedByScanBound = scanFloorMs > startTime.getTime();
+ const hasMore = truncatedByCount || truncatedByScanBound;
+
+ // Count truncation: resume strictly before the last returned window.
+ // Scan-bound truncation: the slice down to scanFloor (inclusive) is fully
+ // covered, so resume strictly before it.
+ const nextBefore = !hasMore
+ ? undefined
+ : truncatedByCount
+ ? data[data.length - 1].createdAt
+ : new Date(scanFloorMs);
+
+ return { data, hasMore, nextBefore };
}
/**
@@ -119,3 +459,134 @@ export async function getRecentAlertHistoriesBatch(
),
);
}
+
+/**
+ * Returns alert firing/recovery transitions (ALERT-boundary crossings) within
+ * [startTime, endTime] for one alert, for drawing chart annotations. Each
+ * transition carries `bucketStart` — the start of the newest bucket the
+ * transitioning evaluation covered — so markers land on the data point that
+ * produced the transition (charts plot buckets at their start, while the
+ * evaluation runs at the bucket end). One window before startTime is fetched
+ * to know the state on entry: if the alert is already firing then, a firing
+ * marker is pinned to startTime so a later in-range recovery isn't orphaned.
+ * PENDING/INSUFFICIENT_DATA count as non-firing, so only ALERT crossings are
+ * reported.
+ */
+export async function getAlertTransitionsInRange({
+ alertId,
+ interval,
+ startTime,
+ endTime,
+}: {
+ alertId: ObjectId;
+ interval: AlertInterval;
+ startTime: Date;
+ endTime: Date;
+}): Promise {
+ const intervalMs = intervalToMs(interval);
+ const lookbackStart = new Date(startTime.getTime() - intervalMs);
+
+ // Per-window state detects crossings; the newest evaluated bucket start
+ // positions the marker where the chart plots that bucket's value. ERROR rows
+ // are failed evaluations, not state observations — excluding them prevents a
+ // query failure mid-firing from drawing a false recovery annotation.
+ const windows = await AlertHistory.aggregate<{
+ _id: Date;
+ states: string[];
+ // One array of bucket-start dates per row in the window (one row per
+ // group for group-by alerts). Push only the dates via a plain field path
+ // and derive the newest in JS — the file's DocumentDB-safe pattern (see
+ // mapGroupedHistories) avoids expression operators inside accumulators,
+ // and pushing dates instead of whole lastValues rows keeps the buffered
+ // payload small. The router caps the scanned span (MAX_HISTORY_SPAN_MS).
+ bucketStarts: Date[][];
+ }>([
+ {
+ $match: {
+ alert: new ObjectId(alertId),
+ createdAt: { $gte: lookbackStart, $lte: endTime },
+ state: { $ne: AlertState.ERROR },
+ },
+ },
+ {
+ $group: {
+ _id: '$createdAt',
+ states: { $push: '$state' },
+ bucketStarts: { $push: '$lastValues.startTime' },
+ },
+ },
+ { $sort: { _id: 1 } },
+ ]);
+
+ // Newest bucket start across the window's rows; null when no row carries
+ // lastValues. Be defensive about missing arrays/fields in case of engine
+ // differences (e.g. DocumentDB).
+ const newestBucketStart = (bucketStarts: Date[][] | undefined): Date | null =>
+ (bucketStarts ?? [])
+ .flat()
+ .reduce(
+ (max, startedAt) =>
+ startedAt != null && (max == null || startedAt > max)
+ ? startedAt
+ : max,
+ null,
+ );
+
+ const transitions: AlertTransition[] = [];
+ // Assume "not firing" before the earliest known window, so an alert whose
+ // history begins already in ALERT still yields a firing marker.
+ let prevIsAlert = false;
+ let enteredRange = false;
+
+ // Pin a firing marker to the range start if the alert was already firing on
+ // entry (carried in from before startTime). The marker is synthetic ("firing
+ // when the window opens"), so it carries no bucketStart of its own.
+ const pinCarryInIfFiring = () => {
+ if (prevIsAlert) {
+ transitions.push({
+ createdAt: startTime.toISOString(),
+ state: AlertState.ALERT,
+ bucketStart: startTime.toISOString(),
+ });
+ }
+ };
+
+ for (const evalWindow of windows) {
+ const isAlert =
+ groupStateToOverallState(evalWindow.states) === AlertState.ALERT;
+ const inRange = evalWindow._id >= startTime;
+
+ if (inRange && !enteredRange) {
+ enteredRange = true;
+ pinCarryInIfFiring();
+ }
+
+ if (inRange && isAlert !== prevIsAlert) {
+ // Fall back to createdAt − interval when the window has no lastValues
+ // (mirrors the evaluation table's fallback for failed evaluations).
+ const derived =
+ newestBucketStart(evalWindow.bucketStarts) ??
+ new Date(evalWindow._id.getTime() - intervalMs);
+ // Floor at startTime: a derived bucket start one bucket earlier than an
+ // edge crossing's createdAt can precede the range start, and a marker
+ // before startTime would render at or left of a carry-in pin (which is
+ // pinned to startTime) — inverting the firing→recovery order.
+ const bucketStart = derived >= startTime ? derived : startTime;
+ transitions.push({
+ createdAt: evalWindow._id.toISOString(),
+ state: isAlert ? AlertState.ALERT : AlertState.OK,
+ bucketStart: bucketStart.toISOString(),
+ });
+ }
+
+ prevIsAlert = isAlert;
+ }
+
+ // No window landed inside the range (e.g. the alert interval is wider than
+ // the dashboard window) but the pre-range state was firing — pin it anyway.
+ if (!enteredRange) {
+ pinCarryInIfFiring();
+ }
+
+ return transitions;
+}
diff --git a/packages/api/src/controllers/alerts.ts b/packages/api/src/controllers/alerts.ts
index 095d317d3b..0e8762e48c 100644
--- a/packages/api/src/controllers/alerts.ts
+++ b/packages/api/src/controllers/alerts.ts
@@ -1,26 +1,37 @@
import {
+ displayTypeSupportsBuilderAlerts,
displayTypeSupportsRawSqlAlerts,
+ isFormulaSourceKind,
validateRawSqlForAlert,
} from '@hyperdx/common-utils/dist/core/utils';
import { isRawSqlSavedChartConfig } from '@hyperdx/common-utils/dist/guards';
-import { sign, verify } from 'jsonwebtoken';
+import { isRangeThresholdType } from '@hyperdx/common-utils/dist/types';
import { groupBy } from 'lodash';
-import ms from 'ms';
+import { Types } from 'mongoose';
import { z } from 'zod';
+import { recordOnboardingTaskCompletion } from '@/controllers/user';
import type { ObjectId } from '@/models';
-import Alert, { AlertSource, IAlert } from '@/models/alert';
+import Alert, {
+ AlertChannel,
+ AlertSource,
+ getAlertChannels,
+ IAlert,
+} from '@/models/alert';
+import Connection from '@/models/connection';
import Dashboard, { IDashboard } from '@/models/dashboard';
import { ISavedSearch, SavedSearch } from '@/models/savedSearch';
+import { Source } from '@/models/source';
import { IUser } from '@/models/user';
import Webhook from '@/models/webhook';
+import { type AlertRefs, deriveAlertDisplayFields } from '@/utils/alerts';
import { Api400Error } from '@/utils/errors';
-import logger from '@/utils/logger';
-import { alertSchema, objectIdSchema } from '@/utils/zod';
+import { internalAlertSchema, objectIdSchema } from '@/utils/zod';
export type AlertInput = Omit<
IAlert,
| 'id'
+ | 'channel'
| 'scheduleStartAt'
| 'savedSearchId'
| 'createdAt'
@@ -30,6 +41,9 @@ export type AlertInput = Omit<
| 'state'
> & {
id?: string;
+ // Exactly one of channel/channels is provided (enforced by alertSchema);
+ // `channels` flows in optionally via IAlert.
+ channel?: AlertChannel;
// Replace the Date-type fields from IAlert
scheduleStartAt?: string | null;
// Replace the ObjectId-type fields from IAlert
@@ -47,9 +61,17 @@ export const validateAlertInput = async (
teamId: ObjectId,
alertInput: Pick<
AlertInput,
- 'source' | 'dashboardId' | 'tileId' | 'savedSearchId' | 'channel'
+ | 'source'
+ | 'dashboardId'
+ | 'tileId'
+ | 'savedSearchId'
+ | 'chartConfig'
+ | 'channel'
+ | 'channels'
>,
) => {
+ const refs: AlertRefs = {};
+
if (alertInput.source === AlertSource.TILE) {
validateObjectId(alertInput.dashboardId, 'Invalid dashboard ID');
@@ -61,6 +83,7 @@ export const validateAlertInput = async (
if (dashboard == null) {
throw new Api400Error('Dashboard not found');
}
+ refs.dashboard = dashboard;
const tile = dashboard.tiles.find(tile => tile.id === alertInput.tileId);
@@ -95,23 +118,131 @@ export const validateAlertInput = async (
if (savedSearch == null) {
throw new Api400Error('Saved search not found');
}
+ refs.savedSearch = savedSearch;
}
- if (alertInput.channel.type === 'webhook') {
- validateObjectId(alertInput.channel.webhookId, 'Invalid webhook ID');
+ if (alertInput.source === AlertSource.INLINE) {
+ const chartConfig = alertInput.chartConfig;
+ if (chartConfig == null) {
+ throw new Api400Error('Chart config is required');
+ }
+
+ if (isRawSqlSavedChartConfig(chartConfig)) {
+ if (!displayTypeSupportsRawSqlAlerts(chartConfig.displayType)) {
+ throw new Api400Error(
+ 'Alerts on Raw SQL charts are only supported for Line, Stacked Bar, or Number display types',
+ );
+ }
- if (
- (await Webhook.findOne({
- _id: alertInput.channel.webhookId,
+ const { errors } = validateRawSqlForAlert(chartConfig);
+ if (errors.length > 0) {
+ throw new Api400Error(
+ `Raw SQL alert query is invalid: ${errors.join(', ')}`,
+ );
+ }
+
+ // Raw SQL configs carry the connection directly; the source reference
+ // is optional metadata for macro expansion ($__sourceTable).
+ validateObjectId(chartConfig.connection, 'Invalid connection ID');
+ const connection = await Connection.findOne({
+ _id: chartConfig.connection,
team: teamId,
- })) == null
- ) {
- throw new Api400Error('Webhook not found');
+ });
+ if (connection == null) {
+ throw new Api400Error('Connection not found');
+ }
+ if (chartConfig.source) {
+ validateObjectId(chartConfig.source, 'Invalid source ID');
+ const source = await Source.findOne({
+ _id: chartConfig.source,
+ team: teamId,
+ });
+ if (source == null) {
+ throw new Api400Error('Source not found');
+ }
+ // The worker executes the query through chartConfig.connection while
+ // expanding $__sourceTable/metricTables from this source. Accepting a
+ // source on a different (even team-owned) connection would query the
+ // wrong database — silently wrong values when the table also exists
+ // there, repeated query failures when it does not.
+ //
+ // Compare as ObjectIds, not strings: objectIdSchema admits every
+ // representation ObjectId.isValid does (uppercase hex, 12-byte
+ // strings), and the Mongo lookups above cast them — a lexical
+ // comparison would reject an equivalent non-canonical ID.
+ if (
+ !new Types.ObjectId(String(source.connection)).equals(
+ chartConfig.connection,
+ )
+ ) {
+ throw new Api400Error(
+ 'Source does not belong to the specified connection',
+ );
+ }
+ }
+ } else {
+ // Builder configs: same display types the alert task can evaluate as a
+ // time series (mirrors the tile-alert rules).
+ if (!displayTypeSupportsBuilderAlerts(chartConfig.displayType)) {
+ throw new Api400Error(
+ 'Inline chart alerts are only supported for Line, Stacked Bar, or Number display types',
+ );
+ }
+
+ validateObjectId(chartConfig.source, 'Invalid source ID');
+ const source = await Source.findOne({
+ _id: chartConfig.source,
+ team: teamId,
+ });
+ if (source == null) {
+ throw new Api400Error('Source not found');
+ }
+
+ // Same formula source-kind gate as dashboard tiles ("Add Formula" is
+ // disabled in the editor for these kinds), so the API cannot persist a
+ // config the editor refuses.
+ if (
+ 'formulas' in chartConfig &&
+ (chartConfig.formulas?.length ?? 0) > 0 &&
+ !isFormulaSourceKind(source.kind)
+ ) {
+ throw new Api400Error(
+ 'Alerts with formulas require a Metric, Log, or Trace source',
+ );
+ }
}
}
+
+ const channels = getAlertChannels(alertInput);
+ if (channels.length === 0) {
+ throw new Api400Error('At least one notification channel is required');
+ }
+
+ const webhookIds = channels
+ .filter(c => c.type === 'webhook')
+ .map(c => c.webhookId);
+ for (const webhookId of webhookIds) {
+ validateObjectId(webhookId, 'Invalid webhook ID');
+ }
+ const uniqueIds = [...new Set(webhookIds)];
+ const found = await Webhook.countDocuments({
+ _id: { $in: uniqueIds },
+ team: teamId,
+ });
+ if (found !== uniqueIds.length) {
+ throw new Api400Error('Webhook not found');
+ }
+
+ return refs;
};
-const makeAlert = (alert: AlertInput, userId?: ObjectId): Partial => {
+// Exported for unit testing the channel-mirroring invariant (see
+// controllers/__tests__/alerts.test.ts) -- otherwise only used internally.
+export const makeAlert = (
+ alert: AlertInput,
+ userId?: ObjectId,
+ refs: AlertRefs = {},
+): Partial => {
// Preserve existing DB value when scheduleStartAt is omitted from updates
// (undefined), while still allowing explicit clears via null.
const hasScheduleStartAt = alert.scheduleStartAt !== undefined;
@@ -125,9 +256,23 @@ const makeAlert = (alert: AlertInput, userId?: ObjectId): Partial => {
: hasScheduleStartAt && alert.scheduleOffsetMinutes == null
? 0
: alert.scheduleOffsetMinutes;
+ const isSavedSearch = alert.source === AlertSource.SAVED_SEARCH;
+ const isTile = alert.source === AlertSource.TILE;
+ const isInline = alert.source === AlertSource.INLINE;
+ const channels = getAlertChannels(alert);
+ // If the input explicitly provides a non-null value, persist it. If the input omits the
+ // field and a referenced entity is available, persist the value derived from the referenced entity.
+ // Otherwise, null.
+ const derivedDisplay = deriveAlertDisplayFields(alert, refs);
+ const displayName = alert.displayName ?? derivedDisplay.displayName ?? null;
+ const tags = alert.tags ?? derivedDisplay.tags ?? null;
return {
- channel: alert.channel,
+ // `channels` is canonical; `channel` mirrors channels[0] so readers that
+ // predate multi-channel support (older task runners mid-rollout) still
+ // notify the first target.
+ channel: channels[0] ?? { type: null },
+ channels,
interval: alert.interval,
...(normalizedScheduleOffsetMinutes != null && {
scheduleOffsetMinutes: normalizedScheduleOffsetMinutes,
@@ -138,7 +283,13 @@ const makeAlert = (alert: AlertInput, userId?: ObjectId): Partial => {
}),
source: alert.source,
threshold: alert.threshold,
- thresholdMax: alert.thresholdMax,
+ // Omitted rather than set to undefined for a non-range comparator, so
+ // updateAlert can $unset the path. Writing null instead would surface on
+ // the create/update responses, which return the document directly, and a
+ // client round-tripping one back into an update would fail validation.
+ ...(isRangeThresholdType(alert.thresholdType) && {
+ thresholdMax: alert.thresholdMax,
+ }),
thresholdType: alert.thresholdType,
...(userId && { createdBy: userId }),
@@ -149,24 +300,56 @@ const makeAlert = (alert: AlertInput, userId?: ObjectId): Partial => {
message: alert.message ?? null,
note: alert.note ?? null,
+ displayName,
+ tags,
+
// Log alerts
- savedSearch: alert.savedSearchId as unknown as ObjectId,
- groupBy: alert.groupBy,
- // Chart alerts
- dashboard: alert.dashboardId as unknown as ObjectId,
- tileId: alert.tileId,
+ savedSearch: isSavedSearch
+ ? ((alert.savedSearchId ?? null) as unknown as ObjectId)
+ : null,
+ groupBy: isSavedSearch ? (alert.groupBy ?? null) : null,
+ // Tile alerts
+ dashboard: isTile
+ ? ((alert.dashboardId ?? null) as unknown as ObjectId)
+ : null,
+ tileId: isTile ? (alert.tileId ?? null) : null,
+
+ // Inline alerts
+ chartConfig: isInline ? (alert.chartConfig ?? null) : null,
+
+ // Multi-window alerting
+ numConsecutiveWindows: alert.numConsecutiveWindows ?? null,
};
};
+// makeAlert omits thresholdMax for a non-range comparator, and Mongoose drops
+// an omitted key from an update rather than clearing the path, so the bound has
+// to be unset explicitly or an alert edited off `between` keeps a stale one and
+// reports a range condition it no longer has. Every update path needs this;
+// $unset is ignored on an upsert insert, so it is safe there too.
+const makeAlertUpdate = (
+ alertInput: AlertInput,
+ userId?: ObjectId,
+ refs: AlertRefs = {},
+) => ({
+ $set: makeAlert(alertInput, userId, refs),
+ ...(!isRangeThresholdType(alertInput.thresholdType) && {
+ $unset: { thresholdMax: 1 },
+ }),
+});
+
export const createAlert = async (
teamId: ObjectId,
- alertInput: z.infer,
+ alertInput: z.infer,
userId: ObjectId,
+ refs: AlertRefs = {},
) => {
- return new Alert({
- ...makeAlert(alertInput, userId),
+ const alert = await new Alert({
+ ...makeAlert(alertInput, userId, refs),
team: teamId,
}).save();
+ recordOnboardingTaskCompletion(userId, 'alert');
+ return alert;
};
// create an update alert function based off of the above create alert function
@@ -174,22 +357,29 @@ export const updateAlert = async (
id: string,
teamId: ObjectId,
alertInput: AlertInput,
+ refs: AlertRefs = {},
+ userId?: ObjectId,
) => {
// should consider clearing AlertHistory when updating an alert?
- return Alert.findOneAndUpdate(
+ const alert = await Alert.findOneAndUpdate(
{
_id: id,
team: teamId,
},
- makeAlert(alertInput),
+ makeAlertUpdate(alertInput, undefined, refs),
{
returnDocument: 'after',
},
);
+ // Editing an alert also completes "set up an alert", not just creating one.
+ if (alert != null) {
+ recordOnboardingTaskCompletion(userId, 'alert');
+ }
+ return alert;
};
-export const getAlerts = async (teamId: ObjectId) => {
- return Alert.find({ team: teamId });
+export const countAlerts = async (teamId: ObjectId) => {
+ return Alert.countDocuments({ team: teamId });
};
export const getAlertById = async (
@@ -225,12 +415,13 @@ export const getDashboardAlertsByTile = async (
};
export const createOrUpdateDashboardAlerts = async (
- dashboardId: ObjectId | string,
+ dashboard: Pick,
teamId: ObjectId,
alertsByTile: Record,
userId?: ObjectId,
) => {
- return Promise.all(
+ const dashboardId = dashboard._id;
+ const result = await Promise.all(
Object.entries(alertsByTile).map(async ([tileId, alert]) => {
const filter = {
dashboard: dashboardId,
@@ -238,18 +429,31 @@ export const createOrUpdateDashboardAlerts = async (
source: AlertSource.TILE,
team: teamId,
};
+ const alertInput = {
+ ...alert,
+ source: AlertSource.TILE,
+ dashboardId: dashboardId.toString(),
+ tileId,
+ };
const oldAlert = await Alert.findOne(filter);
- const alertValues =
+ const alertUpdate =
oldAlert && oldAlert.createdBy
- ? makeAlert(alert)
- : makeAlert(alert, userId);
+ ? makeAlertUpdate(alertInput, undefined, { dashboard })
+ : makeAlertUpdate(alertInput, userId, { dashboard });
- return await Alert.findOneAndUpdate(filter, alertValues, {
+ return await Alert.findOneAndUpdate(filter, alertUpdate, {
new: true,
upsert: true,
});
}),
);
+
+ // Tile alerts never hit the /alerts router, so record here.
+ if (result.length > 0) {
+ recordOnboardingTaskCompletion(userId, 'alert');
+ }
+
+ return result;
};
export const deleteDashboardAlerts = async (
@@ -275,96 +479,82 @@ export const deleteSavedSearchAlerts = async (
});
};
-export const getAlertsEnhanced = async (teamId: ObjectId) => {
- return Alert.find({ team: teamId }).populate<{
- savedSearch: ISavedSearch;
- dashboard: IDashboard;
- createdBy?: IUser;
- silenced?: IAlert['silenced'] & {
- by: IUser;
- };
- }>(['savedSearch', 'dashboard', 'createdBy', 'silenced.by']);
+/** Represents the documents populated and projected by getAlert[s]WithDisplayRefs */
+type AlertWithDisplayRefs = {
+ savedSearch: Pick | null;
+ dashboard: Pick | null;
};
-export const getAlertEnhanced = async (
- alertId: ObjectId | string,
+/** Minimal projections to support deriving alert names from referenced searches and dashboard tiles */
+const DISPLAY_REF_POPULATE = [
+ { path: 'savedSearch', select: 'name tags' },
+ { path: 'dashboard', select: 'name tags tiles.id tiles.config.name' },
+];
+
+/** Get alerts, with a populated (and projected) search or dashboard reference */
+export const getAlertsWithDisplayRefs = async (
teamId: ObjectId,
+ { limit, offset }: { limit: number; offset: number },
) => {
- return Alert.findOne({ _id: alertId, team: teamId }).populate<{
- savedSearch: ISavedSearch;
- dashboard: IDashboard;
- createdBy?: IUser;
- silenced?: IAlert['silenced'] & {
- by: IUser;
- };
- }>(['savedSearch', 'dashboard', 'createdBy', 'silenced.by']);
+ return Alert.find({ team: teamId })
+ .sort({ _id: 1 })
+ .skip(offset)
+ .limit(limit)
+ .populate(DISPLAY_REF_POPULATE);
};
-export const deleteAlert = async (id: string, teamId: ObjectId) => {
- return Alert.deleteOne({
- _id: id,
- team: teamId,
- });
-};
-
-export const generateAlertSilenceToken = async (
+/** Get alert, with a populated (and projected) search or dashboard reference */
+export const getAlertWithDisplayRefs = async (
alertId: ObjectId | string,
teamId: ObjectId | string,
) => {
- const secret = process.env.EXPRESS_SESSION_SECRET;
+ return Alert.findOne({
+ _id: alertId,
+ team: teamId,
+ }).populate(DISPLAY_REF_POPULATE);
+};
- if (!secret) {
- logger.error(
- 'EXPRESS_SESSION_SECRET is not set for signing token, skipping alert silence JWT generation',
- );
- return '';
- }
+/** Represents the documents populated and projected by ALERT_PAGE_POPULATE */
+export type AlertPageRefs = {
+ // `_id` is not in the select below but Mongo returns it anyway
+ savedSearch: Pick | null;
+ dashboard:
+ | (Pick & {
+ tiles: { id: string; config?: { name?: string } }[];
+ })
+ | null;
+ createdBy?: Pick;
+ silenced?: IAlert['silenced'] & {
+ by: Pick;
+ };
+};
- const alert = await getAlertById(alertId, teamId);
- if (alert == null) {
- throw new Error('Alert not found');
- }
+/**
+ * Projections for the internal alerts surfaces (the alerts page and the alert
+ * detail endpoint), kept to what the response actually renders.
+ */
+export const ALERT_PAGE_POPULATE = [
+ { path: 'savedSearch', select: 'name tags' },
+ {
+ path: 'dashboard',
+ select: 'name provisioned tags tiles.id tiles.config.name',
+ },
+ { path: 'createdBy', select: 'email name' },
+ { path: 'silenced.by', select: 'email' },
+];
- const token = sign(
- { alertId: alert._id.toString(), teamId: teamId.toString() },
- secret,
- { expiresIn: '1h' },
+export const getAlertEnhanced = async (
+ alertId: ObjectId | string,
+ teamId: ObjectId,
+) => {
+ return Alert.findOne({ _id: alertId, team: teamId }).populate(
+ ALERT_PAGE_POPULATE,
);
-
- // Slack does not accept ids longer than 255 characters
- if (token.length > 255) {
- logger.error(
- 'Alert silence JWT length is greater than 255 characters, this may cause issues with some clients.',
- );
- }
-
- return token;
};
-export const silenceAlertByToken = async (token: string) => {
- const secret = process.env.EXPRESS_SESSION_SECRET;
-
- if (!secret) {
- throw new Error('EXPRESS_SESSION_SECRET is not set for verifying token');
- }
-
- const decoded = verify(token, secret, {
- algorithms: ['HS256'],
- }) as { alertId: string; teamId: string };
-
- if (!decoded?.alertId || !decoded?.teamId) {
- throw new Error('Invalid token');
- }
-
- const alert = await getAlertById(decoded.alertId, decoded.teamId);
- if (alert == null) {
- throw new Error('Alert not found');
- }
-
- alert.silenced = {
- at: new Date(),
- until: new Date(Date.now() + ms('30m')),
- };
-
- return alert.save();
+export const deleteAlert = async (id: string, teamId: ObjectId) => {
+ return Alert.deleteOne({
+ _id: id,
+ team: teamId,
+ });
};
diff --git a/packages/api/src/controllers/alertsPage.ts b/packages/api/src/controllers/alertsPage.ts
new file mode 100644
index 0000000000..87410fb536
--- /dev/null
+++ b/packages/api/src/controllers/alertsPage.ts
@@ -0,0 +1,170 @@
+import { MAX_ALERT_DISPLAY_NAME_LENGTH } from '@hyperdx/common-utils/dist/types';
+import { escapeRegExp } from 'lodash';
+import type { FilterQuery } from 'mongoose';
+import { z } from 'zod';
+
+import { ALERT_PAGE_POPULATE, type AlertPageRefs } from '@/controllers/alerts';
+import type { ObjectId } from '@/models';
+import Alert, { AlertSource, AlertState, IAlert } from '@/models/alert';
+import { decodeCursor, encodeCursor } from '@/utils/pagination';
+import { objectIdSchema, stringListQueryParam, tagsSchema } from '@/utils/zod';
+
+const MAX_PAGE_LIMIT = 500;
+
+/** The page cursor's payload: the sort key of the last item on the page. */
+const alertsPageCursorSchema = z.object({
+ n: z.string().nullable(),
+ id: objectIdSchema,
+});
+
+type AlertsPageCursor = z.infer;
+
+/** Query parameters schema for the list alerts endpoint. */
+export const alertsPageQuerySchema = z
+ .object({
+ /** Number of alerts to return in one page. Omitted means no paging is applied */
+ limit: z.coerce.number().int().min(1).max(MAX_PAGE_LIMIT).optional(),
+ /** The cursor pointing to the last item of the previous page, if there was a previous page */
+ cursor: z.string().optional(),
+ /** A user ID, to filter for alerts created by a specific user */
+ createdBy: objectIdSchema.optional(),
+ /** A list of tags to filter alerts by. Returns alerts that have any of the specified tags */
+ tag: stringListQueryParam.pipe(tagsSchema).optional(),
+ /** A list of alert sources to filter by. Returns alerts that match any of the specified sources */
+ source: stringListQueryParam
+ .pipe(z.array(z.nativeEnum(AlertSource)).optional())
+ .optional(),
+ /** A list of alert states to filter by. Returns alerts that match any of the specified states */
+ state: stringListQueryParam
+ .pipe(z.array(z.nativeEnum(AlertState)).optional())
+ .optional(),
+ /** A search string to filter alerts by their display name. Substring, case-insensitive match */
+ search: z.string().trim().max(MAX_ALERT_DISPLAY_NAME_LENGTH).optional(),
+ })
+ .superRefine((query, ctx) => {
+ if (query.cursor == null) {
+ return;
+ }
+ // A cursor implies pagination, which requires a limit
+ if (query.limit == null) {
+ ctx.addIssue({
+ code: 'custom',
+ path: ['cursor'],
+ message: 'cursor requires limit',
+ });
+ }
+ if (decodeCursor(query.cursor, alertsPageCursorSchema) == null) {
+ ctx.addIssue({
+ code: 'custom',
+ path: ['cursor'],
+ message: 'invalid cursor',
+ });
+ }
+ });
+
+export type AlertsPageParams = z.infer;
+
+/**
+ * Mongo filters for a single page.
+ */
+export function buildAlertsPageFilter(
+ teamId: ObjectId | string,
+ params: AlertsPageParams,
+): FilterQuery {
+ const filter: FilterQuery = { team: teamId };
+
+ if (params.createdBy != null) {
+ filter.createdBy = params.createdBy;
+ }
+
+ if (params.tag != null) {
+ filter.tags = { $in: params.tag };
+ }
+
+ if (params.state != null) {
+ // OK is the default value when state is missing from the document, so we include
+ // documents without the field as well when filtering for OK state alerts.
+ filter.state = inOrMissing(params.state, AlertState.OK);
+ }
+
+ if (params.source != null) {
+ // SAVED_SEARCH is the default value when source is missing from the document, so we include
+ // documents without the field as well when filtering for saved-search alerts.
+ filter.source = inOrMissing(params.source, AlertSource.SAVED_SEARCH);
+ }
+
+ if (params.search) {
+ // Case-insensitive substring.
+ filter.displayName = {
+ $regex: escapeRegExp(params.search),
+ $options: 'i',
+ };
+ }
+
+ const cursor = params.cursor
+ ? decodeCursor(params.cursor, alertsPageCursorSchema)
+ : null;
+ if (cursor != null) {
+ applyCursor(filter, cursor);
+ }
+
+ return filter;
+}
+
+function inOrMissing(values: T[], schemaDefault: T) {
+ return {
+ $in: values.includes(schemaDefault) ? [...values, null] : values,
+ };
+}
+
+/**
+ * Filters for documents strictly after `cursor` in `{displayName: 1, _id: 1}` order.
+ *
+ * Null and missing displayNames sort first as one block, so which predicate applies
+ * depends on whether the page ended inside that block.
+ */
+function applyCursor(
+ filter: FilterQuery,
+ cursor: AlertsPageCursor,
+): void {
+ // When the previous page ended inside the null block of displayNames,
+ // the next row may be another null displayName or a named one.
+ if (cursor.n === null) {
+ filter.$or = [
+ { displayName: null, _id: { $gt: cursor.id } },
+ { displayName: { $gte: '' } },
+ ];
+ return;
+ }
+
+ // Otherwise all remaining displayNames are non-null strings, so we find documents
+ // with displayNames after the cursor, or displayName equal to the cursor but with
+ // a higher _id.
+ filter.$or = [
+ { displayName: { $gt: cursor.n } },
+ { displayName: cursor.n, _id: { $gt: cursor.id } },
+ ];
+}
+
+export async function getAlertsPage(
+ teamId: ObjectId,
+ params: AlertsPageParams,
+) {
+ const { limit } = params;
+ const query = Alert.find(buildAlertsPageFilter(teamId, params))
+ .sort({ displayName: 1, _id: 1 })
+ .populate(ALERT_PAGE_POPULATE);
+
+ // One row past the page answers "is there more?" without a second count
+ // query. Omitting limit keeps the legacy unpaginated response.
+ const docs = limit == null ? await query : await query.limit(limit + 1);
+ const hasMore = limit != null && docs.length > limit;
+ const data = hasMore ? docs.slice(0, limit) : docs;
+ const last = data.at(-1);
+ const nextCursor =
+ hasMore && last != null
+ ? encodeCursor({ n: last.displayName ?? null, id: last._id.toString() })
+ : undefined;
+
+ return { data, hasMore, nextCursor };
+}
diff --git a/packages/api/src/controllers/connection.ts b/packages/api/src/controllers/connection.ts
index 1bff0d6e79..4231f93c8f 100644
--- a/packages/api/src/controllers/connection.ts
+++ b/packages/api/src/controllers/connection.ts
@@ -1,8 +1,45 @@
+import type { ObjectId } from '@/models';
import Connection, { IConnection } from '@/models/connection';
+import { objectIdSchema } from '@/utils/zod';
+export type ConnectionValidation =
+ | { ok: true }
+ | { ok: false; status: 400 | 403; message: string };
+
+export async function validateConnectionId(
+ connection: unknown,
+ teamId: string | ObjectId | undefined,
+): Promise {
+ const parsed = objectIdSchema.safeParse(connection);
+ if (!parsed.success) {
+ return {
+ ok: false,
+ status: 400,
+ message: 'connection must be a valid connection id',
+ };
+ }
+ if (teamId == null) {
+ return { ok: false, status: 403, message: 'Forbidden' };
+ }
+ const connectionExists = await Connection.exists({
+ _id: parsed.data,
+ team: teamId,
+ });
+ if (connectionExists == null) {
+ return {
+ ok: false,
+ status: 400,
+ message: 'connection must be an existing connection id',
+ };
+ }
+ return { ok: true };
+}
+
+// Returns all connections across all teams. Only intended for instance-level
+// operations (e.g. startup auto-provisioning); user-facing routes must use
+// the team-scoped variants below.
export function getConnections() {
// Never return password back to the user
- // Return all connections in current tenant
return Connection.find({});
}
diff --git a/packages/api/src/controllers/dashboard.ts b/packages/api/src/controllers/dashboard.ts
index 92a93e4c17..c342f20f4a 100644
--- a/packages/api/src/controllers/dashboard.ts
+++ b/packages/api/src/controllers/dashboard.ts
@@ -1,7 +1,9 @@
import {
DashboardWithoutIdSchema,
+ resolveChartPaletteToken,
SavedChartConfig,
Tile,
+ walkRawDashboardTileColors,
} from '@hyperdx/common-utils/dist/types';
import { map, partition, uniq } from 'lodash';
import { z } from 'zod';
@@ -12,9 +14,11 @@ import {
getDashboardAlertsByTile,
getTeamDashboardAlertsByDashboardAndTile,
} from '@/controllers/alerts';
+import { recordOnboardingTaskCompletion } from '@/controllers/user';
import type { ObjectId } from '@/models';
import type { AlertDocument, IAlert } from '@/models/alert';
-import Dashboard from '@/models/dashboard';
+import Dashboard, { IDashboard } from '@/models/dashboard';
+import { resolveAlertDisplayFields } from '@/utils/alerts';
function pickAlertsByTile(tiles: Tile[]) {
return tiles.reduce((acc, tile) => {
@@ -25,6 +29,43 @@ function pickAlertsByTile(tiles: Tile[]) {
}, {});
}
+// Completes only when a dashboard has a tile (an empty shell isn't charting).
+// Shared by every dashboard write path (controllers, external REST v2, MCP).
+export function recordDashboardOnboardingIfHasTiles(
+ userId: string | ObjectId | undefined | null,
+ tiles: { length: number } | null | undefined,
+) {
+ if ((tiles?.length ?? 0) > 0) {
+ recordOnboardingTaskCompletion(userId, 'dashboard');
+ }
+}
+
+/**
+ * Rewrite any legacy `chart-1`..`chart-10` tile colors from #2265 in
+ * an already-serialized dashboard JSON to their hue-named equivalents
+ * before it leaves the server. Keeps the wire format on a single
+ * canonical vocabulary so non-React HTTP clients (CI scripts, stale
+ * bundle tabs during a rolling deploy, the upcoming external API
+ * surface) never have to know about the legacy values, and so a
+ * GET → unmodified PATCH round-trip on a Mongo-seeded legacy doc can
+ * never resurrect the legacy tokens through the strict server-side
+ * `ChartPaletteTokenSchema`. The React-side
+ * `normalizeDashboardTileColors` becomes redundant for the wire path
+ * after this lands but stays in place as defense in depth for
+ * `IS_LOCAL_MODE` and in-memory tile literals.
+ *
+ * Unresolvable strings (stale hexes, hand-edited values, forward-rolled
+ * future tokens) pass through untouched so the user's data is not
+ * silently dropped; the strict schema surfaces a clear error on next
+ * save.
+ */
+function healLegacyDashboardTileColors(dashboard: T): T {
+ return walkRawDashboardTileColors(dashboard, current => {
+ const resolved = resolveChartPaletteToken(current);
+ return resolved ?? current;
+ }) as T;
+}
+
type TileForAlertSync = Pick & {
config?: Pick | { alert?: IAlert | AlertDocument };
};
@@ -41,12 +82,13 @@ function extractTileAlertData(tiles: TileForAlertSync[]): {
}
async function syncDashboardAlerts(
- dashboardId: string,
+ dashboard: Pick,
teamId: ObjectId,
oldTiles: TileForAlertSync[],
newTiles: Tile[],
userId?: ObjectId,
): Promise {
+ const dashboardId = dashboard._id.toString();
const { tileIds: oldTileIds, tileIdsWithAlerts: oldTileIdsWithAlerts } =
extractTileAlertData(oldTiles);
@@ -61,7 +103,7 @@ async function syncDashboardAlerts(
const alertsByTile = pickAlertsByTile(newTiles);
if (Object.keys(alertsByTile).length > 0) {
await createOrUpdateDashboardAlerts(
- dashboardId,
+ dashboard,
teamId,
alertsByTile,
userId,
@@ -88,6 +130,23 @@ async function syncDashboardAlerts(
}
}
+/**
+ * Attach the resolved display name/tags so the chart editor prefills real
+ * values for alerts written before the fields existed.
+ */
+function withResolvedDisplayFields(
+ alert: AlertDocument | undefined,
+ dashboard: Pick,
+) {
+ if (alert == null) {
+ return undefined;
+ }
+ return {
+ ...alert.toJSON(),
+ ...resolveAlertDisplayFields(alert, { dashboard }),
+ };
+}
+
export async function getDashboards(teamId: ObjectId) {
const [_dashboards, alerts] = await Promise.all([
Dashboard.find({ team: teamId })
@@ -104,10 +163,14 @@ export async function getDashboards(teamId: ObjectId) {
...t,
config: {
...t.config,
- alert: alerts[`${d._id.toString()}:${t.id}`]?.[0],
+ alert: withResolvedDisplayFields(
+ alerts[`${d._id.toString()}:${t.id}`]?.[0],
+ d,
+ ),
},
})),
- }));
+ }))
+ .map(healLegacyDashboardTileColors);
return dashboards;
}
@@ -120,13 +183,20 @@ export async function getDashboard(dashboardId: string, teamId: ObjectId) {
getDashboardAlertsByTile(teamId, dashboardId),
]);
- return {
- ..._dashboard?.toJSON(),
- tiles: _dashboard?.tiles.map(t => ({
+ if (_dashboard == null) {
+ return null;
+ }
+
+ return healLegacyDashboardTileColors({
+ ..._dashboard.toJSON(),
+ tiles: _dashboard.tiles.map(t => ({
...t,
- config: { ...t.config, alert: alerts[t.id]?.[0] },
+ config: {
+ ...t.config,
+ alert: withResolvedDisplayFields(alerts[t.id]?.[0], _dashboard),
+ },
})),
- };
+ });
}
export async function createDashboard(
@@ -142,12 +212,14 @@ export async function createDashboard(
}).save();
await createOrUpdateDashboardAlerts(
- newDashboard._id,
+ newDashboard,
teamId,
pickAlertsByTile(dashboard.tiles),
userId,
);
+ recordDashboardOnboardingIfHasTiles(userId, newDashboard.tiles);
+
return newDashboard;
}
@@ -191,7 +263,7 @@ export async function updateDashboard(
if (updates.tiles) {
await syncDashboardAlerts(
- dashboardId,
+ updatedDashboard,
teamId,
oldDashboard?.tiles || [],
updates.tiles,
@@ -199,5 +271,7 @@ export async function updateDashboard(
);
}
+ recordDashboardOnboardingIfHasTiles(userId, updatedDashboard.tiles);
+
return updatedDashboard;
}
diff --git a/packages/api/src/controllers/savedSearch.ts b/packages/api/src/controllers/savedSearch.ts
index 203b86a5b9..ac66b1cb47 100644
--- a/packages/api/src/controllers/savedSearch.ts
+++ b/packages/api/src/controllers/savedSearch.ts
@@ -8,6 +8,8 @@ import { z } from 'zod';
import { deleteSavedSearchAlerts } from '@/controllers/alerts';
import Alert from '@/models/alert';
import { SavedSearch } from '@/models/savedSearch';
+import { resolveAlertDisplayFields } from '@/utils/alerts';
+import logger from '@/utils/logger';
type SavedSearchWithoutId = Omit, 'id'>;
@@ -26,9 +28,10 @@ export async function getSavedSearches(
return savedSearches.map(savedSearch => ({
...savedSearch.toJSON(),
- alerts: alertsBySavedSearchId[savedSearch._id.toString()]?.map(alert => {
- return alert.toJSON();
- }),
+ alerts: alertsBySavedSearchId[savedSearch._id.toString()]?.map(alert => ({
+ ...alert.toJSON(),
+ ...resolveAlertDisplayFields(alert, { savedSearch }),
+ })),
}));
}
@@ -69,11 +72,36 @@ export function updateSavedSearch(
}
export async function deleteSavedSearch(teamId: string, savedSearchId: string) {
- const savedSearch = await SavedSearch.findOneAndDelete({
+ const savedSearch = await SavedSearch.findOne({
_id: savedSearchId,
team: teamId,
});
- if (savedSearch) {
+ if (savedSearch == null) {
+ return null;
+ }
+ // Delete dependent alerts before the parent. Without a transaction (which
+ // requires a replica set), the deletes are not atomic, so this order picks
+ // the least-bad partial-failure mode: if deleteSavedSearchAlerts throws,
+ // nothing is deleted and the caller can retry. If the final deleteOne throws
+ // after the alerts are gone, the search survives without its alerts —
+ // recoverable by re-creating alerts, and strictly better than the reverse
+ // order's failure mode (orphaned alerts pointing at a deleted saved search).
+ await deleteSavedSearchAlerts(savedSearchId, teamId);
+ await SavedSearch.deleteOne({ _id: savedSearchId, team: teamId });
+ // Re-sweep after the parent is gone: a concurrent alert-create targeting this
+ // saved search could land between the two deletes above and orphan itself.
+ // Once the parent no longer exists this second sweep cleans up any such alert
+ // (and is a cheap no-op in the common case). Best-effort: the parent is
+ // already deleted, so the operation has succeeded from the caller's view — a
+ // failure here must not surface as a 500. Log it so the (rare) orphan window
+ // is observable and can be swept later, and still return success.
+ try {
await deleteSavedSearchAlerts(savedSearchId, teamId);
+ } catch (e) {
+ logger.warn(
+ { err: e, savedSearchId, team: teamId },
+ 'Post-delete alert re-sweep failed; a concurrently-created alert may be orphaned for this deleted saved search',
+ );
}
+ return savedSearch;
}
diff --git a/packages/api/src/controllers/sources.ts b/packages/api/src/controllers/sources.ts
index a34f931dff..ad3670cd27 100644
--- a/packages/api/src/controllers/sources.ts
+++ b/packages/api/src/controllers/sources.ts
@@ -1,4 +1,5 @@
-import { SourceKind, SourceSchema } from '@hyperdx/common-utils/dist/types';
+import { SourceKind, SourceSchemaNoId } from '@hyperdx/common-utils/dist/types';
+import mongoose from 'mongoose';
import {
ISourceInput,
@@ -35,8 +36,21 @@ export function getSources(team: string) {
return Source.find({ team });
}
-export function getSource(team: string, sourceId: string) {
- return Source.findOne({ _id: sourceId, team });
+export async function getSource(team: string, sourceId: string) {
+ // Pre-check the sourceId shape so a non-ObjectId input returns null
+ // (the caller's "not found" branch) instead of bubbling a Mongoose
+ // CastError.
+ if (!mongoose.Types.ObjectId.isValid(sourceId)) {
+ return null;
+ }
+ try {
+ return await Source.findOne({ _id: sourceId, team });
+ } catch {
+ // Defense-in-depth: if Mongoose still throws (e.g. a future cast
+ // path), treat it as "not found" so the caller can surface a clean
+ // error.
+ return null;
+ }
}
type DistributiveOmit = T extends T
@@ -59,35 +73,57 @@ export async function updateSource(
const existing = await Source.findOne({ _id: sourceId, team });
if (!existing) return null;
- // Same kind: simple update through the discriminator model
+ // Mongoose sets createdAt to now on replace unless the replacement
+ // document already carries one — preserve the original in both paths.
+ const createdAt: Date | undefined = existing.get('createdAt');
+
+ // findOneAndReplace swaps the whole document, so `team` must be in the
+ // replacement or the source is dropped from every team-scoped query.
if (existing.kind === source.kind) {
- // @ts-expect-error The findOneAndUpdate method has incompatible type signatures but is actually safe
- return getModelForKind(source.kind)?.findOneAndUpdate(
+ // @ts-expect-error The findOneAndReplace method has incompatible type signatures but is actually safe
+ return getModelForKind(source.kind)?.findOneAndReplace(
{ _id: sourceId, team },
- source,
+ { ...source, team, ...(createdAt && { createdAt }) },
{ new: true },
);
}
// Kind changed: validate through Zod before writing since the raw
- // collection bypass skips Mongoose's discriminator validation.
- const parseResult = SourceSchema.safeParse(source);
+ // collection bypass skips Mongoose's discriminator validation. Uses the
+ // no-id schema because callers identify the source via sourceId — any id
+ // in the body is ignored (and _id is preserved from the existing doc).
+ const parseResult = SourceSchemaNoId.safeParse(source);
if (!parseResult.success) {
throw new Error(
`Invalid source data: ${parseResult.error.errors.map(e => e.message).join(', ')}`,
);
}
+ // The raw write below bypasses Mongoose casting, so validate and cast
+ // connection explicitly — otherwise a garbage string is persisted verbatim
+ // and the stored BSON type diverges from documents written via Mongoose.
+ if (!mongoose.Types.ObjectId.isValid(parseResult.data.connection)) {
+ throw new Error('Invalid source data: connection must be a valid id');
+ }
+
// Use replaceOne on the raw collection to swap the entire document
// in place (including the discriminator key). This is a single atomic
// write — the document is never absent from the collection.
const replacement = {
...parseResult.data,
+ connection: new mongoose.Types.ObjectId(parseResult.data.connection),
_id: existing._id,
team: existing.team,
+ ...(createdAt && { createdAt }),
updatedAt: new Date(),
};
- await Source.collection.replaceOne({ _id: existing._id }, replacement);
+ const result = await Source.collection.replaceOne(
+ { _id: existing._id, team: existing.team },
+ replacement,
+ );
+ // Deleted concurrently between the findOne above and the replace: report
+ // not-found instead of returning a hydrated document that no longer exists.
+ if (result.matchedCount === 0) return null;
return getModelForKind(replacement.kind)?.hydrate(replacement);
}
diff --git a/packages/api/src/controllers/team.ts b/packages/api/src/controllers/team.ts
index be3c841a03..6fd5bba1d5 100644
--- a/packages/api/src/controllers/team.ts
+++ b/packages/api/src/controllers/team.ts
@@ -1,13 +1,21 @@
-import { TeamClickHouseSettingsUpdate } from '@hyperdx/common-utils/dist/types';
+import {
+ TagResourceType,
+ TeamClickHouseSettingsUpdate,
+} from '@hyperdx/common-utils/dist/types';
import mongoose from 'mongoose';
import { v4 as uuidv4 } from 'uuid';
import * as config from '@/config';
import type { ObjectId } from '@/models';
+import Alert from '@/models/alert';
import Dashboard from '@/models/dashboard';
import { SavedSearch } from '@/models/savedSearch';
import Team, { type ITeam, type TeamDocument } from '@/models/team';
+export function getTeamInviteUrl(token: string) {
+ return `${config.FRONTEND_URL}/join-team?token=${token}`;
+}
+
const LOCAL_APP_TEAM_ID = '_local_team_';
export const LOCAL_APP_TEAM = {
_id: new mongoose.Types.ObjectId(LOCAL_APP_TEAM_ID),
@@ -17,6 +25,7 @@ export const LOCAL_APP_TEAM = {
hookId: uuidv4(),
apiKey: uuidv4(),
collectorAuthenticationEnforced: false,
+ isMetricsSeriesTableEnabled: false,
toJSON() {
return this;
},
@@ -114,24 +123,45 @@ export function updateTeamClickhouseSettings(
return Team.findByIdAndUpdate(teamId, update, { new: true });
}
-export async function getTags(teamId: ObjectId) {
- const [dashboardTags, savedSearchTags] = await Promise.all([
- Dashboard.aggregate([
- { $match: { team: teamId } },
- { $unwind: '$tags' },
- { $group: { _id: '$tags' } },
- ]),
- SavedSearch.aggregate([
- { $match: { team: teamId } },
- { $unwind: '$tags' },
- { $group: { _id: '$tags' } },
- ]),
- ]);
-
- return [
- ...new Set([
- ...dashboardTags.map(t => t._id),
- ...savedSearchTags.map(t => t._id),
- ]),
+function getCollectionsWithTags(
+ resourceType?: TagResourceType,
+): Pick, 'aggregate'>[] {
+ if (resourceType == null) {
+ return [Alert, Dashboard, SavedSearch];
+ }
+
+ switch (resourceType) {
+ case 'alert':
+ return [Alert];
+ case 'dashboard':
+ return [Dashboard];
+ case 'savedSearch':
+ return [SavedSearch];
+ default:
+ resourceType satisfies never;
+ throw new Error(`${resourceType} is not a valid TagResourceType`);
+ }
+}
+
+/**
+ * Distinct tags applied to the team's entities. Scoped to one kind of entity
+ * when `resourceType` is given.
+ */
+export async function getTags(
+ teamId: ObjectId,
+ resourceType?: TagResourceType,
+) {
+ const distinctTagsPipeline: mongoose.PipelineStage[] = [
+ { $match: { team: teamId } },
+ { $unwind: '$tags' },
+ { $group: { _id: '$tags' } },
];
+
+ const tagGroups = await Promise.all(
+ getCollectionsWithTags(resourceType).map(collection =>
+ collection.aggregate<{ _id: string }>(distinctTagsPipeline),
+ ),
+ );
+
+ return [...new Set(tagGroups.flat().map(t => t._id))];
}
diff --git a/packages/api/src/controllers/timeseriesEngine.ts b/packages/api/src/controllers/timeseriesEngine.ts
new file mode 100644
index 0000000000..7ed0286d41
--- /dev/null
+++ b/packages/api/src/controllers/timeseriesEngine.ts
@@ -0,0 +1,408 @@
+import {
+ ChSql,
+ chSql,
+ concatChSql,
+} from '@hyperdx/common-utils/dist/clickhouse';
+import { supportsPrometheusHttpApi } from '@hyperdx/common-utils/dist/core/clickhouseVersion';
+import {
+ Metadata,
+ MetadataCache,
+} from '@hyperdx/common-utils/dist/core/metadata';
+import {
+ PrometheusMatrixResult,
+ PrometheusVectorResult,
+} from '@hyperdx/common-utils/dist/types';
+
+import { ClickhouseClient } from '@/clickhouse';
+import logger from '@/utils/logger';
+
+export const PROMETHEUS_MAX_EXECUTION_SEC = 30;
+export const PROMETHEUS_MAX_RESULT_ROWS = 100000;
+
+export type TimeSeriesTagsQueryArgs = {
+ client: ClickhouseClient;
+ connectionId: string;
+ databaseName: string;
+ tableName: string;
+ startMs?: number;
+ endMs?: number;
+ limit?: number;
+ match?: string[];
+};
+
+/**
+ * Indicates whether the tags inner table associated with the given TimeSeries table
+ * includes the optional `min_time` and `max_time` columns.
+ */
+async function timeSeriesTagsTableHasTimeBounds({
+ metadata,
+ connectionId,
+ database,
+ table,
+}: {
+ metadata: Metadata;
+ connectionId: string;
+ database: string;
+ table: string;
+}): Promise {
+ try {
+ const columns = await metadata.getTimeSeriesTableColumns({
+ connectionId,
+ databaseName: database,
+ tableName: table,
+ innerTableType: 'Tags',
+ });
+
+ const columnNames = new Set(columns.map(column => column.name));
+ return columnNames.has('min_time') && columnNames.has('max_time');
+ } catch (e) {
+ logger.warn(
+ { err: e, database, table },
+ 'Failed to check if TimeSeries tags table has time bounds columns',
+ );
+ return false;
+ }
+}
+
+// `timeSeriesSelector()` requires both bounds, so we define the minimum and maximum possible times
+// as default bounds when explicit bounds are not provided.
+const TIMESERIES_MIN_TIME_MS = -2208988800000; // 1900-01-01T00:00:00Z
+const TIMESERIES_MAX_TIME_MS = 10413791999999; // 2299-12-31T23:59:59.999Z
+
+/**
+ * Returns a predicate keeping only the series matched by at least one of
+ * the given `match` selectors.
+ */
+function getSeriesSelectorCondition({
+ databaseName,
+ tableName,
+ match,
+ startMs,
+ endMs,
+}: {
+ databaseName: string;
+ tableName: string;
+ match: string[];
+ startMs?: number;
+ endMs?: number;
+}): ChSql {
+ const minTime = { Int64: startMs ?? TIMESERIES_MIN_TIME_MS };
+ const maxTime = { Int64: endMs ?? TIMESERIES_MAX_TIME_MS };
+ const idsPerSelector = match.map(
+ selector => chSql`
+ SELECT id
+ FROM timeSeriesSelector(
+ ${{ String: databaseName }},
+ ${{ String: tableName }},
+ ${{ String: selector }},
+ fromUnixTimestamp64Milli(${minTime}),
+ fromUnixTimestamp64Milli(${maxTime})
+ )`,
+ );
+
+ return chSql`id IN (${concatChSql(' UNION DISTINCT ', idsPerSelector)})`;
+}
+
+/**
+ * Returns SQL predicates restricting tags rows to the series a label lookup
+ * should consider: those overlapping [startMs, endMs], and those matched by
+ * `match`. Empty when neither is given.
+ */
+async function getSeriesFilterConditions({
+ client,
+ connectionId,
+ databaseName,
+ tableName,
+ startMs,
+ endMs,
+ match,
+}: Omit): Promise {
+ const selectors = match?.length ? match : undefined;
+ if (startMs == null && endMs == null && selectors == null) return [];
+
+ // Both filters lean on min_time/max_time, which the tags inner table carries
+ // only when the table was created with store_min_time_and_max_time on.
+ const metadata = new Metadata(client, new MetadataCache());
+ const tableHasTimeBounds = await timeSeriesTagsTableHasTimeBounds({
+ metadata,
+ connectionId,
+ database: databaseName,
+ table: tableName,
+ });
+
+ const conditions: ChSql[] = [];
+
+ if (selectors != null) {
+ // timeSeriesSelector() prunes on min_time/max_time unconditionally, so
+ // without those columns it cannot run. Unlike the bounds it also cannot
+ // degrade to "no filter" — that would answer a different question.
+ if (!tableHasTimeBounds) {
+ throw new Error(
+ 'match[] requires a TimeSeries table created with store_min_time_and_max_time = 1',
+ );
+ }
+ conditions.push(
+ getSeriesSelectorCondition({
+ databaseName,
+ tableName,
+ match: selectors,
+ startMs,
+ endMs,
+ }),
+ );
+ }
+
+ if (tableHasTimeBounds) {
+ if (endMs != null)
+ conditions.push(
+ chSql`(min_time IS NULL OR min_time <= fromUnixTimestamp64Milli(${{ Int64: endMs }}))`,
+ );
+ if (startMs != null)
+ conditions.push(
+ chSql`(max_time IS NULL OR max_time >= fromUnixTimestamp64Milli(${{ Int64: startMs }}))`,
+ );
+ }
+
+ return conditions;
+}
+
+/**
+ * Runs `SELECT DISTINCT AS val FROM timeSeriesTags(...)` with the given
+ * conditions, sorted, and returns the distinct values.
+ */
+async function queryDistinctTagsValues({
+ client,
+ databaseName,
+ tableName,
+ value,
+ conditions,
+ limit,
+}: {
+ client: ClickhouseClient;
+ databaseName: string;
+ tableName: string;
+ value: ChSql;
+ conditions: ChSql[];
+ limit?: number;
+}): Promise {
+ const where = conditions.length
+ ? chSql`WHERE ${concatChSql(' AND ', ...conditions)}`
+ : chSql``;
+ const limitSql = limit
+ ? chSql`LIMIT ${{ Int32: Math.min(limit, PROMETHEUS_MAX_RESULT_ROWS) }}`
+ : chSql``;
+ const query = chSql`
+ SELECT DISTINCT ${value} AS val
+ FROM timeSeriesTags(${{ Identifier: databaseName }}, ${{ Identifier: tableName }})
+ ${where}
+ ORDER BY val
+ ${limitSql}
+ `;
+
+ const resp = await client.query({
+ query: query.sql,
+ query_params: query.params,
+ format: 'JSON',
+ clickhouse_settings: {
+ allow_experimental_time_series_table: 1,
+ max_execution_time: PROMETHEUS_MAX_EXECUTION_SEC,
+ max_result_rows: String(PROMETHEUS_MAX_RESULT_ROWS),
+ },
+ });
+
+ const json = await resp.json<{ val: string }>();
+ return json.data.map(r => r.val);
+}
+
+/**
+ * Queries distinct values for the given label name from the given TimeSeries
+ * table, optionally narrowed to a time range and to `match`'s series selectors.
+ */
+export async function queryLabelValues({
+ labelName,
+ limit,
+ ...args
+}: TimeSeriesTagsQueryArgs & { labelName: string }): Promise {
+ const isMetricName = labelName === '__name__';
+ const value = isMetricName
+ ? chSql`${{ Identifier: 'metric_name' }} `
+ : chSql`${{ Identifier: 'tags' }}[${{ String: labelName }}]`;
+
+ const conditions: ChSql[] = [];
+ if (!isMetricName)
+ conditions.push(
+ chSql`mapContains(${{ Identifier: 'tags' }}, ${{ String: labelName }})`,
+ );
+ conditions.push(...(await getSeriesFilterConditions(args)));
+
+ return queryDistinctTagsValues({ ...args, value, conditions, limit });
+}
+
+/**
+ * Queries the distinct label names carried by any series in the given TimeSeries
+ * table, optionally narrowed to a time range and to `match`'s series selectors.
+ */
+export async function queryLabelNames({
+ limit,
+ ...args
+}: TimeSeriesTagsQueryArgs): Promise {
+ // The engine keeps `__name__` in `metric_name`, not `tags`, so it is folded
+ // back in explicitly.
+ const value = chSql`arrayJoin(arrayConcat([${{ String: '__name__' }}], mapKeys(${{ Identifier: 'tags' }})))`;
+ const conditions = await getSeriesFilterConditions(args);
+
+ return queryDistinctTagsValues({ ...args, value, conditions, limit });
+}
+
+// --------------------------
+// PromQL via table functions (ClickHouse < 26.6)
+// --------------------------
+//
+// Servers without the `prometheus_api_v1` HTTP handler can still evaluate
+// PromQL through `prometheusQuery`/`prometheusQueryRange`. ClickHouse only
+// holds the HTTP API forward-compatible while TimeSeries is in preview, so
+// this path is a fallback for old servers, not the primary route.
+
+const toUnixSeconds = (timestamp: string | number) =>
+ typeof timestamp === 'string'
+ ? new Date(timestamp).getTime() / 1000
+ : Number(timestamp);
+
+export function formatMatrixResponse(
+ rows: { tags: [string, string][]; samples: [string, number][] }[],
+): PrometheusMatrixResult[] {
+ return rows.map(row => ({
+ metric: Object.fromEntries(row.tags),
+ values: row.samples.map(([ts, value]) => [
+ toUnixSeconds(ts),
+ String(value),
+ ]),
+ }));
+}
+
+/**
+ * The name of the per-series samples column, which the engine renamed from
+ * `time_series` to `samples` in TimeSeries schema version 3. The rename
+ * follows the table's pinned version, not the server's, so a v2 table on a new
+ * server still reads `time_series`. `prometheusQueryRange` returns whichever
+ * the table uses. Looked up fresh each call: a throwaway cache, so a table
+ * dropped and recreated at a new version is seen at once.
+ */
+async function timeSeriesSamplesColumn({
+ client,
+ connectionId,
+ databaseName,
+ tableName,
+}: {
+ client: ClickhouseClient;
+ connectionId: string;
+ databaseName: string;
+ tableName: string;
+}): Promise<'samples' | 'time_series'> {
+ const metadata = new Metadata(client, new MetadataCache());
+ const version = await metadata.getTimeSeriesTableVersion({
+ connectionId,
+ databaseName,
+ tableName,
+ });
+ return version >= 3 ? 'samples' : 'time_series';
+}
+
+export function formatVectorResponse(
+ rows: { tags: [string, string][]; timestamp: string; value: number }[],
+): PrometheusVectorResult[] {
+ return rows.map(row => ({
+ metric: Object.fromEntries(row.tags),
+ value: [toUnixSeconds(row.timestamp), String(row.value)],
+ }));
+}
+
+const PROMQL_TABLE_FUNCTION_SETTINGS = {
+ allow_experimental_time_series_table: 1,
+ max_execution_time: PROMETHEUS_MAX_EXECUTION_SEC,
+ max_result_rows: String(PROMETHEUS_MAX_RESULT_ROWS),
+} as const;
+
+export async function queryRangeViaTableFunction({
+ client,
+ connectionId,
+ databaseName,
+ tableName,
+ expr,
+ startMs,
+ endMs,
+ stepSec,
+}: {
+ client: ClickhouseClient;
+ connectionId: string;
+ databaseName: string;
+ tableName: string;
+ expr: string;
+ startMs: number;
+ endMs: number;
+ stepSec: number;
+}): Promise {
+ const samplesColumn = await timeSeriesSamplesColumn({
+ client,
+ connectionId,
+ databaseName,
+ tableName,
+ });
+ const resp = await client.query({
+ query: `SELECT tags, ${samplesColumn} AS samples FROM prometheusQueryRange({db:String}, {table:String}, {expr:String}, fromUnixTimestamp64Milli({startMs:Int64}), fromUnixTimestamp64Milli({endMs:Int64}), toIntervalSecond({stepSec:UInt32})) SETTINGS allow_experimental_time_series_table = 1`,
+ query_params: {
+ db: databaseName,
+ table: tableName,
+ expr,
+ startMs,
+ endMs,
+ stepSec,
+ },
+ format: 'JSON',
+ clickhouse_settings: PROMQL_TABLE_FUNCTION_SETTINGS,
+ });
+ const json = await resp.json();
+ return formatMatrixResponse(json.data);
+}
+
+export async function queryInstantViaTableFunction({
+ client,
+ databaseName,
+ tableName,
+ expr,
+ evalMs,
+}: {
+ client: ClickhouseClient;
+ databaseName: string;
+ tableName: string;
+ expr: string;
+ evalMs: number;
+}): Promise {
+ const resp = await client.query({
+ query: `SELECT tags, timestamp, value FROM prometheusQuery({db:String}, {table:String}, {expr:String}, fromUnixTimestamp64Milli({evalMs:Int64})) SETTINGS allow_experimental_time_series_table = 1`,
+ query_params: { db: databaseName, table: tableName, expr, evalMs },
+ format: 'JSON',
+ clickhouse_settings: PROMQL_TABLE_FUNCTION_SETTINGS,
+ });
+ const json = await resp.json();
+ return formatVectorResponse(json.data);
+}
+
+/**
+ * Whether the connection's ClickHouse is new enough for the Prometheus HTTP
+ * API. `SELECT version()` runs each call (throwaway cache) so an upgraded
+ * server is picked up without a restart; an unknown version is treated as
+ * old, matching the other version-gated features.
+ */
+export async function connectionSupportsPrometheusHttpApi({
+ client,
+ connectionId,
+}: {
+ client: ClickhouseClient;
+ connectionId: string;
+}): Promise {
+ const metadata = new Metadata(client, new MetadataCache());
+ return supportsPrometheusHttpApi(
+ await metadata.getServerVersion({ connectionId }),
+ );
+}
diff --git a/packages/api/src/controllers/user.ts b/packages/api/src/controllers/user.ts
index 55258c2db6..7c80b782be 100644
--- a/packages/api/src/controllers/user.ts
+++ b/packages/api/src/controllers/user.ts
@@ -1,12 +1,27 @@
+import type { OnboardingTaskId } from '@hyperdx/common-utils/dist/types';
+import { isPersistableUserId as isPersistableUserIdHex } from '@hyperdx/common-utils/dist/types';
import mongoose from 'mongoose';
+import { v4 as uuidv4 } from 'uuid';
import type { ObjectId } from '@/models';
import Alert from '@/models/alert';
import User from '@/models/user';
+import logger from '@/utils/logger';
export function findUserByAccessKey(accessKey: string) {
return User.findOne({ accessKey });
}
+/**
+ * Rotates a user's personal access key, immediately revoking the previous one.
+ *
+ * There is exactly one key per user and no grace period: findUserByAccessKey
+ * above is hit uncached on every bearer request (see validateUserAccessKey), so
+ * requests presenting the old key start 401ing the instant this returns.
+ */
+export function rotateUserAccessKey(userId: string | ObjectId) {
+ return User.findByIdAndUpdate(userId, { accessKey: uuidv4() }, { new: true });
+}
+
export function findUserById(id: string) {
return User.findById(id);
}
@@ -20,6 +35,66 @@ export function findUsersByTeam(team: string | ObjectId) {
return User.find({ team }).sort({ createdAt: 1 });
}
+// Type-guard wrapper over the shared 24-hex check; rejects the synthetic
+// `_local_user_` id injected in IS_LOCAL_APP_MODE (see isPersistableUserIdHex).
+function isPersistableUserId(
+ userId: string | ObjectId | undefined | null,
+): userId is string | ObjectId {
+ return userId != null && isPersistableUserIdHex(String(userId));
+}
+
+// Returns null for a non-persistable user so the route reports unchanged
+// default state rather than a write that silently matched nothing.
+export function completeOnboardingTask(
+ userId: string | ObjectId,
+ taskId: OnboardingTaskId,
+) {
+ if (!isPersistableUserId(userId)) {
+ return null;
+ }
+ return User.findByIdAndUpdate(
+ userId,
+ { $addToSet: { 'onboardingData.completedTasks': taskId } },
+ { new: true },
+ );
+}
+
+export function setOnboardingDismissed(
+ userId: string | ObjectId,
+ isDismissed: boolean,
+) {
+ if (!isPersistableUserId(userId)) {
+ return null;
+ }
+ return User.findByIdAndUpdate(
+ userId,
+ { $set: { 'onboardingData.isDismissed': isDismissed } },
+ { new: true },
+ );
+}
+
+// Fire-and-forget recording from an unrelated write path (alert/dashboard save,
+// MCP tool call): must never fail or delay the triggering operation, so errors
+// are swallowed. The $ne skips the write once already recorded — these fire on
+// every save / tool call, so it avoids write amplification on hot paths.
+export function recordOnboardingTaskCompletion(
+ userId: string | ObjectId | undefined | null,
+ taskId: OnboardingTaskId,
+) {
+ if (!isPersistableUserId(userId)) {
+ return;
+ }
+ void User.updateOne(
+ { _id: userId, 'onboardingData.completedTasks': { $ne: taskId } },
+ { $addToSet: { 'onboardingData.completedTasks': taskId } },
+ ).catch(err => {
+ logger.warn(
+ { error: err, userId: userId.toString(), taskId },
+ 'Failed to record onboarding task completion',
+ );
+ });
+}
+
export async function deleteTeamMember(
teamId: string | ObjectId,
userIdToDelete: string,
diff --git a/packages/api/src/controllers/webhook.ts b/packages/api/src/controllers/webhook.ts
new file mode 100644
index 0000000000..a37f41fcbe
--- /dev/null
+++ b/packages/api/src/controllers/webhook.ts
@@ -0,0 +1,145 @@
+import type { ObjectId } from '@/models';
+import Alert from '@/models/alert';
+import Webhook, {
+ type WebhookDocument,
+ WebhookService,
+} from '@/models/webhook';
+import { validateWebhookConfig } from '@/utils/validators';
+
+export interface WebhookInput {
+ name: string;
+ service: WebhookService;
+ url: string;
+ description?: string;
+ queryParams?: Record;
+ headers?: Record;
+ body?: string;
+}
+
+/**
+ * Create a team-scoped webhook. Shared by the internal API, External API v2, and
+ * MCP; callers handle their own duplicate-key errors and secret redaction.
+ */
+export async function createWebhook(
+ team: ObjectId | string,
+ { name, service, url, description, queryParams, headers, body }: WebhookInput,
+) {
+ validateWebhookConfig({ service, url, body });
+
+ return Webhook.create({
+ team,
+ name,
+ service,
+ url,
+ description,
+ queryParams,
+ headers,
+ body,
+ });
+}
+
+export type UpdateWebhookResult =
+ | { status: 'ok'; webhook: WebhookDocument }
+ | { status: 'not_found' }
+ | { status: 'conflict' };
+
+/**
+ * Update (full replace) a team-scoped webhook. Readable fields (description,
+ * body) are set/cleared by presence; write-only fields (headers, queryParams)
+ * are preserved when omitted and cleared on empty {} — or cleared outright when
+ * the destination (url/service) changes, so stored secrets are never forwarded
+ * to a new destination. Shared by External API v2 and MCP.
+ */
+export async function updateWebhook(
+ team: ObjectId | string,
+ webhookId: string,
+ { name, service, url, description, queryParams, headers, body }: WebhookInput,
+): Promise {
+ const existing = await Webhook.findOne({ _id: webhookId, team });
+ if (existing == null) {
+ return { status: 'not_found' };
+ }
+
+ validateWebhookConfig({ service, url, body });
+
+ const destinationChanged =
+ url !== existing.url || service !== existing.service;
+
+ const setFields: Record = { name, service, url };
+ const unsetFields: Record = {};
+
+ if (description === undefined) unsetFields.description = 1;
+ else setFields.description = description;
+
+ if (body === undefined) unsetFields.body = 1;
+ else setFields.body = body;
+
+ if (headers === undefined) {
+ if (destinationChanged) unsetFields.headers = 1;
+ } else if (Object.keys(headers).length === 0) {
+ unsetFields.headers = 1;
+ } else {
+ setFields.headers = headers;
+ }
+
+ if (queryParams === undefined) {
+ if (destinationChanged) unsetFields.queryParams = 1;
+ } else if (Object.keys(queryParams).length === 0) {
+ unsetFields.queryParams = 1;
+ } else {
+ setFields.queryParams = queryParams;
+ }
+
+ const updateOp: Record =
+ Object.keys(unsetFields).length > 0
+ ? { $set: setFields, $unset: unsetFields }
+ : { $set: setFields };
+
+ // Pin to the snapshotted url/service so a concurrent destination change
+ // yields a conflict rather than attaching a secret to the wrong destination.
+ const webhook = await Webhook.findOneAndUpdate(
+ { _id: webhookId, team, url: existing.url, service: existing.service },
+ updateOp,
+ { new: true },
+ );
+
+ if (webhook == null) {
+ const stillExists = await Webhook.exists({ _id: webhookId, team });
+ return stillExists != null
+ ? { status: 'conflict' }
+ : { status: 'not_found' };
+ }
+
+ return { status: 'ok', webhook };
+}
+
+export type DeleteWebhookResult =
+ | { status: 'ok'; webhook: WebhookDocument }
+ | { status: 'not_found' }
+ | { status: 'referenced'; alertCount: number };
+
+/**
+ * Delete a webhook for a team, blocking deletion while alerts still reference it
+ * so we never orphan an alert onto a missing destination.
+ */
+export async function deleteWebhook(
+ team: ObjectId | string,
+ webhookId: string,
+): Promise {
+ // Match on webhookId alone (not channel.type) so a legacy/skewed alert that
+ // still references this webhook also blocks deletion.
+ const alertCount = await Alert.countDocuments({
+ 'channel.webhookId': webhookId,
+ team,
+ });
+ if (alertCount > 0) {
+ return { status: 'referenced', alertCount };
+ }
+
+ const deleted = await Webhook.findOneAndDelete({ _id: webhookId, team });
+ if (deleted == null) {
+ return { status: 'not_found' };
+ }
+
+ return { status: 'ok', webhook: deleted };
+}
diff --git a/packages/api/src/fixtures.ts b/packages/api/src/fixtures.ts
index 5dc05094ac..47d2edafd8 100644
--- a/packages/api/src/fixtures.ts
+++ b/packages/api/src/fixtures.ts
@@ -1,8 +1,10 @@
import { createNativeClient } from '@hyperdx/common-utils/dist/clickhouse/node';
import {
+ AlertChartConfig,
AlertThresholdType,
BuilderSavedChartConfig,
DisplayType,
+ Filter,
RawSqlSavedChartConfig,
SavedChartConfig,
Tile,
@@ -257,6 +259,114 @@ export const executeSqlCommand = async (sql: string) => {
});
};
+// The TimeSeries engine is experimental, and its flag is a *query* setting — it
+// cannot ride along in a CREATE's own SETTINGS clause, which only takes storage
+// settings. Every statement therefore carries it, which is why these tables
+// cannot go through `executeSqlCommand`.
+export const executeTimeSeriesSqlCommand = async (sql: string) => {
+ const client = await getTestFixtureClickHouseClient();
+ return await client.command({
+ query: sql,
+ clickhouse_settings: {
+ allow_experimental_time_series_table: 1,
+ wait_end_of_query: 1,
+ },
+ });
+};
+
+export const dropTimeSeriesTable = async ({
+ table,
+ database = DEFAULT_DATABASE,
+}: {
+ table: string;
+ database?: string;
+}) => executeTimeSeriesSqlCommand(`DROP TABLE IF EXISTS ${database}.${table}`);
+
+export type TimeSeriesFixtureSeries = {
+ metricName: string;
+ /** Labels other than `__name__`, which is derived from `metricName`. */
+ tags: Record;
+ /** Series window in unix seconds; ignored when the table stores no bounds. */
+ startSec: number;
+ endSec: number;
+};
+
+/**
+ * (Re)creates a TimeSeries table and writes `series` straight into its tags
+ * inner table — Prometheus remote-write is the only other way in.
+ *
+ * `storeTimeBounds: false` creates the table with
+ * `store_min_time_and_max_time = 0`, which leaves the tags table without the
+ * min_time/max_time columns a time-bounded lookup reads.
+ *
+ * `withSamples: true` also writes a sample at each series' `startSec` and
+ * `endSec`. A `match[]` lookup matches only series that have a sample in the
+ * window, so on a tags-only table every selector answers nothing.
+ */
+export const seedTimeSeriesTagsTable = async ({
+ table,
+ series,
+ database = DEFAULT_DATABASE,
+ storeTimeBounds = true,
+ withSamples = false,
+}: {
+ table: string;
+ series: TimeSeriesFixtureSeries[];
+ database?: string;
+ storeTimeBounds?: boolean;
+ withSamples?: boolean;
+}) => {
+ if (withSamples && !storeTimeBounds) {
+ throw new Error(
+ 'withSamples needs storeTimeBounds: sample timestamps come from min_time/max_time',
+ );
+ }
+
+ await dropTimeSeriesTable({ table, database });
+ await executeTimeSeriesSqlCommand(
+ `CREATE TABLE ${database}.${table} ENGINE = TimeSeries${
+ storeTimeBounds ? '' : ' SETTINGS store_min_time_and_max_time = 0'
+ }`,
+ );
+
+ const quoted = (v: string) => `'${v.replace(/'/g, "\\'")}'`;
+ const mapLiteral = (tags: Record) =>
+ `map(${Object.entries(tags)
+ .flatMap(([k, v]) => [quoted(k), quoted(v)])
+ .join(', ')})`;
+
+ const columns = storeTimeBounds
+ ? '(metric_name, tags, min_time, max_time)'
+ : '(metric_name, tags)';
+ const values = series
+ .map(s => {
+ const row = [quoted(s.metricName), mapLiteral(s.tags)];
+ if (storeTimeBounds) {
+ row.push(
+ `toDateTime64(${s.startSec}, 3)`,
+ `toDateTime64(${s.endSec}, 3)`,
+ );
+ }
+ return `(${row.join(', ')})`;
+ })
+ .join(', ');
+
+ await executeTimeSeriesSqlCommand(
+ `INSERT INTO TABLE FUNCTION timeSeriesTags('${database}', '${table}') ${columns} VALUES ${values}`,
+ );
+
+ // The engine derives `id` from the tags, so the samples are read back out of
+ // the tags table rather than recomputed here.
+ if (withSamples) {
+ await executeTimeSeriesSqlCommand(
+ `INSERT INTO TABLE FUNCTION timeSeriesData('${database}', '${table}')
+ SELECT id, ts AS timestamp, 1 AS value
+ FROM timeSeriesTags('${database}', '${table}')
+ ARRAY JOIN [min_time, max_time] AS ts`,
+ );
+ }
+};
+
export const clearClickhouseTables = async () => {
if (!config.IS_CI) {
throw new Error('ONLY execute this in CI env 😈 !!!');
@@ -332,15 +442,26 @@ export const bulkInsertLogs = async (
await bulkInsertData(`${DEFAULT_DATABASE}.${DEFAULT_LOGS_TABLE}`, events);
};
-export const bulkInsertMetricsGauge = async (
- metrics: {
- MetricName: string;
- ResourceAttributes: Record
\n', /raw HTML/],
+ 'off-site link': [
+ 'See [x](https://evil.example/p)\n',
+ /Disallowed link target: https:\/\/evil\.example\/p/,
+ ],
+ 'protocol-relative': [
+ 'See [x](//evil.example/p)\n',
+ /Disallowed link target: \/\/evil\.example\/p/,
+ ],
+ 'uppercase protocol': [
+ 'See [x](HTTPS://evil.example/p)\n',
+ /Disallowed link target/,
+ ],
+ 'relative link': [
+ 'See [x](/local/path)\n',
+ /Disallowed link target: \/local\/path/,
+ ],
+ 'host suffix confusion': [
+ 'See [x](https://github.com.evil.example/p)\n',
+ /Disallowed link target/,
+ ],
+ 'bare url in prose': [
+ 'Read more at https://evil.example/p for details.\n',
+ /Disallowed URL in prose: https:\/\/evil\.example\/p/,
+ ],
+ };
+ for (const [label, [body, expected]] of Object.entries(bad)) {
+ const errors = validateBody(body);
+ assert.ok(errors.length > 0, `should reject: ${label}`);
+ assert.match(errors.join('\n'), expected, `wrong reason for: ${label}`);
+ }
+});
+
+test('validateBody does not fire on prose a code span or fence makes literal', () => {
+ // These reddened the publish job while the release shipped with no changelog
+ // section — the worst of both outcomes for a legitimate body.
+ const good = [
+ 'Summary.\n\n```yaml\n# a comment, not a heading\nkey: value\n```\n',
+ 'Wrap the value in `` and it works.\n',
+ 'Use `# heading` syntax in the template.\n',
+ 'The `` form is no longer emitted.\n',
+ 'Copy from `https://clickhouse.com/docs` if you need the upstream page.\n',
+ ];
+ for (const body of good) {
+ assert.deepEqual(validateBody(body), [], `should accept: ${body}`);
+ }
+});
+
+test('validateBody agrees with the render-time allowlist on parsed-URL forms', () => {
+ // A prefix regex rejected these while allowChangelogUrl renders them, so the
+ // publish job reddened for content that was fine. Same algorithm now.
+ for (const url of [
+ 'https://github.com:443/hyperdxio/hyperdx',
+ 'https://user@github.com/hyperdxio/hyperdx',
+ 'https://docs.hyperdx.io',
+ 'https://GitHub.com/x',
+ ]) {
+ assert.deepEqual(validateBody(`See [x](${url}).\n`), [], `accept: ${url}`);
+ }
+});
+
+test('the render-time host allowlist matches this script’s', () => {
+ // Two languages, one policy. Pinned here so a host added to one side cannot
+ // silently make the CI gate stricter (a red release) or looser (a phishing
+ // link that only the drawer blocks).
+ const drawer = readFileSync(
+ join(REPO_ROOT, 'packages/app/src/components/AppNav/WhatsNewDrawer.tsx'),
+ 'utf-8',
+ );
+ const literal = drawer.match(
+ /ALLOWED_LINK_HOSTS = new Set\(\[([^\]]*)\]\)/,
+ )?.[1];
+ assert.ok(
+ literal,
+ 'could not find ALLOWED_LINK_HOSTS in WhatsNewDrawer.tsx',
+ );
+ const appHosts = [...literal.matchAll(/'([^']+)'/g)].map(m => m[1]);
+
+ assert.deepEqual(appHosts.sort(), [...ALLOWED_LINK_HOSTS].sort());
+});
+
+test('validateBody accepts a bullet that thanks an outside contributor', () => {
+ // A handle is plain text and the profile link is on an allowed host, so
+ // thanking someone must not redden the publish job.
+ assert.deepEqual(
+ validateBody('- **Fix**: yes (#1, thanks @alice!).\n'),
+ [],
+ );
+ assert.deepEqual(
+ validateBody(
+ '- **Fix**: yes (#1, thanks [@alice](https://github.com/alice)!).\n',
+ ),
+ [],
+ );
+});
+
+test('validateBody rejects an oversized body', () => {
+ assert.ok(validateBody('x'.repeat(70_000)).length > 0);
+});
+
+test('CLI validate exits 1 with an annotation, 0 on a clean body', () => {
+ const dir = mkdtempSync(join(tmpdir(), 'release-notes-'));
+ const bad = join(dir, 'bad.md');
+ const good = join(dir, 'good.md');
+ writeFileSync(bad, 'See 
\n';
+ assert.ok(validateBody(hidden).length > 0);
+
+ // A properly closed fence is still fine.
+ assert.deepEqual(validateBody('Summary.\n\n```yaml\nkey: value\n```\n'), []);
+});
+
+test('validateBody rejects a fenced ## because parseChangelog is not fence-aware', () => {
+ // The splice would treat it as a section boundary and cut the section in two,
+ // so validate has to agree with the parser rather than with CommonMark here.
+ const fenced = 'Summary.\n\n```markdown\n## Not a real heading\n```\n';
+ assert.ok(validateBody(fenced).length > 0);
+});
+
+test('validateBody rejects CommonMark heading forms that a bare ^## misses', () => {
+ // Up to three leading spaces, and a tab delimiter, both render as real H2s.
+ assert.ok(validateBody(' ## v9.9.9 — Security notice\n').length > 0);
+ assert.ok(validateBody('##\tv9.9.9 — Security notice\n').length > 0);
+ assert.ok(validateBody(' # Forged H1\n').length > 0);
+ // `###` and deeper are the section headings the prompt asks for.
+ assert.deepEqual(validateBody('### ✨ New Features\n\n- **x**: y\n'), []);
+});
+
+test('insertSection drops a non-release section above every release', () => {
+ const file = insertSection(null, OPTS);
+ const withOrphan = file.replace(
+ '## v2.33.0',
+ '## Notes from review\n\nSomething a human typed.\n\n## v2.33.0',
+ );
+ const out = insertSection(withOrphan, {
+ ...OPTS,
+ version: '2.34.0',
+ inputs: 'newer',
+ date: '2026-08-01',
+ body: 'Newer body.',
+ });
+
+ assert.doesNotMatch(out, /Notes from review/);
+ // Real release sections are still preserved.
+ assert.match(out, /## v2\.33\.0/);
+ assert.match(out, /## v2\.34\.0/);
+});
+
+test('insertSection refuses to splice over a non-release heading inside published notes', () => {
+ // The tail of a release split by a hand-added `## ` carries real release
+ // notes. Dropping it would delete them from the committed changelog with
+ // nothing in the run log to show for it.
+ const file = insertSection(null, OPTS);
+ const split = `${file}\n## Notes from review\n\nSomething a human typed.\n`;
+
+ assert.throws(
+ () =>
+ insertSection(split, {
+ ...OPTS,
+ version: '2.34.0',
+ inputs: 'newer',
+ date: '2026-08-01',
+ body: 'Newer body.',
+ }),
+ /Notes from review/,
+ );
+});
+
+test('stripPackageList leaves a half-deleted marker pair alone', () => {
+ const list = `${PACKAGE_LIST_HEADING}\n\n- \`@hyperdx/app\` 1 → 2\n`;
+ const note = 'A maintainer note written below the generated list.';
+
+ // End marker gone: there is no safe cut, so nothing is removed and the
+ // leftover start marker makes validateBody treat the body as a cache miss.
+ const endGone = `Summary.\n\n${PACKAGE_LIST_START}\n\n${list}\n${note}\n`;
+ assert.equal(stripPackageList(endGone), endGone);
+ assert.match(
+ validateBody(stripPackageList(endGone)).join('\n'),
+ /package-list marker/,
+ );
+
+ // Start marker gone: the heading is where the generated block begins.
+ const startGone = `Summary.\n\n${list}\n${PACKAGE_LIST_END}\n\n${note}\n`;
+ const stripped = stripPackageList(startGone);
+ assert.doesNotMatch(stripped, /Package changelogs/);
+ assert.match(stripped, /Summary\./);
+ assert.match(stripped, /A maintainer note written below/);
+ assert.deepEqual(validateBody(stripped), []);
+});
+
+test('CLI latest-version falls back to the heading when the marker is gone', () => {
+ // Deleting the marker is the one edit the design anticipates. Exiting 2 here
+ // made the caller hand the already-released section to the generator as this
+ // release's prior text.
+ const dir = mkdtempSync(join(tmpdir(), 'release-notes-'));
+ const changelog = join(dir, 'CHANGELOG.md');
+ writeFileSync(
+ changelog,
+ insertSection(null, OPTS).replace(
+ /\n/,
+ '',
+ ),
+ );
+
+ const res = runCli(['latest-version', '--changelog', changelog]);
+ assert.equal(res.code, 0);
+ assert.equal(res.stdout, '2.33.0');
+});
+
+test('validateBody rejects raw HTML that GitHub would render', () => {
+ // react-markdown ignores raw HTML, but the committed CHANGELOG.md is rendered
+ // by GitHub with these allowed, and 
\n',
+ 'Click here\n',
+ 'Summary \n',
+ 'Summary \n',
+ ]) {
+ assert.ok(validateBody(body).length > 0, `should reject: ${body}`);
+ }
+ // Prose using angle brackets is not HTML and must still pass.
+ assert.deepEqual(
+ validateBody('Latency is <2ms and `Map
- 
