Skip to content

feat: auto-detect a PromQL source during onboarding - #3310

Merged
karl-power merged 2 commits into
mainfrom
hdx-5595-auto-detect-promql-source
Oct 7, 2026
Merged

karl-power merged 2 commits into
mainfrom
hdx-5595-auto-detect-promql-source

Conversation

@karl-power

Copy link
Copy Markdown
Contributor

Summary

When PromQL is enabled (NEXT_PUBLIC_ENABLE_PROMQL=true), onboarding auto-detection now creates a PromQL source if the connection has a ClickHouse TimeSeries table, the same way it already creates Logs, Traces, Metrics and Sessions sources from the OTel tables.

  • Metadata.getTimeSeriesTables lists tables with engine = 'TimeSeries'. Matching on the engine also skips the TimeSeries table's inner MergeTree tables.
  • pickTimeSeriesTable prefers metrics_ts (the table the collector's PromQL migration creates) in the OTel tables' database, then any metrics_ts, then the only TimeSeries table if there is exactly one.
  • A connection with only a TimeSeries table now gets a PromQL source instead of being sent to manual setup.
  • A failure in TimeSeries detection or PromQL source creation leaves the other sources in place.

How to test

Delete all sources then go to /search page. OTEL and TSE sources are auto-detected and added.

References

  • Linear Issue: HDX-5595 (part 1; a stacked PR adds dev/local provisioning)

@vercel

vercel Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 7, 2026 3:26pm UTC
hyperdx-storybook Ready Ready Preview Oct 7, 2026 3:26pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: c51bc27

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 4
  • Production lines changed: 148 (+ 323 in test files, excluded from tier calculation)
  • Branch: hdx-5595-auto-detect-promql-source
  • Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 127 in total (1.78%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Adds PromQL source auto-detection during onboarding.

The PR appears safe to merge; no new actionable issues were found.

What we checked:

  • PromQL discovery stops OTel setup: The lookup catches its own error and returns an empty list, so OTel discovery can continue.
  • Discovery picks an inner table: The query requires the TimeSeries engine, which excludes inner MergeTree tables.

Summary

Adds automatic PromQL source creation during onboarding when PromQL is enabled.

  • Finds outer TimeSeries tables and prefers prometheus.metrics, then metrics_ts.
  • Keeps OTel onboarding working if TimeSeries discovery or PromQL source creation fails.
  • Adds table-selection, onboarding, and metadata tests.
  • karl-power dismissed the earlier search-page concern because searchable sources are expected to be present. That unnumbered thread is not reposted.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Onboarding] --> B[Find OTel tables]
  A --> C{PromQL enabled?}
  C -->|Yes| D[Find TimeSeries tables]
  C -->|No| E[Skip PromQL]
  D --> F[Pick preferred TimeSeries table]
  B --> G[Create OTel sources]
  F --> H[Try creating PromQL source]
  G --> I{Any sources created?}
  H --> I
  I -->|Yes| J[Link sources and close onboarding]
  I -->|No| K[Manual setup]
Loading

Reviews (4) · Last reviewed commit: "support documented promql table name" · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Auto-detecting a PromQL source during onboarding. The core change is sound: TimeSeries detection and PromQL creation are both failure-isolated, the new getTimeSeriesTables query/caching is type-correct, and the selection helper is well-tested. No correctness, security, or data-integrity defects introduced by this diff. One recommended UX gap (a previously-raised review thread) and a cluster of robustness/test nits remain.

🟡 P2 — recommended

  • packages/app/src/components/OnboardingModal.tsx:519 — A connection with only a TimeSeries table now completes onboarding by creating a Promql-only source and calls setStep('closed'), but /search filters sources to ALLOWED_SOURCE_KINDS = [SourceKind.Log, SourceKind.Trace] (packages/app/src/DBSearchPage.tsx:187), so the user lands on an empty, unselectable source picker instead of the pre-change manual-setup fallback.
    • Fix: Route a PromQL-only onboarding completion to a metrics/dashboard destination that can query the source, rather than closing onto /search.
    • previous-comments, adversarial
🔵 P3 nitpicks (7)
  • packages/app/src/source.ts:520 — The final fallback in pickTimeSeriesTable returns any lone TimeSeries table regardless of name or database, so onboarding can auto-wire a PromQL source to an unrelated single TimeSeries table with no signal it was a guess.
    • Fix: Consider restricting the sole-table fallback to conventional names/databases, or otherwise flag an auto-picked non-conventional table.
  • packages/common-utils/src/core/metadata.ts:656 — The [] returned on a privileges error is cached under ${connectionId}.timeSeriesTables, so if grants widen within the same cache lifetime auto-detection keeps seeing an empty list.
    • Fix: Skip caching the degraded-empty result, or scope it so a later privilege grant can re-query.
    • correctness, reliability, kieran-typescript, maintainability
  • packages/common-utils/src/core/metadata.ts:682 — The privileges check matches on the English substring 'Not enough privileges', so other access-error wordings or codes rethrow instead of degrading to [].
    • Fix: Match on the ClickHouse error code/type rather than a localized message substring.
    • reliability, adversarial
  • packages/app/src/components/OnboardingModal.tsx:326 — getOtelTables has no .catch in the Promise.all, so an OTel query rejection discards an otherwise-detectable PromQL source and sends the user to manual setup, an asymmetry with the wrapped TimeSeries arm.
    • Fix: Decide intentionally whether an OTel detection error should still allow PromQL auto-creation, and guard accordingly.
    • correctness, reliability
  • packages/app/src/__tests__/source.test.ts:84 — No case exercises the collectorNamed.find(preferredDatabase) ?? collectorNamed[0] fall-through where preferredDatabase is supplied but matches no metrics_ts, so a too-loose find would pass.
    • Fix: Add a case with multiple metrics_ts tables and a non-matching preferredDatabase asserting the first collector-named table is returned.
  • packages/app/src/components/__tests__/OnboardingModal.test.tsx:138 — In the PromQL-failure test, createdKinds() is derived from attempted mockCreateSource calls so it still includes the failed Promql call; only the toast text proves the failure was swallowed.
    • Fix: Assert on the resulting created/linked sources instead of attempted-call kinds so the test distinguishes attempted from succeeded.
  • packages/common-utils/src/core/metadata.ts:680 — getTimeSeriesTables returns [] on empty/permission while sibling getOtelTables returns null, giving the detection API two divergent empty conventions.
    • Fix: Document the convention difference in the method JSDoc so future consumers don't assume parity.

Reviewers (8): correctness, testing, maintainability, kieran-typescript, reliability, previous-comments, adversarial, api-contract.

Testing gaps:

  • No test covers getOtelTables rejecting while a TimeSeries table exists (the Promise.all rejection path that drops a valid PromQL source).
  • No test asserts the post-onboarding destination for a PromQL-only result, so the empty-/search outcome is uncaught.
  • The cached-empty-result behavior of getTimeSeriesTables is only asserted indirectly via a stubbed getOrFetch.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 398 passed • 1 skipped • 1673s

Status Count
✅ Passed 398
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@karl-power
karl-power force-pushed the hdx-5595-auto-detect-promql-source branch from 414db2d to 1b8df8a Compare October 7, 2026 09:46
pulpdrew
pulpdrew previously approved these changes Oct 7, 2026

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and worked well locally.

Just confirming whether the following is/will be covered:

  1. "Connect to Demo Server" in IS_LOCAL_MODE will detect a time series table if one is present. Maybe this can be done as part of HDX-5450.
  2. Onboarding in ClickStack Cloud / ClickHouse observability (may be EE-specific modals?) will detect timeseries tables.

Comment thread packages/app/src/source.ts Outdated
@@ -480,6 +480,34 @@ export async function inferTableSourceConfig({
};
}

// The TimeSeries table name used by the collector's PromQL schema migration
// and the dev ClickHouse remote-write handler.
const DEFAULT_TIME_SERIES_TABLE_NAME = 'metrics_ts';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know if this aligns with the table that is created in ClickHouse cloud when an org enables the timeseries engine? Ideally this should work well in Cloud for the Timeseries Private Preview.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. I don't believe a table is created. However, the docs suggest a name here, so I've added that as the preferred table.

@karl-power

karl-power commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor Author

@pulpdrew

"Connect to Demo Server" in IS_LOCAL_MODE will detect a time series table if one is present. Maybe this can be done as part of HDX-5450.

Yep I think we should add this with the demo server changes.

Onboarding in ClickStack Cloud / ClickHouse observability (may be EE-specific modals?) will detect timeseries tables.

Yeah, I planned to do this in the EE PR.

@karl-power
karl-power requested a review from pulpdrew October 7, 2026 15:21
@greptile-apps

greptile-apps Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings could not be posted inline.

  • P2 Review finding ▶

    Confidence Score: 5/5

    The PR appears safe to merge; no new blocking issues were found.

    Summary

    Onboarding now detects ClickHouse TimeSeries tables and creates a PromQL source when PromQL is enabled.

    • Since the previous review, pickTimeSeriesTable now prefers prometheus.metrics before metrics_ts. Tests and the changeset reflect that choice.
    • No new actionable issues were found.
    • karl-power dismissed the earlier concern about PromQL-only connections leaving search empty because searchable sources are expected to be present. That thread is resolved and is not reposted.

@karl-power
karl-power merged commit e61461b into main Oct 7, 2026
30 checks passed

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — c51bc274 Deployed Oct 7, 2026 by vercel[bot]
Preview – hyperdx-storybook — c51bc274 Deployed Oct 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants