Skip to content

feat: stream filter keys and value results to UI - #3348

Merged
kodiakhq[bot] merged 16 commits into
mainfrom
aaron/stream-filter-key-value-results
Oct 16, 2026
Merged

kodiakhq[bot] merged 16 commits into
mainfrom
aaron/stream-filter-key-value-results

Conversation

@knudtty

@knudtty knudtty commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Filter queries no longer wait for all values to arrive. Values are streamed and the UI updates as keys and values arrive. It now behaves that:

  1. Keys are fetched
  2. Keys are returned. The UI updates and shows keys even if they have not yet fetched values.
  3. Values are fetched and begin streaming in. The filters UI updates as new values arrive.
  4. Eventually all value queries finish. The filters UI shows the finished state.

Screenshots or video

Before

export-1791576561547.mp4

After

export-1791575968941.mp4

How to test on Vercel preview

Steps:

  1. Go to logs
  2. Assign a longer time range (1-2 days)
  3. View the filters bar

References

  • Linear Issue:

@changeset-bot

changeset-bot Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 73c8f2a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/app Patch
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 16, 2026 6:18pm UTC
hyperdx-storybook Ready Ready Preview Oct 16, 2026 6:18pm UTC

Request Review

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 125 in total (1.69%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@greptile-apps

greptile-apps Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue was established.

Summary

Filter keys now appear before their values. Values arrive as each lookup finishes, with a loader on filters still waiting.

  • Metadata values arrive as each lookup finishes.
  • Streamed queries keep displayed results until replacements finish.
  • Filter keys appear with a loader while their values load.
  • Filter test helpers wait for groups to settle and keep the chosen source.

Previous threads: the findings about quoted-key loaders, saved metadata settings, cached exact results, and duplicated filter types were manually resolved without explanation. The header-color thread was also resolved; knudtty replied “Fixed,” and the header loader now uses a theme token. All supplied threads were unnumbered.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Fetch filter keys] --> B[Show keys with loaders]
  B --> C[Start value queries]
  C --> D[Each completed query yields values]
  D --> E[Publish values to the cache]
  E --> F[Update matching filters]
  F --> G[Remove loaders when fetching ends]
Loading

Reviews (13) · Last reviewed commit: "Merge branch 'main' into aaron/stream-fi..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 430 passed • 1 skipped • 1784s

Status Count
✅ Passed 430
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

…ey-value-results

# Conflicts:
#	packages/app/src/components/DBSearchPageFilters/hooks.test.tsx
#	packages/app/src/hooks/useMetadata.tsx
Drop the separate useAllKeyValues hook and stream 'all' mode through the
existing useMultipleGetKeyValues query, restoring main's multi-source
behavior (each source's own MVs, the single-source MVs override, limit,
partial results when a source fails).

Partial results only fill an empty panel; values already showing stay
until the new result is complete, so filters don't blank out mid-use.

Also return useFetchFacets' pendingKeys inside data.
Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
@knudtty
knudtty force-pushed the aaron/stream-filter-key-value-results branch from 80a6d11 to f1b1c33 Compare October 14, 2026 21:01
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 549 production lines changed (Tier 2 max: < 250)
  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 7
  • Production lines changed: 549 (+ 745 in test files, excluded from tier calculation)
  • Branch: aaron/stream-filter-key-value-results
  • Author: knudtty

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

5 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 4 minor

2 posted as inline comment(s) on the changed lines. 3 listed below.

Findings outside the changed lines

3 minor
  • 🔵 packages/app/src/components/DBSearchPageFilters/utils.ts:614 — Keys that 'all' mode never answers show a loader for the whole fetch, then vanish → The useMultipleGetKeyValues doc says 'all' mode skips JSON-column keys. Metadata dispatchKeyValueQueries also drops keys that match no strategy and aren't in rawTable. Yet keysToFetch includes JSON sub-paths, and listRequestedFacets lists every requested key as pending until isFetching ends. These rows spin for the full duration and are then dropped. Fix: leave keys that can't be answered out of requestedKeys in 'all' mode (e.g. filter out keys under jsonColumns).
  • 🔵 packages/common-utils/src/core/metadata.ts:1766 — inCompletionOrder repeats the race-over-a-Map loop in mergeStreams → mergeStreams in packages/app/src/hooks/useStreamingQuery.tsx implements the same "race pending promises, delete the winner, yield its value" loop (it is the multi-step version). Per the DRY (REQUIRED) convention, move mergeStreams into common-utils (e.g. core/utils). Then build inCompletionOrder on it, or drop it, so the app imports the shared helper.
  • 🔵 packages/app/src/hooks/useMetadata.tsx:1332 — Exact mode copies client defaults by hand and hardcodes refetchOnWindowFocus: true → Exact mode goes through useStreamingQuery and then undoes its overrides by reading retry/structuralSharing from queryClient.getDefaultOptions(). It also forces refetchOnWindowFocus: true, which ignores any app-level default, and it doesn't restore staleTime. Every new override in useStreamingQuery has to be mirrored here by hand. Fix: let useStreamingQuery take a streaming: false option that skips its overrides, or run exact mode through a plain useQuery path.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Streams filter keys and value results to the filters sidebar: keys render first, then per-key values stream in as each metadata query completes. Reviewed against base f56d760d. 11 reviewers ran (report-only). Findings merged, deduplicated, and re-graded to a ship-blocker bar; cross-reviewer corroboration noted in the <sub> lines.

🔴 P0/P1 — must fix

  • packages/app/src/hooks/useStreamingQuery.tsx:138 — data: query.isError ? undefined : query.data unconditionally blanks the result on error, so a failed refetch after a successful load discards the complete values React Query still holds in query.data; in all mode retry: false means one transient ClickHouse/network error collapses the whole sidebar to "No filters available," a regression from the prior plain useQuery which retained data on error.
    • Fix: return the retained query.data on error when it is a completed (non-partial) result, suppressing only genuinely partial streamed data.
    • correctness, adversarial, api-contract, previous-comments

🟡 P2 — recommended

  • packages/app/src/components/DBSearchPageFilters/utils.ts:46 — all-mode lookups return unquoted keys (e.g. service-name) while requestedKeys carry the quoted form, so the verbatim arrivedByKey.get(sqlKey) misses: the key stays in pendingKeys (spinner) while its values are re-appended through the trailing loop, producing a loaded-but-spinning filter and a transient duplicate entry.
    • Fix: key arrivedByKey by toUiKey(facet.key) and look it up by toUiKey(sqlKey) before adding a pending placeholder; add the quoted-column-arrives-early test.
    • previous-comments, testing
  • packages/app/src/hooks/useMetadata.tsx:510 — the query key includes metadataMVsOverride but not each source's resolved metadataMaterializedViews used in the lookup, so saving a changed rollup table or granularity reuses stale filter values for the 5-minute cache window (pre-existing cache-key gap, unresolved in scope).
    • Fix: include the resolved per-source materialized views in the query key.
🔵 P3 nitpicks (15)
  • packages/app/src/hooks/useStreamingQuery.tsx:130 — the caller's staleTime: 1000*60*5 wins over the hook's staleTime: Infinity via ...options spread order, re-enabling the stale refetch that the Infinity guard exists to prevent and that triggers the blanking above.
    • Fix: apply the load-bearing staleTime after the spread, or merge it deliberately.
    • adversarial
  • packages/app/src/hooks/useStreamingQuery.tsx:85 — keepPreviousData makes isShowingData true after the first load, so partial flushes are skipped and incremental streaming only occurs on an empty first view; later loads wait for the full result, limiting the PR's stated benefit.
    • Fix: confirm this matches intent, or rework the isShowingData gate if incremental display on refetch is wanted.
    • adversarial, correctness, frontend-races
  • packages/app/src/hooks/useMetadata.tsx:556 — the exact branch casts results.flat() as Facet[] without the .toString() the all branch applies, so numeric values are typed as string[] but remain number[] at runtime (HDX-1548); the cast actively suppresses the type error.
    • Fix: stringify values in the exact branch and drop the cast, or widen Facet.value.
    • kieran-typescript, testing, api-contract
  • packages/app/src/components/DBSearchPageFilters/hooks.ts:212 — escapedKeysToFetch.slice(0, maxKeys) mirrors the truncation useMultipleGetKeyValues already applies internally; if the two diverge, pendingKeys strands keys the query never fetched or omits real ones.
    • Fix: return the effective fetched key set from the hook instead of re-slicing in the caller.
    • maintainability
  • packages/app/src/hooks/useStreamingQuery.tsx:90 — the streaming path enforces no timeout tighter than the 1-hour ClickHouse request_timeout, so a hung value query keeps isFetching true and leaves pendingKeys spinning indefinitely with no error surfaced.
    • Fix: bound the typeahead stream with an explicit timeout that surfaces an empty/error state.
    • reliability, frontend-races
  • packages/app/src/hooks/useStreamingQuery.tsx:29 — void iterators[i].return?.() is fire-and-forget, so a rejecting return() during early-exit cleanup becomes an unhandled rejection; throw errors.get(0) is also typed possibly-undefined and positional.
    • Fix: attach .catch(() => {}) to the cleanup and capture the chosen error in a typed local.
    • reliability, kieran-typescript
  • packages/app/src/components/DBSearchPageFilters/utils.ts:11 — FacetEntry re-declares key/value instead of deriving from Facet, leaving three near-identical key/value shapes with divergent value element types.
    • Fix: define FacetEntry = Omit<Facet, 'value'> & { value: (string | boolean)[]; isLoading?: boolean }.
    • previous-comments, maintainability
  • packages/app/src/components/DBSearchPageFilters.tsx:920 — FilterGroup/NestedFilterGroup are not memoized, so each 100ms stream flush reconciles the entire filter list (up to maxKeys subtrees) on cold load even though one key changed.
    • Fix: wrap both in React.memo and memoize their per-group props.
    • performance
  • packages/app/src/components/DBSearchPageFilters.tsx:1332 — structuredClone(_facet) deep-copies each facet's value array only to rewrite facet.key, a cost now re-incurred ~10x/sec during the cold-load stream.
    • Fix: use a shallow { ..._facet } copy.
    • performance
  • packages/common-utils/src/core/metadata.ts:2867 — each strategy promise is wrapped in .catch(() => undefined) and empty results are skipped, so a key whose value query fails silently vanishes from the sidebar, indistinguishable from a key with no values.
    • Fix: mark failed keys with an error/retry affordance rather than dropping them.
    • reliability
  • packages/common-utils/src/core/metadata.ts:2382 — inCompletionOrder/streamAllKeyValues never observe the AbortSignal, so a superseded all stream drains every dispatched query to completion before the generator ends.
    • Fix: race pending promises against signal abort to short-circuit.
    • frontend-races
  • packages/app/src/components/DBSearchPageFilters/utils.ts:56 — the trailing "results the request didn't name" loop re-introduces no-longer-requested keys on a key-set shrink (e.g. toggling "Show more fields" off), so stale facets linger until the new fetch settles.
    • Fix: append only arrived keys whose UI key is still in requestedKeys.
    • frontend-races
  • packages/app/src/hooks/useMetadata.tsx:501 — exactModeOptions reaches into queryClient.getDefaultOptions() to undo useStreamingQuery's hardcoded retry/structuralSharing/refetchOnWindowFocus for the non-streaming path, implicitly coupling to another hook's internals.
    • Fix: have useStreamingQuery accept streaming overrides as explicit opt-ins, or route exact through a plain useQuery.
    • maintainability
  • packages/app/src/hooks/useMetadata.tsx:483 — options?: Omit<UseQueryOptions<any, Error>, 'queryKey'> uses an any data generic, so options forwarded into the now-typed useStreamingQuery<Facet> are unchecked against Facet[].
    • Fix: narrow to UseQueryOptions<Facet[], Error>.
    • kieran-typescript
  • packages/app/src/components/DBSearchPageFilters.tsx:1908 — a reindented Loader still uses raw color="gray" while its new sibling correctly uses var(--color-text-muted).
    • Fix: align the moved Loader to the semantic token if touching the line is in scope.
    • project-standards

Reviewers (11): correctness, adversarial, frontend-races, kieran-typescript, performance, reliability, api-contract, testing, maintainability, project-standards, previous-comments.

Testing gaps:

  • No test for a failed refetch after a successful load — the exact path that blanks the filter list (P1 above); existing tests cover only successful refetch and first-load failure.
  • No test for a quoted column whose values arrive (as an unquoted key) before a slower query — the stuck-spinner/duplicate path.
  • No end-to-end test of the pendingKeys → per-filter loader partial render in DBSearchPageFilters; coverage is unit-level only.
  • No test that the query key invalidates when a source's materialized views (rollup table/granularity) change.
  • No test asserting stream re-render / setQueryData count stays bounded, nor that partial flushes are suppressed under keepPreviousData.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

Comment thread packages/app/src/hooks/useMetadata.tsx Resolved
Comment thread packages/app/src/hooks/useMetadata.tsx Resolved

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, very nice improvement

@kodiakhq
kodiakhq Bot merged commit 85675fc into main Oct 16, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the aaron/stream-filter-key-value-results branch October 16, 2026 18:28

This branch was successfully deployed

2 active deployments
Preview – hyperdx-storybook — 73c8f2a6 Deployed Oct 16, 2026 by vercel[bot]
Preview – hyperdx-oss — 73c8f2a6 Deployed Oct 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants