Skip to content

fix(app): only show distribution labels the data backs - #3230

Merged
jordan-simonovski merged 3 commits into
mainfrom
jordansimonovski/distribution-filtered-values
Sep 28, 2026
Merged

jordan-simonovski merged 3 commits into
mainfrom
jordansimonovski/distribution-filtered-values

Conversation

@jordan-simonovski

@jordan-simonovski jordan-simonovski commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

With a field filtered to one value, Show distribution on that same field showed the selected value as >99% and every other value as <1%. Neither was true: the selected value is exactly 100%, and the other values can't appear under the filter. <1% reads as "rare but present", so it looked like data.

Each label now shows only when the data backs it:

  • 100%: the field's filter allows only this value.
  • >99% and ~N%: estimated from a sample of about 100k rows.
  • <1%: the value was seen in the sample below 1%, or it's missing from a result that was cut off at 100 values, so it ranks below the 100th.
  • No label: the field's own filter rules the value out, or it wasn't seen in a result that wasn't cut off.
  • Known gaps, not fixed here: percentages on fields with more than about 1000 distinct values can come out too high, and for about a second after a selection changes the labels come from the old data.

Before:

distribution-before

After:

distribution-after

Also:

  • Numeric and boolean values now match their percentages. ClickHouse returns them unquoted, so the lookup by string never found them.
  • The toggle tooltip says the percentages come from a sample of rows.

@vercel

vercel Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Sep 28, 2026 10:50pm UTC
hyperdx-storybook Ready Ready Preview Sep 28, 2026 10:50pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8f177d5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/app Patch
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Sep 25, 2026
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Additional context: spans packages but only 49 prod lines — under the 100-line cross-layer bar

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 2
  • Production lines changed: 49 (+ 175 in test files, excluded from tier calculation)
  • Branch: jordansimonovski/distribution-filtered-values
  • Author: jordan-simonovski

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 2 minor

1 posted as inline comment(s) on the changed lines. 2 listed below.

Findings outside the changed lines

2 minor
  • 🔵 packages/app/src/components/__tests__/DBSearchPageFilters.test.tsx:203 — Tests don't pin the cap boundary or the limit + 1 request → The mock ignores limit, and the only test data sizes are 2 and 101. So reverting limit to DISTRIBUTION_VALUE_LIMIT would pass every test, even though it makes the capped branch unreachable in production. So would changing size > DISTRIBUTION_VALUE_LIMIT to >= DISTRIBUTION_VALUE_LIMIT or > 99. Add an assertion that useGetValuesDistribution is called with limit: 101, and a case with exactly 100 entries where a missing value gets no label.
  • 🔵 packages/app/src/components/__tests__/DBSearchPageFilters.test.tsx:221 — The 'own filter' test only checks that the test's own input is passed through → The test injects chartConfig.filters into FilterGroup and then asserts that the same object reaches the mocked hook. That doesn't check the assumption isExcludedByOwnFilter depends on: that the field's filter state ends up in filtersChartConfig (built in packages/app/src/DBSearchPage.tsx). Either test that filtersChartConfig is built from the filter state, or delete this test, since it would still pass if the assumption broke.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Fixes distribution label display logic in search filters.

The PR appears safe to merge, with one non-blocking distribution-label edge case still outstanding.

Fix All in Claude CodeFindings

  1. P2 Rare values lose their label ▶

Summary

The PR changes distribution labels to distinguish filtered-out values from sampled values, adds an exact 100% label for a single allowed value, and normalizes numeric and boolean distribution keys.

  • Adds tests for filtered distribution queries and label behavior.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Facet value] --> B{Excluded by its field filter?}
  B -- Yes --> C[No label]
  B -- No --> D{Present in distribution sample?}
  D -- Yes --> E[Show sampled percentage or exact 100%]
  D -- No --> F{Result capped?}
  F -- Yes --> G[Show less than 1%]
  F -- No --> C
Loading

Reviews (3) · Last reviewed commit: "Merge branch 'main' into jordansimonovsk..."

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: distribution-label display logic in the search-page filters, plus a ClickHouse numeric/boolean key-mapping fix. Intent: show 100% only when the field's own filter allows a single value, <1% only when the sample backs it (rare, or missing from a capped result), and hide the label when the field's filter rules a value out or it's missing from an uncapped result. No P0/P1 issues found; the change is low-risk and well-tested for its happy paths.

🟡 P2 — recommended

  • packages/common-utils/src/core/metadata.ts:2412 — Capped-result detection depends on a fragile cross-file invariant: the caller fetches DISTRIBUTION_VALUE_LIMIT + 1 rows while getPercentage infers capping from distributionData.size > DISTRIBUTION_VALUE_LIMIT, and getValuesDistribution's default limit = 100 silently disables detection for any caller that omits the +1.
    • Fix: Return capping from the query layer (e.g. { distribution, truncated }, mirroring getMetricNames) so the component no longer infers it from Map size.
    • maintainability, correctness, kieran-typescript, testing
🔵 P3 nitpicks (4)
  • packages/app/src/components/DBSearchPageFilters.tsx:598 — The sort comparator reads distributionData.get(...) ?? 0 directly, bypassing getPercentage, so exclusion and capped/uncapped semantics can drift from the display path over time.
    • Fix: Route both the sort and the label through a single shared percentage helper.
  • packages/app/src/components/DBSearchPageFilters.tsx:657 — A value present in the facet options but absent from an uncapped distribution sample renders with no label at all, since getPercentage returns undefined; the author documents this as an accepted sampling trade-off.
    • Fix: Treat an uncapped miss for a known-present, non-excluded option as <1%, or document the facet-vs-sample divergence at the call site.
    • previous-comments, correctness
  • packages/app/src/components/DBSearchPageFilters.tsx:183 — The triple-nested ternary building the label (<1% / 100% / >99% / ~N%) is hard to scan.
    • Fix: Extract a small getPercentageLabel(percentage, isExact) helper with early returns.
  • .changeset/quiet-filters-hide.md:6 — The changeset body is a single run-on summary sentence with no detail paragraph, unlike existing entries.
    • Fix: Use a short imperative summary followed by a detail paragraph.

Reviewers (10): correctness, testing, maintainability, project-standards, kieran-typescript, julik-frontend-races, performance, previous-comments, agent-native, learnings-researcher.

Testing gaps:

  • No test pins the exactly-100-distinct-values boundary (uncapped → hidden) that distinguishes the <1% and hidden branches, so a future edit to the limit + 1 offset or DISTRIBUTION_VALUE_LIMIT could regress silently.
  • No test covers isPercentageExact with a sampled percentage < 99.5 (a sole-included value with a low sampled share renders ~N%, not 100%).
  • Pre-existing (not worsened here): distribution labels are served from the previous filter's sample during the ~1s keepPreviousData refetch window, signalled only by the pulse animation.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 385 passed • 1 skipped • 1574s

Status Count
✅ Passed 385
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@jordan-simonovski jordan-simonovski changed the title fix(app): stop labelling values the field's own filter rules out as <1% fix(app): only show distribution labels the data backs Sep 25, 2026
Comment on lines +655 to +657
// Missing from a capped result means it ranks below the 100th value, so
// under 1%. Missing from an uncapped result means it wasn't seen at all.
return distributionData.size > DISTRIBUTION_VALUE_LIMIT ? 0 : undefined;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Rare values lose their label When a value appears in the facet options but its rare rows are missed by the separate distribution sample, a result with at most 100 entries makes this code show no percentage. That hides the <1% label for a value that is present: an uncapped sample shows only that the value was not sampled, not that it has no matching rows.

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement!

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 128 in total (1.83%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

if (percentage != null) return percentage;
// Missing from a capped result means it ranks below the 100th value, so
// under 1%. Missing from an uncapped result means it wasn't seen at all.
return distributionData.size > DISTRIBUTION_VALUE_LIMIT ? 0 : undefined;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Rare values that really exist lose their "<1%" label on sampled tables

getValuesDistribution (packages/common-utils/src/core/metadata.ts:2362-2377) samples roughly 100k rows once a table is larger than that. So a value missing from a result that wasn't cut off may simply not have been picked by the sample; it doesn't mean the value is absent. The facet options come from getKeyValues over the same filtered chartConfig, so an unselected option in the list does have matching rows. Example: a 10M-row table where a listed status value appears in 0.001% of rows. The sample misses it, and it now shows no label, which looks like 'no rows' even though clicking it returns results. Before this change it showed a correct <1%. Suggested fix: once isExcludedByOwnFilter has ruled out filtered values, fall back to 0 (<1%) for any other missing value. If you want to keep the 'not seen' case, only return undefined when the query wasn't sampled (have getValuesDistribution also return sample_factor, and check it is 1).

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 8f177d5c Deployed Sep 28, 2026 by vercel[bot]
Preview – hyperdx-storybook — 8f177d5c Deployed Sep 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants