Skip to content

fix: replace MD5 with SHA256 for FIPS compatibility - #3409

Merged
wrn14897 merged 24 commits into
ClickHouse:mainfrom
hercynium:bugfix/replace-md5-with-sha256-1
Nov 3, 2026
Merged

wrn14897 merged 24 commits into
ClickHouse:mainfrom
hercynium:bugfix/replace-md5-with-sha256-1

Conversation

@hercynium

@hercynium hercynium commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This patch is to resolve Issue #3390 - when running HyperDX against ClickHouse in FIPS mode the MD5 function isn't available, so viewing some logs fails, because MD5 is not FIPS-compliant.

All uses of MD5 were replaced with equivalent uses of SHA256. This includes in the tests and in comments.

This patch was generated using Cursor, running Composer 2.5 Fast. All relevant tests still pass.

Screenshots or video

NA - no UI changes.

How to reproduce

Starting point:

  • Run ClickHouse in FIPS mode, and configure HyperDX to use it.
  • Load some long log-lines into the relevant table.

On main:

  1. Click on one of the long log entries. You should see the error documented in issue HyperDX compatibility with FIPS ClickHouse #3390.

On this branch:

  1. Click on one of the long log entries. You should see the usual data for the log entry.

References

@changeset-bot

changeset-bot Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 96d1e0e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 5 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/cli Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 19, 2026

Copy link
Copy Markdown

@hercynium is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added external needs-vouch Author needs a maintainer to vouch for them labels Oct 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi @hercynium, thanks for the pull request!

Before we review code from a first-time contributor, a maintainer vouches for them, and you're not on our list yet. There's nothing else to file and this PR stays open — a maintainer will look at it, usually within a day or two, and vouch for you right here if it's a genuine attempt at the problem.

To make that quick, check the PR description has:

  • The issue it addresses (ideally one labelled Accepting PR) and why the change is needed
  • Steps to reproduce the problem and verify the fix, starting from https://play.hyperdx.io or yarn app:dev:local connected to the demo server
  • Before/after screenshots if anything in the UI changed

More detail in our contributing guide.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

4 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 4 minor

1 posted as inline comment(s) on the changed lines. 3 listed below.

Findings outside the changed lines

3 minor
  • 🔵 packages/cli/src/shared/useRowWhere.ts:108 — Moving the null check up leaves dead null guards in the Dynamic branch → value can no longer be null by line 108, because the new check at line 75 returns isNull(...) first, and strValue is now a plain string. So value == null || (line 108) and strValue?.length ?? 0 (line 111) can never take effect. Simplify them to strValue === 'null' and strValue.length > 1000. The same leftover value == null exists at packages/app/src/hooks/useRowWhere.tsx:99.
  • 🔵 packages/app/src/hooks/__tests__/useRowWhere.test.tsx:151 — The 'matches ClickHouse' block only tests crypto-js against constants → This block hashes testCase.input and compares it to expected. The processRowToWhereClause it.each below already checks those same hashes through the real code path. The clickHouse field is never read, and for the log-line case it is a placeholder (<json log line>). Drop this describe block. Alternatively, keep the ClickHouse cross-check as a real integration assertion that runs SELECT lower(hex(SHA256(...))) against a test ClickHouse.
  • 🔵 packages/common-utils/src/core/utils.ts:516 — New exported helper leftUtf8Prefix has no unit test in common-utils → Add direct tests in common-utils core/__tests__/utils.test.ts, since the code-point truncation is the part of this PR that differs from ClickHouse's leftUTF8. Cover limit 0, a string shorter than limit, and astral characters that cross the limit, for example 1200 '😀' with limit 1000, which should give 1000 code points (2000 UTF-16 units). Right now it is only tested indirectly through app/CLI cases, and none of them truncate a multi-unit string at the limit.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

Replaces MD5 with SHA256 in app and CLI row lookups for FIPS-enabled ClickHouse.

  • Web and CLI row lookups use SHA256 for JSON and long text.
  • CLI row lookups compare null values with isNull(...) across types.

No new actionable issues found. Tests were not run.

Reviews (22) · Last reviewed commit: "Merge branch 'main' into bugfix/replace-..." · Reviewed by Greptile

Comment thread packages/app/src/hooks/useRowWhere.tsx Resolved
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This PR is a focused, low-risk swap of MD5 → SHA256 in client-side row-lookup WHERE-clause generation for FIPS-enabled ClickHouse. Independent verification confirmed the two implementations stay in sync: the app (useRowWhere.tsx:66-69) already had the top-level isNull guard, and this PR adds the matching guard to the CLI (useRowWhere.ts:74-77), so the CLI's null-handling change is an alignment fix (and a genuine bug fix for null Date/JSON values that previously produced non-matching SQL), not a regression. The SQL surface was checked for injection: all dynamic values flow through SqlString.format ? placeholders, and the only raw interpolation (${LEFT_UTF8_HASH_CODE_POINTS}) is a trusted integer constant. All three prior-review items (missing changeset, Array.from over-reads, early-break truncation in both app and CLI) are resolved.

🟡 P2 -- recommended

  • packages/common-utils/src/core/utils.ts:54 -- the new exported leftUtf8Prefix helper has no direct unit test; its boundary behavior (limit 0, surrogate-pair/emoji at exactly the 1000 code-point cutoff where it must match ClickHouse leftUTF8) is only exercised transitively via a 600-code-point emoji fixture below the limit.
    • Fix: Add direct leftUtf8Prefix unit tests covering limit 0, a string whose code-point count is exactly LEFT_UTF8_HASH_CODE_POINTS, and a surrogate-pair boundary case.
    • kieran-typescript, performance, previous-comments
🔵 P3 nitpicks (3)
  • packages/cli/src/shared/useRowWhere.ts:108 -- after the new early if (value == null) return isNull(...), the value == null disjunct in the Dynamic branch and the (strValue?.length ?? 0) optional chain on line 111 are now unreachable/dead and misleading.
    • Fix: Simplify to if (strValue === 'null') and if (strValue.length > 1000 || column.length > 1000), mirroring the same cleanup in the app copy's Dynamic branch.
  • packages/common-utils/src/core/utils.ts:54 -- leftUtf8Prefix returns the entire string for a negative limit because n++ === limit can never fire; not reachable from the sole caller (1000) but the helper reads as general-purpose.
    • Fix: Add a if (limit <= 0) return ''; guard to make the non-negative contract explicit.
  • packages/cli/src/__tests__/rowWhere.test.ts:7 -- the Sha256TestCase/Sha256WhereKind types, SHA256_TEST_CASES fixtures, and sha256RowWhere helper are duplicated verbatim across the app and CLI test files and have already drifted (the app set includes a simple JSON object case the CLI set omits).
    • Fix: Co-locate the shared SHA256 ClickHouse-parity fixtures (ideally in common-utils, where the row-WHERE logic is slated to move) and import them from both suites.

Reviewers (7): security, performance, agent-native, previous-comments, project-standards, learnings-researcher, kieran-typescript.

Testing gaps:

  • ClickHouse parity is self-asserted in JS: the expected SHA256 digests are pinned hex constants compared against crypto-js output, with no live-ClickHouse check in CI to catch future drift in toString()/leftUTF8 semantics.
  • The production JSON path slash-escapes values (DBRowTable.tsx .replace(/\//g, '\\/')) before hashing, but the tests hash raw input, so JSON row lookups containing slashes are not covered end-to-end. The CLI (eventQuery.ts) does not apply this escaping — a pre-existing app/CLI divergence worth confirming against a real FIPS deployment, out of scope for this diff.
  • Coverage note: the correctness, testing, maintainability, and adversarial reviewers were dispatched but had not returned at synthesis time; the findings above reflect the seven reviewers that completed plus direct orchestrator verification of the null-handling and SQL-injection surfaces.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
Comment thread .changeset/fips-row-lookup-sha256.md Resolved
Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
@hercynium

Copy link
Copy Markdown
Contributor Author

TBH... I've kind of vibe-coded this and I don't like what I see since the initial commit. I'm going to go back and re-work some things manually. Difficulty is that I don't know TypeScript, but it doesn't seem all that different from other langs I do know.

@hercynium
hercynium force-pushed the bugfix/replace-md5-with-sha256-1 branch from 201f1e3 to 0eeb80a Compare October 19, 2026 20:53
Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
Comment thread packages/app/src/hooks/useRowWhere.tsx Resolved
Comment thread packages/app/src/hooks/useRowWhere.tsx Resolved
Comment thread packages/app/src/hooks/useRowWhere.tsx Resolved
Comment thread packages/cli/src/shared/useRowWhere.ts Resolved
Comment thread .changeset/fips-row-lookup-sha256.md Resolved
let prefix = '';
let n = 0;
for (const ch of value) {
if (n++ === limit) break;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New shared helper leftUtf8Prefix has no unit test of its own in common-utils

leftUtf8Prefix is exported for any caller but is only tested indirectly through the app and CLI row-WHERE hash snapshots. Add direct tests in common-utils __tests__ for: limit larger than the string, exact limit, surrogate pairs (e.g. '😀'.repeat(3) with limit 2 gives 2 emoji, not 1), and limit 0.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add these tests if deemed necessary, but I feel this PR is already touching more than initially planned, all to satisfy the bot.

whereKind: Sha256WhereKind,
hash: string,
): string {
if (whereKind === 'json') {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — JSON test cases say they match ClickHouse, but they hash raw literals instead of what toString(<JSON column>) actually returns

The expected hashes come from SHA256('<literal>'), so the tests only show that crypto-js and ClickHouse hash the same bytes the same way. The real risk is a serialization mismatch. toString() on a JSON column emits compact JSON. Per the comment at packages/app/src/components/DBRowTable.tsx:1848, it also escapes / as \/. So {"key": "value"} (with a space) and the 'actual log line' (with an unescaped github.com/...) would not match a stored row. Fix: build the input the way DBRowTable does (JSON.stringify(obj).replace(/\//g, '\\/')), and take the expected hash from lower(hex(SHA256(toString(Body)))) on a row inserted into a JSON column. Otherwise, drop the 'matches ClickHouse' claim and the clickHouse field.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure the tests are now more robust than they were before. This minor finding is fixable, but seems like it should go into another PR.

@hercynium hercynium changed the title bugfix: replace MD5 with SHA256 for FIPS compatibility fix: replace MD5 with SHA256 for FIPS compatibility Oct 20, 2026
hercynium and others added 3 commits October 23, 2026 10:02
This patch is to resolve Issue ClickHouse#3390 - when running HyperDX against ClickHouse in FIPS mode
the MD5 function isn't available, so viewing some logs fails, because MD5 is not FIPS-compliant.

This patch was generated using Cursor, running Composer 2.5 Fast. All relevant tests still pass.

Co-authored-by: Cursor <cursoragent@cursor.com>
hercynium and others added 8 commits October 23, 2026 10:02
Hoist the ClickHouse leftUTF8-matching helper so app and CLI row lookups
stay in sync and cannot drift silently.

Co-authored-by: Cursor <cursoragent@cursor.com>
Match the app row-WHERE builder so null JSON columns are not hashed as
an empty string.

Co-authored-by: Cursor <cursoragent@cursor.com>
Cover the uniform null guard for a non-JSON column type alongside JSON.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use SHA256_TEST_CASES with it.each in app and CLI suites so ClickHouse
golden hashes stay aligned and duplicated constants are removed.

Co-authored-by: Cursor <cursoragent@cursor.com>
@hercynium
hercynium force-pushed the bugfix/replace-md5-with-sha256-1 branch from dfb4b59 to dd18229 Compare October 23, 2026 14:02
@MikeShi42

Copy link
Copy Markdown
Contributor

/vouch @hercynium FIPS fix for #3390

@github-actions github-actions Bot mentioned this pull request Oct 23, 2026
@MikeShi42
MikeShi42 self-requested a review October 23, 2026 20:47
kodiakhq Bot pushed a commit that referenced this pull request Oct 23, 2026
@hercynium

Copy link
Copy Markdown
Contributor Author

Hello! I'm hoping to get eyes on this PR, and the linked issue. I've been successfully running a patched version of HyperDX against a FIPS build of ClickHouse for the last two weeks, and I'd like to work my PR towards getting merged before something new conflicts requiring re-work of the fix. ☺

I'm open to all feedback. Thanks in advance!

@vercel

vercel Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 3, 2026 4:15am UTC
hyperdx-storybook Ready Ready Preview Nov 3, 2026 4:15am UTC

Request Review

@wrn14897

wrn14897 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Hello! I'm hoping to get eyes on this PR, and the linked issue. I've been successfully running a patched version of HyperDX against a FIPS build of ClickHouse for the last two weeks, and I'd like to work my PR towards getting merged before something new conflicts requiring re-work of the fix. ☺

I'm open to all feedback. Thanks in advance!

@hercynium PR looks good. Could you fix the lint errors then we can merge the PR? Thanks

@hercynium
hercynium requested a review from a team as a code owner November 2, 2026 14:15
@hercynium

Copy link
Copy Markdown
Contributor Author

Thanks, @wrn14897! I fixed the linter issues in the cli but I'm not sure what to do about the errors in the app since none of those are for files this PR changes... Advice?

@hercynium

Copy link
Copy Markdown
Contributor Author

Thanks, @wrn14897! I fixed the linter issues in the cli but I'm not sure what to do about the errors in the app since none of those are for files this PR changes... Advice?

nvm... I found one running the linter locally. Fixed. ☺

},
'600 emoji code points': {
clickHouse: "lower(hex(SHA256(leftUTF8(repeat('😀', 600), 1000))))",
columnMeta: [{ name: 'description', type: 'String' }],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The SHA256 test fixtures are copied word-for-word between the app and CLI tests

Sha256WhereKind, Sha256TestCase, SHA256_TEST_CASES and sha256RowWhere are restated in packages/cli/src/__tests__/rowWhere.test.ts:7-63, so the verified hashes now have to be kept in sync by hand. Move the fixture table into a shared module next to leftUtf8Prefix in common-utils and import it from both tests. Also build the expected SQL with LEFT_UTF8_HASH_CODE_POINTS instead of the hardcoded 1000.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned before, this change already touches more files than I'd like so addressing this can be a different PR.

@wrn14897
wrn14897 merged commit e2754d2 into ClickHouse:main Nov 3, 2026
29 checks passed

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 96d1e0ec Deployed Nov 3, 2026 by vercel[bot]
Preview – hyperdx-storybook — 96d1e0ec Deployed Nov 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external needs-vouch Author needs a maintainer to vouch for them

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HyperDX compatibility with FIPS ClickHouse

3 participants