Skip to content

feat(data-compare): compare 2,000 rows of both sides in key order, paged and filtered - #559

Merged
huyplb merged 2 commits into
mainfrom
feat/data-compare-2000
Oct 26, 2026
Merged

huyplb merged 2 commits into
mainfrom
feat/data-compare-2000

Conversation

@huyplb

@huyplb huyplb commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Phase A of docs/plans/2026-10-17-data-compare-and-sql-tracer.md (Part 1).

Problem

Compare data only saw the page each grid had loaded. Each side pages on its own with OFFSET, so the pages never line up. Every one-sided row on a cut-short side came out unresolved, and paging through a compare wasn't meaningful.

Change

  • Compare 2,000 rows (in the compare view) reads the statement again on both sides, ordered by the key columns. It fetches 2,000 + 1 rows each, the extra row showing whether more follow.
    • /sql/execute takes orderBy (plain column names only, at most 8) and puts the order on the outer query, in each engine's paging syntax.
    • On SQL Server, a plain top-level ORDER BY inside the derived table is dropped, because it's refused there.
    • No new route.
  • Every row the window can decide is classified. A one-sided row is uncertain only when the other side was cut short. With numeric keys (including bigints sent as text) it's still classified when it sorts at or before that side's last key. Text keys stay unresolved past a cut, since engines order text by their own collation.
  • Filter chips with counts for the whole window: All, Changed, Only in source, Only in target, Unresolved, Matching.
  • Paging in the browser: pages of 50 to 500, and Next / Previous difference jumps across pages of matching rows. Sync checkboxes keep their state, because they're keyed by row label.
  • Data migrate reads the compared window. It still writes only when both sides are complete, under the 500-op cap. Raising the cap to 2,000 is Phase B.
  • Redis and MongoDB don't offer the action. Compare needs an ordered tabular result.

Tests

  • sql-page-wrap.service.test.ts: the ordered page on Postgres, MySQL, Oracle, Db2 and SQL Server, including the ORDER BY strip; orderBy validation refuses anything but plain names.
  • sql-execute.service.test.ts: live SQLite. A page comes back in key order with hasNext. It fails without the change, because the statement's own ORDER BY name DESC would win.
  • resultKeyAlign.test.ts: the window boundary, numeric including bigint text, and text staying unresolved. The boundary cases fail on the old code.
  • compareView.test.ts: filter counts, paging, next/previous difference, the slice, and the last-row key.
  • e2e sql-editor-compare-window.test.ts, 4/4: two SQLite files of 2,500 rows with known differences. Exact counts (Changed 3, Only in source 2, Unresolved 2, Matching 1,995, All 2,002), the changed filter, 21 pages of 100, and difference jumps to pages 15 and 20.
  • Full gates: typecheck, eslint . and the security lint are clean. Unit suite: 6559 pass.

Next (Phase B)

Raise migrate's cap to 2,000 through one shared constant, and take the restore snapshot in chunks of 500 keys (Oracle's IN-list limit).

🤖 Generated with Claude Code


Note

Medium Risk
Touches SQL execution paging (orderBy wrapping) across dialects and compare/migrate data paths; mitigated by strict column-name validation and broad tests, but wrong paging SQL would affect any ordered execute calls.

Overview
Adds Compare 2,000 rows so side-by-side compare no longer depends on mismatched grid pages: both connections re-run the statement with orderBy key columns (up to 2,000+1 rows each) via extended POST /sql/execute and dialect-specific wrapSqlForOrderedPage (validated plain column names; SQL Server strips inner ORDER BY when needed).

The compare toolbar gains filter chips with full-window counts (changed, source-only, target-only, unresolved, matching), in-browser paging (50–500 rows), and previous/next difference jumps. useCompareWindow holds fetched windows; compareView slices/filters aligned rows. Key alignment uses leftWindowEnd / rightWindowEnd so numeric keys classify one-sided rows before a truncated side’s last key; text keys stay unresolved past a cut. Data migrate reads the compared window when active. Redis/Mongo subset dialects skip the action.

Coverage: unit tests for ordered paging, alignment boundaries, and compare view; new e2e sql-editor-compare-window.test.ts with seeded SQLite fixtures.

Reviewed by Cursor Bugbot for commit e70bbfe. Bugbot is set up for automated code reviews on this repo. Configure here.

huyplb and others added 2 commits October 25, 2026 18:42
…ged and filtered

The grids page each side on its own with OFFSET, so their pages never line
up and a one-sided row on a cut-short side could only be 'unresolved'.
Compare 2,000 rows reads both sides again ordered by the key columns (the
order goes on the outer query: /sql/execute takes orderBy, plain column
names only), 2,001 rows each to learn whether more follow.

- A row found on one side only is classified when the other side holds
  everything, or (numeric keys) when its key sorts at or before that
  side's last key. Text keys stay unresolved past a cut: engines sort them
  by their own collation.
- Filter chips with counts for the whole window (Changed, Only in source,
  Only in target, Unresolved, Matching), browser-side pages of 50-500,
  and Next / Previous difference across pages. Sync boxes keep their state.
- Data migrate reads the compared window; it still writes only when both
  sides are complete (raising its cap is Phase B).

Phase A of docs/plans/2026-10-17-data-compare-and-sql-tracer.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 12815453-008a-472c-aece-938027be9e0e)

@github-actions

Copy link
Copy Markdown
Contributor

Test IDs

Added (12)

  • sql-result-compare-diff-next-{…}
  • sql-result-compare-diff-prev-{…}
  • sql-result-compare-filter-{…}-{…}
  • sql-result-compare-page-label-{…}
  • sql-result-compare-page-next-{…}
  • sql-result-compare-page-prev-{…}
  • sql-result-compare-page-size-{…}
  • sql-result-compare-view-{…}
  • sql-result-compare-window-clear-{…}
  • sql-result-compare-window-error-{…}
  • sql-result-compare-window-run-{…}
  • sql-result-compare-window-summary-{…}

{…} is a part filled in at run time. The full tree: docs/testing/TEST_IDS.md.

@huyplb
huyplb merged commit aa923db into main Oct 26, 2026
13 checks passed
@huyplb
huyplb deleted the feat/data-compare-2000 branch October 26, 2026 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant