Skip to content

Preserve duplicate Cassandra list values#327

Open
dipeshbabu wants to merge 3 commits into
ekzhu:masterfrom
dipeshbabu:agent/preserve-cassandra-duplicates
Open

Preserve duplicate Cassandra list values#327
dipeshbabu wants to merge 3 commits into
ekzhu:masterfrom
dipeshbabu:agent/preserve-cassandra-duplicates

Conversation

@dipeshbabu

Copy link
Copy Markdown
Contributor

What

Include the monotonic timestamp in the Cassandra clustering key, keep set inserts idempotent with a fixed timestamp, validate existing table schemas, and add duplicate/removal integration regressions.

Why

The previous primary key used only key and value, so repeated ordered values overwrote each other. An empty MinHash repeats the same band hash, which meant its key metadata lost bands and removal left stale bucket entries.

Impact

Cassandra ordered storage now preserves every inserted value while unordered bucket storage remains deduplicated. Tables created with the old primary key fail with a clear rebuild instruction instead of silently retaining unsafe semantics.

Root cause

The ordering timestamp was a regular column rather than part of row identity.

Checks

  • uv run pytest -q test/test_lsh.py (24 passed)
  • uv run pytest -q test/test_integration.py (36 skipped because backend services are unavailable locally; Cassandra CI runs them)
  • uv run ruff check datasketch/storage.py test/test_integration.py
  • uvx pyright (0 errors)
  • git diff --check

@codecov-commenter

codecov-commenter commented Jul 19, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (master@c4df648). Learn more about missing BASE report.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #327   +/-   ##
=========================================
  Coverage          ?   70.74%           
=========================================
  Files             ?       21           
  Lines             ?     2796           
  Branches          ?        0           
=========================================
  Hits              ?     1978           
  Misses            ?      818           
  Partials          ?        0           
Flag Coverage Δ
unittests 70.74% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dipeshbabu
dipeshbabu force-pushed the agent/preserve-cassandra-duplicates branch from 13be921 to 65f4769 Compare July 22, 2026 05:39
@dipeshbabu
dipeshbabu force-pushed the agent/preserve-cassandra-duplicates branch from 65f4769 to df2acb6 Compare July 22, 2026 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants