Skip to content

Preserve duplicate Cassandra list values#3

Closed
dipeshbabu wants to merge 1 commit into
masterfrom
agent/preserve-cassandra-duplicates
Closed

Preserve duplicate Cassandra list values#3
dipeshbabu wants to merge 1 commit into
masterfrom
agent/preserve-cassandra-duplicates

Conversation

@dipeshbabu

Copy link
Copy Markdown
Owner

What

Include the monotonic timestamp in the Cassandra clustering key, keep set inserts idempotent with a fixed timestamp, validate existing table schemas, and add duplicate/removal integration regressions.

Why

The previous primary key used only key and value, so repeated ordered values overwrote each other. An empty MinHash repeats the same band hash, which meant its key metadata lost bands and removal left stale bucket entries.

Impact

Cassandra ordered storage now preserves every inserted value while unordered bucket storage remains deduplicated. Tables created with the old primary key fail with a clear rebuild instruction instead of silently retaining unsafe semantics.

Root cause

The ordering timestamp was a regular column rather than part of row identity.

Checks

  • uv run pytest -q test/test_lsh.py (24 passed)
  • uv run pytest -q test/test_integration.py (36 skipped because backend services are unavailable locally; Cassandra CI runs them)
  • uv run ruff check datasketch/storage.py test/test_integration.py
  • uvx pyright (0 errors)
  • git diff --check

@dipeshbabu
dipeshbabu marked this pull request as ready for review July 19, 2026 03:50
@dipeshbabu

Copy link
Copy Markdown
Owner Author

Superseded by upstream pull request ekzhu#327.

@dipeshbabu dipeshbabu closed this Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant