Skip to content

Per-table dedupe id_field (+ fix cross-table dedupe keyspace collision) #222

Description

@EricAndrechek

Area: dedupe — gap + latent correctness bug · found via WaveHouse-Stats dogfooding

Expected: each table dedupes on its own natural key, in its own keyspace.

Actual: two problems with the single global dedupe key:

  1. One global id_field name applies deployment-wide (internal/config/config.go:105-108) — every table that wants dedupe must carry an identically-named column.
  2. Shared keyspace (latent bug): CheckAndMark keys Pebble on the raw id value with no table namespace (internal/dedupe/embedded.go:29, key := []byte(eventID)). Equal id values across two different tables collide — the second is silently dropped as a "duplicate." Only bites when ids aren't globally unique across tables (e.g. per-table sequential ids), but it's silent data loss when it does.

Impact (Stats): we standardize on globally-unique event_id, so the collision doesn't bite us today — but it's a sharp edge for any multi-table deployment, and the single id_field forces a naming convention on every future table.

Scope: namespace the dedupe key by table (fixes #2 regardless), and allow a per-table id_field (fixes #1). The keyspace fix is the more urgent half.

Related: #219, #220, #221.


From WAVEHOUSE-FEEDBACK.md (WaveHouse-Stats dogfooding); validated by code-read against 0f8826c on 2026-06-04.

Metadata

Metadata

Assignees

Labels

area/dedupeDeduplication (Pebble, ScyllaDB)area/ingestIngest pipeline (Bento, batching, DLQ)breaking-changeBreaking change to public API, CLI, or configbugSomething isn't workingenhancementNew feature or request

Type

No type

Projects

Status
In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions