Skip to content

LowCardinality INSERT corruption on retry/reuse of the same batch object after ambiguous failure #1155

Description

@OrangeFlag

Describe the bug

We are seeing data corruption when retrying the same logical INSERT batch after an ambiguous failure such as read timeout, if the same batch / column objects are reused in ch-go.

The issue affects LowCardinality(String) columns and does not look like row loss:

  • row count stays correct
  • row keys stay correct
  • uniq(trace_id, span_id) stays correct
  • corruption is visible only in LC string metadata columns
  • first insert attempt is clean, retry copies are corrupted

Observed pattern:

  • expected lg-000257 becomes lg-000001
  • expected lg-000258 becomes lg-000002

This points to a stale dictionary / key remapping problem around the 256 boundary.

At unit level this appears to come from proto.ColLowCardinality.Prepare() keeping internal dictionary state across repeated prepares / reuse.

Steps to reproduce

  1. Create a LowCardinality(String) column.
  2. Fill it with 256+ unique values and call Prepare().
  3. Reuse the same column object for another logical batch, or call Prepare() again on the same object.
  4. Encode and decode the column.
  5. Observe mixed / shifted values.

Expected behaviour

Repeated Prepare()+Encode on the same LowCardinality column object should be safe.

Reusing a batch object for retry should not preserve stale LC dictionary state from the previous logical batch.

Code example

in PR

Error log

Current failures:

  • repeated Prepare() on 257 unique values decodes the last row as lg-000001 instead of lg-000257
  • reuse with first batch lg-000001..lg-000256 and second batch lg-000257..lg-000512 decodes second batch as lg-000001..lg-000256
  • key width selection around 255/256/257 is also suspicious

Configuration

Environment

  • Client version: ch-go v0.71.0
  • Language version: Go
  • OS: ubuntu-noble

ClickHouse server

  • ClickHouse Server version: 25.3, 25.8
  • CREATE TABLE statements for tables involved:
  CREATE TABLE traces
  (
      trace_id UUID,
      span_id UInt64,
      tenant LowCardinality(String),
      service LowCardinality(String),
      deployment LowCardinality(String),
      operation LowCardinality(String)
  )
  ENGINE = MergeTree
  ORDER BY (trace_id, span_id)
  • Sample data for all these tables:

Minimal synthetic pattern:

  • first batch: lg-000001..lg-000256
  • second batch: lg-000257..lg-000512

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions