Describe the bug
We are seeing data corruption when retrying the same logical INSERT batch after an ambiguous failure such as read timeout, if the same batch / column objects are reused in ch-go.
The issue affects LowCardinality(String) columns and does not look like row loss:
- row count stays correct
- row keys stay correct
uniq(trace_id, span_id) stays correct
- corruption is visible only in LC string metadata columns
- first insert attempt is clean, retry copies are corrupted
Observed pattern:
- expected
lg-000257 becomes lg-000001
- expected
lg-000258 becomes lg-000002
This points to a stale dictionary / key remapping problem around the 256 boundary.
At unit level this appears to come from proto.ColLowCardinality.Prepare() keeping internal dictionary state across repeated prepares / reuse.
Steps to reproduce
- Create a
LowCardinality(String) column.
- Fill it with 256+ unique values and call
Prepare().
- Reuse the same column object for another logical batch, or call
Prepare() again on the same object.
- Encode and decode the column.
- Observe mixed / shifted values.
Expected behaviour
Repeated Prepare()+Encode on the same LowCardinality column object should be safe.
Reusing a batch object for retry should not preserve stale LC dictionary state from the previous logical batch.
Code example
in PR
Error log
Current failures:
- repeated Prepare() on 257 unique values decodes the last row as lg-000001 instead of lg-000257
- reuse with first batch lg-000001..lg-000256 and second batch lg-000257..lg-000512 decodes second batch as lg-000001..lg-000256
- key width selection around 255/256/257 is also suspicious
Configuration
Environment
- Client version: ch-go v0.71.0
- Language version: Go
- OS: ubuntu-noble
ClickHouse server
- ClickHouse Server version: 25.3, 25.8
- CREATE TABLE statements for tables involved:
CREATE TABLE traces
(
trace_id UUID,
span_id UInt64,
tenant LowCardinality(String),
service LowCardinality(String),
deployment LowCardinality(String),
operation LowCardinality(String)
)
ENGINE = MergeTree
ORDER BY (trace_id, span_id)
- Sample data for all these tables:
Minimal synthetic pattern:
- first batch: lg-000001..lg-000256
- second batch: lg-000257..lg-000512
Describe the bug
We are seeing data corruption when retrying the same logical
INSERTbatch after an ambiguous failure such asread timeout, if the same batch / column objects are reused inch-go.The issue affects
LowCardinality(String)columns and does not look like row loss:uniq(trace_id, span_id)stays correctObserved pattern:
lg-000257becomeslg-000001lg-000258becomeslg-000002This points to a stale dictionary / key remapping problem around the
256boundary.At unit level this appears to come from
proto.ColLowCardinality.Prepare()keeping internal dictionary state across repeated prepares / reuse.Steps to reproduce
LowCardinality(String)column.Prepare().Prepare()again on the same object.Expected behaviour
Repeated
Prepare()+Encodeon the sameLowCardinalitycolumn object should be safe.Reusing a batch object for retry should not preserve stale LC dictionary state from the previous logical batch.
Code example
in PR
Error log
Current failures:
Configuration
Environment
ClickHouse server
Minimal synthetic pattern: