unstable storage test ddl_for_split_tables_with_random_move_table

[wk989898]

What did you do?

https://prow.tidb.net/jenkins/blue/organizations/jenkins/pingcap%2Fticdc%2Fpull_cdc_storage_integration_heavy/detail/pull_cdc_storage_integration_heavy/485/pipeline

What did you expect to see?

No response

What did you see instead?

test pass

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

[wk989898]

Analysis

1. At 2026/03/02 00:49:29 +08:00, ALTER TABLE test.table_5 ADD COLUMN new_col
   INT was applied successfully.
2. At 2026/03/02 00:49:33.648 +08:00, ALTER TABLE test.table_5 DROP COLUMN
   new_col was applied successfully, with commitTs=464619741438017589.
3. Right after that, DML for test.table_5 was still executed as:
   • REPLACE INTO test.table_5 (data, id, new_col) VALUES (...)
   • using the same boundary timestamp (startTs=464619741438017589).
4. MySQL returned Error 1054 (42S22): Unknown column 'new_col' in 'field
   list'.
5. After retries were exhausted (ErrReachMaxTry), the storage consumer exited
   around 00:50:01.
6. Later, at 01:04:01, data check failed and all 5 tables were inconsistent.

DDL event

{
    "Table": "t",
    "Schema": "partition_table",
    "Version": 1,
    "TableVersion": 464641590967926810,
    "Query": "CREATE TABLE `t` (`a` INT,PRIMARY KEY(`a`)) PARTITION BY HASH (`a`) PARTITIONS 5",
    "Type": 3,
    "TableColumns": [
        {
            "ColumnName": "a",
            "ColumnType": "INT",
            "ColumnPrecision": "11",
            "ColumnNullable": "false",
            "ColumnIsPk": "true"
        }
    ],
    "TableColumnsTotal": 1
}

DML event

{"id":0,"database":"test","table":"table_5","pkNames":["id"],"isDdl":false,"type":"INSERT","es":1772383657219,"ts":1772383757482,"sql":"","sqlType":{"id":4,"data":12,"new_col":4},"mysqlType":{"id":"int","data":"varchar","new_col":"int"},"old":null,"data":[{"id":"2376","data":"insert_1772383657_17790_2445","new_col":null}],"_tidb":{"commitTs":464619741438017589}}

For that specific event (id=2376, commitTs=464619741438017589), the key point
is:

Timeline from the logs on 2026-03-02:

1. 00:49:33.558: the DML event is parsed, grouped, and SQL is built as
   REPLACE INTO ... (data,id,new_col) ... for id=2376.
2. 00:49:33.561: that batch is flushed once.
3. 00:49:33.648: DDL succeeds: ALTER TABLE test.table_5 DROP COLUMN new_col
   (same commitTs boundary).
4. 00:49:50.460: the same DML (commitTs=464619741438017589) is appended again
   via fallback (DML events fallback... still append it, with a much higher
   watermark).
5. Replay now runs against post-DDL schema (without new_col), so MySQL returns
   Error 1054: Unknown column 'new_col'.

Root cause

CDC consumer ordering/replay issue at DDL-DML boundary (same
commitTs) + fallback re-append behavior, not that your original INSERT SQL was
wrong.