Migres Go

Keep ClickHouse in sync with MySQL using an online snapshot and continuous binlog replication.

Migres Go copies existing rows, streams committed changes, and resumes from durable state after a restart. It is intended for analytical replicas, reporting databases, and moving MySQL workloads into ClickHouse.

MySQL -> Producer -> durable events -> Transformer -> durable tasks -> Consumer -> ClickHouse

Project status: early implementation with integration and fault recovery coverage. A continuous 24-hour synthetic fault run passed on runtime revision 90fca7f. Production storage fencing and validation against your workload remain release gates. See verification and the roadmap.

Features

• Snapshot only, CDC only, or an online snapshot followed by continuous replication.
• Durable bbolt queues, checkpoints, schema history, and resumable operations.
• Full primary keys, including composite keys, or configured full UNIQUE NOT NULL keys.
• Inserts, updates, deletes, and key changes using row versions and deletion markers.
• Table filters, type conversion, DEFAULT handling, timezones, source character sets, and LowCardinality strings.
• Historical DDL handling and generation rebuilds for incompatible type or key changes.
• ReplacingMergeTree and ReplicatedReplacingMergeTree, with configurable quorum acknowledgement.
• File/position or GTID repositioning, reset, and table resnapshot through a local control socket.
• Graceful shutdown, persistent Docker storage, Kubernetes manifests, and Prometheus metrics.
• Optional debug archives and Teams, Slack, and Telegram notifications.

Try it locally

The demo starts isolated MySQL and ClickHouse services, creates sample orders, and runs Migres. It uses publicly known demo credentials. Only the Migres HTTP endpoint is published, on 127.0.0.1:18080.

git clone https://github.com/arm092/migres-go.git
cd migres-go
docker compose -f examples/compose.yml up -d --build --wait
curl --fail http://127.0.0.1:18080/readyz

Read current rows:

docker compose -f examples/compose.yml exec clickhouse \
  clickhouse-client --user migres --password demo-only-clickhouse \
  --database analytics \
  --query 'SELECT id, customer_id, status, amount FROM orders FINAL WHERE _migres_deleted = 0 ORDER BY id'

Change a source row, then repeat the read:

docker compose -f examples/compose.yml exec mysql \
  mysql -uroot -pdemo-only-root shop \
  -e "UPDATE orders SET status = 'paid' WHERE id = 1;"

See the complete demo for deletes, key changes, status, and cleanup. CDC is asynchronous; repeat the query after the queues catch up.

Use your own databases

Requires Go 1.26.9 to build. Runtime deployment targets Linux, Docker, and Kubernetes. Windows is supported for development; exercise Unix socket control and Linux signals on Linux.

go build -o bin/migres ./cmd/migres
cp config.yml.example config.yml
# Edit hosts, database names, filters, and a unique CDC server ID.
export MYSQL_PASSWORD='your-source-password'
export CLICKHOUSE_PASSWORD='your-target-password'
./bin/migres check --config config.yml
./bin/migres run --config config.yml

Create a dedicated ClickHouse target database with ENGINE = Atomic first. The source needs InnoDB tables, ROW/FULL binlog images, and a stable unique row identity. Online copying requires GTID or a writable prepared signal table. Setup and permissions explain the prerequisites.

Read current state correctly

SELECT id, customer_id, status, amount
FROM analytics.orders FINAL
WHERE _migres_deleted = 0;

Raw tables retain row versions and deletion markers. Use FINAL and the deletion filter for current-state reads. Delivery can be retried; Migres does not promise a globally atomic snapshot, cross-table transaction visibility, or simultaneous publication on every replica. Data guarantees describe the contract.

Documentation

Guide	Contents
Getting started	Prerequisites, permissions, build, and first run
Configuration	YAML, environment variables, filters, tuning, TLS, notifications
Architecture and guarantees	Stages, durable handoffs, snapshots, DDL, replication
Operations and recovery	Status, reposition, reset, resnapshot, signals, troubleshooting
Docker and Kubernetes	Persistent volumes, manifests, probes, upgrades, metrics
Verification	Unit, integration, fault, soak, and performance checks
Runnable examples	Local demo and replicated target configuration
Roadmap	Release gates and planned improvements

Contributing

Bug reports should include database versions, redacted configuration, triggering DDL, operation status, and a minimal reproduction. Do not include passwords, row payloads, private keys, or production debug archives. See CONTRIBUTING.md.

Support development

If Migres helps your work, sponsor Arman on GitHub. Sponsorship supports maintenance, recovery testing, and compatibility work.

License

MIT.