daft-clickhouse

daft-clickhouse is an independent Daft community connector for reading ClickHouse physical tables and appending Arrow batches to existing ClickHouse MergeTree-family tables.

pip install "daft-clickhouse[clickhouse]"

import daft
from daft_clickhouse import read_clickhouse, write_clickhouse

events = read_clickhouse(
    host="localhost",
    database="analytics",
    table="events",
    username="connector",
    password="...",
)

parallel_events = read_clickhouse(
    host="localhost",
    database="analytics",
    table="events",
    partition_column="id",
    split="range",
    topology="direct-local",
    target_tasks=8,
)

write_clickhouse(
    daft.from_pydict({"id": [1], "value": ["new"]}),
    host="localhost",
    database="analytics",
    table="write_events",
    username="connector",
    password="...",
).collect()

Writing is deliberately append-only and currently requires Daft NativeRunner. The target table must already exist, input columns are validated against its discovered schema, and each confirmed batch is reported as source input rows and bytes. The connector does not provide upsert, overwrite, DDL, connector-owned retries, global transactions, or exactly-once guarantees. retry_policy="none" rejects any future connector retry policy rather than silently changing delivery semantics. It does not monkey-patch Daft's existing DataFrame.write_clickhouse method.

The default write path is PyArrow Table -> clickhouse-connect.insert_arrow(). insert_mode="async" is explicit and still waits for the server-side asynchronous insert flush. The accepted target engines are MergeTree, ReplacingMergeTree, SummingMergeTree, AggregatingMergeTree, CollapsingMergeTree, and ReplicatedMergeTree. Other engines, including Distributed targets, are rejected until their complete write contract is certified.

Range splitting is explicit and is certified only for a direct endpoint to one non-replicated local MergeTree-family table. Its first type profile supports Int8/16/32/64, UInt8/16/32/64, Date, Date32, and their nullable forms. The connector issues one filtered MIN/MAX/NULL/count planning aggregate, creates disjoint outer-unbounded ranges, and adds a separate NULL task when needed. target_task_bytes can select the desired task count from estimated decoded Arrow bytes; it is not an on-disk, wire, RSS, or backpressure bound. Multi-task reads are independent queries rather than one shared snapshot.

Daft 0.7.23 does not propagate synchronous consumer early-close into a Python async DataSource task. Read consumers should fully drain results; cancellation and strict end-to-end backpressure remain upstream compatibility limitations. split="auto" is for a direct, single-server endpoint only. split="range" additionally requires topology="direct-local"; neither strategy is certified behind load balancing or ClickHouse Cloud routing.

See writing, consistency, and security for the public contract.