Deterministic source-to-target data reconciliation. Point it at two databases, and it tells you exactly where they disagree — row counts, schemas, totals, and the individual values inside individual rows.
Built for the moment a migration goes live and someone asks "is the new warehouse actually right?" — and for every load after that, when the answer has to keep being yes.
FAILED 17/20 pairs passed, 8/193 checks failed, 41.20s
Failures (3)
* public.orders -> ANALYTICS.ORDERS
row_count: row count mismatch: 1,204 more rows in source (source 8,441,290, target 8,440,086)
aggregate.sum(AMOUNT): sum(AMOUNT) differs: source 41,882,104.55, target 41,879,921.30
* public.customers -> ANALYTICS.CUSTOMERS
qualitative.value_match: 312 row(s) differ in at least one value (99.62% match rate);
most affected: EMAIL (298), REGION (14)
Most "the data moved fine" checks compare row counts. Row counts miss almost everything that actually goes wrong: a rounded decimal, a timezone shift, a join that silently dropped matches, rows duplicated to cover for rows that were lost. reconflow does the cheap structural checks and the expensive row-level ones, and it tells you which is which.
Nothing here is inferred. Every comparison is deterministic and reproducible: the same config against the same data always produces the same verdict. No model, heuristic, or sample decides whether your data matches. reconflow ships an MCP server so an AI assistant can read and explain results — but a model is never in the data path. Remove it and the output is byte-identical.
pip install reconflowDatabase drivers are optional extras — install only what you connect to:
pip install 'reconflow[warehouses]' # SQL Server, Snowflake, Redshift,
# Synapse, PostgreSQL, Oracle
pip install 'reconflow[snowflake]' # or just the one you need
pip install 'reconflow[redshift,synapse]' # several at once
pip install 'reconflow[all]' # everything, including the MCP serverreconflow drivers lists every backend and shows which are installed here.
The repository ships a working example that needs no database at all — two directories of CSV extracts with four deliberate defects:
git clone https://github.com/sumit-gupta03/reconflow && cd reconflow
pip install -e .
reconflow run -c examples/quickstart/reconflow.ymlIt finds a lost row, an extra row, a changed region, and a rounded
lifetime_value, then writes an HTML report you can open in a browser.
If you would rather click than write YAML:
reconflow serve --openSet up the source and target connections, pick a database and schema from the menus, and reconflow pairs the tables by name — on the assumption that a table keeps its name through a migration. Anything present on only one side is listed rather than dropped. Edit any pair to reconcile tables whose names differ, then run it and read the results in the same page.
It binds to 127.0.0.1 and holds nothing: connection details are used for the
request that needs them and discarded, never written to disk and never kept
between requests. Binding elsewhere is possible (--host) but the UI has no
authentication of its own, so put your own in front of it first — the CLI
warns when you do.
When the two sides are not the same shape — five columns here, four there, or a
join needed to line them up — give each side a SELECT instead of a table:
-- source -- target
SELECT id, name, email, balance SELECT id, name, email, balance
FROM customers FROM dim_customerThe two queries need only match each other, not their tables. Every check
runs against them unchanged, so counts, aggregates, and row-level comparison
all behave exactly as they do for a table. In YAML that is source_query: and
target_query: on a pair.
reconflow only ever reads: a query that is not a single SELECT is refused
before it reaches the database.
reconflow init # writes a commented reconflow.yml
reconflow validate # checks the config and tests both connections
reconflow run # reconciles and writes reportsA minimal config:
version: 1
name: warehouse-migration
connections:
source:
driver: postgres
host: ${SOURCE_HOST}
database: app
user: ${SOURCE_USER}
password: ${SOURCE_PASSWORD}
schema: public
target:
driver: snowflake
account: ${SNOWFLAKE_ACCOUNT}
user: ${SNOWFLAKE_USER}
password: ${SNOWFLAKE_PASSWORD}
database: ANALYTICS
schema: PUBLIC
pairs:
- name: customers
source: {schema: public, table: customers}
target: {schema: PUBLIC, table: CUSTOMERS}
keys: [customer_id]
exclude_columns: [load_date, updated_at]
quantitative:
aggregates:
- {column: lifetime_value, functions: [sum, min, max]}
qualitative:
enabled: true
report:
formats: [html, json]
output_dir: ./reconflow-reports
notifications:
- type: slack
webhook_url: ${SLACK_WEBHOOK_URL}
on: [failure, error]Secrets never belong in the file. ${VAR} reads an environment variable and
fails loudly if it is unset; ${VAR:-fallback} supplies a default.
The databases compute these themselves; reconflow only compares the answers. Cheap enough to run across an entire schema on every load.
| Check | What it measures | What a failure means |
|---|---|---|
table_exists |
Presence on both sides | The table never arrived |
row_count |
COUNT(*) per side, after filters |
Rows lost, duplicated, or filtered differently |
column_count |
Column counts | Schemas have diverged |
schema_match |
Which columns exist, and their type families | A named column is missing or was retyped |
null_counts |
NULLs per column | A column is populated differently — the signature of a join that dropped matches |
distinct_counts |
COUNT(DISTINCT col) |
Cardinality changed; often rows duplicated to cover for rows lost |
aggregates |
SUM/MIN/MAX/AVG/COUNT |
The totals disagree — the one that matters most for financial data |
duplicate_keys |
Whether your key is actually unique | Row-level attribution cannot be trusted until fixed |
Off by default and bounded by row_limit, because it costs proportionally more.
With a key, reconflow says "row 4471, column amount: source 1200.00,
target 1200.0", and counts how many rows differ per column. This is the mode
you want.
Without a key, whole rows are matched as units. reconflow can still tell you a row is present on one side only, but it cannot attribute a difference to a column — it cannot distinguish a changed row from a delete plus an insert.
Only one side is held in memory at a time: the source is indexed, the target is streamed against it. Peak cost is one side's rows, not both.
The same data comes back typed differently from different databases.
Decimal("1.00") from PostgreSQL, 1.0 from Snowflake, " 1 " from a CSV
extract — none of these is a data difference, and a tool that reports them as
one is worse than no tool. reconflow normalises numbers across int/float/
Decimal, trims strings, compares type families (VARCHAR ≈ TEXT), and
matches column names case-insensitively. Every rule is explicit and configurable:
qualitative:
trim_strings: true # " x " == "x"
case_insensitive_values: false
null_equals_empty: false # NULL and "" are different findings
float_tolerance: 1.0e-9
decimal_places: 2 # round before comparingAbsolute and percentage allowances are both checked, and either is sufficient. Zero — the default — demands exact equality, which is what a cutover should require. Non-zero belongs in ongoing monitoring where a small, understood drift is expected.
tolerance:
row_count_abs: 0
row_count_pct: 0.0
aggregate_abs: 0.01
min_match_rate: 99.9 # qualitative: % of rows that must match exactly
type_mismatch_is_warning: trueListing two hundred tables by hand is how tables get missed:
discovery:
- source_schema: public
target_schema: PUBLIC
include: ["*"]
exclude: ["*_backup", "tmp_*", "_*"]
target_case: upper # PostgreSQL -> Snowflake naming
target_table_template: "STG_{table}"A source table with no counterpart in the target is reported as a finding, not skipped silently.
report:
formats: [html, json, markdown, csv, junit]
output_dir: ./reconflow-reports
keep_runs: 50
show_values: false # counts still travel when the data must not- html — one self-contained file. No CDN, no external fonts, no network. It renders from an email attachment in five years' time. Opens with failures already expanded, light and dark themes, filter and search.
- json — the machine-readable form. The HTML is a rendering of exactly this, so the two can never disagree.
- markdown — for PR comments and CI job summaries.
- csv — one row per check, for trending results in a warehouse.
- junit — makes a failing reconciliation appear in your CI test tab.
Each run gets a timestamped directory; latest.html points at the newest and
index.html lists the history.
notifications:
- type: slack
webhook_url: ${SLACK_WEBHOOK_URL}
on: [failure, error]
mention: "<!here>"
- type: email
host: ${SMTP_HOST}
sender: reconflow@example.com
recipients: [data-team@example.com]
on: [failure]
attach_report: true
- type: webhook # PagerDuty, Opsgenie, anything
url: ${ALERT_URL}
headers: {Authorization: "Bearer ${ALERT_TOKEN}"}
on: [always]Also teams, console, and file. Triggers are always, failure, error,
warning, success; the default is [failure, error], so a clean run stays
quiet — a tool that pings on every success trains people to ignore it.
Delivery uses the standard library only, and a failed notification never fails the run. A reconciliation that ran correctly but could not reach Slack still told you the truth about your data.
- name: Reconcile
run: reconflow run -c reconflow.yml -f junit -f markdown
env:
SOURCE_PASSWORD: ${{ secrets.SOURCE_PASSWORD }}
SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}Exit codes are the contract:
| Code | Meaning |
|---|---|
0 |
Passed, or passed with warnings |
1 |
At least one check failed or errored — the data is wrong |
2 |
reconflow could not run — bad config, unreachable database |
That distinction matters: a pipeline should treat "the data is wrong" very
differently from "the tool could not look at the data". Use
--fail-on warning to make warnings block too.
No config file needed:
reconflow compare \
--source-url postgresql://user@host/app \
--target-url snowflake://user@account/ANALYTICS \
--source-table customers \
--key customer_id \
--exclude load_date \
--rowsAdd --json to pipe it straight into jq.
reconflow speaks the Model Context Protocol, so an assistant can run a reconciliation and — more usefully — explain the result to whoever has to act on it.
pip install 'reconflow[mcp]'
reconflow mcp -c reconflow.ymlRegister it with an MCP client:
{
"mcpServers": {
"reconflow": {
"command": "reconflow-mcp",
"args": ["-c", "/path/to/reconflow.yml"]
}
}
}Tools exposed: list_drivers, describe_checks, validate_config,
test_connections, run_reconciliation, compare_tables, list_runs,
get_report, explain_pair.
The division of labour is deliberate. reconflow measures; the assistant
explains. Every number a model sees came from the same deterministic engine
the CLI runs. describe_checks returns written reference material so an
explanation matches what a check actually does rather than guessing from its
name. run_reconciliation does not send notifications unless explicitly asked —
exploring a result should not page the on-call team.
from reconflow import load_config, run
from reconflow.report import write_reports
from reconflow.notify import dispatch
config = load_config("reconflow.yml")
report = run(config, only=["customer*"])
print(report.summary.status) # Status.FAIL
for pair in report.failed_pairs:
for check in pair.failed_checks:
print(pair.name, check.name, check.message)
paths = write_reports(report, config.report, config.resolved_output_dir())
dispatch(report, config.notifications, report_path=paths.get("html"))Each connection has its own driver, so any of these can sit on either side.
Primary targets — pip install 'reconflow[warehouses]' installs all six:
| Driver | Notes |
|---|---|
mssql |
SQL Server and Azure SQL, via ODBC |
snowflake |
Role and warehouse selection |
redshift |
Provisioned or Serverless; falls back to the PostgreSQL protocol if the Redshift dialect is absent |
synapse |
Azure Synapse, dedicated or serverless SQL pool |
postgres |
PostgreSQL, CockroachDB, Supabase |
oracle |
via python-oracledb |
Redshift and Synapse are separate drivers rather than aliases for postgres
and mssql, because neither enforces a declared primary key. Treating those
declarations as real would silently mismatch rows, so reconflow requires keys to
be configured explicitly on both.
Also supported: mysql, bigquery, databricks, duckdb, sqlite,
csv (a directory of extracts, queried as SQL), and sql for any other
SQLAlchemy dialect via url:.
Adding a backend means implementing
reconflow.drivers.base.Driver and registering it,
either at import time or through a reconflow.drivers entry point.
Connections carry whatever names you give them; source and target are just
the defaults for tables that do not name one.
connections:
legacy_uk: {driver: mssql, host: ${UK_HOST}, database: OrdersUK, ...}
legacy_eu: {driver: oracle, host: ${EU_HOST}, database: ${EU_SVC}, ...}
warehouse: {driver: synapse, host: ${SYN_HOST}, database: ${POOL}, ...}
defaults:
source_connection: legacy_uk
target_connection: warehouseDifferent tables can come from different systems:
pairs:
- name: uk_customers
source: {connection: legacy_uk, table: customers}
target: {connection: warehouse, table: CUSTOMERS_UK}And when two systems were consolidated into one warehouse table, several sources can be compared against a single target — reconflow adds them together and asks whether the combined source equals the target:
- name: orders
sources:
- {connection: legacy_uk, schema: dbo, table: orders}
- {connection: legacy_eu, schema: ORDERS, table: ORDERS}
target: {connection: warehouse, table: ORDERS}
keys: [order_id]Sources are treated as disjoint, which is what a consolidation means. If a row somehow exists in both, the row-level comparison reports it as a duplicate key rather than quietly absorbing it.
What can and cannot be combined. Row counts, SUM, MIN, MAX, COUNT,
and NULL counts add up across sources and are compared normally. AVG and
COUNT(DISTINCT) do not: an average of averages is wrong unless every source
has the same row count, and the same value may appear in two databases with no
way to deduplicate across two engines. reconflow reports those as skipped,
with the reason, rather than returning a number that looks right and is not.
To get them, compare each source against its slice of the target as a separate
pair — see examples/consolidation.yml.
Existing single-source configs keep working unchanged.
reconflow is read-only against your databases by construction: nothing in the
driver interface writes, and no implementation issues DDL or DML. Table names
and hand-written filter: predicates cannot be passed as bind parameters, so
they are validated before being quoted and embedded — statement terminators,
comment markers, unbalanced quotes, and DDL/DML keywords are refused as
configuration errors. Secrets are redacted from every report, log line, and MCP
response.
docker build -t reconflow -f docker/Dockerfile .
docker run --rm -v "$PWD:/work" -w /work reconflow run -c reconflow.ymlIssues and pull requests welcome — see CONTRIBUTING.md.
pip install -e '.[dev]'
pytest
ruff check reconflow testsThe test suite runs against real databases rather than mocks. A reconciliation tool whose tests never touch a database proves nothing about whether its SQL is correct, and the SQL is where the bugs live.
Two further suites run against real PostgreSQL and real SQL Server, because
SQLite has no true DECIMAL, no BOOLEAN, no DATE, no schemas, does not fold
identifier case, and does not use T-SQL's OFFSET/FETCH or bracket quoting —
all places a cross-engine comparison can go wrong. The SQL Server suite also
covers Azure Synapse, whose driver inherits it. CI runs both on every push; see
CONTRIBUTING.md to run them locally.
ReconFlow was created and is maintained by Sumit Kumar Gupta and Nitish Pradhan. See AUTHORS for the full list of contributors.
Apache License 2.0 — see LICENSE and NOTICE.
Copyright 2026 The ReconFlow Authors.