DataAudit

The open-source trust score for datasets.

Install · Quickstart · Checks · Python API · Extend · CI/CD

---

Most data-quality tools answer a small question:

"Does this dataset satisfy the rules I already wrote?"

Researchers and ML engineers need the bigger one:

"Can I trust this dataset at all?"

DataAudit answers the second. Point it at any tabular dataset — no configuration, no expectation suites — and it returns a single Dataset Health Score backed by a transparent, modular battery of audits for the failure modes that quietly invalidate results: duplicates, train/test leakage, label errors, class imbalance, demographic bias, distribution anomalies, and drift.

$ dataaudit audit loans.csv --target approved --sensitive gender

╭───────────────── DataAudit ─────────────────╮
│ loans.csv                                    │
│ 48,842 rows × 15 cols  ·  target: approved   │
│                                              │
│ Dataset Health Score  81/100  (B-)           │
╰──────────────────────────────────────────────╯

Check          Category      Score  Status
schema         structure       100  ✓ ok
missingness    integrity        96  ✓ ok
duplicates     integrity        73  2 issue(s)
leakage        validity         45  1 issue(s)
label_quality  validity         88  1 issue(s)
distribution   distribution     80  1 issue(s)
fairness       fairness         65  1 issue(s)

leakage
  ✖ [critical] Column 'loan_status' may leak the target (assoc=0.99 ...).
      → Confirm 'loan_status' is available at prediction time; drop it if it encodes the target.
fairness
  ✗ [high] 'gender': disparate impact ratio 0.69 ('female' 57% vs 'male' 82%).
      → Investigate sampling/labeling bias before training.

Why it matters

Published papers and shipped models routinely rest on datasets containing duplicates, mislabeled samples, train/test contamination, class imbalance, demographic bias, outliers, and drift. Each of these can invalidate the result on its own — and none of them are caught by a null-value check. DataAudit makes dataset trust measurable, automatic, and shareable.

Install

pip install dataaudit                 # core
pip install "dataaudit[text]"         # + MinHash/LSH near-duplicate detection
pip install "dataaudit[ai]"           # + LLM-assisted auditing
pip install "dataaudit[all]"          # everything optional

Python 3.9+. Core depends only on the scientific-Python stack (pandas/numpy/scipy/scikit-learn).

Quickstart

# Full audit with a health score
dataaudit audit dataset.csv --target label

# Schema discovery only -> dataset_profile.yaml
dataaudit profile dataset.csv -o dataset_profile.yaml

# Drift between a baseline and a new extract
dataaudit drift reference.csv current.csv

# A shareable HTML report
dataaudit report dataset.csv -o report.html

# LLM-assisted cross-column logic audit (needs ANTHROPIC_API_KEY)
dataaudit ai dataset.csv

# List every available check (built-ins + plugins)
dataaudit list

What it checks

Check	Category	What it finds
schema	structure	Inferred column types; empty / constant / mixed-type / duplicate columns
missingness	integrity	Overall and per-column missing values
duplicates	integrity	Exact duplicate rows; near-duplicate text (MinHash/LSH)
leakage	validity	Target leakage (mutual information, correlation, single-feature predictivity); train/test contamination
label_quality	validity	Likely mislabeled samples via cross-validated confidence (confident-learning style)
distribution	distribution	Class imbalance (entropy, minority ratio); outliers, skew, kurtosis
fairness	fairness	Demographic parity, disparate impact (80% rule), subgroup representation
drift	drift	PSI + KS / chi-square drift vs a reference dataset

Every check is an independent, weighted module that returns a 0–100 sub-score; the Dataset Health Score is their weighted mean (checks that don't apply are skipped, never penalised).

Python API

from dataaudit import audit

report = audit("data.csv", target="label", sensitive=["gender"])

print(report.health_score, report.grade)     # 81.0  B-
print(report.issues_by_severity())            # {'critical': 1, 'high': 1, ...}

for issue in report.issues:
    print(issue.severity.label, issue.code, "->", issue.recommendation)

report.save("report.html")                    # or .json / .yaml

Run a subset, tune thresholds, or sample large data:

from dataaudit import Auditor, Dataset

ds = Dataset.load("big.parquet", backend="pyarrow", target="y").sample(50_000)

report = Auditor(
    include=["leakage", "fairness", "duplicates"],
    options={"duplicates": {"near_threshold": 0.85}},
).run(ds)

Multi-backend loading

CSV/TSV/JSON/Parquet/Feather load out of the box. For other engines, DataAudit materialises to pandas at the boundary:

dataaudit audit data.parquet --backend pyarrow
dataaudit audit data.csv     --backend polars
dataaudit audit data.parquet --backend duckdb

In-memory pandas, polars, and pyarrow objects can be passed directly to Dataset.load(...) / audit(...).

Extending DataAudit

A new audit is a small subclass — register it and it shows up everywhere (CLI, API, reports, scoring):

import pandas as pd
from dataaudit import Check, Issue, Severity, registry
from dataaudit.core.dataset import ColumnType

@registry.register
class FutureDateCheck(Check):
    name = "future_dates"
    category = "integrity"
    weight = 0.5
    description = "Flag datetime columns with values in the future."

    def run(self, ds):
        now = pd.Timestamp.utcnow().tz_localize(None)
        issues = []
        for col in ds.columns_of_type(ColumnType.DATETIME):
            future = pd.to_datetime(ds.df[col], errors="coerce") > now
            if future.any():
                issues.append(Issue(
                    code="future_dates.found",
                    message=f"Column '{col}' has {int(future.sum())} future-dated values.",
                    severity=Severity.MEDIUM,
                    columns=[col],
                    count=int(future.sum()),
                    recommendation=f"Verify timestamps in '{col}'.",
                ))
        return self.result(issues)

Third-party packages can ship checks for everyone by advertising them under the dataaudit.checks entry-point group — no changes to DataAudit required. See docs/PLUGINS.md.

CI/CD

Fail a build when dataset quality regresses:

# .github/workflows/data-quality.yml
- run: pip install dataaudit
- run: dataaudit audit data/train.csv --target label --fail-under 80

--fail-under makes dataaudit audit exit non-zero when the health score drops below the threshold, so dataset quality becomes a checkable contract alongside your tests.

The vision

Imagine a Dataset Health Score: 92/100 — generated by DataAudit badge attached to papers, Kaggle datasets, and Hugging Face cards — a portable, open signal of dataset trust. That's the goal. It sits at the intersection of data engineering, ML reliability, AI safety, and research reproducibility.

Development

git clone https://github.com/dataaudit/dataaudit && cd dataaudit
pip install -e ".[dev,all]"
pytest                 # run the suite
ruff check . && mypy src

See CONTRIBUTING.md. DataAudit ships a benchmark suite of deliberately broken datasets in benchmarks/ — every check must catch its corresponding defect, and every PR must keep them passing.

License

Apache-2.0.