Mutation coverage for a dbt model's tests

[marcociav-exmergo]

Closes #232.

Context

A dbt model can carry unit, generic and singular tests, pass all of them, and still
be wrong in a way none of them can detect. Test count is not test strength, and
nothing in the dbt ecosystem measures the difference. #232 asks dex to answer "do
these tests actually test anything":

1. Plant standard analytics defects in a model's SQL.
2. Run the model's own tests against each mutant on the dev target.
3. Report which defects survive, in defect terms ("your tests would not catch an
   inner join here"), not as diffs.

Both dependencies are closed:

• transform: scaffold dbt unit tests from a model's inputs #215: transform test --scaffold, the unit-test skeleton.
• transform build --verify: run the sweep over the nodes just built #231: transform build --verify, whose build-and-check machinery this reuses.

The design holds the safety spine:

• mutants never touch the project;
• dev target only;
• the batch is priced and confirmed before anything runs;
• a hard cap is stated with its elision;
• no raw rows reach the envelope.

Outcome: transform test --mutate <model> on every connector, dogfooded on DuckDB,
BigQuery and Snowflake. A model with a strong suite reports few survivors, and a
not_null-only model reports most surviving.

Decisions taken

• Surface: transform test --mutate <model> [--max-mutants N] [--target T].
  • --mutate and --scaffold are a mutually exclusive argparse group.
  • --target behaves as it does on build.
• Cap: MAX_MUTANTS = 20. --max-mutants may only lower it. A value above 20
  is refused, naming the ceiling, the same way --scope narrows and never widens.
• API parity: DexEngine.test_mutations(model, *, max_mutants=None, target=None),
  mapped in the CLI parity table.
• Survivors are survivors. Telling apart survivors that are equivalent on today's
  data goes to a follow-up issue, drafted here and opened only after you approve its
  text.
• One PR.

Design

How a mutant runs: in a copy, as an ephemeral model, under dbt test

• The copy. Extract the copy-and-overlay from shadow_parse
  (transform/build.py:297-317) into a context manager shadow_project(project, edits=()).
  shadow_parse keeps its exact behavior on top of it.
• ShadowRun, also in transform/build.py. It uses class DI because it owns
  state (the temp dir and dbt's partial-parse cache across N+1 invocations).
  • Methods: write(rel_path, text), which uses contained_path; strip_run_hooks();
    parse(); compile(select); and test(select, exclude=()), which returns
    _summarize(...).
  • cwd is the real project; dbt artifacts go to the copy. Every argv carries
    --project-dir <copy> --profiles-dir <real> --target-path <copy>/target --log-path <copy>/logs.
    • dbt-duckdb resolves a relative path: against cwd, exactly as
      transform build does.
    • target/ and logs/ follow the flags.
    • The warehouse files are never copied, linked or split from their WAL.
  • Pinned flags: --indirect-selection eager --no-defer --no-favor-state --no-fail-fast.
  • Scrubbed from the environment: DBT_TARGET_PATH, DBT_LOG_PATH, DBT_STATE, DBT_DEFER_STATE, DBT_STORE_FAILURES, DBT_WARN_ERROR, DBT_WARN_ERROR_OPTIONS, DBT_RESOURCE_TYPES, DBT_EXCLUDE_RESOURCE_TYPES, DBT_SELECTOR, DBT_EMPTY, DBT_FULL_REFRESH.
  • Set in the environment: DO_NOT_TRACK=1, because otherwise dbt writes
    .user.yml into the profiles dir.
  • The server-side caps from _build_env(connector, paradigm, ceiling) are
    merged in.
• The mutant file. Every mutant, and the baseline, is written into the copy as
  the model file with this header:
  {{ config(materialized='ephemeral', contract={'enforced': false}, access='protected') }}.
  • The in-file config() beats YAML (dbt/context/context_config.py:165-173).
  • access='protected' stops a public model failing parse
    (dbt/parser/manifest.py:1437-1447).
• Each run is dbt test --select <model>, never build or run:
  • Unit tests are picked up through indirect selection (dbt/graph/selector.py:22-33).
  • Data tests get the mutant inlined as __dbt__cte__<model> (dbt/compilation.py:539-591).
  • Unit tests take the tested model's raw_code whatever its materialization
    (dbt/parser/unit_tests.py:61-86).
  • dbt test cannot run model DDL, so even an ignored override could never
    overwrite your dev relation.
  • dbt build would be wrong here: one failing unit test marks the model skipped,
    and that skip cascades onto its data tests (dbt/task/build.py:33-38,134-148).
  • Before the baseline, the copy's manifest must show the node as ephemeral,
    otherwise the command fails closed.
• Hooks and store_failures.
  • The copy's dbt_project.yml loses on-run-start/on-run-end, so hooks do not
    run N+1 times. A note says so.
  • A test with store_failures would write mutant rows into the warehouse. It is
    passed as --exclude and reported excluded with that reason.
• Unit-test temp tables. Unit tests still create and drop a temp table in the
  dev schema, the same as dbt build today. This is documented.
• dev_target.check runs once, against the real project, first, as in
  commands.build (commands.py:1198).

What gets mutated: the compiled SQL, with references restored

1. Write the original raw code into the copy with the ephemeral header
   appended. The last config() wins for scalar keys
   (dbt_common/contracts/config/base.py:296-317), so is_incremental() is false
   by construction. --full-refresh would not do this, because a model with
   full_refresh: false defeats the flag.
2. compile it and take compiled_code from the manifest.
3. mutation.prepare(compiled_code, dialect=..., parents=..., dbt_ctes=...) parses
   the code in the connector dialect (adapters.get_dialect) and restores
   references:
   • Parents. Each depends_on parent's relation_name is parsed with
     exp.to_table(name, dialect=...) and matched exactly on (catalog, db, name,
     quoting). That covers BigQuery backticks and Snowflake case with no folding.
     The suffix-tolerant match_identifier is not used, because an ambiguous match
     would cross-wire two refs.
   • Emitted Jinja. A match emits {{ ref('m') }}, {{ ref('pkg','m') }},
     {{ ref('m', v=N) }} or {{ source('s','t') }}.
   • Hoisted ephemeral CTEs. Every __dbt__cte__* CTE belonging to an
     ephemeral node is stripped, including hoisted grandparents. Refused if a
     stripped non-parent is still referenced.
   • Unmatched literal tables stay as written, with a note.
4. Every non-placeholder segment of the written file is wrapped in
   {% raw %}...{% endraw %}, so a {{ inside a string literal or comment is not
   re-rendered. Refused if the SQL contains endraw.
5. The identity is the baseline. The identity is the unmutated tree through the
   same generator. It is the harness's own self-check: a round trip that changes
   what a test sees shows up at baseline, not in every mutant.

The mutation library (transform/mutation.py, pure, no I/O)

class	sites	rewrite	suggested test
comparison	EQ/NEQ/GT/GTE/LT/LTE (null-safe pair flips to each other)	to the boundary neighbour	a boundary unit test
predicate	each top-level AND predicate of WHERE/HAVING/QUALIFY (sql_shape.predicates)	drop; negate	expression_is_true or accepted_values
join_type	inner and left joins (sql_shape.joins)	inner to left, left to inner	relationships or a row-count test
case_branch	each WHEN (exp.Case.ifs)	remove it; a sole branch is replaced by default or exp.null() (empty ifs generates invalid CASE END)	accepted_values or not_null on the output
division	exp.Div (keeping typed/safe), exp.SafeDivide, Snowflake DIV0 (round-trips to IFF(b = 0 ...), so the guard's denominator is swapped too)	swap numerator and denominator	a unit test with a known ratio
window_frame	numeric Literal bounds of exp.WindowSpec (interval RANGE bounds skipped)	shift by one	a window unit test
aggregate	SUM, MAX	SUM to MAX, MAX to SUM	a unit test with a multi-row group

• Where sites come from. Sites are enumerated over every exp.Select in the
  tree, not only sql_shape.scopes, so a UNION root is covered.
• Labels. Each site is labelled "CTE x", "final select" or "branch N of the
  final union".
• How a mutant is built. It follows the _variant pattern
  (row_attribution.py:687): copy the tree, find site i in a fixed traversal
  order, apply the change. So mutants never compound.
• Dropped at generation and counted, never run: a mutant that fails to
  re-parse, and, when guards.approved_functions is set, a mutant that fails
  guarded_statement_verdict.
• Ordering and the cap. Mutants are ordered round-robin across classes, with
  stable ids (m01...), then cut to the cap. The elided count per class goes into
  data.cap and a warning, following the rule at row_attribution.py:51 that a
  cap which binds says so.
• Moved to sql_shape. row_attribution._set_predicates becomes the public
  sql_shape.set_predicates, and row_attribution imports it from there.
• Public API: prepare, enumerate_mutants(model, cap), inline_into_test,
  classify, MAX_MUTANTS, EPHEMERAL_HEADER.

Baseline and verdicts

• The expected test set is pinned from the identity compile, together with
  each test's attached model.
  • Eager selection includes a child's relationships test pointing here, which is
    useful. It is reported with attached_model.
  • Downstream models' own tests are out of scope.
• Baseline.
  • Tests that do not pass at baseline are excluded, each with its status and a
    reason. This covers tests that introspect the relation catalog, unit tests that
    depend on overrides (the Jinja they act on is already rendered), and
    store_failures tests.
  • If no test passes, stop with an error that names the statuses and the likely
    fix: transform build --select +<model>.
• Verdicts (classify):
  • killed: a baseline-passing test now fails or warns. caught_by names the
    tests. warn_only is set when every catching test has severity warn.
  • rejected: otherwise, a baseline-passing test now errors. The warehouse or dbt
    refused the mutant, and dbt build would fail too, so it is not a test gap. A
    BigQuery dry run that fails at pricing marks the mutant rejected before any
    spend.
  • survived: otherwise.
  • not_run: the run produced no results, a pinned test is missing from them, or
    the budget stopped the batch.
• No dbt messages are carried, because a unit-test diff prints fixture rows.
  Only test names and statuses appear.
• score = killed / (killed + survived), or null.

Free refusals, before any connection or spend

All of these are refused before dex opens a connection or spends anything:

• the node is not a root-project SQL model: Python models, package models, seeds
  and snapshots are all refused;
• the model sets sql_header, which only materializations emit;
• the compiled SQL is not exactly one query (a Select or a set operation);
• the model has zero tests;
• the model has zero mutation sites;
• --max-mutants is above 20.

Models that are already ephemeral work as they are. A versioned model accepts name
or name.vN; a bare name means the latest version.

Cost

• DuckDB (FREE_LOCAL): no pricing and no handshake. It carries
  skipped_handshake_warning, as build does.
• Billed connectors, priced as one batch:
  • Price every data test's identity compiled_code with adapter.query_estimate:
    a real dry run on BigQuery, a local heuristic on Snowflake.
  • For each mutant, inline_into_test parses the test SQL, replaces the body of
    the CTE aliased __dbt__cte__<model> with the mutant body, and prices the
    result.
    • A plain substring search would miss, because the manifest's model
      compiled_code is post-injection (compilation.py:584-590).
    • Pricing per mutant matters: dropping or flipping a partition predicate can
      raise the bytes scanned, so a multiplied baseline would under-report.
    • A test SQL that will not parse is priced at its baseline cost, and the
      estimate is noted as a partial floor (the compile_estimate convention).
  • Unit tests are unpriced, as today, and a note says so.
  • One command_args.billed_handshake("transform test", adapter, total, per_table={"(baseline)": b, "m01 comparison": x, ...}).
  • If pricing degrades, the command still runs, gated by ceiling and
    confirmation, as commands.build does at :1279.
• Execution:
  • Before each run: if the running spend plus the next run's estimate exceeds the
    confirmed budget, stop. The rest become not_run, stated in a warning.
  • A run with no billing figure counts at its estimate, so the stop rule never
    rounds down.
  • Each run is ledgered as it finishes, through a helper _settle_dbt_spend
    extracted from _shape_build_result's per-paradigm branches
    (commands.py:1936-2006): bytes from adapter_response, seconds from
    execution_time.
  • _record_build_spend gains command=, so rows say transform test.
  • gate.settle() runs in a finally. compute_spend_translation runs once on
    the total.

Payload (MutationCoverageResult, transform/results.py)

• model, target.
• baseline {tests: [{name, kind, attached_model, status}], excluded: [{name, status, reason}]}.
• mutants, survivors first:
  {id, operator, defect, scope, detail, before, after, suggested_test, status, caught_by, warn_only}.
• counts {generated, run, killed, rejected, survived, not_run}.
• score.
• cap {limit, generated, elided}.
• runs.
• spend, lifted to data.spend.

Status is ok when mutants survive, the same as --verify findings. Warnings
carry a pointer line with the survivor count, and a sentence saying that
"survived" means no test told the mutant apart on the dev data or the unit test
fixtures.

Changes by file

Engine (packages/dex-core/src/exmergo_dex_core/)

• sql_shape.py: set_predicates.
• transform/mutation.py (new).
• transform/build.py: shadow_project, ShadowRun. The engine build() is
  unchanged.
• transform/commands.py:
  • _settle_dbt_spend extracted with no behavior change (test_build guards it);
  • _record_build_spend(command=);
  • test_mutations(engine, model, *, max_mutants=None, target=None), which
    mirrors commands.build in order: dev check, deps in the real project,
    ShadowRun, identity compile, refusals, prepare/enumerate, price, handshake,
    baseline, loop, settle;
  • cmd_test dispatching --scaffold or --mutate.
• transform/results.py: MutationCoverageResult.
• engine.py: DexEngine.test_mutations.
• cli.py: the transform test block (:575) and its comment.
• transform/row_attribution.py: imports set_predicates.

Tests (packages/dex-core/tests/, mirroring src)

• tests/test_sql_shape.py: set_predicates.
• tests/transform/test_mutation.py (pure):
  • every operator in the duckdb, bigquery and snowflake dialects: sole CASE
    branch, interval bounds skipped, Div/SafeDivide/DIV0 guard, null-safe pair;
  • prepare with a hoisted grandparent CTE, BigQuery backticks, mixed-case
    Snowflake, a versioned ref, a package ref, a source, an unmatched literal
    table, a union root, and multi-statement SQL refused;
  • {% raw %} wrapping;
  • inline_into_test with a nested WITH;
  • the full classify matrix;
  • round-robin cap and elision.
• tests/transform/test_build.py: shadow_project keeps shadow_parse green, and
  ShadowRun sets argv, cwd and the scrubbed env.
• tests/transform/test_mutation_command.py:
  • With a fake runner (the _fake_runner_factory pattern, test_build.py:168,
    writing artifacts under argv's --target-path):
    • the verb is only parse, compile or test;
    • cwd is the real project, and --project-dir is the copy;
    • each free refusal runs zero test invocations;
    • baseline exclusion and the no-passing-baseline stop;
    • budget stop gives not_run;
    • per-run ledger rows are labelled transform test;
    • the degraded-pricing note.
  • Against real dbt-duckdb, beside test_confirmed_dev_build_runs_dbt_for_real
    (test_build.py:553):
    • generic, singular and unit tests all run against an ephemeral mutant;
    • the appended header beats a model's own materialized='incremental';
    • a public model parses;
    • a strong suite leaves fewer survivors than not_null alone;
    • the project tree hash and the DuckDB relation list are identical before and
      after.
• tests/test_safety_spine.py:
  • Family 2, parametrized over the six metered connectors:
    • an unconfirmed call returns needs_confirmation with the batch total and
      runs no dbt test;
    • a cap above 20 is refused.
  • Family 4: nothing is written into the project.
  • Family 5: a row value in fake dbt output never reaches the envelope.
• tests/test_cli_contract.py: _SUBCOMMAND_PARITY[("transform","test")] maps to
  test_mutations (mutate to model, max_mutants, target), with the scaffold
  path noted. tests/test_engine.py: _NOT_ENGINE_METHODS and _METHOD_NAMES.
• tests/test_spend_parity.py: add the command to billed_runs.
• tests/integration/test_{bigquery,snowflake}_transform.py:
  • the unconfirmed call returns an estimate;
  • a confirmed run on a small model settles a ledger row with
    command="transform test", under integration_budget().

Docs

• AGENTS.md: the command-table row (:96) and the "three things called a test"
  note (:159).
• references/command-contract.md: the surface line (:258), the transform table
  row, and the spend-reporting command list (about :1155).
• references/<connector>.md, all seven: a cost paragraph beside the --verify
  one. DuckDB is free; BigQuery prices per test and per mutant with dry runs;
  Snowflake uses seconds, with the resume-minimum floor.
• skills/transform/SKILL.md: when to reach for --mutate (after writing tests,
  before trusting them), and how to read survivors and suggested_test. Also one
  eval case in skills/transform/evals/evals.json.
• references/methodology.md: a short "Test strength" section.
• CHANGELOG.md [Unreleased] ### Added, in house style