Mutation coverage for a dbt model's tests

AGENTS.md

@@ -94,6 +94,7 @@ credentials and no network.
 | `transform remove <kind> <name> [--edits-file <f>]` | the definition removed, and every read verified gone. Same kinds and same refusals as `rename`. dex authors the removal of the **definition** and refuses while any read survives, naming each with a file and a line; it never rewrites a read, because `{% if var('flag') %}` can be dropped or unguarded and `{{ var('x') }}` in an expression has no value dex may invent, and only you know which. Author those read edits yourself and pass them with `--edits-file`: they are validated and stored in this same plan, so the removal stays atomic without dex guessing at semantics |
 | `transform place <column> --targets <m,m> --expr "<sql>" [--explain]` | where a derived column that several models need should be *defined*: the lowest model in the `ref()` graph that every target descends from and that already projects the inputs the expression reads. The inputs are parsed out of `--expr`, so they cannot disagree with the expression using them. Defines the column there and threads it down every chain, with a `schema.yml` entry at the ancestor and at each target and none at the hops in between. `data.reasoning` names the ancestor, why it is the lowest, which targets descend from it, and the chain, because a proposal you cannot argue with is an imposition. Where there is no common ancestor, where the lowest one lacks an input (dex will not hunt further upstream to pull one down, because that turns one placement into an unbounded rewrite of the graph above it), or where two candidates tie, `data.strategy` is `per_target` and the reason is stated rather than the worse thing being done quietly. `--explain` returns the reasoning and stores no plan. Repo-only and free |
 | `transform test --scaffold <model>` | plan a `unit_tests:` skeleton for the model: a `given` block per `ref()`/`source()` input carrying only the columns the model reads, typed from the exploration cache, and an `expect:` stub that fails until it is filled in (dbt-parse-checked; applied with `transform apply`) |
+| `transform test --mutate <model> [--max-mutants <n>]` | measure the tests the model already has. Plants one standard analytics defect at a time in the model's compiled SQL (a flipped boundary, a dropped or negated filter, a swapped join type, a removed `CASE` branch, an inverted ratio, a shifted window frame, `sum` for `max`), runs the model's own tests against each, and reports which defects nothing caught, each with the test that would catch it. Every mutant builds as an ephemeral model in a throwaway copy of the project, so nothing is written to your tree and no relation is created, replaced or dropped. Dev-target only. Capped at 20 mutants, with `--max-mutants` able to narrow and never widen, and whatever the cap cuts reported per defect class. Refused for free, before any connection, when the model has no tests, is a Python model, belongs to a package, or has no mutable SQL. On a metered connector the whole batch is priced as one number and confirmed once, and a budget that runs out stops the run with the remainder reported as `not_run` |
 | `transform macro [name]` | no name lists the shipped dbt macros; a name proposes scaffolding it into the project's macro directory as a plan (dbt-parse-checked, applied with `transform apply`); re-running diffs the project's copy against the shipped version |
 | `transform build --target dev [--verify] [--for-plan <id>\|--for-plan-file <f>] [--no-install-deps]` | prod-looking targets refused outright; then a free dev-target preflight (refuses when `.dex/config.yml` and the rendered `profiles.yml` disagree, or when the dev database does not exist, naming the fix); then the cost preflight, priced upfront by a free `dbt compile` dry-run of each node (a partial floor when a cold dev target has not built a node's inputs yet; degrades to no estimate when dex cannot open its own connection); runs only with `--confirm` and a budget; cwd pinned to the project dir; auto-runs `dbt deps` when packages are declared but not installed. Each node in `data.nodes` carries dbt's `unique_id` beside a readable `name`, because a generic test's id ends in a content hash and a name taken from the last segment reads as `3249b83c15`, which is no use to a caller asking which of seventeen tests warned. `--verify` runs the `maintain verify` sweep over the nodes this build touched and reports it under `data.verification`, in the same envelope: `ran` is always present (a build that did not verify and one that verified and found nothing are different answers), and a run that swept adds `scope`, `findings`, `finding_count` and `suppressed`. Opt-in on every connector, free ones included. Findings never enter `errors` and never change the status: whether a `row_loss` should gate a pipeline is the caller's policy, and a pointer line in `warnings` names the count so a caller reading warnings alone still learns of them. A build that failed partway still reports the build-status half, which is when naming the failed node and what it took down with it is worth most; row population is suppressed there, and `no_relation` is suppressed always, since dbt's run results are a better authority on what it just built than the catalog is. Cost: the row counts a verdict needs are priced during the build's own pricing pass off the compiled manifest and folded into the same estimate as a `(row counts)` entry, so one `--budget` covers both phases; only a relation the warehouse keeps no count for (any view) costs anything, a cold dev target that cannot be priced upfront says so and is priced again after the build as a phase drawn against the reservation it already holds, and a phase that does not fit returns `ok` with the counts in `data.offer` rather than `needs_confirmation` for a build that is finished and billed. `--verify` folds the connector's dev namespace into its own read scope for the length of the command, since that namespace is refused as a source everywhere else and is exactly what this one is verifying; the widening shows in `connection.target` and nothing is written back to `.dex/config.yml``data.outcome` says what the run *established*, which `success` cannot: `success` is dbt's exit code and an empty selection exits zero, so a build that matched no nodes is `empty_selection`, one that built nothing the change required is `unrelated`, one whose tree no longer matches the plan is `stale`, and `validated` is reserved for a run that built and passed every node the change required. `data.evidence` carries the invocation, the typed per-node statuses, what the selection asked for against what it got, the manifest and run-results digests, the relations generated, and the principal errors. `--for-plan` (a stored plan id) or `--for-plan-file` (an exported plan document, which is what a build sandbox holds) is what adds `coverage`; without either there is no coverage to report and the key is absent rather than empty. `--no-install-deps` refuses a declared-but-uninstalled package by name instead of installing it, before any subprocess and before pricing, which is the right outcome in a sandbox with no network and costs nothing |
 | `transform deps` | install/refresh dbt packages (repo-confined; no warehouse spend) |
@@ -157,7 +158,8 @@ enters the drift baseline and neither raises a dangling-reference guard on
 delete. Note that three separate things are called a test: generic tests
 declared inside a `schema.yml`, unit tests scaffolded by
 `transform test --scaffold` into a `unit_tests:` block, and the files under
-`test-paths` that `test_sql` authors. The semantic commands take a second,
+`test-paths` that `test_sql` authors. `transform test --mutate` measures all
+three at once, since a defect has to get past every one of them to ship. The semantic commands take a second,
 narrower payload instead: `--definitions-file <path|->` with
 `{"definitions": [{"kind", "path", "content"}, ...]}`, where `kind` is
 `semantic_model` or `metric` and `content` is that one definition's YAML body.

CHANGELOG.md

@@ -11,6 +11,46 @@ tag releases both in lockstep, so entries below are keyed by the engine version.
 
 ### Added
 
+- **`transform test --mutate <model>` measures whether a model's tests would
+  catch anything** ([#232]). A suite that passes proves the tests ran, not that
+  they would object if the model were wrong, and no count distinguishes the two:
+  a `not_null` on a surrogate key and a unit test pinning the arithmetic both
+  read as "tested". This plants one standard analytics defect at a time in the
+  model's compiled SQL (a boundary flipped, a filter dropped or negated, an
+  inner join swapped for a left join, a `CASE` branch removed, a ratio inverted,
+  a window frame shifted, `sum` reporting a `max`), runs the model's own generic,
+  singular and unit tests against each, and reports which defects nothing caught.
+  Each finding is written as the defect rather than as a diff, and carries the
+  test that would catch it, because the reader's next action is to write a test.
+
+  **Nothing is written and nothing is materialized.** Every mutant is built in a
+  throwaway copy of the project as an ephemeral model, so dbt inlines it into
+  each test and creates no relation: the project is byte-identical afterwards and
+  the dev namespace holds exactly what it held before. The dogfood checked both
+  on all three warehouses. The run uses `dbt test` rather than `dbt build`,
+  which is load-bearing: under a build, one failing unit test marks the model
+  skipped and that skip cascades onto every data test attached to it, so every
+  mutant would read as caught and nothing would be learned about the data tests.
+
+  **The batch is priced and confirmed once.** On a metered connector each mutant
+  is priced as the statement the warehouse will actually run, by splicing it into
+  each test's compiled SQL rather than multiplying the baseline, because a mutant
+  that drops a partition predicate scans more than the model it came from. One
+  estimate names `(baseline)` and each mutant; one `--budget` covers the run. A
+  budget that runs out partway stops the run and reports the remainder as
+  `not_run` rather than overspending. Capped at 20 mutants, ordered round robin
+  across the defect classes so a cap stays representative, with whatever it cut
+  reported per class.
+
+  **Every verdict is relative to what already passed.** A test failing before
+  anything was mutated is excluded and named, so a suite measured against its own
+  broken tests cannot come back looking clean, and a run where nothing passes at
+  baseline is an error rather than a clean sweep. A mutant the warehouse refuses
+  outright is reported as `rejected` rather than `killed`, since a build would
+  have failed on it anyway and counting it would flatter the suite.
+
+  Also available as `DexEngine.test_mutations(model)`.
+
 - **`maintain verify` reports a built relation whose grain is not unique**
   ([#229]). For every selected model, the intended grain is determined from a
   declared `unique` test, a declared composite `unique_combination_of_columns`

packages/dex-core/src/exmergo_dex_core/cli.py

@@ -573,10 +573,19 @@ def _build_parser() -> argparse.ArgumentParser:
                     # calling one if a caller can ask for it cheaply and disagree.
                     sp.add_argument("--explain", action="store_true", default=False)
                 if group == "transform" and name == "test":
-                    # `test` is scaffold-only for now: the model to derive a
-                    # unit_tests: skeleton from. No bare `transform test`
-                    # mode exists yet, unlike `macro`'s list-when-bare shape.
-                    sp.add_argument("--scaffold", default=None)
+                    # Two modes, and they are opposites: `--scaffold` writes a
+                    # unit test, `--mutate` measures the tests that already
+                    # exist. Mutually exclusive rather than ordered, because
+                    # asking for both in one call names no coherent outcome. No
+                    # bare `transform test` mode exists, unlike `macro`'s
+                    # list-when-bare shape.
+                    mode = sp.add_mutually_exclusive_group()
+                    mode.add_argument("--scaffold", default=None)
+                    mode.add_argument("--mutate", default=None)
+                    # Only ever narrows: the engine ceiling is what keeps a run
+                    # that invokes dbt once per mutant predictable.
+                    sp.add_argument("--max-mutants", type=int, default=None)
+                    sp.add_argument("--target", default=None)
                 if group == "semantic" and name in {"define", "update", "plan"}:
                     sp.add_argument("argument", nargs="?", default=None)
                     sp.add_argument("--edits-file", default=None)

packages/dex-core/src/exmergo_dex_core/engine.py

@@ -93,6 +93,7 @@
         InitResult,
         MacroListResult,
         MacroResult,
+        MutationCoverageResult,
         PlacementResult,
         PlanExportResult,
         PlanListResult,
@@ -1333,6 +1334,30 @@ def build(
             dependencies=dependencies or DependencyPolicy.INSTALL,
         )
 
+    def test_mutations(
+        self,
+        model: str,
+        *,
+        max_mutants: int | None = None,
+        target: str | None = None,
+    ) -> MutationCoverageResult:
+        """Measure a model's tests by planting defects and seeing what they catch.
+
+        Answers what a green suite cannot: whether the tests would notice if the
+        model were wrong. Each mutant is one standard analytics defect, built in
+        a throwaway copy as an ephemeral model, so nothing is written to the
+        project or materialized in the warehouse. Dev target only, and on a
+        billed connector the whole batch is priced and confirmed as one number.
+
+        ``max_mutants`` may only narrow the engine's ceiling, never raise it.
+        """
+
+        from .transform import commands as transform
+
+        return transform.test_mutations(
+            self, model, max_mutants=max_mutants, target=target
+        )
+
     def deps(self) -> DepsResult:
         from .transform import commands as transform
 

packages/dex-core/src/exmergo_dex_core/sql_shape.py

@@ -14,11 +14,18 @@
 a WITH has to accept either. Spread across call sites that is a rename waiting
 to be half-applied; here it is absorbed once.
 
-Everything is a pure read of a parsed tree: nothing mutates, nothing executes,
-nothing opens a connection. The module imports sqlglot at the top, so a caller
-that must survive its absence (the base install carries no dialect engine)
-should reach it behind :func:`~.guards.dialect.ensure_available` and degrade on
-the refusal rather than importing unconditionally.
+Almost everything here is a pure read of a parsed tree. The one exception is
+:func:`set_predicates`, which is the inverse of :func:`predicates`: the reader
+flattens a WHERE across its top-level ANDs, and rebuilding the clause from a
+flattened list is the operation that undoes it. The two belong together, because
+a caller that splits a clause one way and reassembles it another produces a
+statement neither function describes.
+
+Nothing here executes anything or opens a connection. The module imports sqlglot
+at the top, so a caller that must survive its absence (the base install carries
+no dialect engine) should reach it behind
+:func:`~.guards.dialect.ensure_available` and degrade on the refusal rather than
+importing unconditionally.
 """
 
 from __future__ import annotations
@@ -77,6 +84,24 @@ def predicates(select: exp.Select, key: str) -> list[exp.Expression]:
     return flat
 
 
+def set_predicates(select: exp.Select, key: str, preds: list[exp.Expression]) -> None:
+    """Rebuild a WHERE/HAVING/QUALIFY clause from flattened predicates, in place.
+
+    The inverse of :func:`predicates`. An empty list removes the clause outright
+    rather than leaving an empty wrapper, because a ``WHERE`` with nothing under
+    it is not something a generator can print.
+    """
+
+    if not preds:
+        select.set(key, None)
+        return
+    condition = preds[0]
+    for extra in preds[1:]:
+        condition = exp.And(this=condition, expression=extra)
+    wrapper = {"where": exp.Where, "having": exp.Having, "qualify": exp.Qualify}[key]
+    select.set(key, wrapper(this=condition))
+
+
 def text(node: exp.Expression | None) -> str:
     """What a fragment says, normalized by sqlglot's own generator and no further.