Skip to content

transform test --mutate: tell apart survivors that are equivalent on the dev data #466

Description

@marcociav-exmergo

Depends on: #232.

Problem

transform test --mutate reports a surviving mutant as a defect the tests would
not catch. That is always true, and for some survivors it is not the whole
story: a
mutant can be indistinguishable from the original on the data the dev
target currently holds
, in which case no test against that data could catch it,
and the finding is really about the fixtures rather than about the assertions.

The #232 dogfood produced three of these on one model, and they are the ordinary
case rather than a curiosity:

  • an inner join swapped for a left join where every foreign key happens to
    resolve, so the two produce identical rows;
  • a window frame widened from 2 preceding to 3 where no partition holds three
    rows;
  • a dropped status filter whose excluded rows are removed by a join anyway.

All three are reported the same way as a defect that a differently-shaped test
would have caught today, and the two deserve different next actions. The first
kind needs a fixture that reaches the case at all. The second needs an assertion.

Proposal

For each surviving mutant, compare its output against the baseline's with an
aggregate, and label the ones that agree.

  • An aggregate symmetric difference, never rows: a count of rows in one and not
    the other, in both directions. Nothing row-level crosses the envelope, which
    is the same rule row-population attribution already follows.
  • A zero count means "equivalent on current data": the mutant and the model
    produce the same result today, so no test against the dev target could
    distinguish them and only a unit test fixture that reaches the case can.
  • A non-zero count means the data does distinguish them and the tests did not,
    which is the stronger finding and should be ranked above the equivalent ones.

The suggested test should differ accordingly: an equivalent survivor needs a
fixture reaching the boundary, category, or unmatched key; a distinguishable one
needs an assertion over data that already differs.

Cost

Free on DuckDB. On a metered connector it is two aggregate scans per survivor,
which has to be priced into the same batch estimate the run already confirms,
not asked for separately after the fact. Worth considering whether it should be
opt-in on metered connectors given it can double the batch.

Why it was not in #232

It changes the cost model, and #232 was already the most expensive command in
the engine. Shipping the honest version first ("no test told this apart") and
refining it second keeps the two decisions separable.

Acceptance

  • A survivor whose output matches the baseline is labelled equivalent, with the
    comparison that established it.
  • A survivor whose output differs is ranked above the equivalent ones.
  • No row value appears in the envelope, only counts.
  • On a metered connector the comparison is priced into the run's own estimate,
    and the run still confirms once.
  • On DuckDB the labelling is on by default and costs nothing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions