Depends on: #232.
Problem
transform test --mutate reports a surviving mutant as a defect the tests would
not catch. That is always true, and for some survivors it is not the whole
story: a
mutant can be indistinguishable from the original on the data the dev
target currently holds, in which case no test against that data could catch it,
and the finding is really about the fixtures rather than about the assertions.
The #232 dogfood produced three of these on one model, and they are the ordinary
case rather than a curiosity:
- an inner join swapped for a left join where every foreign key happens to
resolve, so the two produce identical rows;
- a window frame widened from 2 preceding to 3 where no partition holds three
rows;
- a dropped
status filter whose excluded rows are removed by a join anyway.
All three are reported the same way as a defect that a differently-shaped test
would have caught today, and the two deserve different next actions. The first
kind needs a fixture that reaches the case at all. The second needs an assertion.
Proposal
For each surviving mutant, compare its output against the baseline's with an
aggregate, and label the ones that agree.
- An aggregate symmetric difference, never rows: a count of rows in one and not
the other, in both directions. Nothing row-level crosses the envelope, which
is the same rule row-population attribution already follows.
- A zero count means "equivalent on current data": the mutant and the model
produce the same result today, so no test against the dev target could
distinguish them and only a unit test fixture that reaches the case can.
- A non-zero count means the data does distinguish them and the tests did not,
which is the stronger finding and should be ranked above the equivalent ones.
The suggested test should differ accordingly: an equivalent survivor needs a
fixture reaching the boundary, category, or unmatched key; a distinguishable one
needs an assertion over data that already differs.
Cost
Free on DuckDB. On a metered connector it is two aggregate scans per survivor,
which has to be priced into the same batch estimate the run already confirms,
not asked for separately after the fact. Worth considering whether it should be
opt-in on metered connectors given it can double the batch.
Why it was not in #232
It changes the cost model, and #232 was already the most expensive command in
the engine. Shipping the honest version first ("no test told this apart") and
refining it second keeps the two decisions separable.
Acceptance
- A survivor whose output matches the baseline is labelled equivalent, with the
comparison that established it.
- A survivor whose output differs is ranked above the equivalent ones.
- No row value appears in the envelope, only counts.
- On a metered connector the comparison is priced into the run's own estimate,
and the run still confirms once.
- On DuckDB the labelling is on by default and costs nothing.
Depends on: #232.
Problem
transform test --mutatereports a surviving mutant as a defect the tests wouldnot catch. That is always true, and for some survivors it is not the whole
story: a
mutant can be indistinguishable from the original on the data the dev
target currently holds, in which case no test against that data could catch it,
and the finding is really about the fixtures rather than about the assertions.
The #232 dogfood produced three of these on one model, and they are the ordinary
case rather than a curiosity:
resolve, so the two produce identical rows;
rows;
statusfilter whose excluded rows are removed by a join anyway.All three are reported the same way as a defect that a differently-shaped test
would have caught today, and the two deserve different next actions. The first
kind needs a fixture that reaches the case at all. The second needs an assertion.
Proposal
For each surviving mutant, compare its output against the baseline's with an
aggregate, and label the ones that agree.
the other, in both directions. Nothing row-level crosses the envelope, which
is the same rule row-population attribution already follows.
produce the same result today, so no test against the dev target could
distinguish them and only a unit test fixture that reaches the case can.
which is the stronger finding and should be ranked above the equivalent ones.
The suggested test should differ accordingly: an equivalent survivor needs a
fixture reaching the boundary, category, or unmatched key; a distinguishable one
needs an assertion over data that already differs.
Cost
Free on DuckDB. On a metered connector it is two aggregate scans per survivor,
which has to be priced into the same batch estimate the run already confirms,
not asked for separately after the fact. Worth considering whether it should be
opt-in on metered connectors given it can double the batch.
Why it was not in #232
It changes the cost model, and #232 was already the most expensive command in
the engine. Shipping the honest version first ("no test told this apart") and
refining it second keeps the two decisions separable.
Acceptance
comparison that established it.
and the run still confirms once.