Keep tests from every declaration of a duplicated column - #16322
Open
CallumFriend-MP wants to merge 3 commits into
Open
CallumFriend-MP wants to merge 3 commits into
CallumFriend-MP wants to merge 3 commits into
Conversation
`column_tests_inner` collects column entries straight into a BTreeMap keyed by column name, so a column declared twice in one `columns:` list lost the earlier declaration's data tests. Parsing exits 0 with no warning, so the test disappears from the manifest and never runs again. dbt-core keeps the tests from both declarations. Collect onto a single entry rather than overwriting; the column's own attributes stay last-wins, matching how its description already resolves. The versioned-columns path in persist_generic_data_tests.rs has the same shape but is left alone: an existing entry there can come from version inheritance, where overriding is the intended behaviour. Fixes dbt-labs#16313. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #16313.
When the same column name appears twice in one
columns:list, the v2 parser keeps only the tests from the last declaration and drops the earlier one's. Parse exits 0 with nothing logged, so a data test disappears from the manifest and stops running.column_tests_innercollects column entries straight into aBTreeMap<String, ColumnTestEntry>keyed by column name, so the later entry evicts the earlier one wholesale. This collects onto a single entry instead. The column's own attributes (quote,tags,legacy_syntax_handling) stay last-wins, which is how itsdescriptionalready resolves under both parsers.This isn't #15841. That one covered tests whose generated names collide across different models, and it's fixed. This reproduces on 2.0.0rc2 which contains it, and here the two names don't collide at all.
Scope
The versioned-columns path in
persist_generic_data_tests.rshas the same insert-keyed-by-name shape. I left it alone on purpose: an existing entry there can come from version inheritance, where overriding is the point. Happy to extend it if you'd rather the two were consistent.Verification
cargo check -p dbt-parseris clean.cargo test -p dbt-parser --libgives 398 passed, 0 failed, 5 ignored, same asmain.Built a binary from this branch and ran it on the repro from #16313:
Surviving column description is
second declarationin all three, so last-wins is preserved.This is one of two options
#16313 sets out two ways to close this. This PR does the first:
I think 2 is the better end state. A column declared twice with two different descriptions is nearly always a mistake, and right now neither engine says anything about it. But it breaks projects that parse clean today, including the one where I found this. So it would want a deprecation warning first and an error later, and it needs to land in both engines together. Otherwise it just swaps one v1/v2 divergence for another.
That's a maintainer call, not something to decide in a PR, so I went with the conservative one here. Happy to swap this for the deprecation-warning version, or stack it on top.
Pre-existing failure
cargo test -p dbt-parser --docfails onresolve::validate_metrics::validate_metric_name(line 43) withE0432: unresolved import dbt_parser::resolve::validate_metric_name. It fails the same way on pristinemain, so it isn't from this PR.