Repository navigation
fix(athena): S3 Tables macros, bucket groups, schema LF-Tags, catalog-qualified columns - #1
Merged
aoelvp94 merged 4 commits intoSep 28, 2026
Conversation
The vendored package predates dbt-adapters 62a5495cf (S3 Tables catalogs). From that commit, applied as is: drop_relation drops an S3 Tables relation through Glue; the table materialization drops and recreates it, since S3 Tables allows no ALTER TABLE RENAME; create_table_as sets no S3 location and skips the direct S3 cleanup for it; incremental skips the Glue table version expiry, which the federated catalog does not serve.
dbt v2's regex match objects do not support indexing (dbt-labs#16451), so bucket_match[1] rendered empty; .group(n) works in dbt v1 and v2.
As dbt-athena does after the DDL; adapter.add_lf_tags_to_database is implemented in dbt-labs#16376.
Athena's information_schema covers only the catalog it is read from, so the columns of a relation in another catalog (S3 Tables, s3tablescatalog/<bucket>) came back empty.
aoelvp94
added a commit
that referenced
this pull request
Sep 27, 2026
…mation_schema Relation and column lookups went to Glue unconditionally. Glue serves only the Data Catalog, so a relation in a federated connector or in S3 Tables (`s3tablescatalog/<bucket>`) came back empty and the next build re-ran its CTAS (TABLE_ALREADY_EXISTS). Reading everything from `information_schema` instead would fix that at the cost of an Athena query per relation, each billed at the 10 MB minimum. Dispatch on the catalog: Glue when it is the Data Catalog, otherwise that catalog's own `information_schema` — Athena's covers only the catalog it is read from, so the catalog belongs in the table reference, not just in a filter. `list_relations` also falls back when the Glue call fails at all, since Lake Formation can deny GetTables on tables that stay queryable through Athena. The `information_schema` queries and their tests are @fpiped's, from #1 and #2; this keeps the Glue fast path in front of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for dbt-labs#16369 found running the whole Athena stack against a real work group; one commit each, so any can be dropped:
62a5495cf(dbt-athena S3 Tables support), applied as is;bucket_match.group(n)instead ofbucket_match[n]: dbt v2 match objects do not index ([v2 Bug]modules.re: indexing a match (m[1],m['name']) renders empty instead of the group dbt-labs/dbt#16451),.group()works in v1 and v2;athena__create_schemacallsadapter.add_lf_tags_to_database(relation)after the DDL, as dbt-athena does (implemented in feat(athena): Part 6 — AthenaAdapter methods behind the dbt-athena macros dbt-labs/dbt#16376);athena__get_columns_in_relationreads"<catalog>".information_schema.columns: Athena'sinformation_schemacovers only its own catalog, so S3 Tables relations came back with no columns.Tested:
cargo nextest run -p dbt-loader(228 passed) on this branch; end to end with the other parts (Iceberg bucket partitions, schema LF-Tags,catalogs.ymlv2s3_tablestable and incremental models built twice). The companion fix forlist_relationsis onathena/part-5-metadata-adapter.