Conversation
…on ones
Resolves five of the nine todo!("Athena") sites — the ones parse can reach
— and leaves the two execution-layer sites in place with comments saying
why.
Filled:
- relation/factory.rs: Athena joins the generic RelationStatic arm with
the other plain SQL adapters. Same change dbt#16274 makes.
- adapter_impl.rs: schema-listing column is `schema_name`, since Athena's
information_schema is Trino's.
- column_builder.rs (two sites): a `build_athena` structurally identical
to `build_exasol`, delegating every adapter-specific decision to
`sql_types`, which already carries ATHENA_KEYS. Deliberately NOT routed
through build_postgres_like, whose Timestamp -> "datetime" rendering
is a legacy quirk that path documents as broken.
- relations/base.rs: microbatch event-time boundaries as explicit
TIMESTAMP literals. The generic arm emits a bare varchar, which Trino
rejects ("Cannot apply operator: timestamp >= varchar"); Trino's
literal syntax also takes no 'T' and no zone offset.
- seed_io.rs: column-name inference is Lowercase, because the Glue
catalog stores column names lowercase whatever the DDL says.
Left as todo!(), now annotated:
- metadata/get_relation.rs: needs an athena_get_relation querying
information_schema over a live connection.
- adapter_impl.rs MetadataAdapter construction: needs an
AthenaMetadataAdapter type against a live connection.
Both are rung-3 work that parse never reaches. Guessing them would be
worse than leaving an honest panic.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dbt-athena 1.11.0 declares `_ALIASES = {"catalog": "database"}`
(connections_legacy.py), which canonicalises node config keys. Athena
leaves the "no aliases" group and gets its own arm.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
fpiped
left a comment
There was a problem hiding this comment.
Tested merged with the rest of the stack against a real workgroup: debug, compile --write-catalog, docs generate work, and with Part 6 the materializations do too.
One divergence that bites: convert_type returns varchar and bigint for Athena where AthenaAdapter.convert_text_type / convert_number_type return string and integer. delete_overlapping_partitions and get_partition_batches only handle integer / string / date / timestamp, so an insert_overwrite model partitioned by a text or bigint column fails with Need to add support for column type varchar on its second run. The seed macros already map string to varchar themselves, so returning dbt-athena's names is safe there.
Landing note: #16374 touches the same Athena arms in adapter_impl.rs and get_relation.rs.
`adapter.convert_type` backs dbt-athena's `convert_text_type` and `convert_number_type`, which return `string` and `integer` with no width distinction. Fusion returned the DDL spellings `varchar` and `bigint`. The macros compare against dbt-athena's names: `get_partition_batches` and `delete_overlapping_partitions` handle only `integer`, `string`, `date` and `timestamp`, so an `insert_overwrite` model partitioned on a text or bigint column failed on its second run with "Need to add support for column type varchar". Mapped at the `convert_type` boundary rather than in `format_arrow_type_as_sql`, which renders DDL for unit-test fixtures and seeds and must keep the Trino spellings and the integer widths (`integer` is 32-bit there, so a collapsed name overflows bigint values). The seed macros reach the DDL spellings through `ddl_data_type`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Part of #16252. Related: #13822. Independent PR; the sequence is at the bottom.
Problem
With an
athenaprofile parsing (#16274),dbt parsepanics at the firsttodo!("Athena")it reaches,relation/factory.rs. Five of the ninetodo!("Athena")arms are on the parse path; the other four are metadata, execution anddbt init, handled in later parts.Solution
Fills the five parse-path arms. Each is the smallest thing that is correct for Athena, with the reason in a comment at the site:
relation/factory.rs: Athena joins the genericRelationStaticarm with the other plain-SQL adapters. Same three lines as feat(athena): AthenaDbConfig profile schema #16274; whichever merges second drops them on rebase.adapter_impl.rs: the schema-listing column isschema_name, since Athena'sinformation_schemais Trino's.column/column_builder.rs(two arms):build_athena, structurally identical tobuild_exasol, delegating every type decision tosql_types, which already carriesATHENA_KEYS. Deliberately not routed throughbuild_postgres_like, whoseTimestamp -> "datetime"rendering that path documents as a legacy quirk.relations/base.rs: microbatch event-time boundaries asTIMESTAMP '...'literals. The generic arm emits a bare varchar, which Trino rejects (Cannot apply operator: timestamp >= varchar); Trino's literal takes noTand no zone offset.seed_io.rs: column-name inference isLowercase, because the Glue catalog stores column names lowercase whatever the DDL says.adapter.convert_typealso returns dbt-athena's names for Athena:convert_text_typegivesstringandconvert_number_typegivesintegerfor whole numbers, with no width distinction. The macros compare against those names, so returning the DDL spellingsvarcharandbigintmade aninsert_overwritemodel partitioned on a text or bigint column fail on its second run inget_partition_batches. Mapped at theconvert_typeboundary rather than informat_arrow_type_as_sql, which renders unit-test fixtures and seed DDL and must keep the Trino spellings and the integer widths.Plus
config_aliases: dbt-athena 1.11.0 declares_ALIASES = {"catalog": "database"}(connections_legacy.py), which canonicalises node config keys. Athena leaves the "no aliases" group and gets its own arm.The two remaining adapter
todo!("Athena")arms (MetadataAdapterconstruction,get_relation) are annotated, not filled; Part 5 fills them.Verification
cargo fmt --check,cargo clippyondbt-adapter,dbt-adapter-core,dbt-df-providers,dbt-schemasclean on the pinned toolchain.dbt parseon a 750-model project completes with 0 errors in 1.8 s;dbt lsexits 0. Without Part 4 it stops at the missingdbt-athenapackage, as feat(athena): AthenaDbConfig profile schema #16274 describes.Feedback wanted:
build_athenamirrorsbuild_exasolrather thanbuild_postgres_like; confirm that is the intended direction for Trino-family adapters.Sequence (independent PRs, each compiles alone against main)
table/incrementalhelpers (athena/parts-6-7-execution); see also feat(athena): Part 6 — AthenaAdapter methods behind the dbt-athena macros #16376Checklist