Conversation
…cros The dbt-athena macro package calls 21 `adapter.*` methods that exist only on the Python `AthenaAdapter`; without them every materialization fails at the first call. This ports them: the pure helpers (`generate_s3_location`, partition formatting, `murmur3_hash`, `TableType`), the Glue / S3 / Athena / STS backed ones, and the two statement runners (`run_query_with_partitions_limit_catching`, `run_operation_with_potential_multiple_runs`). The AWS calls go through the ADBC driver, as the BigQuery adapter does for `copy_table`: a statement with `athena.operation` / `athena.operation.payload` runs the named API call with the connection's credentials and answers with the API response as one JSON cell (dbt-labs/athena, `operations.go`). The composition — catalog ids, which versions to expire, the `swap_table` update and partition move, the docs diff — follows `impl.py` here. Lake Formation stays out: disabled configs are a no-op, enabled ones raise NotSupported. Also what the macros need around them: `relation.render_hive()` / `render_pure()`, `Relation.create(..., s3_path_table_part=...)`, `quote_seed_column`'s `quote_character`, an Athena arm in `build_catalog_relation`, dbt-athena's query comment rules (line comment, none ahead of ALTER / DROP / OPTIMIZE / VACUUM / MSCK), and the time-machine registry entries.
|
@aoelvp94 I saw Your branch has fixes this PR needs whichever way the client question goes: case folding of quoted identifiers in the relation cache, the missing-table probe before |
2de6773 to
eebac86
Compare
resolves #16252 (partial)
Problem
The dbt-athena macro package (#16369) calls 21
adapter.*methods that exist only on the PythonAthenaAdapter(impl.py). None of them existed in the Rust adapter, so every materialization failed: debug builds panicked intime_machine/semantic.rs(Unknown adapter method 'upload_seed_to_s3'), release builds hitUnknown method on adapter object.view.sqlcallsexpire_glue_table_versionsunconditionally, so even a plain view could not run.Solution
crates/dbt-adapter/src/adapter/athena/:mod.rs: the pure methods, ported fromimpl.pyandutils.py:generate_s3_location(all fiveS3DataNaminglayouts),format_value_for_partition,format_one_partition_key,format_partition_keys,murmur3_hash(MurmurHash3 x86_32 asmmh3.hash, checked against the Iceberg reference vectors),is_list,TableType/get_table_type, thepersist_docsstring helpers.driver_ops.rs: the Glue / S3 / Athena / STS calls go through the ADBC driver, as the BigQuery adapter does forcopy_table: a statement withathena.operation/athena.operation.payloadruns the named AWS API call with the connection's credentials and returns the API response as one JSON cell (feat(go): run Glue, S3 and STS operations behind statement options athena#19). The payload and the response are the AWS API JSON shapes; the composition followsimpl.pyhere:CatalogIdresolution as_get_data_catalog(awsdatacatalogands3tablescatalog/*from the STS account id, anything else throughGetDataCatalog), which versionsexpire_glue_table_versionsdeletes, theswap_tableupdate and partition move, the expression batching ofclean_up_partitions, the docs diff ofpersist_docs_to_glue.dispatch.rs: theadapter.<method>entry points, plusrun_query_with_partitions_limit_catchingandrun_operation_with_potential_multiple_runsover the adapter'sexecute.add_lf_tags/apply_lf_grants: a disabled config is a no-op as in dbt-athena; an enabled one raisesNotSupported. Lake Formation is a separate part.Around it, what the macros need from the rest of the adapter:
relation.render_hive()/relation.render_pure()andRelation.create(..., s3_path_table_part=...)(AthenaRelation).quote_seed_column(column, quote_config, quote_character): dbt-athena's third argument.build_catalog_relationfor Athena withoutcatalogs.yml: Glue, table format fromtable_type/table_format._AthenaQueryComment.add:-- /* ... */line comment, none ahead ofALTER/DROP/OPTIMIZE/VACUUM/MSCK(Athena rejects a block comment there; verified against the API).SemanticCategorytable.No new dependencies. Requires a driver with the operations (dbt-labs/athena#19); an older driver rejects the
athena.operationoption and the first materialization fails with that message.Testing
Unit tests for the pure helpers, the dispatch arms, the relation additions, the catalog relation arm and the comment rule.
End to end against a real Athena workgroup with the whole stack (#16365, #16274, #16367, #16368, #16369, #16374, #16370, #16375) on the jaffle_shop project plus Iceberg, incremental (append / insert_overwrite with partitions / Iceberg merge) and snapshot (check strategy) models:
seed,run(views, tables, repeated runs),test,build(28/28),snapshot,docs generate. Checked in Glue / S3, not only exit codes: seed CSVs uploaded undertables/<name>/<uuid>/and removed afterwards, no temp Glue tables left, view versions expire to the newest four, dropped tables leave no S3 data, OPTIMIZE / VACUUM ran as bare statements, the Iceberg table sits undertables/ice_orders/<uuid>withtable_type=ICEBERG.Not reachable in that run:
swap_table(needs thehamodel config, which #16368 does not accept yet), Lake Formation,catalogs.ymlv2 for Athena.Findings on the other parts that the run needed as local fixes are reported on their PRs.
Checklist