Skip to content

feat(athena): Part 6 — AthenaAdapter methods behind the dbt-athena macros - #16376

Open
fpiped wants to merge 1 commit into
dbt-labs:mainfrom
fpiped:feat/athena-adapter-methods
Open

fpiped wants to merge 1 commit into
dbt-labs:mainfrom
fpiped:feat/athena-adapter-methods

Conversation

@fpiped

@fpiped fpiped commented Sep 19, 2026

Copy link
Copy Markdown

resolves #16252 (partial)

Problem

The dbt-athena macro package (#16369) calls 21 adapter.* methods that exist only on the Python AthenaAdapter (impl.py). None of them existed in the Rust adapter, so every materialization failed: debug builds panicked in time_machine/semantic.rs (Unknown adapter method 'upload_seed_to_s3'), release builds hit Unknown method on adapter object. view.sql calls expire_glue_table_versions unconditionally, so even a plain view could not run.

Solution

crates/dbt-adapter/src/adapter/athena/:

  • mod.rs: the pure methods, ported from impl.py and utils.py: generate_s3_location (all five S3DataNaming layouts), format_value_for_partition, format_one_partition_key, format_partition_keys, murmur3_hash (MurmurHash3 x86_32 as mmh3.hash, checked against the Iceberg reference vectors), is_list, TableType / get_table_type, the persist_docs string helpers.
  • driver_ops.rs: the Glue / S3 / Athena / STS calls go through the ADBC driver, as the BigQuery adapter does for copy_table: a statement with athena.operation / athena.operation.payload runs the named AWS API call with the connection's credentials and returns the API response as one JSON cell (feat(go): run Glue, S3 and STS operations behind statement options athena#19). The payload and the response are the AWS API JSON shapes; the composition follows impl.py here: CatalogId resolution as _get_data_catalog (awsdatacatalog and s3tablescatalog/* from the STS account id, anything else through GetDataCatalog), which versions expire_glue_table_versions deletes, the swap_table update and partition move, the expression batching of clean_up_partitions, the docs diff of persist_docs_to_glue.
  • dispatch.rs: the adapter.<method> entry points, plus run_query_with_partitions_limit_catching and run_operation_with_potential_multiple_runs over the adapter's execute.
  • add_lf_tags / apply_lf_grants: a disabled config is a no-op as in dbt-athena; an enabled one raises NotSupported. Lake Formation is a separate part.

Around it, what the macros need from the rest of the adapter:

  • relation.render_hive() / relation.render_pure() and Relation.create(..., s3_path_table_part=...) (AthenaRelation).
  • quote_seed_column(column, quote_config, quote_character): dbt-athena's third argument.
  • build_catalog_relation for Athena without catalogs.yml: Glue, table format from table_type / table_format.
  • Query comments follow _AthenaQueryComment.add: -- /* ... */ line comment, none ahead of ALTER / DROP / OPTIMIZE / VACUUM / MSCK (Athena rejects a block comment there; verified against the API).
  • The 21 names registered in the time-machine SemanticCategory table.

No new dependencies. Requires a driver with the operations (dbt-labs/athena#19); an older driver rejects the athena.operation option and the first materialization fails with that message.

Testing

Unit tests for the pure helpers, the dispatch arms, the relation additions, the catalog relation arm and the comment rule.

End to end against a real Athena workgroup with the whole stack (#16365, #16274, #16367, #16368, #16369, #16374, #16370, #16375) on the jaffle_shop project plus Iceberg, incremental (append / insert_overwrite with partitions / Iceberg merge) and snapshot (check strategy) models: seed, run (views, tables, repeated runs), test, build (28/28), snapshot, docs generate. Checked in Glue / S3, not only exit codes: seed CSVs uploaded under tables/<name>/<uuid>/ and removed afterwards, no temp Glue tables left, view versions expire to the newest four, dropped tables leave no S3 data, OPTIMIZE / VACUUM ran as bare statements, the Iceberg table sits under tables/ice_orders/<uuid> with table_type=ICEBERG.

Not reachable in that run: swap_table (needs the ha model config, which #16368 does not accept yet), Lake Formation, catalogs.yml v2 for Athena.

Findings on the other parts that the run needed as local fixes are reported on their PRs.

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.

…cros

The dbt-athena macro package calls 21 `adapter.*` methods that exist only on
the Python `AthenaAdapter`; without them every materialization fails at the
first call. This ports them: the pure helpers (`generate_s3_location`,
partition formatting, `murmur3_hash`, `TableType`), the Glue / S3 / Athena /
STS backed ones, and the two statement runners
(`run_query_with_partitions_limit_catching`,
`run_operation_with_potential_multiple_runs`).

The AWS calls go through the ADBC driver, as the BigQuery adapter does for
`copy_table`: a statement with `athena.operation` / `athena.operation.payload`
runs the named API call with the connection's credentials and answers with
the API response as one JSON cell (dbt-labs/athena, `operations.go`). The
composition — catalog ids, which versions to expire, the `swap_table` update
and partition move, the docs diff — follows `impl.py` here. Lake Formation
stays out: disabled configs are a no-op, enabled ones raise NotSupported.

Also what the macros need around them: `relation.render_hive()` /
`render_pure()`, `Relation.create(..., s3_path_table_part=...)`,
`quote_seed_column`'s `quote_character`, an Athena arm in
`build_catalog_relation`, dbt-athena's query comment rules (line comment,
none ahead of ALTER / DROP / OPTIMIZE / VACUUM / MSCK), and the time-machine
registry entries.
@fpiped

fpiped commented Sep 20, 2026

Copy link
Copy Markdown
Author

@aoelvp94 I saw athena/parts-6-7-execution and the pointer to this PR in your descriptions. The two branches close the same gap differently: yours stays SQL-only and skips or refuses the Glue / S3 methods; this one keeps dbt-athena's behaviour and, since today, routes the AWS calls through the ADBC driver (dbt-labs/athena#18 and #19), the way the BigQuery adapter hands copy_table to its driver. That keeps seeds, Hive drops, insert_overwrite and version expiry working with no new dependency on the Rust side, at the cost of needing a driver release before any of it runs from a released binary.

Your branch has fixes this PR needs whichever way the client question goes: case folding of quoted identifiers in the relation cache, the missing-table probe before SHOW CREATE TABLE, Column.quoted_hive. Would you open those as Part 7 on top of this one? I can run them against the same work group as the rest of the stack. And if you see a reason to prefer the SQL-only route, better to settle it here than to carry two Part 6s.

@fpiped
fpiped force-pushed the feat/athena-adapter-methods branch from 2de6773 to eebac86 Compare September 20, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Athena adapter for dbt Core 2.0

1 participant