Skip to content

feat(athena): Part 8 — run dbt unit tests on Athena - #16370

Open
aoelvp94 wants to merge 2 commits into
dbt-labs:mainfrom
aoelvp94:athena/part-8-unit-tests
Open

aoelvp94 wants to merge 2 commits into
dbt-labs:mainfrom
aoelvp94:athena/part-8-unit-tests

Conversation

@aoelvp94

@aoelvp94 aoelvp94 commented Sep 18, 2026

Copy link
Copy Markdown

Part of #16252. Related: #13822. Independent PR; the sequence is at the bottom.

Problem

Three Athena-specific gaps in the unit-test path, found by running a real project's 31 unit tests:

  1. Schema inference. The default path materialises an empty temporary table via CTAS and introspects it. On Athena a zero-row Hive CTAS yields a table with no accessible columns (COLUMN_NOT_FOUND: Relation contains no accessible columns), and the probe would land under s3_staging_dir/tables/.
  2. Type names. The Arrow-to-SQL table emitted Postgres spellings for Athena (text, float8, timestamp without time zone), which Trino rejects.
  3. Integer widths. The table collapses every integer width into one name per backend. Trino's integer is 32-bit, so bigint fixtures overflowed (NUMERIC_VALUE_OUT_OF_RANGE).

Solution

  • Athena joins DuckDB and ClickHouse on the query-schema path: the driver returns the result-set schema for ... where false limit 0.
  • sql_types: Athena maps to varchar, double, timestamp, in line with dbt-athena's convert_*_type; bigint / integer / smallint / tinyint are kept distinct.
  • get_field_sql_type_metadata_key Athena arm returns ATHENA:type, the key the driver emits.

Verification

  • cargo fmt --check, cargo clippy -p dbt-adapter, cargo nextest run -p dbt-adapter -E 'test(athena)' clean on the pinned toolchain.
  • On the real project: 15 of 31 unit tests pass; the remaining 16 fail for project reasons (an upstream table absent in that environment, and refs without a given fixture, which dbt-core resolves to the real relation and Fusion refuses). The latter is a documented Fusion/core difference, not raised here.

Feedback wanted: none beyond the code.

Sequence (independent PRs, each compiles alone against main)

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes.

Three Athena-specific gaps in the unit-test path, found by running a real
project's 31 unit tests against Athena:

- Schema inference: the default path materializes an empty temporary table
  via CTAS and introspects it. On Athena a zero-row Hive CTAS yields a table
  with no accessible columns (`COLUMN_NOT_FOUND: Relation contains no
  accessible columns`), and the probe would land under
  `s3_staging_dir/tables/`. Athena joins DuckDB and ClickHouse on the
  query-schema path: the driver returns the result-set schema for
  `... where false limit 0`.
- Type names: the Arrow-to-SQL conversion table emitted Postgres spellings
  for Athena (`text`, `float8`, `timestamp without time zone`), which Trino
  rejects. Athena now maps to `varchar`, `double`, `timestamp`, in line with
  dbt-athena's convert_*_type (whose `string` is a Hive DDL spelling that the
  vendored `ddl_data_type` macro produces where DDL needs it).
- Integer widths: the table collapses every integer width into one name per
  backend. Trino's `integer` is 32-bit, so bigint fixtures overflowed
  (`NUMERIC_VALUE_OUT_OF_RANGE`). Athena keeps `bigint` / `integer` /
  `smallint` / `tinyint`.

Also fills the Athena arm of `get_field_sql_type_metadata_key` with the
`ATHENA:type` key the driver emits (dbt-labs/athena#6).

Against the real project, 15 of 31 unit tests pass; the rest fail on an
upstream table that does not exist in that environment (10) and on refs
without a `given` fixture (6), which dbt-core resolves to the real relation
and Fusion refuses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant