Conversation
Athena tables are Glue entries over S3 prefixes and Athena SQL manages
neither, so dbt-athena drops tables, expires Glue versions, cleans
partitions and stages seeds through the AWS APIs. A client that only has
this driver had no way to do that with the connection's credentials.
Two statement options select an operation in place of the SQL text:
`athena.operation` names it, `athena.operation.payload` carries the JSON
form of the AWS API input; the result is one record with a utf8 `result`
column holding the JSON form of the API output. Paginated calls return all
pages, batch calls are chunked to the API limits (25 / 100 partitions,
1000 keys), a missing Glue table is `{"Table": null}` / `{"Deleted": false}`
rather than an error. Sixteen operations over Glue, S3, STS and the Athena
control plane; the SDK clients are built from the database's existing AWS
config on first use and mockable like the Athena client.
Adds aws-sdk-go-v2 service/glue, service/s3 and service/sts, which moves the
SDK core to v1.47.
This was referenced Sep 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18.
What
Two statement options select an AWS API call in place of the SQL text:
The payload is the JSON form of the AWS API input, the result the JSON form of the output; paginated calls return every page, batch calls are chunked (25 / 100 partitions, 1000 keys), a missing Glue table answers
{"Table": null}/{"Deleted": false}instead of failing.ExecuteUpdateruns the operation and returns 0. Unknown operations and malformed payloads areInvalidArgument; AWS errors areIO, like SQL failures. The Glue / S3 / STS clients are built from the database's existing AWS config on first use and are mockable likeathenaClientAPI. The operation table is in the README.Why
See #18: dbt-athena's materializations manage Glue and S3 directly, and dbt Core 2.0 reaches AWS only through this driver.
Testing
make test-unit(15 new tests with mock Glue / S3 / STS clients: option handling, each operation, pagination, chunking, not-found and error mapping) andmake vet. Exercised end to end through the engine side (dbt-labs/dbt#16376) against a real work group: seeds, views, tables, incremental models and snapshots, with the Glue and S3 side effects checked.go getof the three service modules moved theaws-sdk-go-v2core to v1.47 andsmithy-goto v1.28.go mod tidyfails onmainas well (thedriverbase-go/testutilpseudo-version), sogo.sumcarries only whatgo getadded.