Skip to content

feat(go): run Glue, S3 and STS operations behind statement options - #19

Open
fpiped wants to merge 1 commit into
dbt-labs:mainfrom
fpiped:feat/glue-operations
Open

fpiped wants to merge 1 commit into
dbt-labs:mainfrom
fpiped:feat/glue-operations

Conversation

@fpiped

@fpiped fpiped commented Sep 20, 2026

Copy link
Copy Markdown

Closes #18.

What

Two statement options select an AWS API call in place of the SQL text:

stmt.SetOption("athena.operation", "glue.get_table")
stmt.SetOption("athena.operation.payload", `{"DatabaseName": "analytics", "Name": "orders"}`)
rdr, _, err := stmt.ExecuteQuery(ctx) // one row, utf8 column `result`: {"Table": {...}}

The payload is the JSON form of the AWS API input, the result the JSON form of the output; paginated calls return every page, batch calls are chunked (25 / 100 partitions, 1000 keys), a missing Glue table answers {"Table": null} / {"Deleted": false} instead of failing. ExecuteUpdate runs the operation and returns 0. Unknown operations and malformed payloads are InvalidArgument; AWS errors are IO, like SQL failures. The Glue / S3 / STS clients are built from the database's existing AWS config on first use and are mockable like athenaClientAPI. The operation table is in the README.

Why

See #18: dbt-athena's materializations manage Glue and S3 directly, and dbt Core 2.0 reaches AWS only through this driver.

Testing

make test-unit (15 new tests with mock Glue / S3 / STS clients: option handling, each operation, pagination, chunking, not-found and error mapping) and make vet. Exercised end to end through the engine side (dbt-labs/dbt#16376) against a real work group: seeds, views, tables, incremental models and snapshots, with the Glue and S3 side effects checked.

go get of the three service modules moved the aws-sdk-go-v2 core to v1.47 and smithy-go to v1.28. go mod tidy fails on main as well (the driverbase-go/testutil pseudo-version), so go.sum carries only what go get added.

Athena tables are Glue entries over S3 prefixes and Athena SQL manages
neither, so dbt-athena drops tables, expires Glue versions, cleans
partitions and stages seeds through the AWS APIs. A client that only has
this driver had no way to do that with the connection's credentials.

Two statement options select an operation in place of the SQL text:
`athena.operation` names it, `athena.operation.payload` carries the JSON
form of the AWS API input; the result is one record with a utf8 `result`
column holding the JSON form of the API output. Paginated calls return all
pages, batch calls are chunked to the API limits (25 / 100 partitions,
1000 keys), a missing Glue table is `{"Table": null}` / `{"Deleted": false}`
rather than an error. Sixteen operations over Glue, S3, STS and the Athena
control plane; the SDK clients are built from the database's existing AWS
config on first use and mockable like the Athena client.

Adds aws-sdk-go-v2 service/glue, service/s3 and service/sts, which moves the
SDK core to v1.47.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose the Glue / S3 / STS calls dbt-athena's materializations need

1 participant