[EPIC] Upstream BigQuery driver patches to ADBC Foundry

[serramatutu]

Why

We don't want to drift too much from the upstream because:

• We want to easily bring in new driver features, patches and bug fixes without many merge conflicts
• We don't want to maintain duplicate implementations of features that exist both in our fork and upstream

Upstreaming may require changing some code, adding validations, fixing pre-commit and overall cleanup, as their quality standards are generally higher than ours (oops).

End goal

By the end of this epic, our fork should contain minimal differences from upstream, except some dbt-specific hacks that we know won't get accepted upstream (those should be minimal).

Tasks

Done

• feat(go): add job link to error messages adbc-drivers/bigquery#278
• fix(go): respect empty values for impersonation delegates and scopes adbc-drivers/bigquery#277
• fix(go): sleep on the normal poll path in safeWaitForJob adbc-drivers/bigquery#276
• feat(go): list hidden datasets in AdbcConnectionGetObjects adbc-drivers/bigquery#275
• feat(go): add extra table metadata keys adbc-drivers/bigquery#274
• feat(go): add BIGQUERY:query_id to Arrow schema metadata adbc-drivers/bigquery#272

In-progress

• feat(go): support opting out of Read Storage API, with fallback to row-based iterator adbc-drivers/bigquery#280
  • Depends on: feat(go): support optional job mode with Arrow result format adbc-drivers/bigquery#303
• feat(go): map BIGQUERY:type Arrow metadata to BigQuery FieldType adbc-drivers/bigquery#273

TODO

(NOTE: The PRs show up as merged, but they're merged in our internal repo. I still have to replicate them against upstream).

• Before submitting the next batch of PRs, rebase our driver into all the DONE tasks from upstream. This will require some changes to keep backward compatibility:
  • Statistics keys got changed from CamelCase to snake_case upstream
  • Schema metadata in GetTableSchema got changed from CamelCase to snake_case
  • use_storage_api_disabled_client got renamed to disable_storage_api
• Port over the commits from the legacy bigquery go driver, rebase the latest upstream main bigquery-adbc#1
• feat(go): support integer pseudocolumns bigquery-adbc#2
• Port over the commits from the legacy bigquery go driver, rebase the latest upstream main bigquery-adbc#5
• fix(go/adbc/driver/bigquery): recover from 409 on duplicate job insert bigquery-adbc#13
• feat: port over honoring HTTPS_PROXY/NO_PROXY on token-exchange bigquery-adbc#15
• Port support for 147 to this repo bigquery-adbc#16
• Rebase everything again, keeping backwards compatibility even if some changes get requested upstream
• Remove the backward compatibility layer for old option/metadata names (adbc.bigquery.sql., CamelCase stuff), change dbt to use the new keys.
• Setup validations in our own fork

Out of scope

These will not get accepted upstream. We should come back to them later after the Metadata API revision in ADBC 1.2 (see the discussion in adbc-drivers/bigquery#279)

• Update column descriptions: dbt-labs/bigquery-adbc@38d5543
  • fix: update table columns descriptions race bigquery-adbc#10
• Update table descriptions: dbt-labs/bigquery-adbc@4452a26
• Update policy tags: dbt-labs/bigquery-adbc@908c93e

These PRs are dbt-specific do not need to be upstreamed:

• fix(go): resolve pre-existing lint, staticcheck, and RAT debt bigquery-adbc#7
• [Tracking] CI: Go Test auth step broken on main — BigQuery CI environment has no secrets configured bigquery-adbc#8

These PRs are backports of what already exists upstream, and when we rebase we should just delete the backport commits:

• chore: refresh ADBC build tooling bigquery-adbc#12
• feat(go): handle SCRIPT-type queries returning results (cherry-picked from upstream) bigquery-adbc#14
• feat(go): skip tables in GetObjects that 403 bigquery-adbc#17
• feat(go): fix record batches being dropped bigquery-adbc#18