Skip to content

Bootstrap missing tables for incremental materializations - #2516

Open
albertobruin wants to merge 4 commits into
mainfrom
albertobruin/incremental-create-table
Open

Bootstrap missing tables for incremental materializations#2516
albertobruin wants to merge 4 commits into
mainfrom
albertobruin/incremental-create-table

Conversation

@albertobruin

@albertobruin albertobruin commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

What

Incremental merge, delete+insert, and time_interval materializations now create an empty target from the asset query schema when the target does not exist.

Changes

  • Shared empty-CTAS helper plus dialect-specific implementations across supported SQL platforms
  • Athena catalog check and hook-aware initialization, Oracle dynamic PL/SQL, and Redshift LIKE staging
  • BigQuery bootstrap tables preserve partitioning, clustering, require_partition_filter, and partition_expiration_days
  • Cross-platform regression tests and materialization documentation

How to test

  1. make format
  2. make test
  3. make build-no-duckdb

Risk & rollback

  • Risk: Medium — generated SQL changes across many warehouse dialects, with unit coverage but no live warehouse integration run.
  • Rollback: Revert the merge commit.

Checklist

  • Unit tests added or updated (make test)
  • Builds and lint pass (make build-no-duckdb, make format)
  • Integration tests pass if affected (make integration-test)
  • No unrelated changes included
  • Tested locally

Security

  • No secrets, credentials, or PII in this change
  • No new dependencies with known vulnerabilities
  • Security implications considered

Related issues

N/A

@greptile-apps

greptile-apps Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Fix All in Conductor

Prompt To Fix All With AI
### Issue 1
pkg/clickhouse/materialization.go:83-88
**Bootstrap drops ClickHouse partitioning**

When a `delete+insert`, `merge`, or `time_interval` asset with `partition_by` runs before its target exists, the bootstrap DDL creates the table without the configured partition key. Since subsequent runs skip `CREATE TABLE IF NOT EXISTS`, the target remains permanently unpartitioned until it is manually rebuilt.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "feat: bootstrap incremental materializat..." | Re-trigger Greptile

Comment thread pkg/clickhouse/materialization.go
@greptile-apps

greptile-apps Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Reviews (2): Last reviewed commit: "chore: satisfy Oracle lint" | Re-trigger Greptile

@albertobruin
albertobruin force-pushed the albertobruin/incremental-create-table branch from 03d6ca5 to d7403b9 Compare August 5, 2026 20:18
@greptile-apps

greptile-apps Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Reviews (3): Last reviewed commit: "chore: satisfy Oracle lint" | Re-trigger Greptile

@albertobruin
albertobruin force-pushed the albertobruin/incremental-create-table branch from d7403b9 to 3a283aa Compare August 6, 2026 13:16
@greptile-apps

greptile-apps Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Reviews (4): Last reviewed commit: "chore: satisfy Oracle lint" | Re-trigger Greptile

@greptile-apps

greptile-apps Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Reviews (5): Last reviewed commit: "fix: preserve BigQuery bootstrap options" | Re-trigger Greptile

@crazy-treyn

Copy link
Copy Markdown
Contributor

This is going to be nice, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants