Skip to content

Catalog-aware relational SQL and lake query mode - #145

Open
ajroetker wants to merge 1875 commits into
mainfrom
combine-pr-141-143-144
Open

Catalog-aware relational SQL and lake query mode#145
ajroetker wants to merge 1875 commits into
mainfrom
combine-pr-141-143-144

Conversation

@ajroetker

@ajroetker ajroetker commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Combines #141, #143, and #144 into one integration branch, then extends that base into the current catalog-aware relational SQL and lake-query architecture. The branch now covers three connected design tracks: relational typed row execution, database/namespace/tablespace catalog semantics, and external lake row sources over the same row-plan API.

What This PR Adds

Relational SQL and typed row APIs

  • Lands relational mode as a table profile on the existing Antfly engine: closed schemas, typed cells, relational base-row storage, typed column entries, and derived full-text/vector/sparse/graph/algebraic indexes over that base store.
  • Expands the public row-plan surface for rows/query, rows/batch, rows/aggregate, rows/window, rows/join, rows/lateral, and reusable plan envelopes with CTE metadata and result-schema reporting.
  • Broadens SQL lowering into native typed plans rather than backend SQL text: DDL, row mutations, CTEs, joins, lateral plans, aggregates, windows, row claims, conflict actions, RETURNING, JSON/array/text/datetime/regex expressions, temporal DML, and schema migration metadata.
  • Adds stronger relational constraint coverage: primary/unique metadata, covering payload columns, foreign-key workflows, partial/expression indexes, check/default/generated-column metadata, row-policy/auth DDL, and typed migration/rebuild intent.
  • Splits the SQL adapter into parser/grammar/binder/plan modules and backs it with source fixtures plus parity fingerprints so unsupported SQL remains explicit and fail-closed.

Databases, namespaces, tablespaces, and auth

  • Defines database / namespace / table as the durable catalog model, with /tables/{table} remaining shorthand for default.public.{table}.
  • Adds explicit catalog route semantics and OpenAPI-backed operations for database, namespace, table, table I/O, indexes, backup/restore, documents, and tablespaces.
  • Routes REST, SQL, MCP, A2A, CLI, backup/restore, foreign-source query, and role checks through typed catalog targets instead of compatibility table strings.
  • Integrates qualified resources into the role model: database:name, namespace:db.ns, table:db.ns.table, and tablespace:name, preserving default-public compatibility where needed.
  • Adds SQL catalog/session planning for current_database, search_path, SET/RESET/SHOW search_path, and DISCARD ALL, with session mutation modeled explicitly rather than hidden route rewriting.
  • Makes tablespaces first-class metadata-backed catalog resources with SQL/OpenAPI lifecycle, durable table/database/namespace bindings, rename propagation, dependency checks, and fail-closed placement policy validation.

Lake query mode and serverless row sources

  • Documents and implements the first production shape for lake tables as relational tables with an external base_source, not a separate query system.
  • Adds external lake catalog bindings, pinned source snapshots, serverless manifest base sources, external inventory publication, and resolved inventory helpers.
  • Builds the lake row-source path around object-storage range reads, Parquet footer/page metadata, row-group planning, typed residual filters, aggregate/query routing, and stable external row refs.
  • Supports a growing Parquet scanner surface: flat numeric/text/boolean/timestamp/decimal paths, dictionary/plain pages, nullable columns, Snappy/gzip/Zstd payloads, row-group min/max pruning, object-range coalescing, and cache-keyed reads.
  • Adds Iceberg metadata/inventory scaffolding through object storage: snapshot metadata reads, manifest-list/data/delete manifest planning, partition/stat pruning, cache reuse, and validation.
  • Adds lake sidecar scaffolding for text, sparse, dense, graph, and algebraic materializations, with pinned source validation, candidate hydration, automatic sidecar selection, rebuild/reconcile planning, and explain/cache diagnostics.

Docs Updated

  • zig/RELATIONAL.md is now the canonical relational-mode and SQL parity design, including the typed row-plan contract, constraint/migration semantics, read/write execution, lake materialization relationship, and implementation state.
  • zig/DATABASES.md defines the database/namespace/table mental model, defaulting rules, explicit catalog APIs, tablespace semantics, role-resource mapping, and cross-surface integration for REST, SQL, MCP, A2A, CLI, and auth.
  • zig/LAKES.md defines lake query mode as a relational row source over Parquet/Iceberg/Lance-style data, with serverless manifests, object-store reads, sidecars, cache isolation, explain/rebuild workflows, and remaining engine work.

Remaining Work Called Out In The Docs

  • Finish the efficient typed ColumnBatch hot path so lake scans do not rely on JSON adaptation for broad query execution.
  • Broaden Parquet/Iceberg support for nested/repeated data, page-index pruning, exact large decimals, delete semantics, schema evolution, and broader S3/GCS fixture coverage.
  • Add persistent cache storage and object-store checksum provenance beyond the current in-memory range-cache validation.
  • Wire lake rebuild/reconcile and snapshot-retention/GC workflows into real operator commands/controllers.
  • Teach future tablespace schedulers to consume policy fields beyond the currently supported native placement keys.

Testing

  • zig build
  • git diff --check

Earlier focused gates on this branch also covered generated OpenAPI freshness, SQL/API parity fixtures, catalog/session SQL fixtures, explicit catalog route wiring, MCP/A2A catalog normalization, CLI surface compilation, and qualified role-resource checks.

@ajroetker ajroetker changed the title Combine algebraic HLL, relational mode, and dynamic templates Algebraic HyperLogLog, relational mode for storage, and dynamic templates for algebraic indexes Jun 1, 2026
@ajroetker ajroetker changed the title Algebraic HyperLogLog, relational mode for storage, and dynamic templates for algebraic indexes Combine algebraic HLL, relational storage, and dynamic-template indexes Jun 1, 2026
@ajroetker ajroetker changed the title Combine algebraic HLL, relational storage, and dynamic-template indexes Relational tables and Eigenvector Centrality Metrics Jun 10, 2026
@ajroetker ajroetker changed the title Relational tables and Eigenvector Centrality Metrics Catalog-aware relational SQL and lake query mode Jun 18, 2026
@ajroetker
ajroetker force-pushed the combine-pr-141-143-144 branch 3 times, most recently from 31c5dcb to bd5308e Compare June 26, 2026 00:31
@ajroetker
ajroetker force-pushed the combine-pr-141-143-144 branch from d274de9 to 5897ac2 Compare June 26, 2026 18:01
ajroetker and others added 30 commits June 28, 2026 19:01
# Conflicts:
#	zig/build.zig
#	zig/pkg/antfly/src/api/table_writes.zig
#	zig/pkg/antfly/src/common/mod.zig
#	zig/pkg/antfly/src/metadata/service.zig
#	zig/pkg/antfly/src/storage/db/db.zig
# Conflicts:
#	zig/pkg/antfly/src/api/http_server.zig
#	zig/pkg/antfly/src/api/table_reads.zig
#	zig/pkg/antfly/src/api/table_writes.zig
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant