lakehouse-platform-starter

Production-grade lakehouse reference architecture — runnable, tested, and interview-ready

Live dbt docs · Interview walkthrough · Portfolio · Open Airflow PR #70185

---

Airflow + Cosmos → dbt → Iceberg → OpenLineage → Great Expectations

Thin orchestration, observable pipelines, incremental marts, and backfill-safe design — built to demonstrate senior data engineering execution, not slide-deck architecture.

Table of contents

• Highlights
• Architecture
• Quick start
• Implementation status
• OSS contributions
• Repo layout
• Documentation

Highlights

Two runnable paths	DuckDB for fast CI/local · Trino + Iceberg for credible lakehouse demo
Cosmos orchestration	Per-model Airflow tasks with virtualenv isolation
14 dbt tests	Schema, singular, and incremental mart with partition keys
Quality gate	Great Expectations blocks publish on mart validation failure
Full local stack	MinIO · Iceberg REST · Trino · Airflow · Marquez in Docker Compose
IaC + CI	Terraform bronze module · GitHub Actions · hosted dbt docs

Architecture

flowchart LR
    subgraph ingest [Ingestion]
        A[PyIceberg ingest] --> B[Iceberg bronze.events]
    end
    subgraph transform [Transform]
        B --> T[Trino]
        T --> C[Cosmos dbt TaskGroup]
        B -.->|dev/CI| C2[DuckDB fast path]
    end
    subgraph orchestrate [Orchestration]
        D[Airflow DAG] --> A
        D --> C
        D --> E[Quality gate]
    end
    subgraph observe [Observability]
        D --> F[OpenLineage → Marquez]
        E --> H[Great Expectations]
    end
    C --> I[fct_daily_events mart]

Loading

Design principles

1. Thin orchestration — Airflow schedules and observes; dbt owns transform logic.
2. OpenLineage as contract — every task emits lineage; dbt Cloud jobs are first-class RUN parents.
3. Backfill safety — idempotent DAGs, partition keys, incremental marts, documented runbooks.

See ARCHITECTURE.md and docs/decisions/.

Quick start

Entry point: pip install -r requirements.txt then make pipeline (or make help for all targets).

Path	Command	Time
Fast (no Docker)	make pipeline	~30s
Full lakehouse	docker compose up -d → trigger lakehouse_daily	~2 min
Iceberg CLI	make pipeline-iceberg (stack must be up)	~1 min
Smoke test	make test	~30s

Windows: use .\scripts\pipeline.ps1 or .\scripts\pipeline-iceberg.ps1 instead of make.

Environment variables (DBT_TARGET, DUCKDB_PATH, etc.) are documented in ARCHITECTURE.md.

Fast path — no Docker (~30 seconds)

pip install -r requirements.txt
make pipeline

Bronze ingest → dbt seed/build → Great Expectations → populated fct_daily_events mart.

Full lakehouse stack — Docker

docker compose up -d

Service	URL	Credentials
Airflow	http://localhost:8080	admin / admin
Marquez (lineage)	http://localhost:5000	—
Trino	http://localhost:8090	—
MinIO console	http://localhost:9001	admin / password
Iceberg REST	http://localhost:8181	—

Trigger DAG lakehouse_daily — uses DBT_TARGET=iceberg (PyIceberg → Trino → dbt-trino → Iceberg marts).

make pipeline-iceberg

Makefile targets

Run make help for the full list. Common targets:

Target	Description
pipeline	DuckDB path: ingest → dbt → GE
pipeline-iceberg	Trino/Iceberg path (Docker required)
test	End-to-end smoke test
docs	Generate local dbt docs
lint	dbt compile/parse + DAG syntax check
up / down	Start/stop Docker stack

Interview prep

Rehearse from docs/interview-walkthrough.md — 30-second pitch, demo script, Q&A, and trade-offs.

Implementation status

Component	Status	Notes
dbt transform (staging → int → marts)	✅	Tests, docs site, incremental mart
Local pipeline (ingest → dbt → GE)	✅	make pipeline
Iceberg REST + MinIO	✅	Docker Compose; PyIceberg ingest
dbt-trino on Iceberg	✅	Native path via Trino :8090
Airflow + Cosmos DbtTaskGroup	✅	Per-model tasks; virtualenv execution
OpenLineage + Marquez	✅	Docker Compose
Great Expectations gate	✅	Blocks publish on failure
CI + GitHub Pages dbt docs	✅	Hosted docs
Terraform (S3 + IAM)	✅	Bronze bucket module
Interview walkthrough	✅	docs/interview-walkthrough.md
Meltano ingestion	🔜	PyIceberg simulates bronze today
OpenTelemetry traces	🔜	Lineage via Marquez only
Helm / K8s deploy	🔜	See infra/terraform/

OSS contributions

Item	Link	Status
Flagship issue	apache/airflow#68661	RUN-level OpenLineage for dbt Cloud
Open PR	#70185	dbt Cloud job metadata on OL events
Quick win	#47160	Python 3.12 fork() fix

Playbook: oss/AIRFLOW_CONTRIBUTIONS.md

Repo layout

infra/          Terraform (S3 bronze bucket + IAM)
orchestration/  Airflow DAGs + Cosmos + OpenLineage
transform/      dbt project (staging → int → marts)
storage/        Iceberg catalog + Trino config + DuckDB warehouse
quality/        Great Expectations checkpoint
ingestion/      DuckDB + PyIceberg bronze ingest
tests/          End-to-end pipeline smoke test
docs/           ADRs, runbooks, interview walkthrough

Documentation

Doc	Purpose
CONTRIBUTING.md	Dev setup, PR guidelines
ARCHITECTURE.md	Stack, data flow, env vars
docs/interview-walkthrough.md	Interview demo script + Q&A
docs/decisions/	ADRs (Iceberg, thin orchestration, OpenLineage)
docs/runbooks/backfill-safety.md	Backfill checklist
dbt docs (hosted)	Model lineage + column docs

---

Author: Bobby Ray (br413) · Senior Data Engineer
Portfolio: br413.github.io · License: Apache 2.0

GitHub social preview (repo maintainers)

Export docs/assets/social-preview.svg to PNG (1280×640 recommended) as docs/assets/social-preview.png, then upload under Settings → General → Social preview. The SVG banner above is the source asset.