-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
290 lines (281 loc) · 18.7 KB
/
Copy pathindex.html
File metadata and controls
290 lines (281 loc) · 18.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="google-site-verification" content="tmuS6i3hrTbfIXr8W78PZeYxef0q2Nk73dkIH6csWcY">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Senior Data Engineer & Data Architect | br413</title>
<meta name="description" content="Senior Data Engineer building contract-driven data platforms where pipelines fail loudly, not silently — incremental ETL, medallion lakehouse architecture, and observability with Python, dbt, Airflow, and Terraform.">
<meta name="keywords" content="data engineer, data architect, cloud data platform, lakehouse, ETL pipeline, dbt, Airflow, Terraform, AWS, Snowflake, Databricks">
<meta name="author" content="br413">
<link rel="canonical" href="https://br413.github.io/">
<link rel="sitemap" type="application/xml" title="Sitemap" href="https://br413.github.io/sitemap.xml">
<meta property="og:title" content="Senior Data Engineer & Data Architect | br413">
<meta property="og:description" content="Production data platforms with incremental pipelines, contract-driven quality checks, and lakehouse patterns that survive failure and schema drift.">
<meta property="og:type" content="website">
<meta property="og:url" content="https://br413.github.io/">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Senior Data Engineer & Data Architect | br413">
<meta name="twitter:description" content="Production data platforms with incremental pipelines, contract-driven quality, and lakehouse patterns built for operational scale.">
<link rel="stylesheet" href="styles.css">
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Person",
"name": "Bobby Ray",
"alternateName": "br413",
"jobTitle": "Senior Data Engineer & Data Architect",
"url": "https://br413.github.io/",
"sameAs": [
"https://github.com/br413",
"https://dev.to/bobby_ray_581732c715283b2"
],
"knowsAbout": ["Data Engineering", "Data Architecture", "Cloud Data Platforms", "dbt", "Apache Airflow", "Lakehouse", "ETL", "Terraform"]
}
</script>
</head>
<body>
<header class="site-header">
<nav aria-label="Primary">
<a class="logo" href="#">br413</a>
<ul class="nav-links">
<li><a href="#about">About</a></li>
<li><a href="#platform">Platform</a></li>
<li><a href="#projects">Projects</a></li>
<li><a href="#oss">Open Source</a></li>
<li><a href="#writing">Writing</a></li>
<li><a href="#contact">Contact</a></li>
</ul>
</nav>
</header>
<main>
<section class="hero" id="top">
<p class="eyebrow">Senior Data Engineer & Data Architect</p>
<h1>Data platforms where pipelines fail loudly — not silently</h1>
<p class="lead">
I build production-grade systems that survive partial failures, schema drift, and operational scale:
incremental ingestion with checkpoint recovery, contract-driven data quality, medallion lakehouse
patterns, and CI-validated platform automation on AWS, Azure, Databricks, and Snowflake.
</p>
<ul class="hero-highlights">
<li><strong>End-to-end ownership</strong> — ingestion, transformation, orchestration, and observability</li>
<li><strong>Operational maturity</strong> — idempotent loads, run history, alerts, and documented recovery</li>
<li><strong>Platform thinking</strong> — IaC, governance, lineage, and tests that catch regressions before production</li>
</ul>
<div class="hero-actions">
<a class="btn btn-primary" href="https://github.com/br413/lakehouse-platform-starter">View flagship project</a>
<a class="btn btn-secondary" href="https://br413.github.io/lakehouse-platform-starter/">dbt docs (live)</a>
<a class="btn btn-secondary" href="#platform">How I design platforms</a>
</div>
</section>
<section class="principles">
<blockquote>
Design for failure. Automate repeatable work. Measure data quality.
Document decisions. Keep systems understandable.
</blockquote>
</section>
<section id="about" class="section">
<h2>About</h2>
<p>
I'm <strong>Bobby Ray</strong> (<a href="https://github.com/br413">br413</a>) — a Senior Data Engineer and Data Architect
who turns fragmented sources into <strong>reliable, observable, analytics-ready platforms</strong>.
Senior data engineering, to me, is less about tool count and more about <strong>judgment under constraint</strong>:
choosing trade-offs intentionally, designing for failure before the first incident, and keeping systems
understandable for the team that inherits them.
</p>
<p>
My open-source work spans connected layers — a **runnable lakehouse platform** (Iceberg, Trino, Cosmos, OpenLineage),
a production ETL pipeline with Airflow and dbt, contract-driven quality observability, and platform IaC blueprints.
Together they show how I think about the full data lifecycle, not isolated scripts.
</p>
<div class="signals-grid">
<article class="signal-card">
<h3>System design</h3>
<p>Architecture diagrams, ADRs, and explicit boundaries between ingestion, transformation, quality, and platform layers.</p>
</article>
<article class="signal-card">
<h3>Failure handling</h3>
<p>Checkpoint recovery, idempotent loads, quality gates before bronze, and runbooks for 2 AM incidents.</p>
</article>
<article class="signal-card">
<h3>Operational proof</h3>
<p>pytest coverage, GitHub Actions CI, smoke tests, alert routing, and documented trade-offs in every flagship repo.</p>
</article>
</div>
<div class="skills-grid">
<article class="skill-card">
<h3>Data engineering</h3>
<p>Python, SQL, incremental ingestion, ETL/ELT, API pipelines, checkpointing, idempotent loads</p>
</article>
<article class="skill-card">
<h3>Data architecture</h3>
<p>Medallion lakehouse, bronze/silver/gold layers, lineage, governance, cost modeling</p>
</article>
<article class="skill-card">
<h3>Orchestration</h3>
<p>Apache Airflow, dbt, Prefect, Spark</p>
</article>
<article class="skill-card">
<h3>Cloud & quality</h3>
<p>AWS, Azure, Databricks, Snowflake, data contracts, CI/CD, monitoring, alerting</p>
</article>
</div>
</section>
<section id="platform" class="section">
<h2>Platform approach</h2>
<p class="section-intro">
Senior data platform work is a stack, not a single pipeline. My portfolio models how ingestion,
quality, and platform governance connect in production.
</p>
<div class="platform-flow">
<div class="platform-step">
<span class="platform-label">Flagship</span>
<h3>Lakehouse platform</h3>
<p><a href="https://github.com/br413/lakehouse-platform-starter">lakehouse-platform-starter</a></p>
<p class="platform-detail">Iceberg + Trino + Cosmos dbt + Airflow + OpenLineage + GE — runnable Docker stack, CI, hosted dbt docs.</p>
</div>
<div class="platform-step">
<span class="platform-label">Layer 1</span>
<h3>Ingest & transform</h3>
<p><a href="https://github.com/br413/production-data-pipeline">production-data-pipeline</a></p>
<p class="platform-detail">Incremental API ingestion, PostgreSQL bronze, quarantine/DLQ for bad records, dbt silver/gold, Airflow orchestration, webhook alerts.</p>
</div>
<div class="platform-step">
<span class="platform-label">Layer 2</span>
<h3>Quality & observability</h3>
<p><a href="https://github.com/br413/data-quality-observability">data-quality-observability</a></p>
<p class="platform-detail">YAML data contracts, schema/freshness/RI checks, persisted run history, alert routing.</p>
</div>
<div class="platform-step">
<span class="platform-label">Layer 3</span>
<h3>Platform & governance</h3>
<p><a href="https://github.com/br413/cloud-lakehouse-blueprint">cloud-lakehouse-blueprint</a></p>
<p class="platform-detail">Medallion manifests, Terraform IaC, IAM governance, lineage graph, CI validation.</p>
</div>
</div>
<p class="platform-outcome">
<strong>Outcome:</strong> pipelines that fail loudly, contracts that catch drift early, and platform patterns
that teams can operate without tribal knowledge.
</p>
</section>
<section id="projects" class="section">
<h2>Data engineering portfolio</h2>
<p class="section-intro">Production-style open-source projects with architecture docs, ADRs, tests, CI, and operations runbooks.</p>
<div class="project-grid">
<article class="project-card featured">
<p class="project-tier">Flagship · Runnable lakehouse platform</p>
<h3><a href="https://github.com/br413/lakehouse-platform-starter">lakehouse-platform-starter</a></h3>
<p>Airflow + Cosmos orchestrates PyIceberg ingest into Iceberg bronze, dbt-trino transforms via Trino, OpenLineage feeds Marquez, and Great Expectations gates publish.</p>
<p class="project-signal"><strong>Senior signal:</strong> dual DuckDB/Trino targets, 14 dbt tests, incremental marts, Terraform IaC, CI smoke tests, and <a href="https://br413.github.io/lakehouse-platform-starter/">hosted dbt docs</a>.</p>
<ul class="tags">
<li>Iceberg</li><li>Trino</li><li>dbt</li><li>Airflow</li><li>Cosmos</li><li>OpenLineage</li>
</ul>
<a class="project-link" href="https://github.com/br413/lakehouse-platform-starter/releases/tag/v1.0.0">v1.0.0 release →</a>
</article>
<article class="project-card">
<p class="project-tier">End-to-end pipeline</p>
<h3><a href="https://github.com/br413/production-data-pipeline">production-data-pipeline</a></h3>
<p>Incremental API ingestion with PostgreSQL bronze landing, per-record quarantine/DLQ, dbt silver/gold models, Airflow orchestration, and webhook alerts.</p>
<p class="project-signal"><strong>Senior signal:</strong> checkpoint recovery, idempotent loads, quarantine table for poison-pill triage, and dbt test gates that stop bad data before gold.</p>
<ul class="tags">
<li>Python</li><li>dbt</li><li>Airflow</li><li>PostgreSQL</li><li>ETL</li>
</ul>
<a class="project-link" href="https://github.com/br413/production-data-pipeline/releases/tag/v0.2.1">v0.2.1 release →</a>
</article>
<article class="project-card">
<p class="project-tier">Quality layer · Contracts & observability</p>
<h3><a href="https://github.com/br413/data-quality-observability">data-quality-observability</a></h3>
<p>YAML data contracts with schema, freshness, and referential-integrity checks, persisted history, and alert routing.</p>
<p class="project-signal"><strong>Senior signal:</strong> contract-driven checks with run history — quality regressions are auditable, not one-off surprises.</p>
<ul class="tags">
<li>Data contracts</li><li>Observability</li><li>Airflow</li><li>Python</li>
</ul>
<a class="project-link" href="https://github.com/br413/data-quality-observability">View repository →</a>
</article>
<article class="project-card">
<p class="project-tier">Platform layer · Lakehouse blueprint</p>
<h3><a href="https://github.com/br413/cloud-lakehouse-blueprint">cloud-lakehouse-blueprint</a></h3>
<p>Medallion architecture manifests, Terraform IaC, IAM governance, lineage graph, and CI validation with <code>validate --json</code>.</p>
<p class="project-signal"><strong>Senior signal:</strong> platform decisions encoded in IaC, manifests, and CI — not slide-deck architecture.</p>
<ul class="tags">
<li>Lakehouse</li><li>Terraform</li><li>AWS</li><li>Data architecture</li>
</ul>
<a class="project-link" href="https://github.com/br413/cloud-lakehouse-blueprint">View repository →</a>
</article>
</div>
</section>
<section id="oss" class="section">
<h2>Open-source contributions</h2>
<p class="section-intro">Production operations knowledge contributed upstream to orchestration and analytics tooling teams rely on daily.</p>
<div class="oss-list">
<article class="oss-item">
<h3><a href="https://github.com/dbt-labs/docs.getdbt.com/pull/9781">dbt docs #9781</a> <span class="badge open">open</span></h3>
<p>Fusion telemetry: use <code>duration_ms</code> for slowest-nodes ranking — fixes misleading span wall-clock examples.</p>
</article>
<article class="oss-item">
<h3><a href="https://github.com/meltano/meltano/pull/10253">Meltano #10253</a> <span class="badge open">open</span></h3>
<p><code>elt</code> vs <code>run</code> decision guide for replication workloads — closes maintainer-requested docs gap.</p>
</article>
<article class="oss-item">
<h3><a href="https://github.com/apache/airflow/pull/71158">Airflow #71158</a> <span class="badge merged">merged</span></h3>
<p>Clarify metrics vs traces <code>otel_*</code> config options — merged upstream.</p>
</article>
<article class="oss-item">
<h3><a href="https://github.com/apache/airflow/pull/70171">Airflow #70171</a> <span class="badge open">open</span></h3>
<p>Surface dbt Cloud failure details in Airflow task logs for faster on-call triage.</p>
</article>
<article class="oss-item">
<h3><a href="https://github.com/PrefectHQ/prefect/pull/22500">Prefect #22500</a> <span class="badge merged">merged</span></h3>
<p>Kubernetes readiness vs liveness probes for workflow orchestrators — the kind of ops detail that prevents false-positive deploy rollbacks.</p>
</article>
<article class="oss-item">
<h3><a href="https://github.com/PrefectHQ/prefect/pull/22533">Prefect #22533</a> <span class="badge open">open</span></h3>
<p>Clarify global concurrency limit setup for production Prefect deployments.</p>
</article>
<article class="oss-item">
<h3><a href="https://github.com/dbt-labs/docs.getdbt.com/pull/9606">dbt docs #9606</a> <span class="badge merged">merged</span></h3>
<p>Troubleshooting note for prefixed custom schema names — a deployment pitfall that breaks analytics environments silently.</p>
</article>
</div>
</section>
<section id="writing" class="section">
<h2>Writing</h2>
<p class="section-intro">Technical articles on production pipeline patterns, failure handling, and platform design.</p>
<article class="oss-item">
<h3><a href="https://dev.to/bobby_ray_581732c715283b2/building-a-production-data-pipeline-with-incremental-loading-and-dbt-2e2c">Building a Production Data Pipeline with Incremental Loading and dbt</a></h3>
<p>How to design idempotent ingestion, checkpoint stores, medallion-style transformations, and Airflow orchestration — with failure modes and scale trade-offs.</p>
<p class="writing-repos">Related repos: <a href="https://github.com/br413/production-data-pipeline">production-data-pipeline</a> · <a href="https://github.com/br413/cloud-lakehouse-blueprint">cloud-lakehouse-blueprint</a> · <a href="https://github.com/br413/lakehouse-platform-starter">lakehouse-platform-starter</a></p>
</article>
<article class="oss-item">
<h3><a href="https://dev.to/bobby_ray_581732c715283b2/data-quality-contracts-in-production-pipelines-without-a-separate-platform-team-f3">Data Quality Contracts in Production Pipelines (Without a Separate Platform Team)</a></h3>
<p>Row-level quarantine at ingestion, YAML dataset contracts at the quality boundary, alert routing, and CI enforcement — the follow-up to the incremental pipeline article.</p>
<p class="writing-repos">Related repos: <a href="https://github.com/br413/production-data-pipeline">production-data-pipeline</a> · <a href="https://github.com/br413/data-quality-observability">data-quality-observability</a></p>
</article>
<article class="oss-item">
<h3><a href="https://dev.to/bobby_ray_581732c715283b2/what-i-learned-contributing-to-prefect-dbt-and-airflow-an-honest-oss-retrospective-1ki8">What I Learned Contributing to Prefect, dbt, and Airflow (An Honest OSS Retrospective)</a></h3>
<p>Two merges, four open PRs, and ninety days of building in public — what actually worked for upstream contributions.</p>
<p class="writing-repos">Related repos: <a href="https://github.com/br413/br413">br413</a> · <a href="https://github.com/br413/production-data-pipeline">production-data-pipeline</a> · <a href="https://github.com/br413/data-quality-observability">data-quality-observability</a> · <a href="https://github.com/br413/cloud-lakehouse-blueprint">cloud-lakehouse-blueprint</a></p>
</article>
<article class="oss-item">
<h3><a href="https://dev.to/bobby_ray_581732c715283b2/contract-versioning-in-production-pipelines-registry-cli-and-run-history-13el">Contract Versioning in Production Pipelines: Registry, CLI, and Run History</a></h3>
<p>Git-native contract registry, semver pins, versioned run history, and CI guards — the follow-up to the data quality contracts article.</p>
<p class="writing-repos">Related repos: <a href="https://github.com/br413/data-quality-observability">data-quality-observability</a> · <a href="https://github.com/br413/production-data-pipeline">production-data-pipeline</a></p>
</article>
</section>
<section id="contact" class="section contact">
<h2>Collaboration</h2>
<p>Open to senior data engineering roles, data platform architecture discussions, and technical collaboration with engineering teams building analytics infrastructure, quality tooling, or lakehouse platforms.</p>
<ul class="contact-links">
<li><a href="https://github.com/br413">GitHub</a></li>
<li><a href="https://dev.to/bobby_ray_581732c715283b2">Dev.to</a></li>
<li><a href="https://github.com/br413/lakehouse-platform-starter">Flagship repo</a></li>
<li><a href="https://br413.github.io/lakehouse-platform-starter/">dbt docs (live)</a></li>
<li><a href="https://github.com/br413/br413/discussions">Discussions</a></li>
</ul>
</section>
</main>
<footer class="site-footer">
<p>© 2026 Bobby Ray (br413) · Senior Data Engineer & Data Architect</p>
</footer>
</body>
</html>