D L Narayana D-L-Narayana

👨‍💻 About Me

name        : D L Narayana
location    : Visakhapatnam, India 🇮🇳
education   : B.Tech Computer Science & Engineering · GITAM · Class of 2027 · CGPA 7.97/10
role        : Data Engineer · Full-Stack Developer · AI Product Engineer
focus       : Production-grade data pipelines and web products — real-time CDC lakehouses, batch warehouses,
              analytics dashboards, and apps with LLMs built in
data        : Apache Spark (PySpark · Spark SQL · Structured Streaming) · Apache Kafka · Debezium (CDC) · Apache Airflow
              Parquet lakehouse (Bronze/Silver/Gold) · Star schema & SCD Type 2 · Data-quality gates & quarantine
              PostgreSQL · MongoDB · Docker Compose · pytest · JSON structured logging & pipeline metrics
stack       : TypeScript · React · Next.js · Node.js · Python · Java · SQL · Prisma · PostgreSQL · Tailwind
ai          : LLM integration · RAG pipelines · embeddings & vector search · AI agents · evals & guardrails
fundamentals: Data Structures & Algorithms · OOP · DBMS · Operating Systems · Computer Networks · System Design
exploring   : Databricks & Delta Lake · Snowflake · Hadoop / Hive · Kafka Streams · Data Contracts
principle   : idempotent pipelines, explicit schemas, tests for every transform, metrics for every run

🚀 What I build

End-to-end data pipelines — CDC ingestion from Postgres through Debezium and Kafka into Spark Structured Streaming, batch ETL into star-schema warehouses with SCD Type 2 history, data-quality gates, Airflow orchestration and observable, idempotent jobs. Plus end-to-end products: authentication, REST APIs, relational data models and polished front-ends, increasingly with AI at the core.

💬 Ask me about

Spark internals (lazy evaluation, shuffles, broadcast joins, foreachBatch, checkpoints) · Kafka & CDC semantics · dimensional modeling & SCD2 · data-quality strategy · SQL window functions · Next.js & React architecture · shipping LLM features to production — RAG, agents, evals · data visualization with Recharts.

🧰 Tech Stack

⭐ Featured Projects

🛠️ Data Engineering

Project	What it does	Stack	Verified results	Links
🌊 LakeFlow — Real-Time CDC Lakehouse Pipeline	Streams PostgreSQL inserts/updates/deletes through Debezium and Kafka into Spark Structured Streaming; lands a Bronze → Silver → Gold Parquet lakehouse with idempotent upsert/delete merges, SCD Type 2 history, rule-based data-quality quarantine, batch backfill and an Airflow DAG	PySpark Structured Streaming · Kafka (KRaft) · Debezium · PostgreSQL · Parquet · Airflow · Docker Compose · pytest	510,663 CDC events → Gold in 29.5 s (~17K events/s) · 5,239 rows quarantined · exactly-once checkpointed sinks · 7 tests	Repo
🏬 Retail Lakehouse ETL — Batch Data Warehouse	Schema-enforced PySpark ETL over 1M+ semi-structured sales lines: corrupt-record capture, schema-drift detection, harmonization, window-function dedupe, an 8-rule DQ gate, a star-schema warehouse (SCD2 customer dimension with point-in-time joins, Hive-partitioned fact), Spark SQL marts and a MongoDB serving layer	PySpark · Spark SQL · Parquet · MongoDB · Airflow · pytest	1,009,989 rows in 50.9 s (~20K rows/s) · 981 corrupt lines caught · 9,993 duplicates removed · 1.3 % quarantined with per-rule breakdown · 5 tests	Repo

🧩 Full-Stack Applications

Project	What it does	Stack	Live
🏙️ CityHelp	Civic platform — report city issues, upvote, track on a real-time dashboard & map, analytics on 7-day trends, category mix and department workload	Next.js · Supabase Realtime (PostgreSQL) · Recharts	↗
🏠 StayNest	Airbnb-style booking marketplace — search & filters, wishlists, booking flow with live price breakdown, host analytics (revenue, occupancy, ratings) on a normalized PostgreSQL schema	Next.js · TypeScript · PostgreSQL (Supabase) · Tailwind	↗
🔎 GitHubLens	Analyze any GitHub profile — stars, languages, top repos & a developer score	Next.js · GitHub API · Recharts	↗
🎮 CodeRunner	A code-typing game — race the clock on real snippets, build combos, climb the leaderboard	Next.js · React · Motion	↗

🎨 Frontend & Tooling

Project	What it does	Stack	Live
📄 ResumeForge	Privacy-first career toolkit — live resume builder, real-time ATS scoring, PDF scanner, cover letters; everything runs in the browser	Vanilla JS · HTML · CSS	↗
📊 AlgoViz	DSA visualizer — 6 sorting + 4 pathfinding algorithms with live metrics	React · Vite · Motion	↗
🔐 CryptoLab	Cryptography playground — classical ciphers + real Web Crypto (AES-GCM, RSA, hashing)	React · Vite · Web Crypto	↗

How the data pipelines are built (design principles)

Bronze is immutable and schema-agnostic — raw events are appended once with checkpointing; typing and business rules live downstream where they can be versioned and replayed.
Idempotency over promises — merges use latest-per-key on the WAL commit order (ts_ms, lsn), so replaying offsets or re-running a backfill yields identical tables.
Deletes and history are first-class — REPLICA IDENTITY FULL before-images drive deletes; SCD Type 2 versions carry effective_from / effective_to / is_current.
Quality is a gate, not a filter — failing rows go to a quarantine table with the rule names; runs and DAGs fail when the share crosses a threshold.
Everything observable — JSON structured logs (ELK-ready), StreamingQueryListener throughput/latency, per-stage timings and run metrics files.
Same code for stream and batch — one set of transforms powers foreachBatch streaming and nightly backfills, so one test suite covers both.

🤝 Let's Connect

"Build things that matter. Ship things that work."

Navigation Menu