Rihan Bagwan the-iconic-rihan

👋 About me

I'm a software engineer with 3+ years of experience. For the last year I've focused on data engineering: Python, PySpark and Airflow pipelines that load 15–20 GB a day into ClickHouse, which I also deploy and run myself on Linux with Docker.

name: Rihan Bagwan
role: Data Engineer · DevOps & Linux
location: Mumbai, India
now: Software Development Engineer @ Skillmine Technology Consulting
building:
  - ETL/ELT pipelines from databases, REST APIs and CSV/XLSX/PDF into ClickHouse
  - PySpark batch jobs orchestrated by Airflow (8 DAGs)
  - async ingestion with Celery, Redis and DuckDB
care_about: [data quality, idempotent re-runs, pipelines that recover on their own]
preferred_locations:
  india: [Mumbai, Navi Mumbai, Pune, Hyderabad, Bangalore]
  relocation: [USA, Canada, UK, Ireland, Germany, Netherlands, rest of Europe,
               Singapore, Hong Kong, Japan, Australia, New Zealand]
  gulf: [UAE (Dubai, Abu Dhabi), Saudi Arabia (Riyadh), Qatar (Doha), Bahrain, Kuwait, Oman]
open_to: [Data Engineering, DevOps, Platform Engineering]

📈 Impact at a glance

15–20 GB

_{processed daily
into ClickHouse}

12+

_{production
ETL/ELT pipelines}

−40%

_{failed loads, via
retries & recovery}

−30%

_{rejected records,
via validation}

−25%

_{SQL query
latency}

⭐ Featured project

batch-data-pipeline

An Airflow pipeline that ingests ~1M rows a day, validates and deduplicates them with PySpark, quarantines bad rows with the reason they failed, and loads ClickHouse through an atomic partition swap, so re-running a day never creates duplicates.

✅ Data-quality gate: if more than 5% of a batch is bad, nothing is published
🔁 Idempotent, incremental loads: one run = one day, safe to re-run or backfill
📊 Every run's counts, errors and timings are recorded in a pipeline_runs table
🐳 One-command setup with Docker Compose · ⚙️ GitHub Actions CI with PySpark tests

flowchart LR
    A[Raw CSV] --> B[Airflow DAG]
    B --> C[PySpark<br/>validate + dedupe]
    C --> D[Partitioned<br/>Parquet]
    C --> Q[Quarantine<br/>bad rows + reasons]
    D --> E[(ClickHouse<br/>atomic partition swap)]

🧰 Tech stack

Languages	_{Python · SQL · Bash · YAML}
Data engineering	_{PySpark · Apache Airflow · Apache Kafka · RabbitMQ · Celery · Pandas · NumPy · DuckDB · Parquet · dbt} _{Concepts: ETL / ELT · Batch & streaming · CDC · Data modeling (star / snowflake) · Data quality · Incremental loads · Partitioning · Data lake / lakehouse}
Databases & storage	_{ClickHouse · PostgreSQL · MySQL · MongoDB · Elasticsearch · Redis · SQLite · DynamoDB}
Python backend	_{FastAPI · Django · Flask · SQLAlchemy · Pydantic · REST APIs · GraphQL · WebSockets · asyncio · pytest · Swagger / OpenAPI · Selenium · Playwright}
DevOps & CI/CD	_{Docker · Docker Compose · Kubernetes · GitHub Actions · Jenkins · GitLab CI · Nginx}
Monitoring & observability	_{Prometheus · Grafana · ELK stack (Elasticsearch, Logstash, Kibana)}
AWS	_{S3 · EC2 · Lambda · RDS}
Azure	_{Blob Storage / ADLS Gen2 · Azure DevOps}
Linux & shell tools	_{Ubuntu · systemd · cron · SSH · rsync · grep / sed / awk · htop · apt · vim · tmux}
Developer tools	_{VS Code · PyCharm · Postman · Git · GitHub · Jira · Jupyter}
Engineering practices	_{Microservices · REST API design · Event-driven architecture · Message queues · Caching · CI/CD · Infrastructure as Code · Containerization · OOP & design patterns · System design · Unit & integration testing · Code review · Agile / Scrum}

💼 Experience

	Role	Focus
Nov 2025 – Present	Software Development Engineer · Skillmine Technology Consulting	ETL/ELT into ClickHouse (15–20 GB/day), PySpark, Airflow, Celery, FastAPI, Docker on Linux
Jan 2024 – Nov 2025	Software Development Engineer · Reliance Jio Platforms	Data ingestion & sync, SQL tuning (−25% latency), Django, Python/Bash automation on Linux
Aug 2023 – Jan 2024	Software Development Engineer · Dimensionless Technologies	GPT-3.5 automation (+35% efficiency), LLM data prep, real-time backends (WebSockets, asyncio)

🎓 Education & publication

B.Tech in Computer Science, GH Raisoni College of Engineering and Management, Pune (2019–2023)
Paper: "GitHub Issue Manager System for Rank-Based Profile", ICET-23

Hiring for data engineering or DevOps? I'd be glad to talk: rihanbagwan98@gmail.com

_{Repositories from before 2023 are college and learning projects. I've kept them as a record of where I started.}

Navigation Menu