Distributed data processing of the NYC Yellow Taxi Trip Records (March 2016) using PySpark — covering RDDs, Spark SQL, and DataFrames.
📘 Course: INFT 4836 — Intro to Big Data Analytics | ADA University
NYC Yellow Taxi Trip Records — March 2016
- 12+ million rows of real-world transportation data
- Fields include: VendorID, pickup datetime, trip distance, passenger count, fare amount, payment type
Dataset download: NYC TLC Trip Records — March 2016 Yellow Taxi CSV
Core transformations and actions applied to the raw dataset:
| Operation | Purpose |
|---|---|
filter() |
Remove trips where trip_distance == 0 (cancelled/erroneous rides) |
map() |
Extract (VendorID, TripDistance) tuples |
flatMap() |
Split pickup datetime into [Date, Time] tokens |
reduceByKey() |
Sum total distance driven per vendor |
collect() |
Retrieve results to driver |
Results:
Registered the DataFrame as a temp view taxi_trips and ran three queries:
Query 1 — Average fare and trip count grouped by passenger count:
Query 2 — High-value trips: fare > $50 AND distance > 10 miles:
Query 3 — Payment type join with human-readable labels:
Data Cleaning — Dropped nulls from passenger_count, trip_distance, fare_amount — row count unchanged, confirming no nulls in these columns.
GroupBy + Aggregation — Statistical summaries grouped by VendorID.
Sorting & Insights — Sorted by passenger_count ASC and fare_amount DESC, revealing significant data quality issues:
- 608 trips recorded with 0 passengers, some with fares over $200
- Extreme outlier: 1 trip with 1 passenger charged $429,496.72 for 0.0 miles
Conclusion: The dataset was null-free but contained logical errors and extreme outliers — highlighting the need for business-rule-based cleaning beyond standard null checks.
- Python 3.x
- Java 17 (set
JAVA_HOMEexplicitly to avoid version conflicts) - Apache Spark / PySpark
pip install pysparkspark-submit analysis.pyOr run interactively in a Jupyter notebook.
├── analysis.py # Main PySpark script
├── screenshots/ # Result screenshots
└── README.md
Rashad Hummatov — Computer Engineering, ADA University
GitHub