This repository delivers an end-to-end unsupervised big data pipeline to detect operational aviation anomalies and instrumentation faults during terminal approach phases. By processing large-scale ADS-B telemetry streams, the framework applies distributed density clustering and multi-variate statistical validation to flag safety corridor deviations without relying on historical labels.
- Goal: Identify unstable flight trajectories and transponder sensor failures dynamically to isolate high-risk landing profiles.
- Kinematic Engineering: Computes vertical rate gradients and runway alignments using distributed Spark SQL window functions.
- Spatial Partitioning: Eliminates massive data skew in dense airspaces by isolating flights into a 2-degree geographic grid layer paired with a 10-bucket salting key.
- Vectorized Parallelism: Runs scikit-learn DBSCAN across 200 worker partitions at native C-speed inside isolated worker RAM.
-
Statistical Verification: Validates isolated noise vectors by computing exact Mahalanobis distances (
$D_M$ ) via a broadcasted, Ridge-regularized global covariance matrix.
The repository strictly adheres to clean production-ready modular machine learning layouts:
Big-data-project/
├── BigDataProject.ipynb # Complete production-grade PySpark interactive pipeline
├── .gitignore # Strict exclusion matrix for local JREs and datasets
└── README.md # Global architectural documentation