Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 

Repository files navigation

Air Quality Index Analysis using Big Data Tools

A Big Data project focused on analyzing Air Quality Index (AQI) data using distributed data processing with Apache Spark and Python. This repository demonstrates how large-scale environmental datasets can be cleaned, transformed, and analyzed efficiently using Big Data frameworks.


📌 Project Overview

Air quality analysis involves handling large, complex, and often nested datasets. This project applies Big Data concepts to process AQI data and extract meaningful insights related to air pollution patterns. The implementation highlights the advantages of Spark-based distributed computation over traditional single-machine approaches.

This project was developed as part of an academic Big Data Analytics course.


🎯 Objectives

  • Analyze AQI data using Big Data tools
  • Perform scalable data cleaning and transformation
  • Handle nested and structured datasets using Spark DataFrames
  • Demonstrate real-world use of distributed data processing

🛠️ Tech Stack

  • Python
  • Apache Spark (PySpark)
  • Jupyter Notebook
  • NumPy

📊 Dataset

The dataset consists of air quality–related parameters collected across locations and time periods. It includes structured and nested fields, making it suitable for demonstrating Spark’s schema handling and transformation capabilities.

Typical attributes include:

  • Air quality and pollution metrics
  • Geographical coordinates
  • Time-based observations

⚙️ Implementation

  • Spark session initialization for distributed execution
  • Data ingestion with schema inference and validation
  • Flattening and transformation of nested fields
  • Data cleaning and type casting
  • Analytical queries using Spark DataFrame operations

All implementation details are available in the Jupyter notebook.


📂 Repository Structure

.
├── BDAAQI.ipynb   # Main notebook with full implementation
├── README.md     # Project documentation

▶️ How to Run

  1. Install Python and Apache Spark
  2. Launch Jupyter Notebook
  3. Open BDAAQI.ipynb
  4. Run the cells sequentially

📈 Results

The project demonstrates efficient processing of AQI data using Spark, showcasing how Big Data tools simplify handling large datasets and enable faster analytical workflows.


✅ Conclusion

This repository serves as a practical example of applying Big Data frameworks to environmental data analysis. It emphasizes scalability, structured data handling, and real-world applicability.


👥 Authors

  • Prajwal JB
  • Aashita Narayanpur

Feel free to explore, fork, or adapt this project for learning and experimentation.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages