🚀 Data Processing Pipeline

A complete, modular, and production-ready end-to-end data processing pipeline built with Python. This pipeline ingests raw datasets, cleans and preprocesses data, performs transformations, analyzes it, and generates automated reports with visualizations.

📸 Project Screenshots

Pipeline Execution

Complete pipeline execution with all 6 stages completing successfully

Sample Input Data

Sample sales data with 1,020 rows including dates, products, quantities, and revenue

Generated Visualizations

Revenue Distribution

Histogram showing the distribution of revenue values

Revenue by Product Category

Average revenue across different product categories

Revenue Trend Over Time

Time series analysis showing revenue trends over the dataset period

Correlation Heatmap

Correlation matrix showing relationships between numeric variables

HTML Report

Automated HTML report with insights, statistics, and embedded visualizations

📋 Features

• Modular Architecture: Separate modules for each pipeline stage
• Configuration-Driven: Customize behavior via JSON configuration
• Error Handling: Comprehensive error handling and logging throughout
• Automated Reporting: Generate professional HTML reports with insights
• Visualization: Automatic generation of meaningful plots
• CLI Interface: Easy-to-use command-line interface
• Production-Ready: Follows industry best practices

🏗️ Project Structure

Data Processing Pipeline/
│
├── data/
│   ├── raw/                  # Raw input data
│   └── processed/            # Cleaned and processed data
│
├── output/
│   ├── reports/              # Generated HTML reports
│   └── visuals/              # Generated visualizations
│
├── src/
│   ├── __init__.py
│   ├── data_ingestion.py     # Data loading (CSV/Excel)
│   ├── data_cleaning.py      # Data cleaning and preprocessing
│   ├── data_transformation.py # Feature engineering and transformations
│   ├── data_analysis.py      # Statistical analysis and visualization
│   ├── reporting.py          # Automated report generation
│   └── utils.py              # Utility functions
│
├── config/
│   └── config.json           # Pipeline configuration
│
├── main.py                   # Pipeline orchestrator with CLI
├── generate_sample_data.py   # Sample data generator
├── requirements.txt          # Python dependencies
└── README.md                 # This file

📦 Installation

Prerequisites

• Python 3.8 or higher
• pip (Python package manager)

Setup

1. Clone or download the project

2. Install dependencies

pip install -r requirements.txt

🚀 Quick Start

1. Generate Sample Data (Optional)

If you want to test the pipeline with sample data:

python generate_sample_data.py

This will create a sample dataset at data/raw/sample_data.csv with 1,000 rows of realistic sales data.

2. Run the Pipeline

Basic usage:

python main.py --input data/raw/sample_data.csv

With custom configuration:

python main.py --input data/raw/sample_data.csv --config config/config.json

Save processed data to custom location:

python main.py --input data/raw/sample_data.csv --output output/my_processed_data.csv

With debug logging:

python main.py --input data/raw/sample_data.csv --log-level DEBUG

View help:

python main.py --help

📖 Pipeline Stages

The pipeline executes the following stages sequentially:

Stage 1: Data Ingestion

• Loads data from CSV or Excel files
• Auto-detects file format
• Handles missing or corrupt files gracefully
• Validates data integrity

Stage 2: Data Cleaning

• Standardizes column names (lowercase, underscores)
• Removes duplicate rows
• Handles missing values (fill/drop strategies)
• Fixes data types (dates, numeric, categorical)
• Detects outliers (optional)

Stage 3: Data Transformation

• Feature engineering (datetime extraction, mathematical operations)
• Categorical encoding (label encoding, one-hot encoding)
• Data normalization (min-max, z-score)
• Filtering and aggregations

Stage 4: Data Analysis

• Summary statistics
• Correlation analysis
• Trend detection
• Distribution analysis
• Automated insight generation

Stage 5: Visualization

• Distribution histograms
• Correlation heatmaps
• Time series trends
• Bar charts
• All plots saved automatically to output/visuals/

Stage 6: Reporting

• Generates comprehensive HTML reports
• Includes key insights and statistics
• Embeds visualizations
• Saves to output/reports/

⚙️ Configuration

The pipeline is configured via config/config.json. Key configuration options:

Paths

{
  "paths": {
    "raw_data_dir": "data/raw",
    "processed_data_dir": "data/processed",
    "output_dir": "output",
    "reports_dir": "output/reports",
    "visuals_dir": "output/visuals"
  }
}

Cleaning Strategy

{
  "cleaning": {
    "missing_value_strategy": "fill",
    "missing_value_fill_method": "median",
    "remove_duplicates": true,
    "standardize_columns": true,
    "fix_data_types": true
  }
}

Feature Engineering

{
  "transformation": {
    "feature_engineering": {
      "enabled": true,
      "features": [
        {
          "name": "date_month",
          "type": "datetime_extract",
          "source_column": "date",
          "extract": "month"
        }
      ]
    }
  }
}

Visualization

{
  "visualization": {
    "enabled": true,
    "plots": [
      {"type": "distribution", "column": "revenue"},
      {"type": "bar_chart", "x": "product_category", "y": "revenue"},
      {"type": "trend", "date_column": "date", "value_column": "revenue"},
      {"type": "heatmap"}
    ]
  }
}

📊 Output

After running the pipeline, you'll find:

• Processed Data: data/processed/<filename>_processed.csv
• Visualizations: output/visuals/*.png
• Reports: output/reports/analysis_report_*.html
• Logs: pipeline.log

🔧 Customization

Adding Custom Features

You can customize the pipeline by:

1. Modifying config.json: Change cleaning strategies, add features, adjust visualization settings
2. Using your own data: Point to any CSV or Excel file
3. Extending modules: Add custom transformations or analysis methods

Example: Custom Feature Engineering

Add to config.json:

{
  "transformation": {
    "feature_engineering": {
      "features": [
        {
          "name": "revenue_per_unit",
          "type": "math",
          "operation": "division",
          "columns": ["revenue", "quantity"]
        },
        {
          "name": "price_range",
          "type": "bin",
          "source_column": "unit_price",
          "bins": 5
        }
      ]
    }
  }
}

📝 Using Your Own Data

The pipeline works with any CSV or Excel file. Just ensure:

• File is accessible and not corrupted
• Contains at least some numeric or categorical columns
• For trend analysis: include a date column and numeric value column

Example:

python main.py --input /path/to/your/data.csv

🐛 Troubleshooting

Common Issues

1. Module not found error

Make sure you're running from the project root directory:

cd "Data Processing Pipeline"
python main.py --input data/raw/sample_data.csv

2. Missing dependencies

Reinstall dependencies:

pip install -r requirements.txt --force-reinstall

3. File not found

Check the file path is correct and relative to the project root.

4. Empty output

Check the log file (pipeline.log) for detailed error messages.

📚 Modules Documentation

data_ingestion.py

• DataIngestion class
• Methods: load_csv(), load_excel(), auto_detect_format(), validate_data()

data_cleaning.py

• DataCleaner class
• Methods: standardize_columns(), handle_missing_values(), remove_duplicates(), fix_data_types(), detect_outliers()

data_transformation.py

• DataTransformer class
• Methods: create_features(), filter_data(), group_and_aggregate(), normalize_data(), encode_categorical()

data_analysis.py

• DataAnalyzer class: Statistical analysis
• DataVisualizer class: Plot generation
• Methods: summary_statistics(), correlation_analysis(), trend_detection(), plot_distribution(), etc.

reporting.py

• ReportGenerator class
• Methods: generate_html_report(), create_summary()

utils.py

• Utility functions: setup_logging(), load_config(), validate_directories()

🎯 Best Practices

1. Always review config.json before running the pipeline
2. Check logs for warnings and errors
3. Validate output to ensure data quality
4. Backup raw data before processing
5. Customize features based on your specific dataset

🤝 Contributing

This project is designed to be extensible. You can:

• Add new transformation methods
• Create additional visualization types
• Support more file formats (Parquet, JSON, etc.)
• Add machine learning pipelines
• Implement database connectivity

📄 License

This project is open-source and available for educational and commercial use.

👨‍💻 Author

Built as a comprehensive data processing pipeline demonstrating industry best practices in data engineering.

---

Happy Data Processing! 🚀

📤 How to Upload to GitHub

Prerequisites

• Git installed on your system
• GitHub account

Step-by-Step Instructions

1. Create a New Repository on GitHub

1. Go to GitHub
2. Click the "+" icon in the top-right corner
3. Select "New repository"
4. Fill in the details:
   • Repository name: data-processing-pipeline (or your preferred name)
   • Description: End-to-end data processing pipeline with automated analysis and reporting
   • Visibility: Public or Private (your choice)
   • DO NOT initialize with README, .gitignore, or license (we already have these)
5. Click "Create repository"

2. Initialize Git Repository (if not already done)

Open terminal/command prompt in your project folder:

cd "c:\Users\dnitr\Desktop\COLLEGE\myProjects\Data Processing Pipeline"
git init

3. Add Files to Git

# Add all files
git add .

# Check what will be committed
git status

4. Commit Your Changes

git commit -m "Initial commit: Complete data processing pipeline with modular architecture"

5. Connect to GitHub Repository

GitHub will show you commands after creating the repository. Use these:

# Rename branch to main
git branch -M main

# Add remote repository (replace YOUR_USERNAME with your GitHub username)
git remote add origin https://github.com/YOUR_USERNAME/data-processing-pipeline.git

# Verify remote
git remote -v

6. Push to GitHub

# Push to GitHub
git push -u origin main

You'll be prompted to enter your GitHub credentials:

• Username: Your GitHub username
• Password: Your GitHub Personal Access Token (not your GitHub password)

Note: If you don't have a Personal Access Token:

1. Go to GitHub → Settings → Developer settings → Personal access tokens
2. Click "Generate new token (classic)"
3. Give it a name (e.g., "Git CLI Access")
4. Select scopes: repo (full control of private repositories)
5. Click "Generate token"
6. Copy the token immediately - you won't see it again!

7. Verify Upload

Go to your GitHub repository page and refresh. You should see all your files!

Alternative: Using GitHub Desktop (Easier)

1. Download GitHub Desktop
2. Install and sign in with your GitHub account
3. Click File → Add Local Repository
4. Browse to your project folder and select it
5. If it says "This directory is not a repository", click "create a repository"
6. Fill in the details and click "Create Repository"
7. Click "Publish repository"
8. Choose the repository name and settings
9. Click "Publish Repository"

📝 Updating Your Repository Later

After making changes to your project:

# Check changes
git status

# Add modified files
git add .

# Commit with descriptive message
git commit -m "Updated pipeline configuration and added new features"

# Push to GitHub
git push

🔒 Security Checklist Before Pushing

• .gitignore file is present (already added)
• No sensitive data (API keys, passwords) in code
• No large data files (use Git LFS if needed)
• pipeline.log is in .gitignore (already added)
• Virtual environment folders are ignored (already added)

📦 Recommended GitHub Repository Settings

1. Add Topics (on your repo page):

   • python, data-pipeline, data-processing, data-analysis, pandas, data-science

2. Enable Issues for tracking bugs and features

3. Add License (optional but recommended):

   • Go to your repo → Add file → Create new file
   • Name it LICENSE
   • Choose a license template (MIT is popular)

4. Set Repository Website (if you deploy it)

🎯 Pro Tips

• Commit Often: Make small, focused commits with clear messages
• Use Branches: Create feature branches for new features
• Write Good Commit Messages: Use present tense ("Add feature" not "Added feature")
• Keep README Updated: Update documentation as you add features

---