A robust, Python-based auditing engine designed to process large-scale CSV transaction data — implementing dynamic column detection and professional error logging.

---

📌 Table of Contents

• 🎬 Demo
• 🔷 Overview
• ⭐ Core Features
• 🔧 Technical Implementation
• ⚙️ How It Works
• 💡 Use Cases
• 📂 Project Structure
• 🚀 How to Use
• 🗺️ Future Roadmap
• 👤 Author

---

🎬 Demo

Demo	Platform	Watch
🎥 Live Demo Video	Repo (mp4)	▶ Watch in Repo
💼 LinkedIn Demo	LinkedIn	▶ Watch on LinkedIn

---

🔷 Overview

Universal Data Auditor is a professional Python tool that moves away from hardcoded scripts by implementing dynamic column detection and production-grade error logging for large-scale CSV transaction data.

📄 CSV File → 🔍 Dynamic Column Detection → ✅ Validate → 📋 Log Errors → 📊 Summary Report

No manual code changes needed for different file structures — the auditor adapts automatically.

---

⭐ Core Features

Feature	Description
🔍 Dynamic Column Detection	Automatically identifies file structure from the header. Adapts to any number of columns without manual code changes.
✅ Business Logic Validation	Implements strict data integrity rules. Transactions with zero or negative values are caught using custom ValueError raises.
📋 Production-Grade Logging	All anomalies, missing columns, and validation failures recorded in production_audit.log for post-audit analysis.
🧩 Modular Architecture	Built as a standalone function — easy to integrate into larger automation workflows or CI/CD pipelines.

---

🔧 Technical Implementation

1. Data Integrity

The script uses a try-except block to ensure the entire process doesn't crash due to a single corrupted row. By using if amount <= 0: raise ValueError, the system separates standard data-type errors from specific business-rule violations.

try:
    if amount <= 0:
        raise ValueError(f"Invalid amount: {amount}")
except ValueError as e:
    logging.warning(f"Row {row_num} skipped — {e}")

2. Efficiency

The auditor reads files line-by-line using enumerate(file), ensuring low memory consumption even when processing files with 10,000+ rows.

with open(file_path, 'r') as file:
    for row_num, line in enumerate(file):
        # Process one row at a time — no full file load

---

⚙️ How It Works

┌────────────────────────────────────────────────────────────────┐
│              UNIVERSAL DATA AUDITOR — PIPELINE                  │
│                                                                  │
│  📄 CSV File Input                                              │
│         │                                                        │
│         ▼                                                        │
│  🔍 Dynamic Column Detection                                    │
│    • Read header row automatically                              │
│    • Map columns — no hardcoding needed                         │
│         │                                                        │
│         ▼                                                        │
│  🔄 Row-by-Row Processing (enumerate)                           │
│    • Low memory — reads line by line                            │
│    • Handles 10,000+ rows efficiently                           │
│         │                                                        │
│         ▼                                                        │
│  ✅ Business Logic Validation                                   │
│    ├── Amount ≤ 0 → raise ValueError → Log & Skip              │
│    ├── Missing column → Log warning → Continue                 │
│    └── Corrupted row → try-except catches → Log & Skip         │
│         │                                                        │
│         ▼                                                        │
│  📋 production_audit.log — Full Error Record                    │
│         │                                                        │
│         ▼                                                        │
│  📊 Console Summary Report                                      │
└────────────────────────────────────────────────────────────────┘

---

💡 Use Cases

Use Case	How It Helps
🏢 Business Transaction Auditing	Validate thousands of daily transactions automatically
📊 Data Quality Checks	Catch corrupted, missing, or invalid rows before they cause problems
🔄 CI/CD Data Pipeline Validation	Integrate as a modular function in automated pipelines
📋 Compliance & Reporting	Generate production-grade audit logs for regulatory needs
🔍 CSV Data Cleaning	Pre-process large datasets before analysis or import

---

📂 Project Structure

universal-data-auditor/
│
├── main.py                        # Primary auditing engine — dynamic logic & validation rules
├── audit.log                      # System-generated log — tracks all data discrepancies
├── company_big_data.csv           # Sample large-scale CSV dataset for testing
├── 2026-03-25 20-59-39.mp4       # Live demo video
└── README.md                      # Project documentation

File	Description
main.py	The primary auditing engine containing the dynamic logic and validation rules
audit.log	Example of the system-generated log file tracking data discrepancies
company_big_data.csv	Sample CSV dataset with large-scale transaction data

---

🚀 How to Use

Prerequisites

• ✅ Python 3.11+
• ✅ No external libraries required — uses Python standard library only

Steps

# 1. Clone this repository
git clone https://github.com/muhammadantor/universal-data-auditor

# 2. Navigate into the project
cd universal-data-auditor

# 3. Generate sample data
python data_creator.py

# 4. Run the auditor
python main.py

# 5. Review results
# → Console: summary report
# → audit.log: full breakdown of skipped or failed rows

---

🗺️ Future Roadmap

• Add support for Excel (.xlsx) and JSON formats
• Integrate automated PDF report generation for audit summaries
• Add a CLI (Command Line Interface) for easier file path input

---

👤 Author

Muhammad Antor

AI Automation Engineer | AutomateIQ Labs ⚡

Building scalable AI & automation solutions

---

⭐ If this project helped you, please give it a star!

Built with ❤️ using Python · CSV Processing · Production-Grade Logging