This project is a Databricks-based prototype that profiles ecommerce tables, enriches column metadata, detects PII and sensitive fields, recommends data quality rules, and executes those rules to produce a table-level data quality scorecard.
The project was designed to simulate an internal data platform utility that helps data engineers onboard new tables faster by automatically generating first-pass metadata and quality recommendations.
Synthetic ecommerce data was generated for:
- customers
- orders
- products
The dataset intentionally includes scattered quality issues such as invalid emails, malformed phone numbers, duplicate IDs, null identifiers, invalid categories, negative amounts, and out-of-range numeric values.
- Generate synthetic ecommerce data
- Save raw data and bronze Delta tables
- Profile column-level metadata
- Infer semantic column types
- Detect PII/sensitive fields and assign governance tags
- Recommend data quality rules
- Execute recommended rules
- Generate data quality scorecard
- Databricks
- PySpark
- Delta Lake
- Unity Catalog Volumes
- SQL
- Python
- Column profiling
- Semantic type inference
- PII and sensitive-field detection
- Governance tagging
- Data quality rule recommendation
- Rule execution engine
- DQ scorecard summary
- email must match a valid email format
- phone number must match expected format
- age must be between 0 and 120
- order totals must be non-negative
- order status must be in an accepted list
- IDs should not be null
- likely primary keys should be unique
The scorecard identified issues across customers, orders, and products, including malformed PII fields, invalid categorical values, negative financial amounts, duplicate IDs, and missing key fields.
- Add referential integrity checks between orders and customers
- Integrate live LLM-based description generation
- Use Faker library to generate massive amounts of data
- Add dashboard visualizations for ongoing monitoring