Add comprehensive examples and end-to-end test cases

[kavyasoni]

• config_postgresql.yaml - Complete PostgreSQL sync configuration

• config_mongodb.yaml - MongoDB to BigQuery configuration

• config_smart_sync.yaml - Smart Sync incremental updates config

• run_from_python.py - 6 programmatic usage examples

  1. Simple PostgreSQL sync (DirectRunner)
  2. Smart Sync on Dataflow
  3. MongoDB to BigQuery sync
  4. Dynamic configuration from environment
  5. Multi-source sequential sync
  6. Scheduled sync with error handling

• local_test.sh - 5 automated local test scenarios

• production_deployment.sh - Complete automated deployment

• examples/README.md - Comprehensive examples guide

• tests/integration/conftest.py - Pytest fixtures for integration tests

• tests/integration/test_pipeline.py - 8 comprehensive E2E pipeline tests

  1. Full PostgreSQL to BigQuery pipeline
  2. Smart Sync incremental pipeline
  3. MongoDB to BigQuery pipeline
  4. Auto-schema detection pipeline
  5. Multi-source pipeline
  6. Error handling pipeline
  7. Data validation pipeline
  8. Transformation pipeline

• tests/integration/test_e2e_scenarios.py - 5 real-world scenarios

  1. Daily full refresh scenario
  2. Hourly incremental sync scenario
  3. Initial historical load scenario
  4. Cross-project sync scenario
  5. Empty table first run scenario

• Total files: 12

• Total size: 182KB

• Configuration examples: 3

• Executable scripts: 3

• Python examples: 6

• Test cases: 15 (8 pipeline + 5 scenario + 2 utility)

✅ Production-ready YAML configurations
✅ Python API for programmatic execution
✅ Automated local testing (5 scenarios)
✅ Automated production deployment
✅ Apache Beam TestPipeline integration
✅ Mock database connections
✅ Real-world scenario tests
✅ Comprehensive documentation

Pull Request

Description

Please include a summary of the changes and the related issue. Include relevant motivation and context.

Fixes # (issue)

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Performance improvement
• Code refactoring
• Dependency update

Testing Checklist

• I have tested this code locally with DirectRunner
• I have tested this code with DataflowRunner (if applicable)
• I have added tests that prove my fix is effective or that my feature works
• I have tested with multiple data sources (PostgreSQL/MongoDB/BigQuery)
• I have verified Smart Sync functionality (if applicable)
• I have verified Auto-Schema detection (if applicable)

Documentation Checklist

• I have updated the CLAUDE.md file with relevant changes
• I have updated the README.md file (if applicable)
• I have added/updated docstrings for new/modified functions
• I have updated metadata.json for new parameters (if applicable)
• I have updated requirements.txt for new dependencies (if applicable)

Code Quality Checklist

• My code follows the project's code style
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing unit tests pass locally
• Test coverage remains at acceptable levels
• I have run linting checks (ruff/black)
• I have run type checking (mypy)

Additional Information

Breaking Changes

If this is a breaking change, describe the impact and migration path:

Performance Impact

Describe any performance implications:

Screenshots/Logs

If applicable, add screenshots or logs to help explain your changes:

Deployment Notes

Any special deployment considerations: