Skip to content

task.md

Latest commit

 

History

History
158 lines (123 loc) · 7.37 KB

task.md

File metadata and controls

158 lines (123 loc) · 7.37 KB
Raw

PixTools — Sprint Breakdown

Sprint 1: Foundation & Local Dev Stack ✅

Goal: Project boots locally, DB connects, basic structure in place.

  • Scaffold project dirs (app/, tests/, k8s/, infra/, models/, alembic/)
  • pyproject.toml + requirements.txt (all deps pinned)
  • app/config.py — Pydantic Settings
  • app/models.py — SQLAlchemy Job model
  • app/schemas.py — Pydantic request/response models
  • app/database.py — async engine + session factory
  • app/main.py — FastAPI app factory with lifespan
  • Dockerfile — multi-stage build
  • docker-compose.yaml — FastAPI + Postgres + Redis + RabbitMQ + LocalStack
  • .env.example
  • Verify: docker-compose up boots, /docs loads, DB connects

Sprint 2: Core Pipeline (API → Celery → S3) ✅

Goal: End-to-end flow works — upload image, run tasks, get webhook.

  • app/routers/jobs.py — POST /process + GET /jobs/{id}
  • app/services/s3.py — upload_raw, download_raw, upload_processed
  • app/services/idempotency.py — Redis check/set
  • app/services/dag_builder.py — Canvas chain/chord builder
  • app/tasks/celery_app.py — Celery config + queue routing
  • app/tasks/image_ops.py — convert_jpg, convert_png, convert_webp, convert_avif
  • app/tasks/finalize.py — DB update + webhook POST
  • tests/test_api.py, tests/test_dag_builder.py, tests/test_tasks.py
  • Verify: Upload image → task runs → processed image in S3 → job COMPLETED with presigned URLs

Sprint 3: Frontend (Neobrutalism) ✅

Goal: Upload UI with preview, operation selection, result cards with download.

  • app/static/index.html — single-page layout (upload → processing → results)
  • app/static/style.css — neobrutalism design system
  • app/static/app.js — drag-drop, polling, result card rendering
  • Mount StaticFiles in app/main.py
  • Verify: Upload image via UI → see processing state → download results

Sprint 4: DnCNN ML Integration ✅

Goal: Denoise task runs real inference with dncnn_color_blind.pth.

  • app/ml/dncnn.py — DnCNN model definition (20-layer, 64ch, RGB)
  • app/tasks/ml_ops.py — denoise task with singleton model loading
  • Place dncnn_color_blind.pth in models/
  • ML worker queue isolation (ml_inference_queue, --concurrency=1)
  • Verify: Upload noisy image with ["denoise"] → denoised output in S3

Sprint 5: Production Hardening ✅

Goal: Fault tolerance, observability, and code quality gates.

  • app/logging_config.py — structured JSON logging (python-json-logger)
  • app/services/webhook.py — Implement Circuit Breaker (using pybreaker) for webhook deliveries
  • Health check endpoint (GET /api/health) with DB/Redis/S3 reachability checks
  • Celery Task.on_failure hooks for global error reporting
  • Verify: Logs appear as JSON in console; webhooks trip "Open" status if target is down
  • app/middleware.py — RequestID middleware (correlation IDs)
  • app/routers/health.py — deep dependency health check
  • app/services/webhook.py — circuit breaker (pybreaker)

Sprint 6: Testing (IN PROGRESS)

Goal: 80%+ coverage and automated verification.

  • Add testing dependencies to requirements.txt
  • tests/conftest.py — Shared fixtures (mock DB, S3, client)
  • tests/test_api.py — REST endpoint verification
  • tests/test_tasks.py — Celery task logic verification
  • tests/test_services.py — Webhook and S3 service verification
  • Verify: All tests pass via pytest
  • Dead Letter Queue config in celery_app.py
  • alembic/ — env.py + initial migration
  • tests/test_idempotency.py
  • pyproject.toml — ruff + mypy config
  • Verify: Failing webhook trips circuit breaker, DLQ catches poison messages

Sprint 6: Infrastructure & Deployment

Goal: Full K3s + AWS IaC, monitoring, CI pipeline.

  • k8s/namespace.yaml
  • k8s/rabbitmq/ — deployment + service
  • k8s/redis/ — deployment + service
  • k8s/workers/worker-standard.yaml + worker-ml.yaml
  • k8s/api/ — deployment + service
  • k8s/keda/scaledobject.yaml
  • k8s/monitoring/ — Prometheus, Grafana, celery-exporter configs
  • infra/main.tf — provider, VPC, subnets
  • infra/asg.tf — launch template + ASG + spot + user data (K3s bootstrap)
  • infra/rds.tf — Postgres (app DB + K3s state)
  • infra/s3.tf — image bucket + manifest bucket
  • infra/sns.tf — EventBridge + SNS interruption alerts
  • infra/security_groups.tf
  • .github/workflows/ci.yaml — lint + type check + test + build + tf validate
  • README.md
  • Verify: terraform validate, kubectl apply --dry-run, CI pipeline passes

Sprint 7: S3 Storage Optimization

Goal: Prevent storage bloat by cleaning up orphan files.

  • Implement automated S3 orphan file cleanup
    • Create implementation plan
    • Configure S3 Lifecycle Rules in s3.py
    • Add s3_retention_days to config.py
    • Verify lifecycle policy via CLI/tests

Sprint 8: Anonymous Persistence & History

Goal: Persistent job history across refreshes with expiration awareness.

  • Implement Anonymous Job Persistence
    • Create implementation plan
    • Create Alembic migration for result_keys
    • Update Job model and finalize_job task
    • Update GET /jobs/{id} for dynamic presigning
    • Implement localStorage logic in app.js
    • Add "Expired" badge logic for items > 24h
    • Verify: Clear local storage, run job, refresh page, verify data remains.

Sprint 9: Maintenance & Cleanup

Goal: Declutter codebase and improve maintainability.

  • Remove temporary log files (*.txt, *.log) from root
  • Remove accidental SQLite artifacts (alembic.db)

Sprint 10: Documentation & Handover

Goal: Provide a professional final state for the repository.

  • Draft comprehensive git commit message
  • Expand and professionalize README.md

Sprint 11: Cloud Readiness & Advanced Features

Goal: Final polish and observability before cloud migration.

  • Implement Operation Parameterization (Quality/Resize)
  • Implement ZIP Result Bundling
  • Implement Advanced Image Metadata (EXIF)
  • Implement Custom Webhook Sandbox
  • Implement LGTM Stack (Loki, Grafana, Tempo, Mimir)