v0.1.1

README.md

@@ -27,22 +27,29 @@ This project underwent three major phases of technical improvement:
 
 ![Phase 3 Training History](docs/visualizations/training_history_v3.png)
 
+### Phase 4: Multi-modal Expansion (Image Select ReCAPTCHA)
+- **Goal**: Beyond text recognition, expanding to **Object Classification** based challenges.
+- **Method**: **EfficientNet-B1** backbone for 3x3 grid image classification.
+- **Integration**: Unified both Text and Image CAPTCHA into a single **FastAPI** serving platform.
+
+![Phase 4 Training History](docs/visualizations/training_history_v4.png)
 ---
 
-## 🏗️ Architecture: CRNN + CTC
+## 🏗️ Integrated Architecture: FastAPI Unified Serving
+The system is designed as a modular **FastAPI** application that serves multiple types of CAPTCHA models through a unified inference pipeline.
 
 ```mermaid
-graph LR
-    A[Input Image: 50x200] --> B[CNN: Spatial Features]
-    B --> C[Bridge: Reshape]
-    C --> D[BLSTM: Seq Modeling]
-    D --> E[CTC Loss: Alignment]
-    E --> F[Output: text]
+graph TD
+    A[Client Request] --> B[FastAPI Gateway]
+    B --> C{Captcha Type?}
+    C -- "v1 (Text)" --> D[CRNN + CTC Model]
+    C -- "v2 (Image)" --> E[EfficientNet-B1 Model]
+    D --> F[Unified Response]
+    E --> F
 ```
 
-- **Backbone**: CNN Layers for visual feature extraction.
-- **Sequence**: Bidirectional LSTM for context modeling between characters.
-- **Loss**: CTCLoss to handle unaligned sequences (crucial for CAPTCHAs with varying character widths).
+- **Inference Service**: Separate service layers for `v1` and `v2`, sharing a consistent preprocessing and logging logic.
+- **Scalable Serving**: Models and preprocessing scripts are encapsulated for seamless deployment across local and cloud environments.
 
 ## 📊 Experimental Results (Latest Run)
 
@@ -91,12 +98,18 @@ pip install -r requirements.txt
 ```
 
 ### 3. Deploy to AWS (EC2)
-The project is configured for automated deployment to **AWS EC2** via GitHub Actions & SSH.
-1. Ensure the following are set in GitHub Secrets:
-    - `EC2_HOST` (IP or DNS), `EC2_USERNAME` (e.g. `ubuntu`), `EC2_SSH_KEY` (Private Key)
-    - `DOCKERHUB_USERNAME`, `DOCKERHUB_TOKEN`
-2. Push or merge to the **`main`** branch to trigger the CD pipeline.
-3. The workflow will build/push an image to Docker Hub and then SSH into your EC2 to restart the container on port **80**.
+The project is configured for automated deployment to **AWS EC2** via GitHub Actions & SSH. Both Text and Image CAPTCHA models are bundled into a single Docker image for unified serving.
+
+1. **Integrated Image**: A single Dockerfile handles model weights, metadata, and dependencies for all versions.
+2. **Automated CD**: Push to the `main` branch triggers the GitHub Actions pipeline.
+3. **Serving Environment**: The workflow pushes the image to Docker Hub and restarts the container on EC2 port **80**.
+
+### 4. Unified Model Deployment Pipeline
+To ensure consistent inference between development and production, we use a unified deployment script for both model versions.
+
+- **Standardized Inference**: All models use a `RecaptchaPredictor` pattern (as seen in `scripts/inference.py`) to wrap model loading and preprocessing.
+- **Metadata-Driven Dashboard**: `generate_v1_metadata.py` and `generate_v2_metadata.py` extract performance metrics, which are then used to dynamically populate the web dashboard at runtime.
+
 
 ### 3. Run Experiments
 ```bash

app/services/captcha_v2_service.py

@@ -10,11 +10,10 @@
 if torch.backends.mps.is_available():
     DEVICE = torch.device('mps')
 
-# Paths are relative to project root usually, but let's make them robust
+# Paths
 BASE_DIR = Path(__file__).parent.parent.parent
 MODEL_PATH = BASE_DIR / "models" / "cnn_best_model.pth"
 METADATA_PATH = BASE_DIR / "models" / "model_metadata_v2.json"
-CLASSES_PATH = BASE_DIR / "models" / "v2_classes.txt"
 
 class CaptchaV2Service:
     _instance = None
@@ -26,53 +25,54 @@ def __new__(cls):
         return cls._instance
 
     def __init__(self):
-        if self._initialized: return
+        if self._initialized: 
+            return
 
         self.model = None
-        self.classes = []
+        self.classes = [
+            "Bicycle", "Bridge", "Bus", "Car", "Chimney", 
+            "Crosswalk", "Hydrant", "Motorcycle", "Palm", 
+            "Stair", "Traffic Light"
+        ]
         self.transform = transforms.Compose([
             transforms.Resize((224, 224)),
             transforms.ToTensor(),
-            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
+            transforms.Normalize(
+                mean=[0.485, 0.456, 0.406], 
+                std=[0.229, 0.224, 0.225]
+            )
         ])
 
         self.load_model()
         self._initialized = True
 
     def load_model(self):
         try:
-            # Load Classes from metadata
+            # Load classes from metadata if available
             if METADATA_PATH.exists():
                 with open(METADATA_PATH, "r") as f:
                     meta = json.load(f)
-                    self.classes = meta.get("classes", [])
+                    self.classes = meta.get("classes", self.classes)
                     print(f"✅ [V2 Service] Loaded {len(self.classes)} classes from metadata")
-
-            if not self.classes:
-                # Fallback classes (11 classes as detected in model weights)
-                self.classes = [
-                    "Bicycle", "Bridge", "Bus", "Car", "Chimney", 
-                    "Crosswalk", "Hydrant", "Motorcycle", "Other", "Palm", "Stair"
-                ]
-                print(f"⚠️ [V2 Service] Using fallback 11 classes")
 
-            # Load Model: cnn_best_model.pth is EfficientNet-B1
+            # Load Model: EfficientNet-B1 with specific classifier structure
             self.model = models.efficientnet_b1(weights=None)
-
-            # Match the nested classifier structure found in the state_dict (classifier.1.1)
             num_ftrs = self.model.classifier[1].in_features
+
+            # Match the exact structure used during training
             self.model.classifier[1] = nn.Sequential(
-                nn.Identity(), 
+                nn.Dropout(0.2),
                 nn.Linear(num_ftrs, len(self.classes))
             )
 
             if MODEL_PATH.exists():
                 state_dict = torch.load(MODEL_PATH, map_location=DEVICE)
+
                 # Handle 'model.' prefix if present
                 new_state_dict = {}
                 for k, v in state_dict.items():
                     if k.startswith("model."):
-                        new_state_dict[k[6:]] = v
+                        new_state_dict[k.replace("model.", "", 1)] = v
                     else:
                         new_state_dict[k] = v
 
@@ -94,19 +94,26 @@ def predict(self, img_path: Path):
         Returns the class name (str) or None if prediction fails.
         """
         if not self.model: 
-            # Try reloading if missing (e.g. usage before startup)
             self.load_model()
-            if not self.model: return None
+            if not self.model: 
+                return None
 
         try:
-            img = Image.open(img_path).convert('RGB')
-            img_tensor = self.transform(img).unsqueeze(0).to(DEVICE)
+            # Load and preprocess image
+            image = Image.open(img_path).convert('RGB')
+            image_tensor = self.transform(image).unsqueeze(0).to(DEVICE)
 
+            # Inference
             with torch.no_grad():
-                outputs = self.model(img_tensor)
-                _, pred_idx = torch.max(outputs, 1)
-
-            return self.classes[pred_idx.item()]
+                outputs = self.model(image_tensor)
+                probabilities = torch.nn.functional.softmax(outputs, dim=1)
+                confidence, predicted_idx = torch.max(probabilities, 1)
+
+            idx = predicted_idx.item()
+            conf_score = confidence.item() * 100
+            result_class = self.classes[idx]
+
+            return result_class
         except Exception as e:
             print(f"❌ [V2 Service] Prediction error for {img_path}: {e}")
             return None

models/model_metadata_v2.json

@@ -1,13 +1,15 @@
 {
-    "accuracy": "72.98%",
-    "char_accuracy": "72.98%",
-    "loss": "1.1722",
-    "loss_value": "1.1722",
-    "precision": "0.67",
-    "recall": "0.73",
-    "f1_score": "0.70",
-    "type": "EfficientNet-B0 (Re-eval)",
-    "timestamp": "2026-01-28T01:00:10",
+    "accuracy": "98.02%",
+    "char_accuracy": "98.02%",
+    "precision": "0.98",
+    "recall": "0.98",
+    "f1_score": "0.98",
+    "loss_value": "N/A",
+    "type": "EfficientNet-B1 (Verified)",
+    "loss": "CrossEntropy",
+    "features": "Image Classification",
+    "preprocessing": "Resize (224x224), Normalization",
+    "timestamp": "2026-01-28T13:21:02",
     "classes": [
         "Bicycle",
         "Bridge",
@@ -17,9 +19,10 @@
         "Crosswalk",
         "Hydrant",
         "Motorcycle",
-        "Other",
         "Palm",
-        "Stair"
+        "Stair",
+        "Traffic Light"
     ],
-    "dataset_path": "/Users/parkyoungdu/Downloads/samples_v2/images"
+    "total_samples": 10390,
+    "valid_predictions": 10390
 }