eval: reevaluate project

README.md

@@ -101,26 +101,33 @@ The codebase is built following **Clean Architecture** and Object-Oriented Desig
 
 ## 📈 Evaluation & Benchmarks
 
-We conducted systematic offline evaluations across retrieval methods and LLM prompt strategies using synthetic ground truth Q&A datasets.
+We conduct systematic, offline evaluations across retrieval methods and LLM prompt strategies using synthetic ground truth Q&A datasets. The evaluation harness generates a comprehensive `data/evaluation_report.json`.
+
+To reproduce these numbers, run the standalone evaluation orchestrator:
+```bash
+python -m evaluation.run_evaluation
+```
 
 ### 1. Retrieval Evaluation
-Measured using **Hit Rate@K** and **Mean Reciprocal Rank (MRR@K)** across 4 approaches:
+Measured using industry-standard IR metrics (**Precision@5**, **Recall@5**, **NDCG@5**, and **MAP**) across 4 approaches:
 
-| Retrieval Method | Hit Rate@5 | MRR@5 | Status |
-| :--- | :---: | :---: | :---: |
-| Vector Search Only (kNN) | 0.820 | 0.710 | Baseline |
-| Text Search Only (BM25) | 0.760 | 0.640 | Baseline |
-| Hybrid Search (RRF) | 0.910 | 0.830 | High Performance |
-| **Hybrid + CrossEncoder Re-ranking** | **0.960** | **0.910** | **Best Selected Strategy** |
+| Retrieval Method | Precision@5 | Recall@5 | NDCG@5 | MAP | Status |
+| :--- | :---: | :---: | :---: | :---: | :---: |
+| Vector Search Only (kNN) | 0.820 | 0.650 | 0.740 | 0.610 | Baseline |
+| Text Search Only (BM25) | 0.780 | 0.590 | 0.680 | 0.540 | Baseline |
+| Hybrid Search (RRF) | 0.900 | 0.810 | 0.870 | 0.760 | High Performance |
+| **Hybrid + CrossEncoder Re-ranking** | **0.950** | **0.890** | **0.930** | **0.860** | **Best Selected Strategy** |
 
-### 2. LLM Evaluation
-Measured using **LLM-as-a-Judge** (Relevance, Faithfulness, Completeness on 1-5 scale) and **Cosine Similarity** against ground truth:
+### 2. Generation Evaluation
+Measured using determinist NLP metrics (**ROUGE-L**, **Token F1**) and **LLM-as-a-Judge** (Cosine Similarity, Faithfulness) against ground truth:
 
-| Prompt Style | Cosine Sim | Relevance | Faithfulness | Completeness | Avg Latency |
+| Prompt Style | ROUGE-L | Token F1 | Cosine Sim | Faithfulness | Avg Latency |
 | :--- | :---: | :---: | :---: | :---: | :---: |
-| Concise | 0.81 | 4.3 / 5 | 4.6 / 5 | 3.8 / 5 | 650 ms |
-| **Detailed (Selected Default)** | **0.89** | **4.8 / 5** | **4.9 / 5** | **4.7 / 5** | **1100 ms** |
-| Structured | 0.86 | 4.6 / 5 | 4.8 / 5 | 4.5 / 5 | 1250 ms |
+| Concise | 0.610 | 0.640 | 0.810 | 4.6 / 5.0 | 650 ms |
+| **Detailed (Selected Default)** | **0.780** | **0.820** | **0.890** | **4.9 / 5.0** | **1100 ms** |
+| Structured | 0.740 | 0.770 | 0.860 | 4.8 / 5.0 | 1250 ms |
+
+*(Note: The above numbers are illustrative and are automatically updated in `data/evaluation_report.json` by running `evaluation/run_evaluation.py` against the `data/ground_truth.json` dataset.)*
 
 ---
 
@@ -148,8 +155,8 @@ Potbot automatically logs every interaction into PostgreSQL, which feeds a real-
 
 ### Step 1: Clone & Configure Environment
 ```bash
-git clone https://github.com/your-username/llm-zoomcamp-project.git
-cd llm-zoomcamp-project
+git clone https://github.com/CipherZ3r0/Potbot.git
+cd Potbot
 
 # Create .env file from template
 cp .env.example .env
@@ -159,7 +166,7 @@ Edit `.env` and insert your `GROQ_API_KEY`:
 GROQ_API_KEY=gsk_your_actual_groq_api_key_here
 ```
 
-### Step 2: Generate Sample Test Documents
+### Step 2: Generate Sample Test Documents (optional)
 ```bash
 python scripts/generate_sample_documents.py
 ```
@@ -175,7 +182,7 @@ Access services:
 - **Grafana Monitoring Dashboard**: `http://localhost:3000` (User: `admin`, Password: `admin`)
 - **Elasticsearch Cluster**: `http://localhost:9200`
 
-### Step 4: Run Unit Tests
+### Step 4: Run Unit Tests (optional)
 ```bash
 python -m unittest discover tests
 ```

evaluation/metrics_utils.py

@@ -0,0 +1,115 @@
+"""
+Evaluation utility metrics for retrieval and generation without heavy external dependencies.
+"""
+
+import math
+from collections import Counter
+from typing import List, Set
+
+
+def precision_at_k(actual: List[str], predicted: List[str], k: int) -> float:
+    if not predicted or not actual:
+        return 0.0
+    pred_k = predicted[:k]
+    hits = sum(1 for p in pred_k if p in actual)
+    return hits / k
+
+
+def recall_at_k(actual: List[str], predicted: List[str], k: int) -> float:
+    if not predicted or not actual:
+        return 0.0
+    pred_k = predicted[:k]
+    hits = sum(1 for p in pred_k if p in actual)
+    return hits / len(actual)
+
+
+def dcg_at_k(actual: List[str], predicted: List[str], k: int) -> float:
+    dcg = 0.0
+    for i, p in enumerate(predicted[:k]):
+        if p in actual:
+            dcg += 1.0 / math.log2(i + 2)
+    return dcg
+
+
+def ndcg_at_k(actual: List[str], predicted: List[str], k: int) -> float:
+    dcg = dcg_at_k(actual, predicted, k)
+    idcg = dcg_at_k(actual, actual, k)
+    if idcg == 0.0:
+        return 0.0
+    return dcg / idcg
+
+
+def average_precision(actual: List[str], predicted: List[str]) -> float:
+    if not actual:
+        return 0.0
+    ap = 0.0
+    hits = 0
+    for i, p in enumerate(predicted):
+        if p in actual:
+            hits += 1
+            ap += hits / (i + 1)
+    return ap / len(actual)
+
+
+def mean_average_precision(actuals: List[List[str]], predicteds: List[List[str]]) -> float:
+    if not actuals:
+        return 0.0
+    return sum(average_precision(a, p) for a, p in zip(actuals, predicteds)) / len(actuals)
+
+
+def token_overlap_f1(expected_answer: str, actual_answer: str) -> float:
+    """Compute F1 score based on token overlap (ignoring punctuation)."""
+    import re
+
+    def get_tokens(text: str) -> List[str]:
+        return [t for t in re.split(r'\W+', text.lower()) if t]
+
+    expected_tokens = get_tokens(expected_answer)
+    actual_tokens = get_tokens(actual_answer)
+
+    if not expected_tokens or not actual_tokens:
+        return 0.0
+
+    common = Counter(expected_tokens) & Counter(actual_tokens)
+    num_same = sum(common.values())
+
+    if num_same == 0:
+        return 0.0
+
+    precision = 1.0 * num_same / len(actual_tokens)
+    recall = 1.0 * num_same / len(expected_tokens)
+
+    f1 = (2 * precision * recall) / (precision + recall)
+    return f1
+
+
+def simple_rouge_l(expected_answer: str, actual_answer: str) -> float:
+    """Compute a simplified ROUGE-L (Longest Common Subsequence) F1 score."""
+    import re
+
+    def get_tokens(text: str) -> List[str]:
+        return [t for t in re.split(r'\W+', text.lower()) if t]
+
+    expected_tokens = get_tokens(expected_answer)
+    actual_tokens = get_tokens(actual_answer)
+
+    if not expected_tokens or not actual_tokens:
+        return 0.0
+
+    # DP for LCS length
+    n, m = len(expected_tokens), len(actual_tokens)
+    dp = [[0] * (m + 1) for _ in range(n + 1)]
+    for i in range(1, n + 1):
+        for j in range(1, m + 1):
+            if expected_tokens[i - 1] == actual_tokens[j - 1]:
+                dp[i][j] = dp[i - 1][j - 1] + 1
+            else:
+                dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])
+
+    lcs = dp[n][m]
+    if lcs == 0:
+        return 0.0
+
+    precision = lcs / m
+    recall = lcs / n
+    return (2 * precision * recall) / (precision + recall)