Rework triage into a robust, structured pipeline

.gitignore

@@ -22,3 +22,7 @@ dist-ssr
 *.njsproj
 *.sln
 *.sw?
+
+# Local agent/editor tooling
+.claude/
+scratch/

IMPROVEMENTS.md

@@ -0,0 +1,91 @@
+# Customer Inbox Triage — Improvements
+
+A review of the triage tool against Relay AI's goal (let small teams handle more volume
+without adding headcount), the top 3 issues found, and the improvement implemented.
+
+## How I tested
+
+- Ran every message in `sample-messages.json` plus new examples through the actual triage
+  modules (deterministic harness — see `tests/`).
+- Drove the running app in a browser (paste message → Analyze → inspect results).
+
+## Top 3 areas for improvement
+
+### 1. The AI categorization is brittle, non-deterministic, and crashes with no key — *the core product*
+
+- The prompt was `"Categorize this customer support message: <msg>"` with **no system
+  prompt, no category list, and no output format**, at `temperature: 0.7`. The same
+  message could get different categories on re-runs.
+- The category was then recovered by **substring-matching the model's free-text prose**
+  (`content.includes('billing')`, checked billing-first). A reply like *"this is not
+  billing, it's technical"* would mislabel as Billing; prose with none of the keywords
+  fell through to "Unknown." The raw model ramble was shown to the user as "reasoning."
+- **Crash:** the Groq client was constructed at module load with the API key. With no key
+  the SDK **throws during construction**, before React mounts — so the **entire app
+  white-screened**. The `try/catch` only wrapped the API *call*, so the "mock mode (no API
+  key needed)" advertised in the README was **unreachable**.
+
+### 2. Urgency scoring was backwards — it scored surface features, not meaning
+
+In the harness, **7 of 10** sample messages were mislabeled. `"Server down now"`,
+`"Database connection lost"`, and `"Our production server is down"` all scored **Low**,
+while a cheerful `"Thank you!!!"` scored **High**. Why:
+
+- each `!` added +30; **short** messages *subtracted* (terse emergencies looked calm);
+  ALL-CAPS subtracted; questions subtracted;
+- **weekends / after-hours subtracted** — a Saturday-2am outage was *deprioritized*, and
+  the score depended on `new Date()`, making results non-deterministic and untestable.
+
+For a tool whose entire value is surfacing what's urgent, this is the worst failure mode.
+
+### 3. Routing & escalation were wrong, and there was no real "routing"
+
+- `Feature Request → "Ask user to check billing portal"` (copy-paste bug);
+  `Technical Problem → "restart your browser"` regardless of severity;
+  `getRecommendedAction` ignored urgency entirely.
+- `shouldEscalate` was `message.length > 100` and **was never called** — a 41-char
+  *"data breach, customer info exposed"* didn't escalate, but a 123-char thank-you did.
+- Nothing assigned a team/queue, although the product's pitch is to *route* messages.
+
+## What I implemented
+
+A **robust, structured, single-pass triage engine**. These three issues are one pipeline,
+so fixing them together gives the biggest lift.
+
+| File | Change |
+|------|--------|
+| `src/utils/llmHelper.js` | One Groq call with a Relay-AI **system prompt**, the category enum, an explicit **urgency rubric**, **JSON mode** (`response_format: json_object`), and `temperature: 0` → deterministic, structured output. Output is **validated** (unknown enum values → safe defaults) and enriched. **Lazy client init** + try/catch → a missing/invalid key degrades to a local heuristic instead of crashing. |
+| `src/utils/urgencyScorer.js` | Rewritten to score **business impact** (outage, data/security, blocked access, churn/legal, time pressure) instead of punctuation/length/time-of-day. Exposes `hasCriticalSignal()` used as a **hard floor**. |
+| `src/utils/templates.js` | Correct per-category **action + owning team**, sharpened when urgency is High (e.g. "page the on-call engineer"). `shouldEscalate` now keys off urgency + critical signal, not length. |
+| `src/pages/AnalyzePage.jsx` | Consumes the single structured result; adds **input validation**, an **escalation banner**, **route-to team**, **confidence**, and a **heuristic-mode notice**. |
+
+**Design note — why a hybrid (LLM + deterministic floor):** the LLM understands meaning
+(so it fixes the urgency inversion), but a pure-LLM verdict can't guarantee an SLA. The
+`hasCriticalSignal` floor means a clearly critical message can *never* be reported below
+`High`, even on an LLM miss or in offline mode. The same function powers the no-key
+heuristic, so the app is useful and safe before a key is ever configured.
+
+## Results
+
+Before → after on the previously-broken cases (now covered by `npm test`, **14/14**):
+
+| Message | Before | After |
+|---------|--------|-------|
+| `Server down now` | Low · "restart your browser" | **High · Escalate · Technical Support · page on-call** |
+| `Database connection lost` | Low | **High · Escalate** |
+| `Thank you so much!!! … amazing!!!` | High | **Low · no escalation** |
+| `Could you add an export to CSV feature?` | "check billing portal" | **Feature Request · Product backlog** |
+| *(no API key)* | **blank white screen** | **renders, heuristic mode** |
+
+## What I'd do next (not in this change)
+
+- **Move the Groq call to a small backend proxy.** `dangerouslyAllowBrowser: true` ships
+  the API key to every browser. A serverless function keeps the key server-side and adds
+  rate-limiting/auditing. This is the #1 production blocker.
+- **History/Dashboard polish:** History sorts alphabetically by message text instead of by
+  recency; Dashboard's "avg/day" divides by a hard-coded 7. Surface `routedTeam`/`escalate`
+  in History too.
+- **Feedback loop:** let agents correct a verdict and feed corrections back as few-shot
+  examples to raise accuracy over time.
+- **Confidence-gated auto-routing:** auto-route high-confidence messages; send low-confidence
+  ones to a human review queue.

README.md

@@ -62,12 +62,19 @@ Support teams waste time manually reading and triaging customer messages. This t
 
 1. **Paste Message**: User pastes a customer support message into the text area
 2. **Analyze**: Click "Analyze Message" to process the input
-3. **Classification**: The app runs three processes in parallel:
-   - **Category Classification** (LLM): Uses Groq AI (Llama 3.3 70B) to categorize the message
-   - **Urgency Scoring** (Rule-based): Applies simple rules to determine urgency
-   - **Recommendation** (Template-based): Maps category to a recommended action
-4. **Display Results**: Shows category, urgency tag, recommended action, and AI reasoning
-5. **History**: All analyses are saved to localStorage and viewable in the History tab
+3. **Triage**: A single structured Groq call (Llama 3.3 70B, JSON mode, `temperature: 0`)
+   returns `{ category, urgency, escalate, confidence, reasoning }`. The result is then
+   validated and enriched deterministically:
+   - **Urgency floor**: a clearly critical message (outage, data/security, blocked
+     access) can never be reported below `High`, even if the model under-rates it.
+   - **Routing**: the category maps to an owning team and a concrete next step.
+   - **Escalation**: `High` urgency or a critical signal flags the message for
+     immediate human attention.
+4. **Graceful fallback**: with no API key (or if the call fails), the app degrades to a
+   local keyword/impact heuristic instead of crashing — results are tagged "heuristic."
+5. **Display Results**: Shows category, urgency, route-to team, escalation, confidence,
+   recommended action, and reasoning.
+6. **History**: All analyses are saved to localStorage and viewable in the History tab.
 
 
 ## Example Test Messages

package.json

@@ -7,7 +7,8 @@
     "dev": "vite",
     "build": "vite build",
     "lint": "eslint .",
-    "preview": "vite preview"
+    "preview": "vite preview",
+    "test": "node tests/triage.test.mjs"
   },
   "dependencies": {
     "groq-sdk": "^0.37.0",

src/pages/AnalyzePage.jsx

@@ -1,13 +1,12 @@
 import { useState, useEffect } from 'react'
 import ReactMarkdown from 'react-markdown'
-import { categorizeMessage } from '../utils/llmHelper'
-import { calculateUrgency } from '../utils/urgencyScorer'
-import { getRecommendedAction } from '../utils/templates'
+import { triageMessage, isUsingMockMode } from '../utils/llmHelper'
 
 function AnalyzePage() {
   const [message, setMessage] = useState('')
   const [results, setResults] = useState(null)
   const [isLoading, setIsLoading] = useState(false)
+  const [error, setError] = useState('')
 
   useEffect(() => {
     // Check for example message from home page
@@ -18,32 +17,33 @@ function AnalyzePage() {
     }
   }, [])
 
+  const validate = (text) => {
+    const trimmed = text.trim()
+    if (!trimmed) return 'Please enter a message to analyze.'
+    if (trimmed.split(/\s+/).length < 2) {
+      return 'Please enter a more complete message (at least a few words) so triage can be accurate.'
+    }
+    return ''
+  }
+
   const handleAnalyze = async () => {
-    if (!message.trim()) {
-      alert('Please enter a message to analyze')
+    const validationError = validate(message)
+    if (validationError) {
+      setError(validationError)
       return
     }
-
+    setError('')
     setIsLoading(true)
     setResults(null)
-    
+
     try {
-      // Run categorization (LLM call)
-      const { category, reasoning } = await categorizeMessage(message)
-
-      // Calculate urgency (rule-based)
-      const urgency = calculateUrgency(message)
-
-      // Get recommended action (template-based)
-      const recommendedAction = getRecommendedAction(category)
-
+      // One structured triage call returns category, urgency, routing and escalation.
+      const triage = await triageMessage(message)
+
       const analysisResult = {
         message,
-        category,
-        urgency,
-        recommendedAction,
-        reasoning,
-        timestamp: new Date().toISOString()
+        ...triage,
+        timestamp: new Date().toISOString(),
       }
 
       setResults(analysisResult)
@@ -52,9 +52,9 @@ function AnalyzePage() {
       const history = JSON.parse(localStorage.getItem('triageHistory') || '[]')
       history.push(analysisResult)
       localStorage.setItem('triageHistory', JSON.stringify(history))
-    } catch (error) {
-      console.error('Error analyzing message:', error)
-      alert('Error analyzing message. Please try again.')
+    } catch (err) {
+      console.error('Error analyzing message:', err)
+      setError('Something went wrong while analyzing the message. Please try again.')
     } finally {
       setIsLoading(false)
     }
@@ -63,6 +63,7 @@ function AnalyzePage() {
   const handleClear = () => {
     setMessage('')
     setResults(null)
+    setError('')
   }
 
   return (
@@ -74,20 +75,32 @@ function AnalyzePage() {
             Paste a customer support message below to automatically categorize and prioritize.
           </p>
 
+          {isUsingMockMode() && (
+            <div className="mb-4 text-sm bg-amber-50 border border-amber-200 text-amber-800 rounded-lg p-3">
+              <span className="font-semibold">Heuristic mode:</span> no Groq API key is
+              configured, so results come from local keyword/impact rules. Add{' '}
+              <code className="bg-amber-100 px-1 rounded">VITE_GROQ_API_KEY</code> for AI-powered triage.
+            </div>
+          )}
+
           {/* Input Section */}
           <div className="mb-4">
             <label className="block text-sm font-semibold text-gray-700 mb-2">
               Customer Message
             </label>
             <textarea
               value={message}
-              onChange={(e) => setMessage(e.target.value)}
+              onChange={(e) => {
+                setMessage(e.target.value)
+                if (error) setError('')
+              }}
               placeholder="Paste customer message here..."
               className="w-full border border-gray-300 rounded-lg p-3 h-40 focus:ring-2 focus:ring-blue-500 focus:border-blue-500"
               disabled={isLoading}
             />
-            <div className="text-sm text-gray-500 mt-1">
-              {message.length} characters
+            <div className="flex justify-between text-sm mt-1">
+              <span className="text-red-600">{error}</span>
+              <span className="text-gray-500">{message.length} characters</span>
             </div>
           </div>
 
@@ -128,23 +141,50 @@ function AnalyzePage() {
         {results && (
           <div className="bg-white rounded-lg shadow-md p-6">
             <h2 className="text-xl font-bold text-gray-900 mb-4">Analysis Results</h2>
-
+
+            {results.escalate && (
+              <div className="mb-4 flex items-center gap-2 bg-red-50 border border-red-300 text-red-800 rounded-lg p-3">
+                <span className="text-lg">🚨</span>
+                <span className="font-semibold">Escalate now —</span>
+                <span>needs immediate human attention, not the normal queue.</span>
+              </div>
+            )}
+
             <div className="space-y-4">
-              <div>
-                <div className="text-sm font-semibold text-gray-600 mb-1">Category</div>
-                <div className="inline-block bg-blue-100 text-blue-800 px-4 py-2 rounded-lg font-semibold">
-                  {results.category}
+              <div className="grid grid-cols-1 sm:grid-cols-2 gap-4">
+                <div>
+                  <div className="text-sm font-semibold text-gray-600 mb-1">Category</div>
+                  <div className="inline-block bg-blue-100 text-blue-800 px-4 py-2 rounded-lg font-semibold">
+                    {results.category}
+                  </div>
                 </div>
-              </div>
 
-              <div>
-                <div className="text-sm font-semibold text-gray-600 mb-1">Urgency Level</div>
-                <div className={`inline-block px-4 py-2 rounded-lg font-semibold ${
-                  results.urgency === 'High' ? 'bg-red-200 text-red-900' :
-                  results.urgency === 'Medium' ? 'bg-yellow-200 text-yellow-900' :
-                  'bg-green-200 text-green-900'
-                }`}>
-                  {results.urgency}
+                <div>
+                  <div className="text-sm font-semibold text-gray-600 mb-1">Urgency Level</div>
+                  <div className={`inline-block px-4 py-2 rounded-lg font-semibold ${
+                    results.urgency === 'High' ? 'bg-red-200 text-red-900' :
+                    results.urgency === 'Medium' ? 'bg-yellow-200 text-yellow-900' :
+                    'bg-green-200 text-green-900'
+                  }`}>
+                    {results.urgency}
+                  </div>
+                </div>
+
+                <div>
+                  <div className="text-sm font-semibold text-gray-600 mb-1">Route To</div>
+                  <div className="inline-block bg-indigo-100 text-indigo-800 px-4 py-2 rounded-lg font-semibold">
+                    {results.routedTeam}
+                  </div>
+                </div>
+
+                <div>
+                  <div className="text-sm font-semibold text-gray-600 mb-1">Confidence</div>
+                  <div className="inline-block bg-gray-100 text-gray-800 px-4 py-2 rounded-lg font-semibold">
+                    {Math.round((results.confidence ?? 0) * 100)}%
+                    <span className="ml-2 text-xs font-normal text-gray-500">
+                      ({results.source === 'ai' ? 'AI' : 'heuristic'})
+                    </span>
+                  </div>
                 </div>
               </div>
 
@@ -170,9 +210,8 @@ function AnalyzePage() {
             <div className="mt-6 pt-4 border-t border-gray-200">
               <button
                 onClick={() => {
-                  const text = `Category: ${results.category}\nUrgency: ${results.urgency}\nRecommendation: ${results.recommendedAction}\n\nReasoning: ${results.reasoning}`
+                  const text = `Category: ${results.category}\nUrgency: ${results.urgency}\nRoute to: ${results.routedTeam}\nEscalate: ${results.escalate ? 'Yes' : 'No'}\nRecommendation: ${results.recommendedAction}\n\nReasoning: ${results.reasoning}`
                   navigator.clipboard.writeText(text)
-                  alert('Results copied to clipboard!')
                 }}
                 className="bg-gray-100 text-gray-700 px-4 py-2 rounded-lg hover:bg-gray-200 font-semibold"
               >