From 1612c789e13f5bad3c74bdeebb3e5994adb42b86 Mon Sep 17 00:00:00 2001 From: Trevor Rukwava Date: Wed, 17 Jun 2026 18:27:36 -0400 Subject: [PATCH] Rework triage into a robust structured pipeline Replaces the brittle categorizer, the backwards urgency scorer, and the length-based routing with a single structured AI call plus deterministic safety nets. Problems fixed: - No-API-key white screen: the Groq client was built at module load and threw before React mounted, making the advertised "mock mode" unreachable. Client is now lazy; a missing/failed key degrades to a local heuristic. - Categorization was a free-text prompt at temp 0.7 parsed by substring matching. Now one Groq call with a system prompt, category enum, JSON mode and temp 0, with output validation and safe defaults. - Urgency scored punctuation/length/time-of-day, inverting reality (7/10 sample messages mislabeled; "server down now" -> Low, "thanks!!!" -> High). Rewritten to score business impact, with a critical-signal floor that can never report a clear outage/security/access issue below High. - Routing sent Feature Requests to the billing portal and escalated on message length. Now correct per-category action + owning team, and escalation keyed off urgency/critical signal. Also: input validation, escalation/route-to/confidence in the UI, a heuristic-mode notice, a committed test (npm test, 14/14), and an IMPROVEMENTS.md writeup. Co-Authored-By: Claude Opus 4.8 --- .gitignore | 4 + IMPROVEMENTS.md | 91 ++++++++++++ README.md | 19 ++- package.json | 3 +- src/pages/AnalyzePage.jsx | 125 ++++++++++------ src/utils/llmHelper.js | 294 +++++++++++++++++++------------------ src/utils/templates.js | 96 ++++++++---- src/utils/urgencyScorer.js | 111 +++++++++----- tests/triage.test.mjs | 62 ++++++++ 9 files changed, 550 insertions(+), 255 deletions(-) create mode 100644 IMPROVEMENTS.md create mode 100644 tests/triage.test.mjs diff --git a/.gitignore b/.gitignore index a547bf3..776f4c1 100644 --- a/.gitignore +++ b/.gitignore @@ -22,3 +22,7 @@ dist-ssr *.njsproj *.sln *.sw? + +# Local agent/editor tooling +.claude/ +scratch/ diff --git a/IMPROVEMENTS.md b/IMPROVEMENTS.md new file mode 100644 index 0000000..0161f5d --- /dev/null +++ b/IMPROVEMENTS.md @@ -0,0 +1,91 @@ +# Customer Inbox Triage — Improvements + +A review of the triage tool against Relay AI's goal (let small teams handle more volume +without adding headcount), the top 3 issues found, and the improvement implemented. + +## How I tested + +- Ran every message in `sample-messages.json` plus new examples through the actual triage + modules (deterministic harness — see `tests/`). +- Drove the running app in a browser (paste message → Analyze → inspect results). + +## Top 3 areas for improvement + +### 1. The AI categorization is brittle, non-deterministic, and crashes with no key — *the core product* + +- The prompt was `"Categorize this customer support message: "` with **no system + prompt, no category list, and no output format**, at `temperature: 0.7`. The same + message could get different categories on re-runs. +- The category was then recovered by **substring-matching the model's free-text prose** + (`content.includes('billing')`, checked billing-first). A reply like *"this is not + billing, it's technical"* would mislabel as Billing; prose with none of the keywords + fell through to "Unknown." The raw model ramble was shown to the user as "reasoning." +- **Crash:** the Groq client was constructed at module load with the API key. With no key + the SDK **throws during construction**, before React mounts — so the **entire app + white-screened**. The `try/catch` only wrapped the API *call*, so the "mock mode (no API + key needed)" advertised in the README was **unreachable**. + +### 2. Urgency scoring was backwards — it scored surface features, not meaning + +In the harness, **7 of 10** sample messages were mislabeled. `"Server down now"`, +`"Database connection lost"`, and `"Our production server is down"` all scored **Low**, +while a cheerful `"Thank you!!!"` scored **High**. Why: + +- each `!` added +30; **short** messages *subtracted* (terse emergencies looked calm); + ALL-CAPS subtracted; questions subtracted; +- **weekends / after-hours subtracted** — a Saturday-2am outage was *deprioritized*, and + the score depended on `new Date()`, making results non-deterministic and untestable. + +For a tool whose entire value is surfacing what's urgent, this is the worst failure mode. + +### 3. Routing & escalation were wrong, and there was no real "routing" + +- `Feature Request → "Ask user to check billing portal"` (copy-paste bug); + `Technical Problem → "restart your browser"` regardless of severity; + `getRecommendedAction` ignored urgency entirely. +- `shouldEscalate` was `message.length > 100` and **was never called** — a 41-char + *"data breach, customer info exposed"* didn't escalate, but a 123-char thank-you did. +- Nothing assigned a team/queue, although the product's pitch is to *route* messages. + +## What I implemented + +A **robust, structured, single-pass triage engine**. These three issues are one pipeline, +so fixing them together gives the biggest lift. + +| File | Change | +|------|--------| +| `src/utils/llmHelper.js` | One Groq call with a Relay-AI **system prompt**, the category enum, an explicit **urgency rubric**, **JSON mode** (`response_format: json_object`), and `temperature: 0` → deterministic, structured output. Output is **validated** (unknown enum values → safe defaults) and enriched. **Lazy client init** + try/catch → a missing/invalid key degrades to a local heuristic instead of crashing. | +| `src/utils/urgencyScorer.js` | Rewritten to score **business impact** (outage, data/security, blocked access, churn/legal, time pressure) instead of punctuation/length/time-of-day. Exposes `hasCriticalSignal()` used as a **hard floor**. | +| `src/utils/templates.js` | Correct per-category **action + owning team**, sharpened when urgency is High (e.g. "page the on-call engineer"). `shouldEscalate` now keys off urgency + critical signal, not length. | +| `src/pages/AnalyzePage.jsx` | Consumes the single structured result; adds **input validation**, an **escalation banner**, **route-to team**, **confidence**, and a **heuristic-mode notice**. | + +**Design note — why a hybrid (LLM + deterministic floor):** the LLM understands meaning +(so it fixes the urgency inversion), but a pure-LLM verdict can't guarantee an SLA. The +`hasCriticalSignal` floor means a clearly critical message can *never* be reported below +`High`, even on an LLM miss or in offline mode. The same function powers the no-key +heuristic, so the app is useful and safe before a key is ever configured. + +## Results + +Before → after on the previously-broken cases (now covered by `npm test`, **14/14**): + +| Message | Before | After | +|---------|--------|-------| +| `Server down now` | Low · "restart your browser" | **High · Escalate · Technical Support · page on-call** | +| `Database connection lost` | Low | **High · Escalate** | +| `Thank you so much!!! … amazing!!!` | High | **Low · no escalation** | +| `Could you add an export to CSV feature?` | "check billing portal" | **Feature Request · Product backlog** | +| *(no API key)* | **blank white screen** | **renders, heuristic mode** | + +## What I'd do next (not in this change) + +- **Move the Groq call to a small backend proxy.** `dangerouslyAllowBrowser: true` ships + the API key to every browser. A serverless function keeps the key server-side and adds + rate-limiting/auditing. This is the #1 production blocker. +- **History/Dashboard polish:** History sorts alphabetically by message text instead of by + recency; Dashboard's "avg/day" divides by a hard-coded 7. Surface `routedTeam`/`escalate` + in History too. +- **Feedback loop:** let agents correct a verdict and feed corrections back as few-shot + examples to raise accuracy over time. +- **Confidence-gated auto-routing:** auto-route high-confidence messages; send low-confidence + ones to a human review queue. diff --git a/README.md b/README.md index 340381a..511ca43 100644 --- a/README.md +++ b/README.md @@ -62,12 +62,19 @@ Support teams waste time manually reading and triaging customer messages. This t 1. **Paste Message**: User pastes a customer support message into the text area 2. **Analyze**: Click "Analyze Message" to process the input -3. **Classification**: The app runs three processes in parallel: - - **Category Classification** (LLM): Uses Groq AI (Llama 3.3 70B) to categorize the message - - **Urgency Scoring** (Rule-based): Applies simple rules to determine urgency - - **Recommendation** (Template-based): Maps category to a recommended action -4. **Display Results**: Shows category, urgency tag, recommended action, and AI reasoning -5. **History**: All analyses are saved to localStorage and viewable in the History tab +3. **Triage**: A single structured Groq call (Llama 3.3 70B, JSON mode, `temperature: 0`) + returns `{ category, urgency, escalate, confidence, reasoning }`. The result is then + validated and enriched deterministically: + - **Urgency floor**: a clearly critical message (outage, data/security, blocked + access) can never be reported below `High`, even if the model under-rates it. + - **Routing**: the category maps to an owning team and a concrete next step. + - **Escalation**: `High` urgency or a critical signal flags the message for + immediate human attention. +4. **Graceful fallback**: with no API key (or if the call fails), the app degrades to a + local keyword/impact heuristic instead of crashing — results are tagged "heuristic." +5. **Display Results**: Shows category, urgency, route-to team, escalation, confidence, + recommended action, and reasoning. +6. **History**: All analyses are saved to localStorage and viewable in the History tab. ## Example Test Messages diff --git a/package.json b/package.json index d348c42..d8af0bd 100644 --- a/package.json +++ b/package.json @@ -7,7 +7,8 @@ "dev": "vite", "build": "vite build", "lint": "eslint .", - "preview": "vite preview" + "preview": "vite preview", + "test": "node tests/triage.test.mjs" }, "dependencies": { "groq-sdk": "^0.37.0", diff --git a/src/pages/AnalyzePage.jsx b/src/pages/AnalyzePage.jsx index 64517d7..7df2ce4 100644 --- a/src/pages/AnalyzePage.jsx +++ b/src/pages/AnalyzePage.jsx @@ -1,13 +1,12 @@ import { useState, useEffect } from 'react' import ReactMarkdown from 'react-markdown' -import { categorizeMessage } from '../utils/llmHelper' -import { calculateUrgency } from '../utils/urgencyScorer' -import { getRecommendedAction } from '../utils/templates' +import { triageMessage, isUsingMockMode } from '../utils/llmHelper' function AnalyzePage() { const [message, setMessage] = useState('') const [results, setResults] = useState(null) const [isLoading, setIsLoading] = useState(false) + const [error, setError] = useState('') useEffect(() => { // Check for example message from home page @@ -18,32 +17,33 @@ function AnalyzePage() { } }, []) + const validate = (text) => { + const trimmed = text.trim() + if (!trimmed) return 'Please enter a message to analyze.' + if (trimmed.split(/\s+/).length < 2) { + return 'Please enter a more complete message (at least a few words) so triage can be accurate.' + } + return '' + } + const handleAnalyze = async () => { - if (!message.trim()) { - alert('Please enter a message to analyze') + const validationError = validate(message) + if (validationError) { + setError(validationError) return } - + setError('') setIsLoading(true) setResults(null) - + try { - // Run categorization (LLM call) - const { category, reasoning } = await categorizeMessage(message) - - // Calculate urgency (rule-based) - const urgency = calculateUrgency(message) - - // Get recommended action (template-based) - const recommendedAction = getRecommendedAction(category) - + // One structured triage call returns category, urgency, routing and escalation. + const triage = await triageMessage(message) + const analysisResult = { message, - category, - urgency, - recommendedAction, - reasoning, - timestamp: new Date().toISOString() + ...triage, + timestamp: new Date().toISOString(), } setResults(analysisResult) @@ -52,9 +52,9 @@ function AnalyzePage() { const history = JSON.parse(localStorage.getItem('triageHistory') || '[]') history.push(analysisResult) localStorage.setItem('triageHistory', JSON.stringify(history)) - } catch (error) { - console.error('Error analyzing message:', error) - alert('Error analyzing message. Please try again.') + } catch (err) { + console.error('Error analyzing message:', err) + setError('Something went wrong while analyzing the message. Please try again.') } finally { setIsLoading(false) } @@ -63,6 +63,7 @@ function AnalyzePage() { const handleClear = () => { setMessage('') setResults(null) + setError('') } return ( @@ -74,6 +75,14 @@ function AnalyzePage() { Paste a customer support message below to automatically categorize and prioritize.

+ {isUsingMockMode() && ( +
+ Heuristic mode: no Groq API key is + configured, so results come from local keyword/impact rules. Add{' '} + VITE_GROQ_API_KEY for AI-powered triage. +
+ )} + {/* Input Section */}