Add threat-report correlation#62
Draft
seth-goodwin wants to merge 9 commits into
Draft
Conversation
ES-retrieval-only addon for threat correlation; the host LLM reasons, guided by a vendored tradecraft rubric. diamond_search returns blind candidate stubs (no scores) from semantic_text kNN + IOC anchors + BM25; get_report batch-fetches full text by id. No LLM calls, no creds beyond ES. Retrieval vendored from the Kibana IntelligenceHub engine. NOTE: requires a custom threat-report corpus index (not stock Elastic) — see src/correlation/README.md. Self-contained: new files + 2 additive lines in server.ts.
The transparent A/B sibling to the blind correlation path. diamond_search_analyst returns scored candidates (per-vertex match scores, overlap, thin-coverage signal) for analyst-supervised triage. correlation_input_check is a display-only gate: the host LLM passes its per-vertex case summaries + self-rated signal (HIGH/PARTIAL/NONE), the analyst reviews a stoplight checkpoint and decides whether to search — "Search this case" injects a proceed message (app.sendMessage, updateModelContext fallback) carrying the vertex queries so the LLM runs the search immediately; "I'll revise first" returns to chat. Two inline views: the stoplight input gate + a ranked collapsed-diamond triage list with a thin-coverage backfill nudge. No LLM calls in the tools (host reasons). The blind diamond_search/get_report path is unchanged. Self-contained: new view dirs + scored types/method + gate tool; no server.ts change.
From the triage view, the analyst selects report(s) or a cluster and hits Synthesize → app.sendMessage tells the host to get_report the full text, synthesize per the vendored rubric, then call render_correlation with the result. render_correlation is a pure pass-through (no ES, no synthesis — the host does all reasoning) that hands the structured CorrelationFindings to a new deep-dive view: BLUF + correlation-signal stoplight, lead cards with collapsed/expanded diamonds + weighted evidence (supporting/counter), no-match list, synthesis + next steps. Completes the gate → search → triage → deep-dive flow. Additive: new view + render tool + triage-view selection; blind path and prior tools unchanged.
manifest.json: register the 5 correlation tools (diamond_search, get_report, diamond_search_analyst, correlation_input_check, render_correlation). skills/threat-correlation/SKILL.md: host-LLM steering — analyst-led path (input gate → scored triage → get_report → render) primary, blind path alternate. src/tools/correlation.test.ts: cover all 5 tools (pass-through tools assert no service calls; blind vs. scored stubs; input-gate branches; UI resources).
correlation-input view: the "Search this case" button drove a new turn via app.sendMessage(), unsupported on Claude Desktop — removed it for a passive prompt; the conversation is the control surface. SKILL.md: gate now offers Full run (autonomous, ~10-candidate cap) vs Analyst-led (interactive) — both use the same 5 tools. analytics: correlation views reported viewId "threat-hunt"; extend VIEW_IDS and label each view correctly.
Route authoritative correlation through the ti-correlation Kibana workflow (correlate -> get_correlation_run -> render_correlation); keep diamond_search, diamond_search_analyst, get_report, and correlation_input_check as analyst exploration aids and mark host-driven synthesis deprecated. Roll the canvas generator into src/canvas (emitCorrelationCanvas): stamp a run's render-shape findings into correlation-report.canvas.tmpl, opt-in via CORRELATION_CANVAS_DIR. Add dev CLIs (gen-correlation-canvas, dump-run-findings, smoke-correlate). Hygiene: fix stale server integration tests (correlation tools + views), drop a hardcoded personal canvas path from gen-correlation-canvas, and document correlation/canvas env vars in .env.example. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop internal codenames from comments (no behavior change): remove the "Mustard" sort-key reference and generalize "threat-intel-ingest" corpus wording in correlationService and .env.example. Index pattern stays env-configurable via TI_REPORTS_INDEX_PATTERN (default ti-reports*). Co-authored-by: Cursor <cursoragent@cursor.com>
seth-goodwin
changed the title
ClusterCredentials now requires sslVerify; the correlation dev/smoke scripts built creds without it, breaking `tsc --noEmit`. Default to true (verify), overridable via ELASTIC_SSL_VERIFY=false for self-signed dev clusters. Co-authored-by: Cursor <cursoragent@cursor.com>
seth-goodwin
force-pushed
the
threat-correlation
branch
from
July 13, 2026 19:55
fae9f01 to
1bf524a
Compare
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an end-to-end threat-report correlation capability to the example security MCP app: correlate a case — a stored corpus report or pasted raw text — against the report corpus and return structured Diamond-model findings.
The authoritative path is a server-side
ti-correlationKibana Workflow: retrieval (exact anchor + Diamond-vertex kNN + BM25) → Sonnet triage → Opus synthesis with a rank-leak firewall. The app triggers the workflow and pollsti-correlationsfor the authoritative findings. The retrieval tools are kept as analyst exploration aids only.What's added
correlate,get_correlation_run,diamond_search,diamond_search_analyst,get_report,correlation_input_check,render_correlation.correlationService.ts— retrieval, triage-pool shaping, workflow trigger/poll.correlation-input,correlation(scored triage),correlation-report(Diamond render with colored vertices, smoking-gun vs supporting evidence, BLUF, gaps, source links).correlation-report.canvas.tmpl+gen-correlation-canvas.ts— paint-by-numbers Cursor canvas generated from a run record.skills/threat-correlation/SKILL.md.correlation.test.ts+ integration updates (full suite: 399 passing).Configuration
TI_REPORTS_INDEX_PATTERN(defaultti-reports*) — the report corpus the tools search.Notes
extracted.diamond.*fields (produced by a compatible ingest/enrichment pipeline).TODO
Test plan
npm testgreen (399 tests)correlatewith a storedreport_idreturns findings from the workflowcorrelatewithraw_textruns the raw-text pathrender_correlationproduces the report view / canvas from a run recorddiamond_search,diamond_search_analyst,get_report) work against a populated corpusMade with Cursor