Skip to content

feat(risk): add listRiskResultsForAgent for AI Insights with redacted match#2922

Merged
simplesagar merged 8 commits into
mainfrom
worktree-insights-risk-tools
May 22, 2026
Merged

feat(risk): add listRiskResultsForAgent for AI Insights with redacted match#2922
simplesagar merged 8 commits into
mainfrom
worktree-insights-risk-tools

Conversation

@simplesagar
Copy link
Copy Markdown
Member

@simplesagar simplesagar commented May 19, 2026

Summary

Adds tools for the dashboard's AI Insights sidebar to reason about risk overview, policies, and findings — without ever loading raw secret content into the model context.

  • New endpoint risk.results.listForAgent mirrors risk.results.list but replaces match with match_redacted, an opaque token <redacted len=N sha=XXXXXXXX> where sha is sha256(orgID || 0x00 || match) truncated to the first 8 hex chars. Per-org salting means identical secrets dedupe within an org but never correlate across orgs. shadow_mcp findings pass match through verbatim (server URL / stdio command — already shown unmasked in the dashboard). start_pos/end_pos are coarsened to a single position_known boolean to remove reconstruction signals.
  • <InsightsConfig> is now mounted on SecurityOverview and PolicyCenter with risk-aware suggestion prompts and page-specific contextInfo.
  • Insights system prompt gains a risk/policy vocabulary block plus a HARD RULE against echoing match_redacted values.

The existing risk.results.list endpoint is unchanged — the dashboard UI still uses it for the click-to-reveal flow on the findings table.

Why this shape

The safest of the options discussed: dedicated redacted endpoint over a ?redact=true flag, so the LLM cannot opt out of redaction. The sha prefix gives agents enough signal to dedupe within an org (e.g. "12 findings of the same OPENAI_KEY across 5 chats") without ever holding a secret in context. Mixing orgID into the hash with a NUL separator is defense-in-depth: even if a future code path widens the endpoint surface beyond the current org-scoped access control, the fingerprints can never correlate the same secret across orgs.

TODOs

  • Configure the speakeasy-team-gram MCP toolset (production Gram instance) to expose the new + safe-to-reuse risk tools. Include:
    • listRiskResultsForAgent (new, redacted)
    • listRiskResultsByChat (already safe — no match field)
    • listRiskPolicies
    • getRiskPolicy
    • getRiskCapabilities
    • getRiskPolicyStatus
    • listShadowMCPApprovals
    • Do not add the mutators (approveShadowMCP, revokeShadowMCPApproval, triggerRiskAnalysis, create/update/deleteRiskPolicy).
  • Smoke-test in dev after the toolset is updated: open AI Insights on /security and /security/policies, try each suggestion prompt, confirm no match_redacted values appear in assistant output.

Non-TODOs (deliberately deferred)

  • Telemetry — every Goa method already gets its own trace span via middleware.TraceMethods, and the Insights sidebar tags requests with X-Gram-Source: dashboard-ai-insights. Those two signals together answer "how often does the AI Insights feature use risk tools" better than a per-endpoint counter would. Revisit only if we want a hard SLI panel.
  • HMAC-secret salting — the current per-org salt uses orgID directly, which prevents cross-org fingerprint correlation. A secret-keyed HMAC would additionally defend against an attacker with raw DB read access guessing fingerprints offline, but that threat model isn't in scope at this layer today. Revisit if redaction outputs ever flow to a less-trusted store.
  • User-facing docs — internal-only surface for now; revisit when the AI Insights feature is documented externally.

Test plan

  • mise gen:goa-server + mise gen:sdk regenerate cleanly
  • mise build:server — green
  • go test ./server/internal/risk/... — all tests pass including 9 new ones:
    • Integration (results_test.go): TestListRiskResultsForAgent_RedactsGitleaksMatch, _ShadowMCPPassthrough, _DeterministicFingerprintWithinOrg, _Unauthorized
    • White-box (redact_internal_test.go): TestRedactMatch_FingerprintDiffersAcrossOrgs, _FingerprintDeterministicWithinOrg, _ShadowMCPIgnoresSalt, _EmptyMatchCollapses, _NULSeparatorPreventsBoundaryAmbiguity
  • cd client/sdk && pnpm build — green
  • cd client/dashboard && pnpm tsc -p tsconfig.app.json --noEmit — 3 unrelated pre-existing errors (Table.Body / AppPortal JSX-component issues), zero new errors from this change
  • Security review (/security-review): no findings (authz delegation verified, sha256 prefix non-reversible at the threat-model boundary, shadow_mcp passthrough matches existing dashboard exposure, redacted shape exposes no new side channels)
  • Manual smoke test in dev once the toolset is updated (see TODOs)

🤖 Generated with Claude Code

@simplesagar simplesagar requested review from a team as code owners May 19, 2026 14:22
@simplesagar simplesagar added enhancement New feature or request go Pull requests that update go code labels May 19, 2026
Copy link
Copy Markdown

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented May 19, 2026

🦋 Changeset detected

Latest commit: a0194d2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
server Minor
dashboard Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel
Copy link
Copy Markdown

vercel Bot commented May 19, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
gram-docs-redirect Ready Ready Preview, Comment May 22, 2026 2:26pm

Request Review

@simplesagar simplesagar requested a review from vishalg0wda May 19, 2026 14:32
@simplesagar simplesagar force-pushed the worktree-insights-risk-tools branch from 64b8965 to d0cb9bc Compare May 19, 2026 14:40
@simplesagar simplesagar added the preview Spawn a preview environment label May 19, 2026
@simplesagar
Copy link
Copy Markdown
Member Author

@claude review

Copy link
Copy Markdown

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Code review skipped — your organization's overage spend limit has been reached.

Code review is billed via overage credits. To resume reviews, an organization admin can raise the monthly limit at claude.ai/admin-settings/claude-code.

Once credits are available, comment @claude review on this pull request to trigger a review.

@speakeasybot
Copy link
Copy Markdown
Collaborator

speakeasybot commented May 19, 2026

🚀 Preview Environment (PR #2922)

Preview URL: https://pr-2922.dev.getgram.ai

Component Status Details Updated (UTC)
⏳ Database Pending Waiting for db-init job 2026-05-22 14:33:03.
✅ Images Available Container images ready 2026-05-22 14:33:01.

Gram Preview Bot

@claude
Copy link
Copy Markdown

claude Bot commented May 19, 2026

Claude encountered an error after 0s —— View job


I'll analyze this and get back to you.

simplesagar and others added 6 commits May 22, 2026 06:25
… match

Adds a redacted variant of risk.results.list designed for MCP / AI assistant
consumption. The raw `match` field is replaced with `match_redacted`, an
opaque token of the form `<redacted len=N sha=XXXXXXXX>`, where the sha
prefix is deterministic so agents can dedupe identical secrets across chats
without ever seeing secret content. `shadow_mcp` findings pass `match`
through verbatim because the value is a non-sensitive server identifier
already shown unmasked in the dashboard.

Also wires `<InsightsConfig>` overrides into the Security Overview and Policy
Center pages with risk-aware suggestions, and extends the Insights system
prompt with a hard rule against echoing `match_redacted` values.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's nolintlint flags //nolint suppressions that the underlying linter
wouldn't have raised — gosec didn't actually warn on int32(len(match))
in the test fixture, so the directive is dead weight. Removing it
satisfies nolintlint without changing test behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mixes the active organization ID into sha256(match) with a NUL byte
separator so the redacted fingerprint can never correlate the same
secret across organizations. Within an org the fingerprint stays
deterministic so agents can still dedupe findings of the same secret
across chats.

Defense in depth: the endpoint is already gated by org:admin and the
auth context is project-scoped, so cross-org reads aren't possible
today. The salt removes the dependency on access-control correctness
for the cross-org non-correlation property.

Adds white-box tests for the helper (cross-org differs, same-org
deterministic, shadow_mcp ignores salt, empty collapses, NUL separator
prevents boundary ambiguity). The integration test environment hard-
codes mockidp.MockOrgID across instances so the cross-org assertion
has to live at the unit-test layer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
main's risk-overview-analytics PR added category, rule_id, unique_match,
from, and to filter fields to listRiskResults. The agent endpoint
constructs a ListRiskResultsPayload literal directly, so exhaustruct
rightly flagged the new fields as missing. Add them to the agent
endpoint's payload too (the LLM benefits from the same filters — and
from/to in particular lets the agent honor the dashboard's selected
date range) and thread them through.

Also reverts mise.lock to main's version; the only diff was the
autogenerated doc-URL header (mise.jdx.dev vs mise.en.dev) from a
locally-stale mise install, which CI rebuilds clean and was failing
git:porcelain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@simplesagar simplesagar added this pull request to the merge queue May 22, 2026
Merged via the queue into main with commit 3b8bfb4 May 22, 2026
56 of 58 checks passed
@simplesagar simplesagar deleted the worktree-insights-risk-tools branch May 22, 2026 14:55
@github-actions github-actions Bot locked and limited conversation to collaborators May 22, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

enhancement New feature or request go Pull requests that update go code preview Spawn a preview environment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants