Skip to content

fix(agent-runtime): a spam verdict withholds the reply instead of sending it - #715

Open
kmonsoe wants to merge 1 commit into
mainfrom
fix/spam-verdict-withholds-reply
Open

fix(agent-runtime): a spam verdict withholds the reply instead of sending it#715
kmonsoe wants to merge 1 commit into
mainfrom
fix/spam-verdict-withholds-reply

Conversation

@kmonsoe

@kmonsoe kmonsoe commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

A cold pitch to hello@getmunin.com got marked as spam and answered — with a polite note telling the sender to forward it to the very address they had just written to.

What was actually broken

Not instruction-following. Control flow.

Spam isn't a tool the agent calls; it's a separate audit pass (packages/agent-runtime/src/audit.ts) invoked after generation but before delivery. It marked the conversation spam via changeStatus, returned, and then the handler unconditionally posted the reply it had already generated. The model did what it was told; the runtime ignored its own verdict.

shouldRespond already skips non-open conversations, so the leak was only ever the in-flight turn.

The fix

runAuditPass now reports the spam verdict alongside the handover reason, and delivery is gated on it. On spam the reply is withheld and parked as a draft_reply instead of discarded — so if the classification was wrong, the reply already exists and a human sends it with one click. No regeneration, and a human looks at it before it goes out, which matters precisely because the trigger is a classifier that just proved unreliable.

Parking needs a way to author a draft without requesting a handover (requestHandover sets needsHumanAttention, which contradicts "this is spam"), so conv.setDraftReply and POST /v1/conversations/:id/draft-reply are new. The endpoint replaces any existing draft rather than stacking, and pre-checks the conversation so an unknown id is a 404 rather than a poisoned outer transaction.

Prompt

system.ts already forbade redirecting people off-channel and volunteering contact details — but the paragraph opened with "When you flag a conversation for a human," so the model read the whole rule as conditional on handover. The pitch reply wasn't a handover, so nothing bound.

That rule is now unconditional, and adds the bit that is genuinely new: never name a contact address the sender has already written to — inbound mail has by definition already reached the right inbox. Plus a short rule that pitches get a two-sentence decline with no redirect, or a handover if they look like real business.

Follow-up not in this PR

Live orgs keep whatever prompt is already in their KB — ensureDocument only seeds when the document is absent, so this reaches new orgs only. Prod, dev and local each need a manual KB edit.

Testing

  • pnpm -F @getmunin/agent-runtime test — 302 pass, including three new cases: spam withholds and parks, a non-spam audit action still sends, and a failed park still withholds.
  • pnpm -F @getmunin/backend-core test — 1339 pass, including two new integration tests for the endpoint (replace-not-stack, 404 on unknown id).
  • pnpm typecheck, lint, openapi:generate clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_011HwhYb3yU3qnGjtFRodQxm

…ding it

The audit pass runs after generation but before delivery, so mark_spam
flipped the conversation to spam and then posted the generated reply
anyway. The verdict now gates delivery: the reply is withheld and parked
as a draft_reply, so a misclassification is one click from recovery.

Adds conv.setDraftReply + POST /v1/conversations/:id/draft-reply to author
a draft without requesting a handover.

Also lifts the seed prompt's no-redirect rule out of the handover
paragraph, where it never bound on replies that weren't handovers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011HwhYb3yU3qnGjtFRodQxm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant