Intent drift is when a model's effective behavior diverges from its original instructions over the course of a conversation — without any single turn containing an overtly rule-breaking request. This repo is a practitioner's map of how it happens and how to detect it.
Also referred to in different corners of the literature as goal drift, policy drift, context drift, or trajectory drift — same underlying phenomenon, different name depending on whether you're coming from safety research, agent evaluation, or applied red teaming.
Every automated red-teaming framework — garak, PyRIT, deepteam, promptfoo — scores a model prompt by prompt. That's the right tool for catching known payload classes fast, at scale, in CI. It structurally cannot catch intent drift, because the failure doesn't live in any one prompt — it lives in the slope of a conversation. A model can pass every automated probe in your pipeline and still be fully compromised by turn 12 of a real session, because nothing in the pipeline ever ran 12 turns.
Intent drift describes the gap between what a model was instructed to do and what it is effectively doing, once that gap has been opened gradually rather than crossed in one step. Three properties distinguish it from an ordinary jailbreak:
- It's cumulative, not instantaneous. The divergence builds across turns; no individual message would trip a static content filter.
- It's measured against a moving reference. Each turn is locally consistent with the one before it — the drift is only visible against the original system prompt or goal, not against the immediately preceding turn.
- It's a property of the conversation, not the model weights. The same model, same weights, same safety training can look perfectly aligned in a fresh session and fully drifted twenty turns into another one.
This matters operationally because most safety evaluation — benchmark scores, red-team reports, guardrail configs — is implicitly single-turn. A model can score well on every static benchmark and still drift in production, because production is where multi-turn sessions actually happen.
Detection is a genuinely different engineering problem from content filtering — you're tracking a trend across a sequence, not classifying a single input. A few approaches, roughly in order of implementation cost:
Embed the system prompt / original stated goal once at session start. After each turn, embed the model's response and compute similarity against that fixed reference — not against the previous turn, which would only catch abrupt jumps, not gradual erosion.
goal_vector = embed(system_prompt)
score_t = cosine_similarity(embed(response_t), goal_vector)
Track score_t as a rolling series per session rather than a single number. A steady downward trend across turns is a stronger signal than any single low score — that's the actual "drift" in intent drift.
Point an LLM-as-judge at the full transcript, not the final message, and ask it to score alignment between the conversation's current effective goal and its stated original goal. This catches the case content-filtering misses entirely: a session that ends on an innocuous message but got there by extracting the same information across five "partial" answers instead of one refused one.
Periodically re-inject a probe adjacent to your baseline refusal category mid-session, and compare the response against the same probe run fresh at session start. If the boundary has moved, that's the finding — independent of whether anything overtly harmful was said.
Track the topic/embedding distribution of a session in fixed-size windows and measure distributional shift (e.g. a divergence metric) between the first window and each subsequent one. This is the same primitive used for data/concept drift detection in classical ML monitoring, applied to conversation state instead of a feature distribution.
No tool is purpose-built for intent-drift detection yet, but the primitives exist in adjacent LLM-observability platforms and can be repurposed:
| Tool | What it gives you toward drift detection |
|---|---|
| Langfuse | Full session tracing — the raw data a goal-consistency scorer needs |
| Phoenix (Arize) | OTEL-native tracing, good for instrumenting a custom drift metric into an existing pipeline |
| WhyLabs langkit | Statistical distribution-shift detection, built for exactly the drift-monitoring primitive above, not yet LLM-conversation-specific |
| Evidently AI | ML monitoring with drift metrics; usable as a backend for session-level anomaly scoring |
None of these ship an "intent drift" metric out of the box today — that gap is a large part of why this repo exists.
Named at a conceptual level: this is a map for talking about the problem, not an operator's manual. Each pattern below is a class of drift-inducing mechanism, not a walkthrough.
| Pattern | What it is | Related Research |
|---|---|---|
| Gradual Drift ("boiling frog") | Incremental escalation across turns rather than a single crossing of the refusal boundary. | Crescendo (Russinovich, Salem, Eldan — Microsoft, 2024) |
| Role Transformation / Persona Anchoring | A persona established early is leveraged for its obligations later. | Generalization of DAN-family jailbreaks; interacts with Many-shot Jailbreaking (Anthropic, 2024) |
| Language / Register Variation | Shifting language or formality mid-conversation, off the distribution safety training mostly covers. | Low-Resource Languages Jailbreak GPT-4 (Yong et al., 2023) |
| Context Window Erosion | Long benign context dilutes the relative weight of system-level instructions. | Shares its enabling condition with many-shot jailbreaking |
| Trust Escalation ("foot-in-the-door") | Small approved requests cited back as precedent for a larger one. | Freedman & Fraser (1966) compliance-escalation effect, applied to LLMs |
| Authority / Urgency Injection (mid-session) | A claimed permission change introduced after rapport is established, not as an opening move. | Related to instruction-override research |
| Narrative Commitment / Sunk-Cost Framing | Refusal framed as breaking continuity in an established collaborative fiction. | Overlaps with role-play jailbreak literature |
| Chain-of-Thought Steering | The model's own reasoning trace is nudged toward a self-generated justification for compliance. | Follows Unit 42's "Bad Likert Judge" (2024) research direction |
| Cross-Session Memory / Tool-State Poisoning | An instruction planted in one agent session activates in a later, unrelated one. | Temporal extension of indirect prompt injection (Greshake et al., 2023) |
Testing for these is a different shape of exercise than running an automated probe suite: fewer conversations, each deliberately driven toward one pattern, with the signal being where a boundary moved rather than whether it eventually did. The operational detail is intentionally out of scope here — this repo is a map of the problem space, not a runbook.
- ASR is measured per-prompt. Automated frameworks report attack success rate against individual probes. There's no widely-adopted equivalent for "policy divergence over N turns," so drift is structurally under-measured, not just under-tested.
- Context resets between probes. Scanner architectures parallelize for speed — each probe typically starts fresh. That's exactly the condition under which drift cannot occur, since drift needs accumulated context.
- Judge models inherit the same blind spot. An LLM-as-judge grading a transcript usually scores the final message rather than the trajectory — see Trajectory-level judge scoring above for the fix.
None of this is undefeatable by design — it's under-tooled, not unsolvable:
- Session-level anomaly detection over per-message filtering, using the detection approaches above.
- Periodic re-grounding — re-inject the system prompt's core constraints at intervals instead of relying on it holding weight for the entire context window.
- Persona-boundary invariance testing — explicitly test whether the refusal boundary moves under role-play framing; if it does, that's the finding, independent of any specific harmful output.
- Treat long agentic sessions as a distinct threat surface from single-shot chat — see the runtime/agentic security sections of ai-security-resources for the tool/memory-poisoning half of this problem.
- Crescendo: Multi-Turn LLM Jailbreak Attack (Russinovich, Salem, Eldan — Microsoft, 2024)
- Many-shot Jailbreaking (Anil et al. — Anthropic, 2024)
- Low-Resource Languages Jailbreak GPT-4 (Yong, Menghini, Bach, 2023)
- Prompt Injection Attacks Against LLM-Integrated Applications (Greshake et al., 2023)
- OWASP GenAI Red Teaming Guide — its four-phase methodology maps closely onto multi-turn evaluation
- Palo Alto Networks Unit 42 research on multi-turn narrative and self-evaluation jailbreaks ("Deceptive Delight," "Bad Likert Judge") — unit42.paloaltonetworks.com
- ai-security-resources — the full AI security handbook this taxonomy sits alongside: automated red-teaming frameworks, runtime guardrails, model scanning, standards, and career resources.
- prompt-injection-ctf — a hands-on, no-signup CTF to build intuition for several of the patterns above interactively.
- AgentInjectionBench — a benchmark for the agentic-injection half of this problem (tool-output injection, goal hijacking).
Pattern proposals, detection methodology, corrected/added citations, and tooling notes are all welcome — open a PR. Keep additions conceptual, not one-off payloads: a pattern belongs here if it exploits how models generalize across turns, not because a specific phrasing happened to work once.
Maintained by @ppradyoth.