interp — ask your coding agent why a language model said that

A Claude Code skill (any agent can drive the JSON CLI) that turns a coding agent into a mechanistic interpretability assistant. Instead of guessing about a model's internals, your agent measures them — logit lens, direct logit attribution, activation patching, induction heads, SAE features, and steering — and writes a grounded, figure-backed answer. Every technique is validated against published circuits, and it runs on a laptop CPU.

You:   Why does GPT-2 complete "When John and Mary went to the store, John
       gave a drink to" with " Mary" and not " John"?

Agent: [runs `interp explain ...`]
       It's the IOI circuit. " Mary" is undecidable through layer 8 (rank 265),
       collapses to rank 14 at layer 9, and is the top guess from layer 10 on
       (final P ≈ 0.48). Three heads write " Mary" over " John": L9H9 (+1.66),
       L9H6 (+1.32), L10H0 (+0.88). Causally, head patching surfaces the
       S-inhibition heads L8H6/L8H10 and the negative name mover L10H7 as the
       heads carrying the signal — matching Wang et al. 2022.

---

Why this exists

LLM coding agents are great at using models and terrible at explaining them — ask one why a model predicted a token and you get plausible-sounding fiction. Mechanistic interpretability has the right tools to answer for real, but they live in research notebooks, not in a form an agent can call. interp packages the core mech-interp toolkit behind a single CLI and a SKILL.md, so any agent can answer "why?" with numbers it computed itself.

Install

As an agent skill (Claude Code, etc.): drop this folder into your skills directory (e.g. .claude/skills/interp/). The agent reads SKILL.md and learns when and how to use it. Then, from the skill's directory, once:

./run.sh setup

As a standalone CLI / library:

pip install -e .          # or: pip install -r requirements.txt
interp explain "..." " Mary" --baseline " John" --corrupt "..."

Honest timing (GPT-2 small on a laptop CPU): setup is a cold start of ~8 minutes — a torch install plus a ~500 MB model download. Single techniques (logit lens, DLA, attention, induction) take seconds to a minute. explain runs the full patching scans (~330 forward passes) and takes a few minutes. Runtime output goes to a gitignored out/ directory; a committed sample report lives at reports/explain.md.

What it does

Command	Technique	Question it answers
logit-lens	Logit lens	How does the prediction form across layers?
dla	Direct logit attribution	Which heads/MLPs write the answer? (correlational)
patch / patch-heads	Activation patching	What does the model causally depend on?
induction	Induction-head detection	Which heads do in-context copying?
attention	Attention patterns	Where does a head look?
sae	SAE features (Neuronpedia)	Which interpretable features fire on this text?
steer	Contrastive activation steering	Does moving along a direction change behavior?
explain	All of the above	Full grounded report + figures for one prediction

Add --json to any command for machine-readable output (what the agent parses).

Validated against published results

This repo's correctness contract: the tools are trusted only because they independently recover circuits other researchers documented. ./run.sh test checks this every run.

Result	Published source	What interp recovers
IOI name-mover heads	Wang et al. 2022	DLA top heads include L9H9, L9H6, L10H0
IOI causal heads	Wang et al. 2022	Head patching surfaces S-inhibition heads L8H6, L8H10 and negative name mover L10H7
Induction heads	Olsson et al. 2022	Top heads include L5H5, L5H1, L6H9, L7H2, L7H10
Logit-lens trajectory	nostalgebraist 2020	Answer rank: ~43k → 14 (L9) → 0 (L10)

(Residual-stream patching at the swapped name token giving recovery ≈ 1.0 is kept as a sanity check — the input token where the minimal pair differs must trivially restore the behavior, so it validates the machinery, not the circuit.)

$ ./run.sh test
......                                                                   [100%]
6 passed in 327.94s (0:05:27)

(One test queries Neuronpedia and skips offline: 5 passed, 1 skipped. CI runs with -m "not network".)

Examples

# How a prediction forms, layer by layer
interp logit-lens "The Eiffel Tower is in the city of" --target " Paris"

# Which heads cause the indirect-object prediction (causal)
interp patch-heads \
  "When John and Mary went to the store, John gave a drink to" \
  "When John and Mary went to the store, Mary gave a drink to" \
  " Mary" " John"

# Which interpretable SAE features fire on a sentence (no GPU)
interp sae "The doctor said the patient was fine"

# Detect induction heads (--fig reproduces the heatmap above)
interp induction --fig figures/induction_heads.png

See examples/ioi.md for a full walkthrough that reproduces the IOI circuit step by step.

How it works

• Built on TransformerLens for hooks, caching, and the residual-stream decomposition.
• SAE features come from the Neuronpedia API — pre-computed features + human explanations, so no SAE training or GPU is needed.
• Default model is GPT-2 small (every circuit above was discovered on it); pass --model <name> for any TransformerLens-supported model.
• Inference-only and CPU by default, so figures and numbers reproduce exactly. MPS is opt-in (--device mps or INTERP_DEVICE=mps): it is faster on Apple silicon, but TransformerLens warns the MPS backend "may produce silently incorrect results", so trust CPU numbers.

Honest limitations

• DLA is correlational; only patching is causal. The skill instructs the agent to keep this distinction and not state a causal claim from DLA alone.
• Steering with a single mean-difference vector is coarse and frequently trades the target behavior for degenerate repetition — interp surfaces this rather than hiding it. (See the companion study steering-audit for a full reliability analysis of when steering silently fails.)
• SAE explanations are auto/crowd-generated (Neuronpedia) and are a guide, not ground truth.
• Activation patching is O(layers × positions) forward passes — fast on GPT-2, slower on large models.

Citation

If you use this in research or teaching:

@software{bano2026interp,
  author = {Bano, Azra},
  title  = {interp: mechanistic interpretability for coding agents},
  year   = {2026},
  url    = {https://github.com/azrabano23/interp}
}

Built on the shoulders of the interpretability community — TransformerLens (Nanda), Neuronpedia (Lin & Bloom), and the IOI / induction-head / logit-lens work cited above.

License

MIT — see LICENSE.