feat: AI-powered node summaries, multi-select, node explain, and context-aware chat agent
Overview
This issue proposes four interconnected AI features that elevate the explorer from
a passive visualization tool into an interactive compiler education and debugging
assistant. The features share a common AI provider configuration panel and a
unified right-hand AI sidebar layout.
Feature 1 — Universal "Summarise Graph" Button
Behaviour
- A button labelled ✦ Summarise sits in the graph toolbar, alongside the
existing PyTorch Names toggle and fullscreen controls.
- Pressing it sends all visible node labels (MLIR op names, block argument names,
SSA values, yield terminators) to a user-configured AI provider.
- The API call returns a short 1–5 word plain-English summary for every node in
a single batched request.
- Summaries are never generated automatically. They are only produced on
explicit button press.
- On hover, a tooltip appears on each node showing its summary beneath the existing
label. The MLIR label is still visible (not replaced).
- A loading spinner/overlay appears on the button while the request is in flight.
Individual nodes can show a subtle pulsing border while awaiting their summary.
- Summaries persist across snapshot changes until the button is pressed again or
the provider is changed.
AI Provider Configuration
- A settings popover (gear icon next to the Summarise button) opens an inline
form:
- Provider base URL — text input, e.g.
https://api.openai.com/v1
- API Key — password input, stored in
localStorage, never sent to the
backend
- Model — text input with a small preset dropdown
(gpt-4o-mini, gpt-4o, claude-3-5-haiku, custom)
- Save button persists to
localStorage
- All AI requests are made directly from the browser to the configured endpoint so
the backend never touches credentials.
- If no provider is configured, pressing Summarise opens the settings popover
instead of making a request.
Acceptance Criteria
Feature 2 — Multi-Node Selection (Ctrl+Click)
Behaviour
- Holding Ctrl (or ⌘ on macOS) and clicking a node adds or removes it
from the active selection set. A single click without the modifier deselects all
and selects only the clicked node (existing behaviour preserved).
- Selected nodes receive a visible ring highlight (accent colour distinct from the
existing focus/trace highlight).
- The multi-selection badge in the toolbar shows
N selected when more than one
node is chosen.
- The existing pane-click handler clears the multi-selection.
- Keyboard shortcut Escape also clears the selection.
- When multiple nodes are selected, the context menu (right-click) is suppressed
to avoid ambiguity; a persistent Selection Actions bar appears at the bottom
of the graph panel offering bulk actions (initially: Clear selection; future:
Explain selected, Export selected IDs).
Acceptance Criteria
Feature 3 — Node "Explain" Context Menu Action
Behaviour
- The right-click context menu on every node gains a second action: Explain
(below the existing Trace option).
- Pressing Explain sends a prompt to the configured AI provider containing:
- The node's MLIR operation label (
rawLabel)
- Its type (block argument, arithmetic op, region container, yield, return)
- Its direct upstream operands and downstream consumers (node labels only)
- The full current MLIR snapshot text
- The current PyTorch source code
- The response is a medium-detail paragraph (approx. 80–150 words) written in
plain English, explaining what the operation does, why the compiler emitted it,
and how it relates to the source-level tensor operation.
- Explanation results appear in the AI Sidebar (see Layout section below),
replacing any previous explanation. The sidebar auto-opens if closed.
- A loading skeleton is shown in the sidebar while the request is in flight.
- The explained node receives a subtle amber glow border (distinct from Trace
orange and Focus teal) while its explanation is displayed.
Acceptance Criteria
Feature 4 — Context-Aware Chat Agent
Behaviour
- A Chat tab in the AI Sidebar opens a conversation interface.
- Every message automatically includes as hidden system context:
- The current MLIR snapshot (the full text of the selected pass file)
- The current PyTorch source code
- A compact serialisation of the visible graph: node IDs, labels, types, and
edges
- Users can ask free-form questions such as:
- "Why is there a
tensor.empty before the linalg.fill?"
- "What does
linalg.generic do in this model?"
- "Which operations correspond to the ReLU?"
- The chat maintains a conversation history during the session. History is cleared
when the snapshot changes.
- A Clear button resets the conversation.
- Code blocks in responses are syntax-highlighted (MLIR/Python).
- The chat uses the same provider configuration as Features 1 and 3.
Acceptance Criteria
Proposed UI Layout
┌──────────────────────────────────────────────────────────────────────────────┐
│ SIDEBAR (left, 280px) │ EDITOR (centre) │ GRAPH PANEL (right, flex) │
│ PyTorch Templates │ │ │
│ Optimization Timeline │ │ ┌─────────────────────────────┐ │
│ │ │ │ GRAPH TOOLBAR │ │
│ │ │ │ [PyTorch Names ☐] [✦ Sum.] │ │
│ │ │ │ [⚙] [☀/☾] [Fullscreen] │ │
│ │ │ └─────────────────────────────┘ │
│ │ │ │
│ │ │ ┌─────────────────────────────┐ │
│ │ │ │ REACT FLOW GRAPH │ │
│ │ │ │ │ │
│ │ │ │ [Node]──▶[Node]──▶[Node] │ │
│ │ │ │ │ │
│ │ │ │ ── Selection bar (N sel.) ──│ │
│ │ │ └─────────────────────────────┘ │
│ │ │ │
│ │ │ ┌─────────────────────────────┐ │
│ │ │ │ AI SIDEBAR (collapsible, │ │
│ │ │ │ slides in from right ~320px)│ │
│ │ │ │ │ │
│ │ │ │ [Explain] [Chat] [✕] │ │
│ │ │ │─────────────────────────────│ │
│ │ │ │ EXPLAIN TAB: │ │
│ │ │ │ Node: linalg.generic │ │
│ │ │ │ ───────────────────────── │ │
│ │ │ │ <AI explanation paragraph> │ │
│ │ │ │ │ │
│ │ │ │ CHAT TAB: │ │
│ │ │ │ ┌─────────────────────┐ │ │
│ │ │ │ │ assistant: ... │ │ │
│ │ │ │ │ you: ... │ │ │
│ │ │ │ └─────────────────────┘ │ │
│ │ │ │ [input field] [Send] │ │
│ │ │ │ [Clear conversation] │ │
│ │ │ └─────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────┘
AI Provider Settings Popover (⚙ next to Summarise button)
┌──────────────────────────────┐
│ ⚙ AI Provider Settings │
│ │
│ Base URL │
│ [_________________________] │
│ │
│ API Key │
│ [••••••••••••••••••••••••] │
│ │
│ Model │
│ [gpt-4o-mini ▾ ] │
│ │
│ [Cancel] [Save] │
└──────────────────────────────┘
Node Hover Tooltip with Summary (Feature 1)
┌─────────────────┐
│ arith.addf │ ← existing MLIR label
│ ─────────────── │
│ adds two floats │ ← AI summary (1–5 words, italic, muted)
└─────────────────┘
Technical Notes
- All AI calls are made client-side (browser
fetch) using the user-supplied base
URL and API key in the standard OpenAI-compatible /chat/completions format.
This keeps the backend stateless with respect to AI credentials.
- Credentials are stored in
localStorage under a namespaced key
(xcomp.ai.config). They are never logged or sent to the project backend.
- Summary and explanation state live in React (
useState/useRef) and are keyed
by node ID. They are intentionally ephemeral (not persisted to localStorage)
so stale explanations do not accumulate across sessions.
- Multi-select state should be managed independently of React Flow's built-in
selection model to avoid conflicts with the fitView, drag, and pan handlers.
- The chat context serialisation should be capped (e.g. first 8 000 tokens of MLIR
text) to stay within typical model context windows without truncating the Python
source or graph structure.
Labels
enhancement · ai · ux · graph · frontend
Milestone
v0.2 — AI Layer
feat: AI-powered node summaries, multi-select, node explain, and context-aware chat agent
Overview
This issue proposes four interconnected AI features that elevate the explorer from
a passive visualization tool into an interactive compiler education and debugging
assistant. The features share a common AI provider configuration panel and a
unified right-hand AI sidebar layout.
Feature 1 — Universal "Summarise Graph" Button
Behaviour
existing PyTorch Names toggle and fullscreen controls.
SSA values, yield terminators) to a user-configured AI provider.
a single batched request.
explicit button press.
label. The MLIR label is still visible (not replaced).
Individual nodes can show a subtle pulsing border while awaiting their summary.
the provider is changed.
AI Provider Configuration
form:
https://api.openai.com/v1localStorage, never sent to thebackend
(
gpt-4o-mini,gpt-4o,claude-3-5-haiku, custom)localStoragethe backend never touches credentials.
instead of making a request.
Acceptance Criteria
Feature 2 — Multi-Node Selection (Ctrl+Click)
Behaviour
from the active selection set. A single click without the modifier deselects all
and selects only the clicked node (existing behaviour preserved).
existing focus/trace highlight).
N selectedwhen more than onenode is chosen.
to avoid ambiguity; a persistent Selection Actions bar appears at the bottom
of the graph panel offering bulk actions (initially: Clear selection; future:
Explain selected, Export selected IDs).
Acceptance Criteria
N selectedbadge visible in graph toolbar.Feature 3 — Node "Explain" Context Menu Action
Behaviour
(below the existing Trace option).
rawLabel)plain English, explaining what the operation does, why the compiler emitted it,
and how it relates to the source-level tensor operation.
replacing any previous explanation. The sidebar auto-opens if closed.
orange and Focus teal) while its explanation is displayed.
Acceptance Criteria
Python source.
Feature 4 — Context-Aware Chat Agent
Behaviour
edges
tensor.emptybefore thelinalg.fill?"linalg.genericdo in this model?"when the snapshot changes.
Acceptance Criteria
Proposed UI Layout
AI Provider Settings Popover (⚙ next to Summarise button)
Node Hover Tooltip with Summary (Feature 1)
Technical Notes
fetch) using the user-supplied baseURL and API key in the standard OpenAI-compatible
/chat/completionsformat.This keeps the backend stateless with respect to AI credentials.
localStorageunder a namespaced key(
xcomp.ai.config). They are never logged or sent to the project backend.useState/useRef) and are keyedby node ID. They are intentionally ephemeral (not persisted to
localStorage)so stale explanations do not accumulate across sessions.
selection model to avoid conflicts with the
fitView, drag, and pan handlers.text) to stay within typical model context windows without truncating the Python
source or graph structure.
Labels
enhancement·ai·ux·graph·frontendMilestone
v0.2 — AI Layer