Skip to content

AI-powered node summaries, multi-select, node explain, and context-aware chat agent #1

Description

@TiElixir

feat: AI-powered node summaries, multi-select, node explain, and context-aware chat agent

Overview

This issue proposes four interconnected AI features that elevate the explorer from
a passive visualization tool into an interactive compiler education and debugging
assistant. The features share a common AI provider configuration panel and a
unified right-hand AI sidebar layout.


Feature 1 — Universal "Summarise Graph" Button

Behaviour

  • A button labelled ✦ Summarise sits in the graph toolbar, alongside the
    existing PyTorch Names toggle and fullscreen controls.
  • Pressing it sends all visible node labels (MLIR op names, block argument names,
    SSA values, yield terminators) to a user-configured AI provider.
  • The API call returns a short 1–5 word plain-English summary for every node in
    a single batched request.
  • Summaries are never generated automatically. They are only produced on
    explicit button press.
  • On hover, a tooltip appears on each node showing its summary beneath the existing
    label. The MLIR label is still visible (not replaced).
  • A loading spinner/overlay appears on the button while the request is in flight.
    Individual nodes can show a subtle pulsing border while awaiting their summary.
  • Summaries persist across snapshot changes until the button is pressed again or
    the provider is changed.

AI Provider Configuration

  • A settings popover (gear icon next to the Summarise button) opens an inline
    form:
    • Provider base URL — text input, e.g. https://api.openai.com/v1
    • API Key — password input, stored in localStorage, never sent to the
      backend
    • Model — text input with a small preset dropdown
      (gpt-4o-mini, gpt-4o, claude-3-5-haiku, custom)
    • Save button persists to localStorage
  • All AI requests are made directly from the browser to the configured endpoint so
    the backend never touches credentials.
  • If no provider is configured, pressing Summarise opens the settings popover
    instead of making a request.

Acceptance Criteria

  • Summarise button visible in graph toolbar.
  • Settings popover with base URL, API key, model selector, and save.
  • Summaries stored per node ID and shown on hover.
  • No summary is generated without an explicit button press.
  • Spinner shown during generation; errors shown as dismissible toast.
  • Summaries survive PyTorch Names toggle, dark/light mode, and graph pan/zoom.
  • Summaries are cleared when a new snapshot is selected.

Feature 2 — Multi-Node Selection (Ctrl+Click)

Behaviour

  • Holding Ctrl (or on macOS) and clicking a node adds or removes it
    from the active selection set. A single click without the modifier deselects all
    and selects only the clicked node (existing behaviour preserved).
  • Selected nodes receive a visible ring highlight (accent colour distinct from the
    existing focus/trace highlight).
  • The multi-selection badge in the toolbar shows N selected when more than one
    node is chosen.
  • The existing pane-click handler clears the multi-selection.
  • Keyboard shortcut Escape also clears the selection.
  • When multiple nodes are selected, the context menu (right-click) is suppressed
    to avoid ambiguity; a persistent Selection Actions bar appears at the bottom
    of the graph panel offering bulk actions (initially: Clear selection; future:
    Explain selected, Export selected IDs).

Acceptance Criteria

  • Ctrl+click adds/removes nodes from selection set.
  • Single click without modifier resets to single-node selection.
  • Selected nodes show distinct ring colour.
  • N selected badge visible in graph toolbar.
  • Pane click and Escape clear multi-selection.
  • Context menu disabled during multi-selection; Selection Actions bar shown.
  • Does not conflict with existing Focus, Trace, or Collapse behaviours.

Feature 3 — Node "Explain" Context Menu Action

Behaviour

  • The right-click context menu on every node gains a second action: Explain
    (below the existing Trace option).
  • Pressing Explain sends a prompt to the configured AI provider containing:
    • The node's MLIR operation label (rawLabel)
    • Its type (block argument, arithmetic op, region container, yield, return)
    • Its direct upstream operands and downstream consumers (node labels only)
    • The full current MLIR snapshot text
    • The current PyTorch source code
  • The response is a medium-detail paragraph (approx. 80–150 words) written in
    plain English, explaining what the operation does, why the compiler emitted it,
    and how it relates to the source-level tensor operation.
  • Explanation results appear in the AI Sidebar (see Layout section below),
    replacing any previous explanation. The sidebar auto-opens if closed.
  • A loading skeleton is shown in the sidebar while the request is in flight.
  • The explained node receives a subtle amber glow border (distinct from Trace
    orange and Focus teal) while its explanation is displayed.

Acceptance Criteria

  • "Explain" option in right-click context menu.
  • Prompt correctly includes node context, operand neighbours, MLIR text, and
    Python source.
  • Response displayed in AI Sidebar with the node label as the heading.
  • Sidebar auto-opens on Explain.
  • Explained node border distinguished from Trace and Focus states.
  • Error shown in sidebar if the AI call fails.
  • Dismissing the sidebar clears the explained node highlight.

Feature 4 — Context-Aware Chat Agent

Behaviour

  • A Chat tab in the AI Sidebar opens a conversation interface.
  • Every message automatically includes as hidden system context:
    • The current MLIR snapshot (the full text of the selected pass file)
    • The current PyTorch source code
    • A compact serialisation of the visible graph: node IDs, labels, types, and
      edges
  • Users can ask free-form questions such as:
    • "Why is there a tensor.empty before the linalg.fill?"
    • "What does linalg.generic do in this model?"
    • "Which operations correspond to the ReLU?"
  • The chat maintains a conversation history during the session. History is cleared
    when the snapshot changes.
  • A Clear button resets the conversation.
  • Code blocks in responses are syntax-highlighted (MLIR/Python).
  • The chat uses the same provider configuration as Features 1 and 3.

Acceptance Criteria

  • Chat tab in AI Sidebar; conversation interface with input and send button.
  • MLIR text, Python source, and graph summary injected as system context.
  • Conversation history preserved within a snapshot session.
  • History cleared on snapshot change.
  • Code blocks syntax-highlighted in responses.
  • Clear button resets conversation.
  • Shared provider config (base URL, API key, model).

Proposed UI Layout

┌──────────────────────────────────────────────────────────────────────────────┐
│ SIDEBAR (left, 280px)  │  EDITOR (centre)  │  GRAPH PANEL (right, flex)     │
│  PyTorch Templates     │                   │                                │
│  Optimization Timeline │                   │ ┌─────────────────────────────┐ │
│                        │                   │ │ GRAPH TOOLBAR               │ │
│                        │                   │ │ [PyTorch Names ☐] [✦ Sum.] │ │
│                        │                   │ │ [⚙] [☀/☾] [Fullscreen]    │ │
│                        │                   │ └─────────────────────────────┘ │
│                        │                   │                                │
│                        │                   │ ┌─────────────────────────────┐ │
│                        │                   │ │ REACT FLOW GRAPH            │ │
│                        │                   │ │                             │ │
│                        │                   │ │  [Node]──▶[Node]──▶[Node]  │ │
│                        │                   │ │                             │ │
│                        │                   │ │ ── Selection bar (N sel.) ──│ │
│                        │                   │ └─────────────────────────────┘ │
│                        │                   │                                │
│                        │                   │ ┌─────────────────────────────┐ │
│                        │                   │ │ AI SIDEBAR (collapsible,    │ │
│                        │                   │ │ slides in from right ~320px)│ │
│                        │                   │ │                             │ │
│                        │                   │ │ [Explain] [Chat]   [✕]     │ │
│                        │                   │ │─────────────────────────────│ │
│                        │                   │ │ EXPLAIN TAB:                │ │
│                        │                   │ │  Node: linalg.generic       │ │
│                        │                   │ │  ─────────────────────────  │ │
│                        │                   │ │  <AI explanation paragraph> │ │
│                        │                   │ │                             │ │
│                        │                   │ │ CHAT TAB:                   │ │
│                        │                   │ │  ┌─────────────────────┐    │ │
│                        │                   │ │  │ assistant: ...      │    │ │
│                        │                   │ │  │ you: ...            │    │ │
│                        │                   │ │  └─────────────────────┘    │ │
│                        │                   │ │  [input field] [Send]       │ │
│                        │                   │ │  [Clear conversation]       │ │
│                        │                   │ └─────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────┘

AI Provider Settings Popover (⚙ next to Summarise button)

┌──────────────────────────────┐
│ ⚙ AI Provider Settings       │
│                              │
│ Base URL                     │
│ [_________________________]  │
│                              │
│ API Key                      │
│ [••••••••••••••••••••••••]  │
│                              │
│ Model                        │
│ [gpt-4o-mini          ▾ ]   │
│                              │
│          [Cancel]  [Save]    │
└──────────────────────────────┘

Node Hover Tooltip with Summary (Feature 1)

┌─────────────────┐
│ arith.addf      │  ← existing MLIR label
│ ─────────────── │
│ adds two floats │  ← AI summary (1–5 words, italic, muted)
└─────────────────┘

Technical Notes

  • All AI calls are made client-side (browser fetch) using the user-supplied base
    URL and API key in the standard OpenAI-compatible /chat/completions format.
    This keeps the backend stateless with respect to AI credentials.
  • Credentials are stored in localStorage under a namespaced key
    (xcomp.ai.config). They are never logged or sent to the project backend.
  • Summary and explanation state live in React (useState/useRef) and are keyed
    by node ID. They are intentionally ephemeral (not persisted to localStorage)
    so stale explanations do not accumulate across sessions.
  • Multi-select state should be managed independently of React Flow's built-in
    selection model to avoid conflicts with the fitView, drag, and pan handlers.
  • The chat context serialisation should be capped (e.g. first 8 000 tokens of MLIR
    text) to stay within typical model context windows without truncating the Python
    source or graph structure.

Labels

enhancement · ai · ux · graph · frontend

Milestone

v0.2 — AI Layer

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions