Skip to content

Inline sandboxed HTML previews in chat (```html-preview fences) with visual-response guidance #88

Description

@ashwin-pc

Summary

Add first-class inline HTML previews in chat: fenced ```html-preview code blocks in assistant messages render as sandboxed, auto-sized, interactive figures inline in the transcript — plus system-prompt guidance that teaches the agent to use them well (text explains, visuals demonstrate).

This is implemented and validated in a downstream fork; filing to track landing it upstream.

Motivation

Long analytical assistant responses (post-mortems, design decisions, multi-finding reviews) read as walls of prose. Mermaid helped for static structure, but cannot express the things that actually earn a visual: state change, time, interaction, quantities, and UI mockups (e.g. "throw the error and watch it get lost", "drag through releases until the version-skew crash", "find your own cost crossover point").

pi-web already has all the machinery: HTML artifacts preview inline in a sandboxed iframe. But artifacts require file juggling, don't stream with the message, and pull the reader out of the conversation flow. A fenced block keeps the figure's source in the transcript (quotable, diffable) and renders it in place.

Design

Renderer (src/markdown/render.ts):

  • ```html-preview fences are enhanced after markdown render (same pattern as Mermaid): the <pre> is wrapped in a .htmlPreview container with an iframe[srcdoc].
  • Security: identical model to HTML artifact previews — sandbox="allow-scripts" without allow-same-origin, so scripts run in an opaque origin with no access to the app origin, its storage, or APIs.
  • Auto-sizing handshake: a small script is appended to the srcdoc that reports content height and intrinsic width (right edge of widest body child + body right padding/margin) via postMessage. The parent matches frames by contentWindow identity and clamps height to min(70vh, 720px).
  • Shrink-wrap + interleaving: the container narrows to the reported content width, so small intrinsically-sized figures (display:inline-block, flex rows) sit inside prose like figures instead of stretching full-width. Previews ≤140px tall get a compact style (lighter chrome, tighter margins) via .htmlPreview--compact.
  • Source toggle: a hover button flips between rendered preview and the original code block (re-tagged language-html for highlighting, keeps the copy button; source view restores full message width).
  • Max source size 50 KB; oversized or empty blocks stay as plain code.

Guidance (contexts/web-ui.md, injected UI context): a "Rich visual responses" section encoding the usage standard we converged on:

  1. Conclusion first — decision/answer in ≤2 sentences at the top.
  2. Text explains, visuals demonstrate — 2–3 sentences of what/why per concept, then one small interactive figure right after its prose; words inside figures are labels (≤6 words), never narrative.
  3. One concept, one widget — interleave; no monolithic multi-tab explorables.
  4. Escalation ladder — markdown/tables → Mermaid for static structure ("if it could be a Mermaid diagram, it must be") → html-preview only for state/time/interaction/quantities/mockups. Code stays code; checklists stay lists.
    Plus a genre table (incident → simulation; design decision → comparison table + scrubbable tradeoff; architecture → Mermaid; review → lists) and a compact embedded exemplar (a control that changes an animated outcome) for the model to imitate.

Implementation (downstream)

File Change
src/markdown/render.ts enhanceInlineHtmlPreviews() + resize/width postMessage listener + source toggle
src/styles/messages.css .htmlPreview, .htmlPreviewFrame, .htmlPreviewToggle, --compact variant, [hidden] handling
server/mock.ts mock reply exercising scripts + the sizing handshake
tests/e2e/pi-web.spec.ts e2e: renders sandboxed, script executed inside frame, auto-height growth, shrink-wrap width, compact class, source toggle round-trip — passing on desktop/tablet/mobile
contexts/web-ui.md usage standard described above

Notes / limitations

  • Previews render when the completed message is markdown-rendered (streaming shows the raw fence, same as Mermaid).
  • One message listener total; frames tracked in a Set, pruned on disconnect.
  • srcdoc + injected reporter script means full-document snippets (<html>…</html>) work; the reporter appended after </html> is tolerated by browsers.

Happy to open a PR with the downstream implementation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestimplemented-downstreamImplemented in a downstream fork; track when upstream lands the same change

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions