Skip to content

refactor: tool-based RAG pipeline replacing fixed HyDE/inject pipeline - #204

Merged
nerualauren merged 1 commit into
mainfrom
tool-rag
Apr 4, 2026
Merged

refactor: tool-based RAG pipeline replacing fixed HyDE/inject pipeline#204
nerualauren merged 1 commit into
mainfrom
tool-rag

Conversation

@nerualauren

@nerualauren nerualauren commented Apr 2, 2026

Copy link
Copy Markdown
Collaborator

Note: This PR was assembled and reviewed by Claude Code. Lauren has reviewed the design and conversation but has not yet read the full diff line-by-line.

Summary

Replace the fixed retrieval pipeline (HyDE -> Pinecone -> inject docs -> single LLM call) with dynamic tool-calling via Anthropic's streaming tool runner. Claude now decides when and what to search, can refine queries, look up full articles, and browse by metadata.

  • Backend: tools.py extracts shared tool definitions from mcp_server.py, returning (model_output, ui_output) tuples. chat.py rewritten as a generator yielding SSE events via the Anthropic streaming tool runner, replacing the callback/Queue/Thread pattern. Removed llms.py, callbacks.py, HyDE, fixed doc injection.
  • Frontend (both web/ and stampy-ui/): block-based rendering (thinking, tool_use+tool_result, text), turn-based SSE protocol, citations extracted from tool results, cross-turn citation reference resolution, markdown+KaTeX support.
  • Settings: Anthropic-only model list with dateless IDs (claude-sonnet-4-6 default), max_response_tokens only includes thinking budget for thinking-capable models.

Design spec: docs/superpowers/specs/2026-03-30-tool-based-rag-design.md

Review fixes included

  • XML attribute escaping in format_block (quoteattr)
  • Error SSE events on exception (was silent stream termination)
  • React perf: memoized CitationMarkdown (was recreating component types every render)
  • Token counting now includes model_output in tool_result blocks
  • Chat tool get_doc capped at 2KB (MCP keeps 130KB)
  • _search copies blocks instead of mutating in place
  • Restored production CHATBOT_URL with localhost auto-detection for dev
  • Cleaned up dead code: search_followups, callbacks.py, accidental screenshot

Known follow-ups

  • Frontend duplication between web/ and stampy-ui/ SSE handling
  • findCitations called per streaming delta (perf at scale)
  • ui_output bloat in history payloads
  • No max_turns limit on tool runner
  • _schema_from_fn doesn't handle list[T] items or complex types

Test plan

  • cd api && pipenv run pytest passes (verified)
  • Local dev: mprocs starts API + web, chat works with tool calls visible
  • Multi-turn conversation preserves citation numbering across turns
  • Thinking models show collapsible thinking blocks
  • Non-thinking models don't waste tokens on thinking budget
  • Tool errors surface as error events in the UI
  • stampy-ui localhost detection hits local API in dev, prod URL in production

🤖 Generated with Claude Code

nerualauren pushed a commit to nerualauren/stampy-ui that referenced this pull request Apr 2, 2026
Lauren
refactor: tool-based RAG UI with block-based rendering
1eec2ba 
Update chat UI to support the new tool-based RAG pipeline from
stampy-chat (StampyAI/stampy-chat#204).

- Block-based rendering: thinking, tool_use+tool_result (merged), text
- Turn-based SSE protocol: handle 'turn' events with tool results
- Citations extracted from tool_result ui_output blocks
- Cross-turn citation reference resolution via priorCitations
- Markdown rendering with react-markdown, KaTeX math support
- Memoized CitationMarkdown component (avoids recreating component
  types per render causing unmount/remount flicker)
- Tool calls displayed with friendly names in collapsible details
- Model list updated to dateless Anthropic IDs (claude-sonnet-4-6 etc.)
- CHATBOT_URL: production default with localhost auto-detection for dev
- Payload size limit raised to 200KB for block-structured history
@nerualauren nerualauren mentioned this pull request Apr 2, 2026
Merged
7 tasks
nerualauren pushed a commit to nerualauren/stampy-ui that referenced this pull request Apr 2, 2026
Lauren
refactor: tool-based RAG UI with block-based rendering
29c2dcf 
Update chat UI to support the new tool-based RAG pipeline from
stampy-chat (StampyAI/stampy-chat#204).

- Block-based rendering: thinking, tool_use+tool_result (merged), text
- Turn-based SSE protocol: handle 'turn' events with tool results
- Citations extracted from tool_result ui_output blocks
- Cross-turn citation reference resolution via priorCitations
- Markdown rendering with react-markdown, KaTeX math support
- Memoized CitationMarkdown component (avoids recreating component
  types per render causing unmount/remount flicker)
- Tool calls displayed with friendly names in collapsible details
- Model list updated to dateless Anthropic IDs (claude-sonnet-4-6 etc.)
- CHATBOT_URL: production default with localhost auto-detection for dev
- Payload size limit raised to 200KB for block-structured history
- Removed: useIsMobile/mobile prop (unused), DOMPurify (replaced by
  react-markdown's React-native rendering)
- Kept upstream's useGlossaryInjection and glossary popup support
nerualauren pushed a commit to nerualauren/stampy-ui that referenced this pull request Apr 2, 2026
Lauren
refactor: tool-based RAG UI with block-based rendering
1ddfd4b 
Update chat UI to support the new tool-based RAG pipeline from
stampy-chat (StampyAI/stampy-chat#204).

- Block-based rendering: thinking, tool_use+tool_result (merged), text
- Turn-based SSE protocol: handle 'turn' events with tool results
- Citations extracted from tool_result ui_output blocks
- Cross-turn citation reference resolution via priorCitations
- Markdown rendering with react-markdown, KaTeX math support
- Memoized CitationMarkdown component (avoids recreating component
  types per render causing unmount/remount flicker)
- Tool calls displayed with friendly names in collapsible details
- Parallel tool_use blocks matched to tool_results by id (not adjacency)
- Model list updated to dateless Anthropic IDs (claude-sonnet-4-6 etc.)
- CHATBOT_URL: production default with localhost auto-detection for dev
- Payload size limit raised to 200KB for block-structured history
- Glossary injection skips pre and details elements (prevents corrupting
  tool result XML display)
- Removed: useIsMobile/mobile prop (unused), DOMPurify (replaced by
  react-markdown's React-native rendering)
- Kept upstream's useGlossaryInjection and glossary popup support
Lauren
refactor: tool-based RAG pipeline replacing fixed HyDE/inject pipeline
022d533 
Replace the fixed retrieval pipeline (HyDE query -> Pinecone search ->
inject docs into prompt -> single LLM call) with dynamic tool-calling
via Anthropic's streaming tool runner. Claude now decides when and what
to search, can refine queries, look up full articles, and browse by
metadata.

Design spec: docs/superpowers/specs/2026-03-30-tool-based-rag-design.md

Backend:
- New tools.py: shared tool definitions returning (model_output, ui_output)
  tuples, used by both MCP server and chat pipeline
- chat.py rewritten as a generator yielding SSE events directly,
  replacing the callback/Queue/Thread pattern
- Removed: llms.py (multi-provider LLM abstraction), callbacks.py,
  HyDE generation, fixed doc injection
- mcp_server.py becomes thin wrappers around tools.py
- Monotonic citation IDs across tool calls and conversation turns
- Error events yielded on exception instead of silent stream termination
- Tool names logged in interaction records

Frontend (both web/ and stampy-ui/):
- Block-based rendering: thinking, tool_use+tool_result, text blocks
- Turn-based SSE protocol replacing phase-based loading states
- Citations extracted from tool result ui_output
- Cross-turn citation reference resolution via priorCitations
- Markdown rendering with KaTeX math support (stampy-ui)
- Tool calls displayed with friendly names in collapsible details

Settings:
- Anthropic-only model list (dateless IDs: claude-sonnet-4-6, etc.)
- Default model: claude-sonnet-4-6
- max_response_tokens includes thinking budget only for thinking models
- Removed: enable_hyde, hyde_max_tokens, multi-provider settings

Review fixes:
- XML attribute escaping in format_block (quoteattr)
- Error SSE events on exception in run_query
- React perf: memoized CitationMarkdown component (was recreating
  component types on every render via withCitationProcessing)
- Token counting includes model_output in tool_result blocks
- Chat tool get_doc capped at 2KB (MCP keeps 130KB default)
- _search copies blocks instead of mutating in place
- Restored production CHATBOT_URL (localhost auto-detected for dev)
- Cleaned up: dead search_followups, accidental screenshot, callbacks.py
@LeMurphant
LeMurphant self-requested a review April 3, 2026 12:19
nerualauren pushed a commit to StampyAI/stampy-ui that referenced this pull request Apr 4, 2026
Lauren @nerualauren
refactor: tool-based RAG UI with block-based rendering
cbcdf77 
Update chat UI to support the new tool-based RAG pipeline from
stampy-chat (StampyAI/stampy-chat#204).

- Block-based rendering: thinking, tool_use+tool_result (merged), text
- Turn-based SSE protocol: handle 'turn' events with tool results
- Citations extracted from tool_result ui_output blocks
- Cross-turn citation reference resolution via priorCitations
- Markdown rendering with react-markdown, KaTeX math support
- Memoized CitationMarkdown component (avoids recreating component
  types per render causing unmount/remount flicker)
- Tool calls displayed with friendly names in collapsible details
- Parallel tool_use blocks matched to tool_results by id (not adjacency)
- Model list updated to dateless Anthropic IDs (claude-sonnet-4-6 etc.)
- CHATBOT_URL: production default with localhost auto-detection for dev
- Payload size limit raised to 200KB for block-structured history
- Glossary injection skips pre and details elements (prevents corrupting
  tool result XML display)
- Removed: useIsMobile/mobile prop (unused), DOMPurify (replaced by
  react-markdown's React-native rendering)
- Kept upstream's useGlossaryInjection and glossary popup support
@nerualauren
nerualauren merged commit 2b57e49 into main Apr 4, 2026
1 check passed
@nerualauren
nerualauren deleted the tool-rag branch April 4, 2026 03:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LeMurphant LeMurphant LeMurphant approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nerualauren @LeMurphant