A compiled knowledge base for AI engineering — following the Karpathy LLM Wiki pattern.
Raw sources (docs, sessions, meetings, papers, books) go into raw/ (append-only). Claude compiles them into data/wiki/ — structured, interlinked, contradiction-flagged. A local MCP server exposes the wiki to any agent at runtime.
Mental model: raw/ = source code. Claude = compiler. data/wiki/ = executable output.
Agent engineering accumulates a lot of hard-won knowledge: which framework to use and when, what thresholds actually work in production, which patterns look good in docs but fail at runtime. Without a system, you re-derive these decisions from scratch on every project.
Librarian captures that knowledge once, structures it for retrieval, and surfaces it before you start — via slash commands in Claude Code, MCP tools in running agents, or a graph UI for exploration.
Sessions ──────────────────────────────► data/raw/sessions/ ─┐
Docs (.claude/skills/, CLAUDE.md) ─────► data/raw/claude-docs/ │
Repo scraper (core/scrape_repos.py) ─────► data/raw/repos/ │
Google Drive ──MCP──────────────────────► data/raw/gdrive/ ├──► /ingest ──► data/wiki/
Notion ────────MCP──────────────────────► data/raw/notion/ │
Meetings, PDFs, web, books, articles ──► data/raw/*/ ─┘
data/wiki/
│
┌───────────────────┼──────────────────┐
▼ ▼ ▼
MCP server /adk-context graph UI
(search_wiki, (agent builder (React + Cytoscape
read_page, briefing) + chat agent)
get_domain_briefing)
│
▼
Claude Code · custom agents · any MCP client
Key invariants:
raw/is append-only — never edit after dropdata/wiki/is Claude-maintained — humans correct factual errors only- Conflicts are flagged, never silently overwritten
data/wiki/private/is gitignored — company-specific pages live here locally
| Use case | How |
|---|---|
| Pre-build briefing | /adk-context or get_domain_briefing("langgraph") — load accumulated patterns before starting a new agent |
| Grounded Q&A | /query "what's our approach to agent memory?" — answers from your own compiled experience, not generic docs |
| Decision records | Ingest an ADR or meeting note → wiki auto-creates a type: decision page with tradeoffs and cross-links |
| Runtime agent knowledge | Wire search_wiki + read_page as MCP tools in any agent — they self-brief from the KB at query time |
| Research synthesis | Drop papers, book quotes, article captures into raw/ → ingest compiles contradictions, cross-links, and concept pages |
| Team knowledge base | Shared wiki with multi-user ingest; MCP server as team-wide retrieval API |
| Codebase onboarding | Scrape a new repo's CLAUDE.md + docs → ingest → instant grounded answers about architecture decisions |
/ingest # full pipeline: Notion + GDrive + sessions/docs scrape + repos → wiki
/lint # health check after ingesting
That's it — this is the command people forget. Full pipeline mode pulls Notion + GDrive via
MCP, scrapes local sources (make scrape + core/scrape_repos.py), and compiles every changed
file into data/wiki/ in one shot. Other variants:
/ingest data/raw/sessions/ # targeted compile of one data/raw/ dir
/ingest resolve # walk through flagged conflicts in data/wiki/_conflicts.md
/seed-kb --ingest # scrape this machine's sessions/docs/bookmarks, then ingest
# 1. Install
uv sync # core deps
make install-api # FastAPI + sentence-transformers + Anthropic client
make install-ui # npm install for React UI
# 2. Configure
cp .env.example .env
# Fill in:
# ANTHROPIC_API_KEY — required for /ingest, /query, /lint slash commands
# and for the graph UI chat agent (#96)
# NOTION_API_KEY — optional, for Notion connector
# LINEAR_API_KEY — optional, for Linear connector
# 3. Open in Claude Code
code . # CLAUDE.md loads automatically; MCP server auto-startsFirst run — seed from your existing docs:
/ingest data/raw/claude-docs/ # compile any existing .claude/ docs
/ingest data/raw/sessions/ # compile Claude Code session history (batches of 20)
Then query:
/query "what's the right approach for agent memory with LangGraph?"
/adk-context # curated briefing before any ADK or LangGraph build session
Drop any new raw source and ingest it immediately:
# Example: paste a meeting transcript
# data/raw/meetings/2026-07-05-architecture-review.md
/ingest data/raw/meetings/2026-07-05-architecture-review.mdFull sync — pull new Notion/Drive content, scrape sessions, compile everything new:
/ingest # full pipeline: Notion + GDrive + sessions + repos → wiki
/lint # health check: orphans, stale pages, dead links, unresolved conflicts
Optionally scrape repos you're actively working on:
# Add repos to data/raw/repos/repos.txt, then:
uv run python core/scrape_repos.py
/ingest data/raw/repos//adk-context # briefing: decisions + patterns + concepts for your domain
# or
get_domain_briefing("langgraph") # via MCP in the agent itself
/ingest resolve # guided conflict resolution: read both claims, pick correct, mark resolved
/lint # confirm all conflicts closed
/code-review # correctness + simplification pass
The contracts review dimension (via uv run --project ~/workspace/guacamayo review-cli run) checks the diff against SANYI.md automatically.
| Component | What it does |
|---|---|
raw/ |
Append-only input drop zone — sessions, docs, meetings, PDFs, books, articles, repo scrapes |
data/wiki/ |
LLM-compiled knowledge — one .md per concept, pattern, or decision; structured frontmatter, wikilinks |
data/wiki/private/ |
Company-specific pages, gitignored — same format, local only |
CLAUDE.md |
Compiler contract — schema rules, ingest checklist, conflict policy, domain taxonomy |
core/ |
Scrapers: scrape_sessions.py (Claude/Codex), scrape_claude_docs.py (workspace docs), scrape_repos.py (repos) |
app/mcp_server/ |
FastMCP server: search_wiki (hybrid FTS + semantic + backlink rank), read_page, list_domain, get_domain_briefing |
app/backend/ |
FastAPI — wiki graph API, DuckDB-cached embeddings + UMAP layout, streaming Anthropic chat agent |
app/frontend/ |
React + Vite graph UI — Cytoscape.js force-directed graph, chat panel, live wiki watch |
.claude/skills/ |
/ingest, /query, /lint, /adk-context, /seed-kb — Claude Code slash commands |
Pre-configured in .claude/settings.json — available to Claude Code and any MCP client automatically.
uv run python app/mcp_server/server.pyTools:
| Tool | Description |
|---|---|
search_wiki(query, domain, limit, expand) |
Hybrid search: FTS + cosine similarity + backlink rank. domain matches directory or domain tag. expand=True also returns pages one typed relationship hop away (spike) |
read_page(path_or_title) |
Read a page by path or fuzzy title match |
list_domain(domain) |
All pages in a domain, sorted by backlink count |
list_pages(tag, directory) |
Filter pages by tag or directory |
get_domain_briefing(domain) |
All pages in a domain concatenated — decisions first, then patterns, concepts |
Hybrid search blends three signals: text match relevance (FTS), semantic similarity (sentence-transformers, if installed), and inbound backlink count. Falls back to FTS-only if sentence-transformers is absent.
Graph-aware retrieval — expand=True walks the typed relationship subgraph
(prerequisite-for / extends) one hop out from the results to answer "what else do I need
to understand these hits?" It deliberately ignores untyped wikilinks: those run ~24 per page
and would drown a context window, while typed edges are sparse and author-asserted. The
rationale, measurements, and rejected alternatives are in
Wiki Graph Engineering.
make api # FastAPI backend on :8000 (wiki graph + semantic edges + UMAP + chat)
make ui # Vite dev server on :5173The backend caches embeddings and UMAP layout in DuckDB — only recomputed when wiki pages change.
- One page per concept — find the existing page or create one; never scatter knowledge
- Conflicts are flagged, not overwritten — contradictions go to
data/wiki/_conflicts.mdfor human review - Every page has at least one backlink — orphan pages are a lint error
updated:is set on every write — stale pages (>60 days) are surfaced by/lint- Sources are cited — every wiki page lists the
raw/files it was compiled from - Private stays private — company-specific pages go to
data/wiki/private/(gitignored), notdata/wiki/projects/
| Tag | Covers |
|---|---|
adk |
Google Agent Development Kit — patterns, APIs, deployment |
langgraph |
LangGraph state machines, CRAG, checkpointers, edges |
rag |
Retrieval-augmented generation, embedders, rerankers, chunking |
memory |
Agent memory patterns — short-term, long-term, episodic, semantic |
mcp |
Model Context Protocol, MCP server design, tool schemas |
voice |
Voice agent patterns, BIDI streaming, session management |
eval |
Evaluation harnesses, LLM judges, golden sets, metrics |
infra |
Deployment, CI/CD, observability, caching |
llm |
LLM API patterns, prompt engineering, context management |
deep-agents |
Deep Agents harness — middleware, StateBackend, StoreBackend |
context-management |
Prefix caching, session compaction, history pruning |
| Directory | Source | Scraper |
|---|---|---|
raw/sessions/ |
Claude Code + Codex session JSONL | core/scrape_sessions.py |
raw/claude-docs/ |
.claude/ docs from workspace projects |
core/scrape_claude_docs.py |
raw/repos/ |
CLAUDE.md, skills, docs from configured repos | core/scrape_repos.py + repos.txt |
raw/notion/ |
Notion pages | MCP connector |
raw/gdrive/ |
Google Drive docs | MCP connector |
raw/linear/ |
Linear issues + projects | core/ingest_linear.py |
raw/pdfs/ |
PDF text extracts | core/ingest_pdf.py |
raw/meetings/ |
Meeting transcripts | Manual drop |
raw/web/ |
Web article captures | Manual drop or URL mode |
raw/books/ |
Book quotes + notes | Manual drop (see CLAUDE.md for format) |
raw/articles/ |
Article highlights + notes | Manual drop or URL mode |
| Command | What it does |
|---|---|
/ingest [path | url | resolve] |
Compile raw sources → wiki; resolve mode for conflict resolution |
/query <question> |
Grounded answer from compiled wiki; optionally files answer as new page |
/lint |
Health check — orphans, dead links, stale pages, unresolved conflicts |
/adk-context [domain] |
Curated briefing for a build session |
/seed-kb |
Scrape sessions + docs then prompt to ingest |
The repo includes the SANYI change-contract system for detecting architectural decay across PRs.
Layers for this repo:
- Bianyi (ever-changing): wiki page content, ingest prompts, domain taxonomy config
- Jianyi (bounded): MCP tool schemas, wiki frontmatter spec, manifest format
- Buyi (invariant): source attribution (never serve without grounded source), manifest dedup, conflict flagging, MCP read-only constraint
SANYI.md is the hand-maintained contract; CLAUDE.md carries the ingest checklist and conflict policy. The contracts review dimension reads SANYI.md and enforces all three layers automatically on every diff.
- Karpathy llm-wiki gist — original compiled-wiki pattern