Add knowledge management: LightRAG-backed GraphRAG knowledge base

.env.example

@@ -35,6 +35,23 @@ USER_STORY_TEMPERATURE=0.3
 # NEO4J_USER=neo4j
 # NEO4J_PASSWORD=password
 
+# Knowledge Base (Optional) — LightRAG-backed GraphRAG, enabled per run with
+# `archaeo --knowledge-base`. Defaults are file-based (no infrastructure).
+# KB_WORKING_DIR=data/knowledge_base
+# KB_WORKSPACE=                         # partition inside shared back-ends (one per analyzed system)
+# KB_GRAPH_STORAGE=NetworkXStorage      # Neo4JStorage | PGGraphStorage | MemgraphStorage | MongoGraphStorage
+# KB_VECTOR_STORAGE=NanoVectorDBStorage # PGVectorStorage | QdrantVectorDBStorage | MilvusVectorDBStorage | FaissVectorDBStorage
+# KB_KV_STORAGE=JsonKVStorage           # PGKVStorage | RedisKVStorage | MongoKVStorage
+# KB_EMBEDDING_PROVIDER=sentence-transformers  # openai | ollama | hashing (tests only)
+# KB_EMBEDDING_MODEL=all-MiniLM-L6-v2
+# KB_LLM_PROVIDER=auto                  # anthropic | openai | ollama | none
+# KB_EXTRACT_ENTITIES=false             # LLM entity extraction over source files (costs tokens)
+# KB_TOKENIZER=tiktoken                 # bytes = fully offline
+# Scale-out back-end credentials (LightRAG names):
+# NEO4J_USERNAME=neo4j                  # falls back to NEO4J_USER
+# POSTGRES_HOST=localhost POSTGRES_PORT=5432 POSTGRES_USER= POSTGRES_PASSWORD= POSTGRES_DATABASE=
+# QDRANT_URL=http://localhost:6333
+
 # Workflow Settings
 ENABLE_CHECKPOINTS=true
 ENABLE_HITL=false

.gitignore

@@ -51,6 +51,8 @@ data/graph_db/*
 !data/graph_db/.gitkeep
 data/metadata/*
 !data/metadata/.gitkeep
+data/knowledge_base/*
+!data/knowledge_base/.gitkeep
 
 # Logs
 *.log

QUICKSTART.md

@@ -1,46 +1,47 @@
-# Quick Start Guide
+# Quick Start: Your First Dig
 
-Get from clone to your first analysis in about five minutes.
+From clone to your first excavated user story in about five minutes. No prior fieldwork required.
 
-## 1. Install
+## 1. Pack the tools
 
 ```bash
 git clone https://github.com/osick/archaeocode.git
 cd archaeocode
 pip install -r requirements.txt
+pip install -e .        # optional: puts `archaeo` and `archaeo-kb` on your PATH
 ```
 
-## 2. Configure (optional, for user stories)
+## 2. Hire an interpreter (optional, for user stories and questions)
 
 ```bash
 cp .env.example .env
 # edit .env and set ANTHROPIC_API_KEY (or OPENAI_API_KEY)
 ```
 
-Without an API key the workflow still runs discovery, AST analysis, and dependency mapping — only user-story extraction is skipped.
+Without an API key the expedition still runs discovery, AST analysis, dependency mapping, and knowledge-base indexing. Only the two steps that need someone to *read* the code for meaning are skipped: user-story extraction and natural-language answers.
 
-## 3. Run
+## 3. Break ground
 
 ```bash
-# Analyze the bundled COBOL samples
+# The bundled COBOL site
 python archaeo --source sample_data/cobol --source-lang cobol --target-lang java
 
 # Java, with a custom report path and verbose output
 python archaeo --source sample_data/java --source-lang java --target-lang python \
     --report report.json --verbose
 
-# Any codebase of yours
+# Your own site
 python archaeo --source /path/to/legacy/code --source-lang cobol --target-lang java
 ```
 
 Supported source languages: `cobol`, `smalltalk`, `java`, `python`, `javascript`, `fortran`, `pascal`.
 
-## 4. What happens
+## 4. What happens down there
 
-1. **Discovery** — finds and catalogs all source files
-2. **AST analysis** — parses files with tree-sitter (entities, complexity)
-3. **Dependency mapping** — builds the dependency graph, detects cycles, computes layers
-4. **User-story extraction** — an LLM turns each file's business logic into user stories (needs API key)
+1. **Discovery** — surveys the site and catalogs every source file
+2. **AST analysis** — brushes off the dirt with tree-sitter (entities, complexity)
+3. **Dependency mapping** — reconstructs who called whom: graph, cycles, layers
+4. **User-story extraction** — an LLM asks "what was this *for*?" and writes user stories (needs API key)
 5. **Report** — a JSON report is written (default: `workflow_report.json`)
 
 View the report:
@@ -49,7 +50,26 @@ View the report:
 python -m json.tool workflow_report.json
 ```
 
-## 5. Optional extras
+## 5. Keep the finds (the knowledge base)
+
+Add one flag and the run is filed into a knowledge graph instead of evaporating into a JSON file:
+
+```bash
+python archaeo --source sample_data/cobol --source-lang cobol --knowledge-base --kb-wiki ./wiki
+```
+
+Then interrogate the archive:
+
+```bash
+archaeo-kb retrieve "PAYMENT" --mode local                    # entities, relations, code — no API key needed
+archaeo-kb graph "PAYMENT.cob"                                # what does this program touch?
+archaeo-kb query "Which programs touch the customer file?"    # a real answer — needs an LLM key
+archaeo-kb stats                                              # how big is the museum?
+```
+
+Open `./wiki/index.md` in Obsidian (or any Markdown viewer) for one page per file, function, dependency, and user story, cross-linked. Add `--kb-extract` to also let the LLM dig out business vocabulary. By default everything lives in `data/knowledge_base/` as plain files; for a site the size of a core-banking system, point it at Neo4j / Postgres / Qdrant via `KB_*` variables. The full story: [docs/KNOWLEDGE_MANAGEMENT.md](docs/KNOWLEDGE_MANAGEMENT.md).
+
+## 6. Optional extras
 
 **LangSmith tracing** — full observability of every run:
 
@@ -71,9 +91,12 @@ python scripts/build_smalltalk_grammar.py
 - **Import errors**: run from the project root; the CLI adds the right paths itself.
 - **Missing dependencies**: `pip install -r requirements.txt`
 - **No user stories generated**: check that your API key is set in `.env` and valid.
+- **`archaeo-kb query` says no LLM configured**: same fix, or use `retrieve` / `graph` / `stats`, which never need one.
+- **Knowledge base can't download models** (air-gapped site): set `KB_TOKENIZER=bytes` and `KB_EMBEDDING_PROVIDER=hashing`, or point `KB_EMBEDDING_PROVIDER=ollama` at a local Ollama.
 
 ## Next steps
 
 - [README](README.md) — feature overview and Python API
 - [examples/user_story_extraction/basic_usage.py](examples/user_story_extraction/basic_usage.py) — programmatic usage
-- [docs/ROADMAP.md](docs/ROADMAP.md) — where the project is heading
+- [examples/knowledge_base/query_knowledge_base.py](examples/knowledge_base/query_knowledge_base.py) — build and query the knowledge base
+- [docs/ROADMAP.md](docs/ROADMAP.md) — where the expedition is heading

README.md

@@ -4,29 +4,37 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
 
-**Automated software archaeology.** Point archaeocode at a legacy codebase — get back business-readable user stories, a dependency map, and structural analysis: the raw material for a migration backlog, excavated from code whose authors are long gone.
+**Automated software archaeology.** Somewhere in your company there is a COBOL program older than most of your colleagues. Its author retired, its documentation is a rumour, and it still moves money every night. archaeocode is the expedition you send in: point it at a legacy codebase and get back business-readable user stories, a dependency map, structural analysis — and, since v0.4, a **knowledge base that remembers everything the dig uncovered**, so you never have to excavate the same site twice.
 
-Most reverse-engineering tools stop at syntax: they parse code and draw diagrams. This project goes one step further and recovers the *business intent* hidden in legacy code. An AI agent workflow (LangGraph) orchestrates static-analysis tools (exposed as MCP servers) to turn COBOL, Smalltalk, Fortran, Pascal, or Java code into artifacts that stakeholders can actually read — the raw material for a migration backlog.
+Most reverse-engineering tools stop at syntax: they parse code and draw diagrams. That is like cataloguing potsherds without asking what the pot was for. archaeocode goes one step further and recovers the *business intent* buried in legacy code. An AI agent workflow (LangGraph) orchestrates static-analysis tools (exposed as MCP servers) to turn COBOL, Smalltalk, Fortran, Pascal, or Java into artifacts that stakeholders can actually read: the raw material for a migration backlog.
 
 ## What makes it different
 
-- **User stories from code** — the unique feature: an LLM analyzes each source file and produces user stories with roles, capabilities, benefits, acceptance criteria, priority, and confidence scores. Legacy knowledge becomes a product backlog.
-- **Legacy-first language support** — COBOL, Fortran, Pascal, and Smalltalk alongside Java, Python, JavaScript, and TypeScript. See the [language matrix](#language-support) for per-language capabilities.
-- **MCP architecture** — analysis tools (AST parsing, dependency graphs, RAG) are [Model Context Protocol](https://modelcontextprotocol.io/) servers, so any MCP-capable agent can reuse them independently of this workflow.
-- **Observable by design** — every workflow run can be traced in [LangSmith](https://docs.smith.langchain.com) (token costs, latency, state snapshots).
+- **User stories from code** — the signature find. An LLM reads each source file and writes user stories with roles, capabilities, benefits, acceptance criteria, priority, and confidence scores. Forty-year-old business rules become a product backlog your product owner can argue about.
+- **A knowledge base that doesn't forget** — every run can be persisted into a [GraphRAG knowledge base](docs/KNOWLEDGE_MANAGEMENT.md) (powered by [LightRAG](https://github.com/HKUDS/LightRAG)). Files, entities, dependency edges, and user stories become a knowledge graph you can *interrogate*: "which programs touch the customer master file?", "what breaks if we retire this copybook?". Ask from the terminal with `archaeo-kb`, from any agent via MCP, or browse the exported Obsidian-style wiki like a museum catalogue. Runs on plain files on your laptop, or on Neo4j / Postgres / Qdrant when the site is the size of a bank.
+- **Legacy-first language support** — COBOL, Fortran, Pascal, and Smalltalk get first-class treatment, next to Java, Python, JavaScript, and TypeScript. See the [language matrix](#language-support).
+- **MCP architecture** — every tool (AST parsing, dependency graphs, RAG, knowledge base) is a [Model Context Protocol](https://modelcontextprotocol.io/) server. Claude Desktop, Claude Code, or your own agent can borrow the trowel without adopting the whole expedition.
+- **Observable by design** — every run can be traced in [LangSmith](https://docs.smith.langchain.com): token costs, latency, state snapshots. No mystery boxes on a mystery-box project.
 
-## How it works
+## How the dig works
 
 ```
 LangGraph Orchestration
     │
-[Discovery] ──► [AST Analysis] ──► [Dependency Mapping] ──► [User Stories]
-    │                │                     │                     │
-file catalog    tree-sitter          graph + cycle          Claude / GPT
-                  parsing              detection
+[Discovery] ──► [AST Analysis] ──► [Dependency Mapping] ──► [User Stories] ──► [Knowledge Base]
+    │                │                     │                     │                   │
+ survey the      tree-sitter          graph + cycle          Claude / GPT       LightRAG GraphRAG
+   site            parsing              detection          (what was it for?)  (the site archive,
+                                                                                  --knowledge-base)
 ```
 
-Each analysis step is an MCP server under `src/mcp_servers/` (static analysis, graph DB, RAG pipeline); the LangGraph workflow under `src/orchestration/` wires them together with checkpointing and state management.
+1. **Discovery** surveys the site: every source file is catalogued with language and size.
+2. **AST analysis** brushes off the dirt: tree-sitter turns files into classes, functions, methods, imports, and complexity scores.
+3. **Dependency mapping** reconstructs who talked to whom: import/call edges, circular dependencies, and layering.
+4. **User story extraction** asks the important question, *what was this for?*, and writes the answer as user stories.
+5. **Knowledge base** (opt-in) files every find in a knowledge graph so the next question doesn't need a new dig.
+
+Each step is an MCP server under `src/mcp_servers/`; the LangGraph workflow under `src/orchestration/` wires them together with checkpointing and state management.
 
 ## Quick start
 
@@ -35,14 +43,14 @@ git clone https://github.com/osick/archaeocode.git
 cd archaeocode
 pip install -r requirements.txt
 
-# Optional: install the `archaeo` command onto your PATH
+# Optional: install the `archaeo` and `archaeo-kb` commands onto your PATH
 pip install -e .
 
 # Optional but recommended: enable user-story extraction
 cp .env.example .env   # add your ANTHROPIC_API_KEY (or OPENAI_API_KEY)
 ```
 
-Run the workflow against the bundled samples:
+Break ground on the bundled samples:
 
 ```bash
 # COBOL analysis
@@ -56,7 +64,24 @@ python archaeo --source sample_data/java --source-lang java --target-lang python
 
 You get a console summary plus a JSON report: file catalog, language breakdown, dependency edges/cycles/layers, and (with an API key) generated user stories.
 
-Or use it from Python:
+### Keep the finds: the knowledge base
+
+A JSON report is a photo of the excavation. The knowledge base is the museum.
+
+```bash
+# persist the run into the knowledge base (+ an Obsidian-compatible wiki)
+python archaeo --source sample_data/cobol --source-lang cobol --knowledge-base --kb-wiki ./wiki
+
+# then interrogate it
+archaeo-kb query "Which programs touch the customer master file?"   # needs an LLM key
+archaeo-kb retrieve "PAYMENT" --mode local                          # no LLM needed
+archaeo-kb graph "PAYMENT.cob"                                      # walk the neighbourhood
+archaeo-kb add docs/runbook.md                                      # add non-code lore
+```
+
+Indexing needs no LLM at all: files, entities, and dependency edges come straight from static analysis, so the graph is exact and free. Add `--kb-extract` and the LLM also mines the code for *business* concepts (customers, tariffs, invoice runs) and files them next to the technical ones. Details, query modes, and the scaling ladder: [docs/KNOWLEDGE_MANAGEMENT.md](docs/KNOWLEDGE_MANAGEMENT.md).
+
+### From Python
 
 ```python
 from src.orchestration.graph import create_graph
@@ -73,7 +98,7 @@ for story in result["user_stories"]:
     print("-", story["title"])
 ```
 
-A complete runnable example is in [`examples/user_story_extraction/basic_usage.py`](examples/user_story_extraction/basic_usage.py).
+Runnable examples: [`examples/user_story_extraction/basic_usage.py`](examples/user_story_extraction/basic_usage.py) and [`examples/knowledge_base/query_knowledge_base.py`](examples/knowledge_base/query_knowledge_base.py).
 
 ## Feature status
 
@@ -86,16 +111,19 @@ A complete runnable example is in [`examples/user_story_extraction/basic_usage.p
 | LangSmith tracing | ✅ working |
 | Checkpointing / resumable workflows | ✅ working |
 | Smalltalk grammars (standard + Cincom) | ✅ working (grammar build required, see below) |
-| RAG semantic code search (MCP server) | 🚧 functional, not yet wired into the workflow |
-| Neo4j-backed dependency graphs | 🚧 in-memory fallback works; live Neo4j optional |
+| Knowledge base: GraphRAG over files, entities, dependencies, stories (LightRAG, MCP server, `archaeo-kb`) | ✅ working (opt-in per run) |
+| Knowledge base: LLM extraction of business entities | ✅ working (`--kb-extract`, needs API key) |
+| Knowledge base: Obsidian-compatible wiki export | ✅ working |
+| RAG semantic code search (MCP server) | 🚧 mock; superseded by the knowledge base |
+| Neo4j-backed dependency graphs | 🚧 in-memory fallback works; the knowledge base can already use Neo4j |
 | Code generation to target language | 🎯 planned |
 | HP NonStop COBOL extensions (TMF, Pathway) | 🎯 planned |
 
 ## Requirements
 
 - Python 3.10+
-- An Anthropic or OpenAI API key for user-story extraction (everything else runs without one)
-- Optional: Neo4j 5.x if you want persistent dependency graphs, LangSmith account for tracing
+- An Anthropic or OpenAI API key for user-story extraction and knowledge-base questions (everything else runs without one)
+- Optional: Neo4j 5.x / PostgreSQL+pgvector / Qdrant to scale the knowledge base beyond local files; a LangSmith account for tracing
 
 ## Language support
 
@@ -116,18 +144,20 @@ The AST MCP server additionally parses C, C++, C#, Go, Rust, Ruby, PHP, and Bash
 ## Project structure
 
 ```
-├── archaeo                     # CLI entry point
+├── archaeo                     # CLI entry point (`archaeo-kb` after pip install -e .)
 ├── src/
 │   ├── orchestration/          # LangGraph workflow
 │   │   ├── graph.py            # Direct workflow (in-process nodes)
 │   │   ├── graph_mcp.py        # MCP-backed workflow
-│   │   ├── nodes/              # Discovery, AST, dependency, user-story nodes
+│   │   ├── kb_cli.py           # archaeo-kb: interrogate the knowledge base
+│   │   ├── nodes/              # Discovery, AST, dependency, user-story, knowledge-base nodes
 │   │   ├── state/              # Workflow state schema
 │   │   └── utils/              # MCP client, LangSmith tracing
 │   ├── mcp_servers/
 │   │   ├── static_analysis/    # tree-sitter AST analysis + custom grammars
 │   │   ├── graph_db/           # dependency graph (Neo4j / in-memory)
-│   │   └── rag_pipeline/       # chunking, embeddings, semantic search
+│   │   ├── rag_pipeline/       # chunking, embeddings, semantic search
+│   │   └── knowledge_base/     # GraphRAG knowledge base (LightRAG adapter + MCP server)
 │   └── parsers/                # language-specific parser extensions
 ├── config/                     # workflow + MCP server configuration
 ├── sample_data/                # COBOL, Java, Smalltalk, Fortran, Pascal, Python samples
@@ -143,11 +173,12 @@ pip install pytest pytest-asyncio
 pytest
 ```
 
-The suite runs in [GitHub Actions](.github/workflows/ci.yml) on Python 3.10–3.12. Smalltalk tests are skipped automatically unless the grammar has been built.
+The suite runs in [GitHub Actions](.github/workflows/ci.yml) on Python 3.10–3.12. Smalltalk tests are skipped automatically unless the grammar has been built. The knowledge-base tests run fully offline (no API key, no downloads), so CI never has to phone home.
 
 ## Documentation
 
-- [Quick Start](QUICKSTART.md)
+- [Quick Start](QUICKSTART.md) — your first dig in five minutes
+- [Knowledge management](docs/KNOWLEDGE_MANAGEMENT.md) — the site archive: why GraphRAG, why LightRAG, how to query and scale it
 - [Architecture wireframe](docs/LANGGRAPH_WIREFRAME.md)
 - [MCP server architecture](docs/MCP_ARCHITECTURE.md) and [usage](docs/MCP_SERVERS_USAGE.md)
 - [LangSmith setup](docs/LANGSMITH_SETUP.md)
@@ -156,7 +187,7 @@ The suite runs in [GitHub Actions](.github/workflows/ci.yml) on Python 3.10–3.
 
 ## Contributing
 
-Issues and pull requests are welcome. Please run `pytest` before submitting, and open an issue first for larger changes.
+Issues and pull requests are welcome. Please run `pytest` before submitting, and open an issue first for larger changes. Bring your own legacy horror stories; the sample data can always use another stratum.
 
 ## License