Develop

.github/agents/Agent-Developer-Assistant.md → AGENTS.md

@@ -1,15 +1,6 @@
----
-name: Agent Developer Assistant
-description: >
-  A specialized coding assistant for the research-assistant repository.
-  Maintains persistent project knowledge, tracks development history via git,
-  and enforces simplicity-first architecture principles for a hierarchical
-  multi-agent deep research system.
----
-
-# Agent Developer Assistant
+# AGENTS.md — Development Guide
 
-You are a senior software engineer embedded in the **research-assistant** project — an AI-powered deep research system built on a hierarchical multi-agent orchestration architecture (Python/FastAPI backend, React/TypeScript frontend). You combine deep familiarity with this specific codebase with strong opinions about simplicity, concurrency safety, and architectural elegance.
+This is a working guide for AI coding assistants in the **research-assistant** project — an AI-powered deep research system built on a hierarchical multi-agent orchestration architecture (Python/FastAPI backend, React/TypeScript frontend). It encodes simplicity-first architecture principles, concurrency safety, and project conventions.
 
 ---
 
@@ -19,44 +10,44 @@ You are a senior software engineer embedded in the **research-assistant** projec
 
 ### 1. Load Persistent Memory
 
-Read `.github/agents/Agent-Developer-Assistant/memory/MEMORY.md` in full before doing anything else. This is your compressed, curated knowledge of the project's history, decisions, and patterns. If the file does not exist yet, create it with an empty structure.
+Read `docs/MEMORY.md` in full before doing anything else. This is the compressed, curated knowledge of the project's history, decisions, and patterns. If the file does not exist yet, create it with an empty structure.
 
 ***CRITICAL***
-***This AND ONLY THIS is your source of truth for project context! This is the only file you need to maintain to preserve project knowledge across sessions.***
+***This AND ONLY THIS is the source of truth for project context! This is the only file that must be maintained to preserve project knowledge across sessions.***
 
 ### 2. Load Project Knowledge Map
 
-Read `.github/agents/Agent-Developer-Assistant/memory/PROJECT-KNOWLEDGE.md` in full. This is your working map of the physical layout, key modules, current state of the codebase, and known technical debt. If the file does not exist yet, create it by exploring the repository structure.
+Read `docs/PROJECT-KNOWLEDGE.md` in full. This is the working map of the physical layout, key modules, current state of the codebase, and known technical debt. If the file does not exist yet, create it by exploring the repository structure.
 
 ***CRITICAL***
-***This is your current mental model of the codebase. It should be accurate and up-to-date at all times. Update it surgically whenever you discover something is outdated or when significant changes land. This is the file you refer to when asked about where things are or how they connect.***
+***This is the current mental model of the codebase. It should be accurate and up-to-date at all times. Update it surgically whenever something is discovered to be outdated or when significant changes land. Refer to it when asked about where things are or how they connect.***
 
 ### 3. Examine Recent Git History
 
-Run `git log --oneline -20` and `git log --stat -5` to orient yourself on what has changed recently. Cross-reference against your MEMORY.md to identify anything new or unexpected. Do not narrate this to the user unless something notable stands out.
+Run `git log --oneline -20` and `git log --stat -5` to orient on what has changed recently. Cross-reference against `docs/MEMORY.md` to identify anything new or unexpected. Do not narrate this to the user unless something notable stands out.
 
 ### 4. Orient and Engage
 
-With memory and git history loaded, you are ready to assist. Briefly acknowledge what you know about the current state of the project if it's relevant to the user's first message, then proceed.
+With memory and git history loaded, briefly acknowledge what you know about the current state of the project if it's relevant to the user's first message, then proceed.
 
 ---
 
 ## Memory Management
 
 ### MEMORY.md — Compressed Session Insights
 
-**Location:** `.github/agents/Agent-Developer-Assistant/memory/MEMORY.md`
+**Location:** `docs/MEMORY.md`
 
 **Purpose:** Long-term, compressed notes about the project. Think of it as an engineering journal — decisions made, patterns discovered, problems solved, anti-patterns to avoid, and important context that would be useful to re-load in a future session.
 
-**Format:** Organized by date (YYYY-MM-DD) with concise bullet points. Entries are not appended after every message — only when you learn something worth preserving. Good triggers for writing:
+**Format:** Organized by date (YYYY-MM-DD) with concise bullet points. Entries are not appended after every message — only when something worth preserving is learned. Good triggers for writing:
 
 - A non-obvious architectural decision was made or explained
 - A bug was found and fixed that reveals an underlying constraint
 - A pattern was discovered (good or bad) that recurs across the codebase
 - A significant refactor or new feature landed
 - A performance or concurrency concern was identified or resolved
-- Context that would save future-you meaningful time
+- Context that would save future sessions meaningful time
 
 Keep entries compressed. Prefer facts over narrative. Prefer implications over descriptions.
 
@@ -74,9 +65,9 @@ Keep entries compressed. Prefer facts over narrative. Prefer implications over d
 
 ### PROJECT-KNOWLEDGE.md — Living Architecture Map
 
-**Location:** `.github/agents/Agent-Developer-Assistant/memory/PROJECT-KNOWLEDGE.md`
+**Location:** `docs/PROJECT-KNOWLEDGE.md`
 
-**Purpose:** Your current, accurate map of the codebase — where things live, what they do, how they connect, and what the current development state is. Load fully at the start of each session. Update surgically during a session when you discover something is outdated or when significant changes land.
+**Purpose:** The current, accurate map of the codebase — where things live, what they do, how they connect, and what the current development state is. Load fully at the start of each session. Update surgically during a session when something is discovered to be outdated or when significant changes land.
 
 **Sections to maintain:**
 
@@ -93,7 +84,7 @@ Keep entries compressed. Prefer facts over narrative. Prefer implications over d
 
 ## Coding Philosophy
 
-You hold these principles as non-negotiable constraints that shape every recommendation and implementation you make:
+These principles are non-negotiable constraints that shape every recommendation and implementation:
 
 ### Simplicity First
 
@@ -159,20 +150,20 @@ Good triggers for asking:
 - It's unclear whether an existing abstraction should be extended or a new one created
 - The desired behavior at edge cases or failure modes hasn't been specified
 
-Ask directly and concisely. Batch related questions into a single message rather than asking one at a time. Do not ask about things you can reasonably infer from the codebase or from prior context in `MEMORY.md`.
+Ask directly and concisely. Batch related questions into a single message rather than asking one at a time. Do not ask about things that can be reasonably inferred from the codebase or from prior context in `docs/MEMORY.md`.
 
 ### Before Modifying Code
 
 1. Read the relevant file(s) — do not modify what you haven't read
 2. Understand the existing pattern before deviating from it
-3. Check `MEMORY.md` for notes about that module or subsystem
+3. Check `docs/MEMORY.md` for notes about that module or subsystem
 4. Consider whether the change respects the concurrency model
 
 ### When Suggesting Changes
 
 - Prefer the minimal change that achieves the goal
 - If a refactor is needed, scope it tightly — do not clean up surrounding code that wasn't part of the task
-- If you discover tech debt while working, note it in `PROJECT-KNOWLEDGE.md` rather than fixing it in-scope unless instructed
+- If tech debt is discovered while working, note it in `docs/PROJECT-KNOWLEDGE.md` rather than fixing it in-scope unless instructed
 - If a change alters external behavior (API shape, WebSocket events, CLI interface, MCP tools, config variables), update the relevant file(s) in `docs/` before considering the task complete
 
 ### Code Style
@@ -204,6 +195,8 @@ Ask directly and concisely. Batch related questions into a single message rather
 | `backend/mcp_client.py` | MCP protocol client and `MCPServerRegistry` |
 | `backend/context.py` | `ResearchContext`: intermediate results and cross-step state |
 | `backend/search_result_store.py` | Per-session RAG store with in-process cosine ranking |
+| `backend/long_term_memory.py` | `AsyncLongTermMemory`: persistent cross-session Neo4j store + `KnowledgeGraph` (GraphRAG) |
+| `backend/session_store.py` | Session registry + disk persistence (atomic checkpoint files in `logs/sessions/`) |
 | `backend/config.py` | Pydantic config from environment variables |
 | `backend/models.py` | Shared data models (ResearchPlan, ResearchStep, Message, etc.) |
 | `src/App.tsx` | WebSocket message dispatch, top-level state wiring |
@@ -220,14 +213,14 @@ Ask directly and concisely. Batch related questions into a single message rather
 
 After completing meaningful work in a session:
 
-1. **Update MEMORY.md** if you learned something worth preserving — new architectural insight, a pattern, a resolved ambiguity, a discovered constraint. Keep it compressed.
+1. **Update `docs/MEMORY.md`** if something worth preserving was learned — new architectural insight, a pattern, a resolved ambiguity, a discovered constraint. Keep it compressed.
 
-2. **Update PROJECT-KNOWLEDGE.md** if the codebase changed in a way that makes your previous map inaccurate — new files added, modules refactored, data flows changed.
+2. **Update `docs/PROJECT-KNOWLEDGE.md`** if the codebase changed in a way that makes the previous map inaccurate — new files added, modules refactored, data flows changed.
 
 3. **Update any relevant documentation in `docs/`** if the change affects user-facing or developer-facing behavior. Each file in `docs/` maps to a subsystem — update only the files relevant to what changed. Good triggers: API contract changes, new WebSocket events, new CLI flags, MCP server changes, orchestration behavior changes, new configuration variables. Do not rewrite docs for internal refactors that leave external behavior unchanged.
 
 4. **Update `backend/smoke_test.py`** if the change affects any core behavior, API contract, or pipeline interaction. Smoke tests are the canary for correctness in the running system — they must reflect the current reality of how the system behaves under real conditions.
 
 5. **Update top level README.md** if the change is significant enough to warrant it — new features, major refactors, architectural changes, or anything that would be important for a new engineer to know when first exploring the repo. This README is generally a high-level overview, so only update it for significant changes that affect the overall understanding of the project. Do not add minor details or internal refactors to the README.
 
-Do not append boilerplate entries. Only write what future-you would actually find useful.
+Do not append boilerplate entries. Only write what future sessions would actually find useful.

README.md

@@ -20,7 +20,7 @@ An AI-powered deep research system built on a hierarchical multi-agent architect
 - In-process RAG: `SearchResultStore` with sentence-transformer embeddings (cosine ranking) scoped per session
 - In-process long-term memory: `AsyncLongTermMemory` persists cross-session findings to Neo4j (graph database). Two layers: flat vector store (Neo4j vector indexes) for raw evidence + `KnowledgeGraph` (GraphRAG) layer for typed entities, directed triples with confidence scores, and LLM-generated community clusters. Relationships are deduplicated and include temporal tracking (`last_confirmed`, `confirmation_count`). Supports hierarchy (`IS_A`), contradiction (`CONTRADICTS`), and provenance (`SOURCED_FROM → Source`) edges. Graph context is recalled during planning; communities are updated post-synthesis only when enough new facts have landed (`GRAPH_COMMUNITY_MIN_MUTATIONS`); confidence is decayed over time (`CONFIDENCE_DECAY_HALF_LIFE`). A `prune()` method archives stale low-confidence relationships and orphaned entities.
 - Self-optimization mode: agent reads session logs and the knowledge graph, then updates its own research playbook (`backend/instructions/RESEARCH-METHODS.md`)
-- MCP servers: UA rotation, per-domain rate limiting (2s), concurrency cap (3), 429 retry, 50KB output cap
+- MCP servers: UA rotation, per-domain rate limiting (2s), concurrency cap (3), 429 retry, 100KB scrape output cap
 
 ## Quick Start
 
@@ -44,7 +44,7 @@ make bench        # Run benchmark script
 | `make restart` | Stop + restart backend + MCP servers |
 | `make restart-all` | Stop + restart everything |
 | `make init` | Install backend venv and frontend node_modules |
-| `make clean` | Remove `.venv` and `node_modules` |
+| `make clean` | Remove backend `venv` and frontend `node_modules` / `dist` |
 
 **Local development (without Docker):**
 
@@ -68,31 +68,45 @@ Create `backend/.env`:
 AGENT_MODE=research                  # research | chat | self-optimization
 
 # ── Model backend ─────────────────────────────────────────────
-MODEL_BACKEND=ollama                 # openai | ollama | bedrock | azure | gcp | huggingface
+MODEL_BACKEND=ollama                 # openai | azure | aws | bedrock | gcp | ollama | huggingface | anthropic
+
+# The research pipeline selects a "heavy" model for planning/synthesis and a
+# "light" model for search/analyst work. Each backend has its own pair of
+# *_HEAVY_MODEL / *_LIGHT_MODEL vars. A legacy single *_MODEL field also exists
+# and is used by the /chat endpoint and CLI chat mode.
 
 # Ollama
 OLLAMA_BASE_URL=http://localhost:11434
-OLLAMA_MODEL=nemotron-3-nano
+OLLAMA_HEAVY_MODEL=nemotron-3-nano
+OLLAMA_LIGHT_MODEL=nemotron-3-nano
 
 # OpenAI (or any OpenAI-compatible endpoint)
 # OPENAI_API_KEY=sk-...
-# OPENAI_MODEL=gpt-4o
+# OPENAI_HEAVY_MODEL=gpt-5.2
+# OPENAI_LIGHT_MODEL=gpt-5-nano
 # OPENAI_BASE_URL=https://api.openai.com/v1
 
-# AWS Bedrock (uses default boto3 credential chain if AWS_API_KEY is unset)
-# AWS_MODEL=global.anthropic.claude-sonnet-4-5-20250929-v1:0
+# AWS Bedrock — two backends:
+#   MODEL_BACKEND=aws     → OpenAI-compatible gateway (needs AWS_BASE_URL + AWS_API_KEY)
+#   MODEL_BACKEND=bedrock → native boto3 (needs AWS_REGION; default credential chain)
+# AWS_HEAVY_MODEL=global.anthropic.claude-sonnet-4-6
+# AWS_LIGHT_MODEL=global.anthropic.claude-haiku-4-5-20251001-v1:0
 # AWS_API_KEY=...
 # AWS_BASE_URL=...
+# AWS_REGION=us-east-1
 
 # Azure OpenAI
 # AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
 # AZURE_OPENAI_API_KEY=...
 # AZURE_OPENAI_DEPLOYMENT=your-deployment-name
+# AZURE_HEAVY_MODEL=gpt-5.2
+# AZURE_LIGHT_MODEL=gpt-5-nano
 # AZURE_API_VERSION=2024-02-15-preview
 
 # GCP Vertex AI
 # GCP_ENDPOINT=https://us-central1-aiplatform.googleapis.com/v1
-# GCP_MODEL=gemini-2.5-pro
+# GCP_HEAVY_MODEL=gemini-2.5-pro
+# GCP_LIGHT_MODEL=gemini-2.5-flash
 # GCP_API_KEY=...
 
 # HuggingFace (local TGI endpoint)
@@ -152,23 +166,21 @@ See [docs/API_SERVER.md](docs/API_SERVER.md) for the full environment variable r
 Long-term memory is handled in-process by `backend/long_term_memory.py`, backed by Neo4j. It has two layers:
 
 - **Flat store** (`:Memory` nodes with vector index): raw evidence and claims, recalled via cosine similarity over Neo4j vector indexes.
-- **Knowledge graph**: typed nodes (`:Entity`, `:Community`, `:Source`) and edges (`:RELATES_TO`, `:IS_A`, `:CONTRADICTS`, `:SOURCED_FROM`, `:MEMBER_OF`). Features: relationship deduplication with merge-on-conflict, temporal tracking, hierarchy traversal, contradiction detection, URL-level provenance, cross-session path finding, confidence decay, and graph pruning. Nine REST endpoints under `/graph/` expose the graph for external inspection and maintenance.
+- **Knowledge graph**: typed nodes (`:Entity`, `:Community`, `:Source`) and edges (`:RELATES_TO`, `:IS_A`, `:CONTRADICTS`, `:SOURCED_FROM`, `:MEMBER_OF`). Features: relationship deduplication with merge-on-conflict, temporal tracking, hierarchy traversal, contradiction detection, URL-level provenance, cross-session path finding, confidence decay, and graph pruning. Twelve REST endpoints under `/graph/` expose the graph for external inspection and maintenance.
 
 There is no separate memory MCP server.
 
 ## AI Coding Assistant
 
-This repo ships with a custom **Agent Developer Assistant** for GitHub Copilot. It is a senior-engineer-level coding agent with deep knowledge of this codebase — architecture, conventions, concurrency model, known bugs, and historical decisions. Use it for implementing features, debugging, refactoring, or answering questions about how the system works.
-
-**To use it:** open GitHub Copilot Chat in VS Code and select the `Agent Developer Assistant` agent.
+This repo ships with a root-level **`AGENTS.md`** developer guide for AI coding assistants. It encodes senior-engineer-level knowledge of this codebase — architecture, conventions, concurrency model, known bugs, and historical decisions — so any capable coding agent can implement features, debug, refactor, or answer questions about how the system works.
 
 At the start of each session the agent automatically:
 
-1. Loads its persistent memory from `.github/agents/Agent-Developer-Assistant/memory/MEMORY.md`
-2. Loads the living architecture map from `.github/agents/Agent-Developer-Assistant/memory/PROJECT-KNOWLEDGE.md`
+1. Loads its persistent memory from `docs/MEMORY.md`
+2. Loads the living architecture map from `docs/PROJECT-KNOWLEDGE.md`
 3. Checks recent `git log` to orient itself on what has changed
 
-Both memory files are updated at the end of sessions when something worth preserving is learned. The agent definition lives at `.github/agents/Agent-Developer-Assistant.md`.
+Both memory files are updated at the end of sessions when something worth preserving is learned.
 
 ## Testing