Milestone 4: hybrid retrieval and context packs

.github/workflows/ci.yml

@@ -25,17 +25,17 @@ jobs:
           python -m pip install -e . --no-deps
           python -m pip install pytest pytest-asyncio pydantic pydantic-settings click PyYAML fastapi structlog ruff mypy mcp httpx
 
-      - name: Run Milestone 0-3 tests
-        run: PYTHONPATH=src python -m pytest tests/test_m0_baseline.py tests/test_m1_services.py tests/test_m2_registry.py tests/test_citations.py tests/test_m3_citations.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_mcp_tools.py tests/test_ui_handlers.py
+      - name: Run Milestone 0-4 tests
+        run: PYTHONPATH=src python -m pytest tests/test_m0_baseline.py tests/test_m1_services.py tests/test_m2_registry.py tests/test_citations.py tests/test_m3_citations.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_mcp_tools.py tests/test_ui_handlers.py tests/test_m4_hybrid_retrieval.py
 
-      - name: Run Milestone 0-3 lint baseline
-        run: ruff check src/coderag/generation src/coderag/models src/coderag/services src/coderag/config.py src/coderag/api/schemas.py src/coderag/api/routes.py src/coderag/mcp/handlers.py src/coderag/mcp/tools.py src/coderag/ui/grounding.py src/coderag/ui/repository_metadata.py src/coderag/ui/handlers.py tests/test_m0_baseline.py tests/test_m1_services.py tests/test_m2_registry.py tests/test_citations.py tests/test_m3_citations.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_mcp_tools.py tests/test_ui_handlers.py scripts/evalfly_smoke.py
+      - name: Run Milestone 0-4 lint baseline
+        run: ruff check src/coderag/generation src/coderag/indexing/vectorstore.py src/coderag/models src/coderag/retrieval src/coderag/services src/coderag/config.py src/coderag/api/schemas.py src/coderag/api/routes.py src/coderag/mcp/handlers.py src/coderag/mcp/tools.py src/coderag/cli.py src/coderag/ui/grounding.py src/coderag/ui/repository_metadata.py src/coderag/ui/handlers.py tests/test_m0_baseline.py tests/test_m1_services.py tests/test_m2_registry.py tests/test_citations.py tests/test_m3_citations.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_mcp_tools.py tests/test_ui_handlers.py tests/test_m4_hybrid_retrieval.py scripts/evalfly_smoke.py
 
-      - name: Run Milestone 0-3 type baseline
-        run: PYTHONPATH=src mypy src/coderag/generation src/coderag/models src/coderag/services src/coderag/config.py src/coderag/api/schemas.py src/coderag/api/routes.py src/coderag/mcp/handlers.py src/coderag/ui/grounding.py src/coderag/ui/repository_metadata.py src/coderag/ui/handlers.py tests/test_m0_baseline.py tests/test_m1_services.py tests/test_m2_registry.py tests/test_citations.py tests/test_m3_citations.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_ui_handlers.py scripts/evalfly_smoke.py
+      - name: Run Milestone 0-4 type baseline
+        run: PYTHONPATH=src mypy src/coderag/generation src/coderag/indexing/vectorstore.py src/coderag/models src/coderag/retrieval src/coderag/services src/coderag/config.py src/coderag/api/schemas.py src/coderag/api/routes.py src/coderag/mcp/handlers.py src/coderag/ui/grounding.py src/coderag/ui/repository_metadata.py src/coderag/ui/handlers.py tests/test_m0_baseline.py tests/test_m1_services.py tests/test_m2_registry.py tests/test_citations.py tests/test_m3_citations.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_ui_handlers.py tests/test_m4_hybrid_retrieval.py scripts/evalfly_smoke.py
 
       - name: Compile changed runtime modules
-        run: python -m py_compile src/coderag/__init__.py src/coderag/config.py src/coderag/main.py src/coderag/cli.py src/coderag/indexing/embeddings.py src/coderag/generation/*.py src/coderag/models/*.py src/coderag/services/*.py src/coderag/api/schemas.py src/coderag/api/routes.py src/coderag/mcp/handlers.py src/coderag/ui/grounding.py src/coderag/ui/repository_metadata.py src/coderag/ui/handlers.py tests/test_m0_baseline.py tests/test_m1_services.py tests/test_m2_registry.py tests/test_citations.py tests/test_m3_citations.py tests/test_api_routes.py tests/test_ui_handlers.py scripts/evalfly_smoke.py
+        run: python -m py_compile src/coderag/__init__.py src/coderag/config.py src/coderag/main.py src/coderag/cli.py src/coderag/indexing/embeddings.py src/coderag/indexing/vectorstore.py src/coderag/generation/*.py src/coderag/models/*.py src/coderag/retrieval/*.py src/coderag/services/*.py src/coderag/api/schemas.py src/coderag/api/routes.py src/coderag/mcp/handlers.py src/coderag/mcp/tools.py src/coderag/ui/grounding.py src/coderag/ui/repository_metadata.py src/coderag/ui/handlers.py tests/test_m0_baseline.py tests/test_m1_services.py tests/test_m2_registry.py tests/test_citations.py tests/test_m3_citations.py tests/test_api_routes.py tests/test_ui_handlers.py tests/test_m4_hybrid_retrieval.py scripts/evalfly_smoke.py
 
       - name: Build package
         run: python -m build

.pi/.specsafe-state.json

@@ -120,6 +120,26 @@
           "turns": 0
         }
       }
+    },
+    {
+      "sliceId": "CRAG-20260630-M4-HYBRID-CONTEXT-PACKS",
+      "workspaceId": "CodeRAG",
+      "sessionId": "omp-session",
+      "beganAt": "2026-06-30T20:39:16.458Z",
+      "endedAt": "2026-06-30T20:58:56.662Z",
+      "outcome": "PASS",
+      "costSummary": {
+        "externalMemoryCalls": 0,
+        "externalMemoryCost": 0,
+        "subagentTokens": {
+          "input": 0,
+          "output": 0,
+          "cacheRead": 0,
+          "cacheWrite": 0,
+          "cost": 0,
+          "turns": 0
+        }
+      }
     }
   ]
 }

docs/superpowers/plans/2026-06-30-m4-hybrid-retrieval-context-packs-contract.md

@@ -0,0 +1,135 @@
+# Milestone 4 Contract: Hybrid Retrieval and Context Packs
+
+**Slice:** `CRAG-20260630-M4-HYBRID-CONTEXT-PACKS`
+
+**Branch:** `feat/m4-hybrid-retrieval-context-packs`
+
+**Goal:** Modernize CodeRAG beyond vector-only retrieval by adding explainable hybrid retrieval and structured context packs that work without LLM generation.
+
+## Scope
+
+- Add local lexical retrieval over indexed chunks.
+- Preserve existing semantic vector retrieval and query behavior.
+- Merge vector + lexical candidates with score components.
+- Add path/name metadata boosting.
+- Deduplicate same-file overlapping ranges.
+- Add deterministic token/context budgeting.
+- Add `ContextPack` schema and shared service method.
+- Expose context packs via MCP, REST, and CLI.
+- Extend deterministic tests, EvalFly smoke checks, and CI M4 gates.
+
+## Non-goals
+
+- No code graph MVP in M4.
+- No autonomous editing behavior.
+- No new model/provider abstraction beyond using existing embeddings/vector store.
+- No production-ready claim.
+- No broad README rewrite beyond later M6.
+
+## Required Interfaces
+
+```python
+@dataclass
+class ScoreBreakdown:
+    vector: float | None
+    lexical: float | None
+    path: float
+    name: float
+    combined: float
+
+@dataclass
+class RetrievalBudget:
+    max_chunks: int
+    max_tokens: int
+    max_chunks_per_file: int
+
+class HybridRetriever:
+    def retrieve(
+        self,
+        query: str,
+        repo_id: str,
+        top_k: int,
+        budget: RetrievalBudget | None = None,
+    ) -> list[RetrievedChunk]: ...
+```
+
+`RetrievedChunk` may be extended with optional metadata fields for score components, token estimate, and retrieval sources without breaking existing callers.
+
+```python
+class RetrievalService:
+    def search_hybrid(self, repo_id: str, query: str, top_k: int = 10) -> list[RetrievedChunk]: ...
+    def get_context_pack(self, repo_id: str, query: str, top_k: int = 8, max_tokens: int = 4000) -> ContextPack: ...
+```
+
+## Acceptance Criteria
+
+### AC-M4.1 Hybrid retrieval
+
+**Pass requires**
+- Semantic vector candidates remain included when embeddings/vectorstore are available.
+- Lexical candidates are computed locally from indexed chunk text.
+- Combined score includes vector, lexical, path, name, and combined components.
+- Ranking order is explainable from score components.
+
+**Fail if**
+- Retrieval remains vector-only.
+- Lexical-only exact matches cannot surface without semantic score.
+- Results have no score explanation.
+
+### AC-M4.2 Dedupe and budget
+
+**Pass requires**
+- Same-file overlapping line ranges are merged/deduped.
+- Budget enforces max chunks, max estimated tokens, and max chunks per file deterministically.
+- Top relevant result is preserved when within budget.
+
+**Fail if**
+- Duplicate vector+lexical hits consume multiple slots.
+- Budget fields are ignored.
+
+### AC-M4.3 Context packs
+
+**Pass requires**
+- Context packs are generated without invoking an LLM.
+- Pack includes query, repo id, snippets, citation candidates, score/ranking reasons, token estimate, budget, and capability metadata.
+- MCP, REST, and CLI expose the pack as structured JSON; CLI also exposes markdown.
+
+**Fail if**
+- Context pack generation requires cloud or local generation.
+- Pack lacks provenance for why files/snippets were selected.
+
+### AC-M4.4 Compatibility
+
+**Pass requires**
+- Existing `query_code` behavior still returns answer/citations/evidence/grounded.
+- Existing `search_code` compatibility is preserved or a migration test proves the shape remains usable.
+- API/MCP/CLI tests cover new outputs.
+
+**Fail if**
+- Existing M0-M3 tests are weakened or removed.
+- MCP/CLI/API disagree on context-pack result shape.
+
+## Required Verification
+
+Run before closing the slice:
+
+```bash
+bun run /Users/sebastian/.omp/omp-pantheon/skills/specsafe/bin/specsafe.ts status
+PYTHONPATH=src .venv/bin/python -m pytest tests/test_m4_hybrid_retrieval.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_mcp_tools.py tests/test_m3_citations.py
+.venv/bin/python -m ruff check src/coderag/retrieval src/coderag/models src/coderag/services src/coderag/indexing/vectorstore.py src/coderag/api/routes.py src/coderag/api/schemas.py src/coderag/mcp/handlers.py src/coderag/mcp/tools.py src/coderag/cli.py tests/test_m4_hybrid_retrieval.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_mcp_tools.py tests/test_m3_citations.py scripts/evalfly_smoke.py
+PYTHONPATH=src .venv/bin/mypy src/coderag/retrieval src/coderag/models src/coderag/services src/coderag/indexing/vectorstore.py src/coderag/api/routes.py src/coderag/api/schemas.py src/coderag/mcp/handlers.py src/coderag/mcp/tools.py tests/test_m4_hybrid_retrieval.py tests/test_api_routes.py tests/test_mcp_handlers.py tests/test_mcp_tools.py tests/test_m3_citations.py scripts/evalfly_smoke.py
+.venv/bin/python scripts/evalfly_smoke.py --suite smoke
+.venv/bin/python -m py_compile src/coderag/retrieval/*.py src/coderag/models/*.py src/coderag/services/*.py src/coderag/indexing/vectorstore.py src/coderag/api/routes.py src/coderag/api/schemas.py src/coderag/mcp/handlers.py src/coderag/mcp/tools.py src/coderag/cli.py tests/test_m4_hybrid_retrieval.py
+.venv/bin/python -m build
+```
+
+Then run the local CI suite with M0-M4 tests.
+
+## Review Gate
+
+A reviewer must approve that:
+- hybrid retrieval is not vector-only;
+- context packs do not call generation;
+- score/budget/dedupe tests are behavior-level;
+- public adapters remain JSON-safe;
+- M4 does not implement graph/autonomous behavior.

docs/superpowers/plans/2026-06-30-m4-m7-execution-plan.md

@@ -0,0 +1,166 @@
+# CodeRAG M4-M7 Execution Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Complete CodeRAG Milestones 4 through 7 without changing the product into an autonomous coding agent.
+
+**Architecture:** Keep API, MCP, CLI, and UI as adapters over shared services. Add hybrid retrieval and context packs first, then graph metadata, then stabilize public interfaces/docs, then prove release readiness with deterministic evidence.
+
+**Tech Stack:** Python 3.11+, FastAPI, Click, MCP/FastMCP, SQLite registry, ChromaDB behind the existing vector-store boundary, Tree-sitter/Python AST where available, pytest, ruff, mypy, EvalFly, GitHub Actions.
+
+## Global Constraints
+
+- CodeRAG remains a RAG/code-intelligence product; it must not edit files, create commits for users, open PRs for users, or run autonomous issue-solving loops.
+- Default quickstart must not require a local generative LLM.
+- Retrieval-only flows must not require cloud or local LLM credentials.
+- Any cloud generation path must remain profile-gated and documented; no blanket local-first privacy claim.
+- No production-ready claim is allowed without fresh tests, EvalFly evidence, branch E2E/run records where feasible, GitHub CI, and reviewer validation.
+- Existing public query behavior remains compatible unless a migration is documented and tested.
+- Each milestone uses its own branch, SpecSafe slice, contract file, TDD implementation, multiple atomic commits, guarded push per commit, PR, CI watch, review, merge, and local main sync.
+
+---
+
+## Milestone 4: Hybrid Retrieval and Context Packs
+
+**Branch:** `feat/m4-hybrid-retrieval-context-packs`
+
+**Slice:** `CRAG-20260630-M4-HYBRID-CONTEXT-PACKS`
+
+**Goal:** Replace vector-only retrieval with explainable hybrid retrieval and add no-generation context packs.
+
+**Files:**
+- Create: `src/coderag/retrieval/hybrid.py`
+- Create: `src/coderag/models/context.py`
+- Modify: `src/coderag/retrieval/retriever.py`
+- Modify: `src/coderag/services/retrieval.py`
+- Modify: `src/coderag/indexing/vectorstore.py`
+- Modify: `src/coderag/api/schemas.py`
+- Modify: `src/coderag/api/routes.py`
+- Modify: `src/coderag/mcp/handlers.py`
+- Modify: `src/coderag/mcp/tools.py`
+- Modify: `src/coderag/cli.py`
+- Test: `tests/test_m4_hybrid_retrieval.py`
+- Test: `tests/test_api_routes.py`
+- Test: `tests/test_mcp_handlers.py`
+- Test: `tests/test_mcp_tools.py`
+- Test: `tests/test_m3_citations.py` only if query result shapes need CLI coverage reuse.
+- Modify: `evals/config.json`
+- Modify: `.github/workflows/ci.yml`
+
+**Interfaces:**
+- `VectorStore.get_repo_chunks(repo_id: str) -> list[Chunk]`: returns indexed chunks with content from Chroma documents/metadatas.
+- `HybridRetriever.retrieve(query: str, repo_id: str, top_k: int, max_tokens: int | None = None) -> list[RetrievedChunk]`: returns deduped, scored, explained chunks.
+- `RetrievalService.get_context_pack(repo_id: str, query: str, top_k: int = 8, max_tokens: int = 4000) -> ContextPack`.
+- MCP tool: `get_context_pack(repo_id, query, top_k=8, max_tokens=4000)`.
+- CLI command: `coderag context-pack <repo_id> <query> --format json|markdown`.
+
+**Acceptance:**
+- Hybrid retrieval combines vector candidates and local lexical candidates.
+- Each retrieved chunk exposes score components and `retrieval_sources` through metadata-compatible fields.
+- Dedupe merges overlapping same-file ranges.
+- Context pack generation invokes no LLM.
+- Context pack includes query, repository id, files/snippets, citation candidates, ranking reasons, token estimate, budget, and profile/capability metadata.
+- EvalFly includes retrieval/context-pack smoke assertions.
+
+---
+
+## Milestone 5: Code Intelligence Graph MVP
+
+**Branch:** `feat/m5-code-intelligence-graph`
+
+**Slice:** `CRAG-20260630-M5-CODE-GRAPH`
+
+**Goal:** Add deterministic Python and TS/JS symbol graph with blast-radius APIs.
+
+**Files:**
+- Create: `src/coderag/graph/models.py`
+- Create: `src/coderag/graph/extractor.py`
+- Create: `src/coderag/services/graph.py`
+- Modify: `src/coderag/services/registry.py` for graph tables or cached graph records.
+- Modify: `src/coderag/api/schemas.py`
+- Modify: `src/coderag/api/routes.py`
+- Modify: `src/coderag/mcp/handlers.py`
+- Modify: `src/coderag/mcp/tools.py`
+- Modify: `src/coderag/cli.py`
+- Test: `tests/test_m5_graph.py`
+- Test: API/MCP/CLI tests for `find_symbol`, `find_references`, `get_blast_radius`.
+- Modify: `evals/config.json`
+- Modify: `.github/workflows/ci.yml`
+
+**Interfaces:**
+- `GraphService.build_graph(repo_id: str) -> CodeGraph`.
+- `GraphService.find_symbol(repo_id: str, name: str) -> list[SymbolRecord]`.
+- `GraphService.find_references(repo_id: str, name: str) -> list[ReferenceRecord]`.
+- `GraphService.get_blast_radius(repo_id: str, path: str | None = None, symbol: str | None = None) -> BlastRadius`.
+
+**Acceptance:**
+- Python fixture extracts modules, classes, functions, methods, imports, and test associations.
+- TS/JS fixture extracts imports, exported functions/classes, and references with deterministic regex/Tree-sitter-safe fallback.
+- Unsupported languages return explicit capability fallback, not fake graph confidence.
+- Incremental cache uses file content hash.
+- CLI/API/MCP expose graph operations with structured JSON.
+
+---
+
+## Milestone 6: Production Interfaces and Docs
+
+**Branch:** `feat/m6-production-interfaces-docs`
+
+**Slice:** `CRAG-20260630-M6-PRODUCTION-INTERFACES`
+
+**Goal:** Stabilize user-facing contracts and align docs with tested behavior.
+
+**Files:**
+- Modify: `src/coderag/api/schemas.py`
+- Modify: `src/coderag/api/routes.py`
+- Modify: `src/coderag/mcp/tools.py`
+- Modify: `src/coderag/mcp/handlers.py`
+- Modify: `src/coderag/cli.py`
+- Modify: `README.md`
+- Create/modify docs under existing docs path only if needed: privacy/profile matrix, migration notes, troubleshooting.
+- Test: contract snapshot tests for CLI/API/MCP representative outputs.
+- Modify: `evals/config.json`
+- Modify: `.github/workflows/ci.yml`
+
+**Acceptance:**
+- MCP tools include index/update/list/query/search_hybrid/get_context_pack/find_symbol/find_references/get_blast_radius/verify_citations where implemented.
+- CLI exposes equivalent useful commands.
+- REST schemas cover outputs and error shapes.
+- README positions CodeRAG as verified codebase context engine, not autonomous agent.
+- Docs do not claim untested production readiness or blanket privacy.
+
+---
+
+## Milestone 7: Production Release Candidate
+
+**Branch:** `feat/m7-production-release-candidate`
+
+**Slice:** `CRAG-20260630-M7-PRODUCTION-RC`
+
+**Goal:** Prove the product works end to end and only then mark release-candidate readiness.
+
+**Files:**
+- Modify: `evals/config.json`
+- Create: fresh `evals/runs/<run-id>.json` and `evals/reports/<run-id>.md` if the existing EvalFly harness writes them.
+- Create: `docs/superpowers/plans/2026-06-30-m7-release-run-record.md`.
+- Modify: `README.md` only for evidence badges/claims that are actually supported.
+- Modify CI if a release gate is missing.
+
+**Acceptance:**
+- Clean install path is exercised from fresh venv or equivalent isolated environment.
+- CI-equivalent local transcript passes.
+- EvalFly smoke, retrieval, citations, graph, interfaces/security smoke cases pass or unsupported cases are explicitly excluded with evidence and user-visible wording.
+- Branch E2E/run record covers CLI/API/MCP flows; UI flow is included only if Gradio dependencies are installable in the environment.
+- Final reviewer validates AC-0 through AC-10.
+- Release checklist in roadmap is all PASS or explicitly out of release scope.
+
+---
+
+## Execution Rules
+
+- Use TDD for behavior changes: red test, expected failure, green implementation, full affected gate.
+- Do not weaken existing M0-M3 tests or CI gates.
+- Do not add autonomous editing behavior.
+- Commit messages use English semantic style observed in repo history.
+- Every commit includes `Spec-Slice: <active slice>` trailer and is pushed immediately with the guarded push wrapper.
+- Remote PR/merge operations use the guarded GitHub wrapper with preview before `--i-approve`.