Graph-Routed Memory (GRM)

This repository contains the implementation of Graph-Routed Memory (GRM): a hybrid graph-and-vector memory router for retrieval-augmented question answering. The artefact provides a controlled comparison between:

• Vector-only retrieval (baseline)
• GRM graph-routed retrieval (graph-first routing, with optional composed facts)

The system exposes a FastAPI backend and a Streamlit dashboard to ingest a dataset, run queries, and run an evaluation on a 50-question HotpotQA dev subset with token/cost logging.

---

Features

• FastAPI backend

  • /ingest to load a HotpotQA subset into memory (graph + vector index)
  • /query to answer questions using either Vector or GRM mode
  • /health to verify the service is running

• Streamlit dashboard

  • Single Query mode (Vector vs GRM)
  • Evaluation mode (runs both modes on a dataset subset and exports CSV)
  • Overview charts for F1, tokens, and latency (with value labels)

• Evaluation outputs

  • Per-question CSV export containing:
    • Vector/GRM answers
    • EM/F1
    • token usage (prompt/completion/total)
    • latency and cost estimates

---

Repository layout

• src/grm/ — core GRM implementation and FastAPI service (grm.api)
• ui/streamlit_app/ — Streamlit dashboard (app.py) and backend client (api_client.py)
• scripts/ — utility scripts (e.g., tuning compose_facts)
• data/
  • data/hotpot/hotpot_dev_50.json — 50-question HotpotQA dev subset (recommended for evaluation)
  • other dataset files may exist locally but are not required for the 50Q run

---

Requirements

Local (WSL / Linux / macOS)

• Python 3.11
• A working virtual environment (recommended)
• OpenAI key for OpenAI mode (optional if using Echo mode)

Docker (recommended for submission/demo)

• Docker Desktop (Windows) with WSL2 integration enabled, or Docker Engine on Linux

---

Quickstart (Docker Compose)

1. Create a local .env in the repo root (do not commit it):

OPENAI_API_KEY=sk-...

2. Start API + UI:

docker compose up --build

3. Open:

• API docs: http://localhost:8000/docs
• Streamlit UI: http://localhost:8501

4. In Streamlit:

• Expand Dataset ingestion
• Dataset path: data/hotpot/hotpot_dev_50.json
• Limit questions: 50
• Embedder: st
• Click Ingest dataset
• Switch to Evaluation
  • Dataset: Hotpot dev (50 Q)
  • LLM: OpenAI
  • Max questions: 50
  • GRM compose_facts: 4
  • Click Run
• Download results using Download per-question metrics (CSV)

Stop services:

Ctrl+C
docker compose down

---

Quickstart (Local)

1) Setup environment

cd graph-routed-memory
python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install uv
uv pip install -e ".[dev]"

2) Start the backend

From the repository root (recommended to keep dataset paths consistent):

export OPENAI_API_KEY="sk-..."   # optional if using OpenAI mode
python -m uvicorn src.grm.api:app --reload --host 127.0.0.1 --port 8000

Check:

curl http://127.0.0.1:8000/health

3) Start the dashboard

Open a second terminal:

source .venv/bin/activate
streamlit run ui/streamlit_app/app.py --server.port 8501

Open:

• http://localhost:8501

Then ingest and run evaluation as described in the Docker section.

---

Evaluation workflow (50-question HotpotQA dev subset)

The recommended evaluation uses the bundled 50-question subset:

• File: data/hotpot/hotpot_dev_50.json
• Ingestion: Embedder = st (sentence-transformers)
• Evaluation: LLM = OpenAI, compose_facts = 4, max_snippets = 1

The Streamlit Evaluation page computes:

• Answer EM
• Answer F1
• Mean token usage (prompt/completion/total)
• Mean latency and cost estimates
• Token savings (absolute and %)

Results can be exported to CSV for dissertation tables/plots.

---

Notes and limitations

• Exact Match (EM) often equals 0 in this generative QA setup because answers may be paraphrased or written as full sentences. F1 is the primary accuracy signal used to compare overlap with gold answers.
• OpenAI embeddings for ingestion were attempted but were not used in the final workflow because ingestion could take too long and trigger timeouts in the project environment. The final evaluation uses sentence-transformers embeddings (st) to keep ingestion reproducible and practical.
• FactScore is not integrated into the submitted artefact evaluation pipeline. The evaluation reports benchmark-grounded EM/F1 and token/latency/cost from API responses.
• The submitted artefact is a single-agent QA pipeline with a modular memory layer (router + retrieval + generator). It does not implement multi-agent conversational orchestration (e.g., AutoGen-style multi-agent dialogues).

---

Security

• Never commit your .env file.
• Do not paste API keys into issues, screenshots, or logs.
• If a key is accidentally shared, revoke/rotate it immediately.

---

License

This project is provided as an MSc dissertation artefact.