Observable vLLM Text-to-SQL Agent

Ask your data in English. A self-correcting LangGraph agent writes the SQL, runs it, checks the answer, and fixes itself — served by a 30B MoE model on a single H100, with every layer instrumented so you can see exactly where a request spends its time.

Demo

Serving dashboard (Grafana) — reacting under load	Agent trace (Langfuse) — generate→verify→revise

A real self-correcting run — the loop earning its keep on a live BIRD question:

$ curl -s -X POST localhost:8001/answer \
    -d '{"question":"What is the coordinates location of the circuits for Australian grand prix?","db":"formula_1"}'
{ "sql": "SELECT DISTINCT c.lat, c.lng FROM circuits c JOIN races r ON c.circuitId = r.circuitId
          WHERE r.name = 'Australian Grand Prix'",
  "rows": [[-34.9272, 138.617]], "iterations": 2, "ok": true }
# ← generate omitted DISTINCT (11 duplicate rows); verify flagged it; revise added DISTINCT → matches gold

What This Project Demonstrates

• Production-style LLM serving — vLLM hosting a Qwen3-30B-A3B MoE (30.5B total / 3.3B active) on one H100, with inference flags chosen for the workload, not defaults.
• A self-correcting agent that earns its keep — a LangGraph verify → revise loop whose value is measured, not assumed (per-iteration eval pass rate proves it).
• Two complementary observability planes — Prometheus + Grafana for serving health (latency percentiles, throughput, KV-cache headroom) and Langfuse for per-request agent traces.
• Metric-grounded performance engineering — a documented SLO-diagnosis loop ("saw X → hypothesized Y → changed Z → result W") with before/after evidence.
• Rigorous evaluation — execution-accuracy scoring on canonicalized row sets, separating capability from latency.
• Clean, layered architecture — strict dependency rules, no import cycles (see ARCHITECTURE.md).

Architecture

An analyst's question becomes SQL via a LangGraph state machine wrapping one vLLM-served model; the SQL runs read-only against a SQLite warehouse (BIRD-bench). The serving plane (vLLM /metrics → Prometheus → Grafana) answers "is the box slow, and where in the lifecycle?"; the agent plane (LangGraph spans → Langfuse) answers "why was this request slow, and why did it revise?" Full diagrams and the layer/dependency contract are in ARCHITECTURE.md.

question ─▶ FastAPI :8001 ─▶ LangGraph(generate→execute→verify→[revise]) ─▶ SQLite
                    │                    │
              Langfuse :3001       vLLM :8000 ──/metrics──▶ Prometheus :9090 ─▶ Grafana :3000

Quick Start

git clone git@github.com:MGhanayim/observable-vllm-text2sql-agent.git && cd observable-vllm-text2sql-agent
uv sync                                   # install pinned deps
cp .env.example .env                      # set VLLM_MODEL / (optional) Token Factory dev backend / Langfuse keys
uv run python scripts/load_data.py        # download BIRD subset → data/bird/
docker compose up -d                      # Prometheus + Grafana + Langfuse

bash scripts/start_vllm.sh                # serve the model (H100) — or point .env at a hosted API
uv run uvicorn agent.server:app --port 8001

Forward ports 3000 9090 3001 8000 8001 over SSH to reach the UIs (Grafana admin/admin).

Results

All numbers from the real Qwen3-30B-A3B on a single H100 (full writeup in REPORT.md).

Metric	Value
Execution accuracy (30-question BIRD set)	40.0% (12/30) — respectable zero-shot (fine-tuned SOTA ≈ 60–70%)
Pass rate: iter 0 → final	33.3% → 40.0% — the verify→revise loop adds +6.7 pp, and saturates at the first revise
Latency	p95 11.7 s @ 4 RPS (sustainable); collapses at the 10 RPS target
SLO (P95 < 5 s @ ≥10 RPS)	Missed — and fully diagnosed: structurally unreachable on one H100 (each request fans out to 2–4 vLLM calls)

The diagnostic highlight: the two observability planes together showed the GPU sitting idle (40/64 sequences, 34% KV, zero queue) while requests took minutes — proving the bottleneck was agent-side SQL execution and thread-pool concurrency, not the model. That cross-plane diagnosis — and a MAX_ITERATIONS=2 tuning that cut p95 ~40% with zero accuracy loss — is the heart of REPORT.md §3–§4.

Tech Stack

• Serving: vLLM (OpenAI-compatible) · Qwen3-30B-A3B-Instruct-2507 (MoE) · 1×H100
• Agent: LangGraph + LangChain 1.x · langchain-openai · FastAPI / Uvicorn
• Observability: Prometheus + Grafana (serving) · Langfuse 4.x (agent traces)
• Data / eval: BIRD-bench (SQLite) · execution-accuracy eval · async load driver
• Tooling: Python 3.11+ · uv · Docker Compose

Project Structure

agent/ (the app, layered L0–L4) · evals/ + load_test/ (offline harnesses) · infra/ + docker-compose.yml (observability stack) · scripts/ (serving + data). Full tree with layer tags in ARCHITECTURE.md (§3).

Project Docs

Doc	Purpose
SPEC.md	Functional requirements & design constraints
ARCHITECTURE.md	Architecture, dependency rules, diagrams
REPORT.md	The writeup — results & SLO diagnosis

License

MIT