Skip to content
BranchesTags

Repository files navigation

Agentic RAG on GCP

Multi-tenant retrieval-augmented chat. React + FastAPI + LangGraph + PostgreSQL/pgvector, deployable to Cloud Run.

Parse → chunk → embed → hybrid retrieve (dense + keyword + RRF) → rerank → grade → corrective web search when the evidence is thin → generate with citations.

Layer Choice
Frontend React (Vite), SSE token streaming, JWT auth
Backend FastAPI, SQLAlchemy 2, Alembic
Agent LangGraph: memory → rewrite → retrieve → grade →⟨web⟩→ augment → stream
Generate OpenAI gpt-5.6-luna, streamed, with a web_search tool
Embed tongyi-embedding-vision-flash (768-d, Alibaba Model Studio, Singapore)
Rerank qwen3-rerank cross-encoder
Retrieve pgvector HNSW cosine + Postgres FTS (GIN) + reciprocal rank fusion
Memory short-term window + rolling summary + embedded long-term facts
Store Cloud SQL Postgres 16 + pgvector, or local Docker
Auth Per-user accounts, scrypt passwords, HS256 JWT, admin role

Without API keys the app still runs. Retrieval uses deterministic offline embeddings and answers are explicitly labelled offline — they are never passed off as model output. See Degraded mode.

Quick start

cp .env.example .env
sed -i "s|^SECRET_KEY=.*|SECRET_KEY=$(openssl rand -hex 32)|" .env
docker compose up --build

Open http://localhost:8080 and create an account. The first account registered becomes the administrator. Turn off ALLOW_REGISTRATION afterwards.

Local development without Docker for the app:

make setup     # backend + frontend dependencies
make dev       # postgres, migrations, API with reload
make frontend  # Vite on :5173, in another terminal
Command What it does
make test Backend + frontend tests, no services required
make test-pg Integration tests against real Postgres (pgvector, GIN)
make harness RAG evaluation with the regression gate
make lint ruff + eslint
make user e=you@example.com Create an admin account from the CLI
make stats Index health, grouped by embedding model

System architecture

flowchart LR
  subgraph Client
    UI[React UI<br/>JWT session]
  end

  subgraph Edge["GCP"]
    FE[Cloud Run<br/>nginx + static]
    API[Cloud Run<br/>FastAPI]
  end

  subgraph Agent["LangGraph"]
    M[memory]
    Q[rewrite]
    R[retrieve]
    G{grade}
    W[web]
    A[augment]
  end

  subgraph Data
    PG[(Cloud SQL<br/>Postgres 16 + pgvector)]
    GCS[Cloud Storage]
  end

  subgraph Models
    OA[OpenAI<br/>gpt-5.6-luna]
    DS[DashScope<br/>embed + rerank]
    WEB[Tavily / DuckDuckGo]
  end

  UI --> FE --> API
  API --> M --> Q --> R --> G
  G -->|weak or empty| W --> A
  G -->|good| A
  A -->|stream| OA
  R --> PG
  M --> PG
  R --> DS
  W --> WEB
  API --> GCS
  OA -.->|SSE tokens| UI
Loading

The agent

stateDiagram-v2
  [*] --> memory
  memory --> rewrite
  rewrite --> retrieve
  retrieve --> grade
  grade --> web: weak / empty / user asked
  grade --> augment: good
  web --> augment
  augment --> [*]
Loading
Node What it does
memory Recent turns (token-budgeted), rolling summary, semantically recalled facts
rewrite Resolves pronouns against history — "what about its pricing?" retrieves nothing on its own
retrieve Dense + keyword, RRF fused, cross-encoder reranked, scoped to the caller
grade Scores the evidence; weak or empty reroutes to the web (corrective RAG)
web Tavily or DuckDuckGo, on user request or as a correction
augment Builds the prompt with passages, memory, web results and retrieval notes

Where generation lives. The compiled graph ends at augment; token streaming runs outside it in run_agent. LangGraph's stream yields state updates per node, not tokens from inside one, so putting generation in a node would mean buffering the whole answer and losing the streaming UX. The routing stays declarative and the token path stays direct.

Request path

sequenceDiagram
  participant U as User
  participant FE as React
  participant API as FastAPI
  participant LG as LangGraph
  participant V as pgvector
  participant LLM as gpt-5.6-luna

  U->>FE: sign in
  FE->>API: POST /api/auth/login → JWT
  U->>FE: message + optional file
  FE->>API: POST /api/documents (202 pending)
  API-->>FE: poll until status=ready
  FE->>API: POST /api/chat/stream (Bearer)
  API->>LG: memory → rewrite → retrieve → grade → augment
  LG->>V: dense + keyword + RRF + rerank (scoped by user_id)
  LG->>LLM: stream tokens + tool calls
  LLM-->>FE: token by token (SSE, 15s heartbeats)
  API->>V: persist message, sources, trace, token counts
Loading

Database

PostgreSQL 16 + pgvector. Relational data and vectors share one database, so a similarity query can filter by tenant in the same statement.

erDiagram
  users ||--o{ conversations : owns
  users ||--o{ documents : owns
  users ||--o{ memories : owns
  users ||--o{ traces : owns
  conversations ||--o{ messages : has
  conversations ||--o{ documents : scopes
  documents ||--o{ chunks : has

  users {
    string id PK
    string email UK
    string password_hash
    string role
    int token_version
    bool is_active
  }
  conversations {
    string id PK
    string user_id FK
    string title
    text summary
  }
  messages {
    string id PK
    string conversation_id FK
    string role
    text content
    jsonb sources
    int tokens_in
    int tokens_out
  }
  documents {
    string id PK
    string user_id FK
    string conversation_id FK
    string filename
    string storage_path
    string content_hash
    string status
    text error
  }
  chunks {
    string id PK
    string document_id FK
    text content
    tsvector content_tsv
    vector embedding
    string embedding_model
    int embedding_dim
  }
  memories {
    string id PK
    string user_id FK
    string conversation_id FK
    text content
    vector embedding
    string embedding_model
  }
  traces {
    string id PK
    string user_id FK
    text query
    text answer
    int tokens_in
    int tokens_out
    float score
    bool degraded
  }
Loading

Indexes that matter

Index Why
ix_chunks_embedding_hnsw HNSW cosine, m=16, ef_construction=200
ix_memories_embedding_hnsw Long-term recall was doing a sequential scan with no index
ix_chunks_content_tsv (GIN) On a stored generated column — to_tsvector(content) per row forced a full scan on every keyword search
ix_messages_conversation_created Every transcript load is "this chat, in order"
ix_conversations_user_updated The sidebar query
ix_documents_user_hash Upload deduplication by content hash

hnsw.ef_search is set per connection (HNSW_EF_SEARCH, default 100) — it is the main recall-versus-latency dial. Raise it before touching m.

Two invariants

Tenancy. Every retrieval query filters on documents.user_id. There is no code path that reads chunks the caller does not own, and tests/test_postgres.py proves it against a real database.

Vector comparability. Dense search matches on embedding_model as well as distance. Cosine distance between vectors from two different models is meaningless; comparing them returns confident nonsense. Rows embedded by a different model are excluded until reindexed:

make stats                                    # chunks grouped by embedding model
curl -X POST .../api/documents/<id>/reindex   # re-embed with the current model

Degraded mode

The app runs without API keys, and says so.

Missing Behaviour
OPENAI_API_KEY Retrieval runs; the reply is the retrieved context, prefixed "Offline mode". The UI shows a retrieval only badge.
DASHSCOPE_API_KEY Deterministic offline embeddings tagged offline-hash-v1, and a stopword-filtered lexical reranker. Both are flagged degraded in the trace.

What it will not do is silently fake a real vector. The previous implementation caught every exception in the embedding path and returned locally hashed vectors, so a bad key, a wrong region or a transient 503 wrote garbage into the same column as real embeddings — with no error, and no way to tell them apart afterwards. Now a configured-but-failing provider raises (STRICT_EMBEDDINGS, mandatory in production).

DashScope regions

tongyi-embedding-vision-flash is served only from Singapore (dashscope-intl.aliyuncs.com). multimodal-embedding-v1 is Beijing only. Set DASHSCOPE_REGION=intl or cn; the app validates the pairing at startup and refuses to boot on a mismatch rather than failing every embedding call at runtime.

EMBEDDING_DIM is likewise checked against the model's fixed output width, because the pgvector column is fixed-width and a mismatch would fail on insert.

Security

Control Implementation
Authentication scrypt (RFC 7914) passwords, HS256 JWT access + refresh, stdlib only
Session revocation users.token_version — password change or sign-out-everywhere invalidates every issued token immediately
Authorisation Ownership checked in one place (api/deps.py) and again in retrieval SQL
Enumeration Login pads failed attempts to a fixed floor; unknown emails still run a hash; foreign ids return 404, not 403
Uploads Type allowlist, magic-byte sniffing, size cap, content-hash dedup
Path containment Every read re-validates against the storage root; symlinks and .. are defeated by resolve()
Rate limits Per-user token buckets on chat/search/upload, IP-keyed on auth
Spend Daily token quota enforced in Postgres, so it holds across instances
Transport HSTS, nosniff, DENY framing, explicit CORS origins (wildcards rejected)
Errors 500s return a request id, never a stack trace or internal message

Two fixed vulnerabilities

Arbitrary file read. image_path used to arrive in the chat request body and flow into Path(path).read_bytes(). A client could send /etc/passwd, or a gs:// URL for any bucket the service account could reach, and the bytes were base64-encoded and shipped to a third-party embedding API — a blind file read with exfiltration. The field is gone from the schema; the image is now derived server-side from a document row already checked for ownership, and storage.assert_safe_path re-validates on every read so a future caller that forgets fails closed.

No tenancy. Every endpoint was unauthenticated and unscoped: GET /api/chats returned every conversation in the database and GET /api/traces dumped every user's prompts and answers to anyone who asked.

Known trade-offs

  • Tokens live in localStorage, which is XSS-readable. Mitigated by 30-minute access tokens, server-side revocation and a strict CSP. httpOnly cookies would need a same-site deployment plus CSRF protection.
  • Rate limiting is in-process. With N Cloud Run instances the effective limit is N × the configured rate. It guards against one runaway client; the daily token quota is the actual spend control. Swap in a Redis-backed RateLimiter for exact distributed limits.

Testing

177 backend tests   offline, SQLite, ~5s
 10 integration     real Postgres + pgvector (make test-pg)
 17 frontend        vitest
 18 eval questions  make harness
File Covers
test_auth.py scrypt, JWT (incl. alg:none and tamper rejection), endpoints
test_tenancy.py Cross-tenant isolation on every resource
test_security.py Path traversal, symlinks, upload validation, config validation
test_services.py Embeddings, rerank, RRF, chunking, vector literals
test_chat.py SSE streaming, rate limits, quota, CRAG routing
test_harness.py The metrics and gates themselves
test_postgres.py pgvector, HNSW/GIN indexes, tenancy in raw SQL

The suite needs no services and no network. Postgres-only tests are marked postgres and skipped unless TEST_DATABASE_URL is set.

Deploying to GCP

cd infra/terraform
cp terraform.tfvars.example terraform.tfvars   # edit project, origin, environment
terraform init && terraform apply

Then build, migrate and deploy:

gcloud builds submit --config infra/cloudbuild.yaml \
  --substitutions=_REGION=us-central1,_SQL_INSTANCE=$(terraform output -raw sql_connection_name)

What the Terraform gives you:

  • Cloud SQL with private IP only, regional HA, PITR and deletion protection in production
  • A VPC connector so Cloud Run reaches the database privately
  • The backend is not publicly invokable — only the frontend's service account can call it (expose_backend_publicly opts out, for running the harness)
  • Per-secret IAM rather than a project-wide secretAccessor
  • A generated JWT signing key that never enters tfvars or your shell history
  • cpu_idle = false so SSE streams are not throttled between tokens
  • 900s request timeout — long generations exceed Cloud Run's 300s default

Migrations run as a Cloud Run job, not in the container command. With more than one instance, alembic upgrade head on startup means every replica races it, and a failed migration takes the service down instead of stopping the release. The build fails if the migration fails, before any traffic shifts.

After the first apply

gcloud run jobs execute luna-migrate --region us-central1 --wait
# register the first account (it becomes admin), then:
#   allow_registration = false   → terraform apply

Configuration

Every setting is in .env.example with an explanation. The ones that will bite you:

Variable Note
SECRET_KEY Production refuses to start on the default or under 32 chars
CORS_ORIGINS Explicit origins only; production requires https
DASHSCOPE_REGION Must match the embedding model's region
EMBEDDING_DIM Must match the model; validated at startup
STRICT_EMBEDDINGS Cannot be disabled in production
DAILY_TOKEN_QUOTA The real spend ceiling
RETRIEVAL_GRADE_THRESHOLD Below this, corrective web search fires
HNSW_EF_SEARCH Recall/latency dial, applied per connection

Production startup validation is deliberately fatal: a misconfigured deployment should fail to boot, not serve traffic with the wrong CORS policy and a default signing key.

Operations

See docs/RUNBOOK.md for incident procedures.

Logs are structured JSON (LOG_FORMAT=json), carry a request id, and correlate with Cloud Trace. Secrets are redacted by the formatter.

# error rate
gcloud logging read 'resource.labels.service_name="luna-backend" severity>=ERROR' --limit 50

# turns that ran on fallback providers
curl -H "authorization: Bearer $ADMIN_JWT" "$API/api/traces/all?degraded_only=true"

About

Agentic RAG on GCP (React, FastAPI, LangGraph, pgvector)

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages