A governance-first reference implementation of enterprise RAG.
Every answer should be able to prove four things: what it retrieved, why this user was allowed to see it, which approved model produced it, and that the system passed its eval gates before release.
flowchart TD
U["User question + identity"] --> R
subgraph DB["Postgres — row-level security"]
R["Hybrid retrieval<br/>vector + full-text, RRF fused"]
ACL[("doc_acl")]
R -. "entitlements scope the<br/>candidate set, not a post-filter" .-> ACL
end
R --> CTX["Retrieved chunks<br/>tagged by classification"]
CTX --> GW
subgraph GOV["Governed model access"]
GW{{"gateway.complete<br/>one choke point"}}
REG[("policy/models.yaml<br/>registry + ceiling")]
RULES[("policy/routing.yaml<br/>routing rules, deny-by-default")]
GW --> REG
GW --> RULES
end
GW -->|"allowed"| MODEL["Approved model"]
GW -->|"denied"| REFUSE["Refusal<br/>no approved model permitted"]
MODEL --> ANSWER["Answer + citations"]
ANSWER --> LEDGER[("Evidence ledger<br/>hash-chained, append-only")]
REFUSE --> LEDGER
style DB fill:#eef4ff,stroke:#5b7fd1
style GOV fill:#fff3e6,stroke:#d19a5b
style LEDGER fill:#f3eefc,stroke:#8a5bd1,color:#000000
Every arrow is enforced where it's drawn, not merely documented: permissions inside Postgres via row-level security (Phase 4), model access through one gateway checked against a registry ceiling and a deny-by-default rule file (Phase 5), and every answer — allowed or refused — written to a tamper-evident ledger (Phase 6).
- Permissions are enforced before retrieval, never by filtering results afterward.
- Measure before optimizing. The eval gate lands before any retrieval tuning.
- Fail closed. If policy can't be evaluated, the answer is "no", with a logged reason.
- Retrieved text is data, not instructions.
- Governance is framework-agnostic. LangChain/LangGraph run the pipeline; policy, evidence and evals don't depend on them.
Measured on the golden set at each phase. Every number below came from a run in this repo, not from an estimate.
The first two columns are the six-document seed corpus scored against 10 items. The last is the full corpus — 70 documents, 148 chunks — scored against 44 items across five personas.
| Naive (Phase 1) | Hybrid retrieval (Phase 3) | Enforced, full corpus (Phase 4) | |
|---|---|---|---|
| Items | 10 | 10 | 44 |
| Context leak | 10/10 | 10/10 | 0/44 |
| Retrieval misses | 0/10 | 0/10 | 0/44 |
| Superseded doc in context | 10/10 | 0/10 | 2/44 |
| Fabricated citations | — | — | 0/44 |
| Ungrounded figures | — | — | 0/44 |
| Answer-tier failures | G05, G06 | G04, G06 | G32 (known) |
Four findings are worth more than the table.
A contractor got a production rollback command. Riley's entitlements are public only. Asked how to roll back the payments API after a bad deploy, the naive pipeline retrieved the on-call runbook and the model repeated the deployctl command without hesitation — context_leak, answer_forbidden_content and no_refusal on a single item.
Improving retrieval made a leak worse. In Phase 1, Sam (engineering, no compensation access) asked for the Senior Engineer II salary band. The confidential document reached the model's context, but the model declined to repeat the figures, so only context_leak fired. After Phase 3 improved retrieval quality — same corpus, same permissions, same model — the model answered with the numbers.
Nothing about the exposure changed. Retrieval just got better at surfacing what it had already been handed. Every retrieval improvement is an amplifier on whatever exposure already exists, which is precisely why permissions have to be enforced before retrieval rather than after. Judged on the answer alone, the Phase 1 version of that item passes and ships. A model's good manners are not an access control.
A three-position ranking miss produced a confident, wrongly-sourced answer. Expanding the corpus pushed the travel policy's receipt rule just outside a five-chunk window. Asked the receipt threshold, the model answered "$75" — a figure appearing nowhere in its context — and attached a real source tag to a real retrieved document that says nothing about receipts. The answer looked fully cited. Raising TOP_K to 8 restored the source and the correct $50.
That failure produced two checks the gate now runs on every item: citation_unsupported (a tag naming a document never retrieved) and ungrounded_value (a currency or percentage figure appearing nowhere in the context). Both are deterministic and cost nothing.
Blocking and allowing are tested as a pair. One item asks Dana for the engineering budget and must be refused; another asks Priya for the same figure and must be answered. A system that simply over-blocked would pass the first and fail the second. Of the 44 items, 9 are denials and 3 are confidential-but-entitled reads.
Caveats. The naive baseline was measured on six documents; enforcement and retrieval quality were verified on seventy. Version demotion is not exclusion — a retired document still reaches context for the public persona, whose nine visible documents nearly fill a TOP_K=8 window; the answers cite the active version regardless.
One known failure remains, recorded in evals/baseline.yaml: the model attributed correct content from one document to another. Neither deterministic check catches misattribution — the tag is real and that document was retrieved — so it is tagged for the LLM judge in Phase 7 rather than worked around.
Full runs: docs/baseline-phase2.json (naive) and docs/phase4-enforced.json (enforced).
docs/FINDINGS.md has the longer write-up: what each measurement turned out to mean, the three checks a green gate has to survive, and what is still open.
| Phase | Learning objective | Ships |
|---|---|---|
| 1 | LangChain primitives | Naive RAG with citations over a synthetic corpus (done) |
| 2 | Evaluation discipline | Golden set scorer + GitHub Actions ratchet gate (done) |
| 3 | Retrieval quality | Hybrid (pgvector + full-text) with RRF, version-aware ranking, comparison harness (done) |
| 4 | Permission-aware retrieval | Owned schema, ACL propagation, Postgres row-level security, RLS suite (done) |
| 5 | Governed model access | Model registry, data-classification routing policy, gateway choke point, cache, cost attribution (done) |
| 6 | Evidence and observability | Hash-chained append-only evidence ledger, correlation ids, audit and tamper-detection tooling (done) |
| 7 | Agentic retrieval and red team | 8-class prompt-injection suite with context fencing measured, LLM attribution judge, demo console (partial — LangGraph rewrite/grade loop not built) |
| Why | |
|---|---|
| Docker | Runs Postgres 16 with pgvector. Docker Desktop on macOS/Windows, Engine on Linux. |
| Python 3.11+ | 3.12 is what CI uses. |
| Ollama | Embeddings and self-hosted inference. Needs ~6GB free for llama3.1:8b. |
| AWS account | Optional. Only for the cloud-routing half of the demo; everything else runs without it. |
git clone https://github.com/Lanier-Developments/ControlPlane.git
cd ControlPlane
cp .env.example .env # defaults work if Ollama is on this machine
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e .
make up # Postgres + pgvector
make pull-models # nomic-embed-text + llama3.1:8b (a few GB)
make migrate # owned schema, RLS policies, rag_app role
make ingest # 70 documents -> ~148 chunks (a minute or two)Run these in order. Each answers a different question, and if one fails the later ones aren't meaningful.
make policy-test # registry and routing rules — no database, no model, seconds
make rls-test # the database actually enforces entitlements
make eval # retrieval tier over 44 golden items
make eval-full # adds answer-quality checks (calls the model; several minutes)make eval should report 0/44 context leaks and 0 missing sources. make eval-full carries two known failures documented in evals/baseline.yaml — that's expected, not a broken install.
make api # console at http://localhost:8000
make ask Q="How many PTO days do I get?" USER=sam # or the CLI
make verify # recompute the evidence chain
make usage # spend and policy decisions, 24hPersonas are sam, dana, priya, riley and marcus — each with different entitlements, listed in evals/personas.yaml.
The demo is more interesting with a second provider, because that's what makes the routing policy visible. Any Bedrock-accessible Claude model works.
- Create an IAM user with
bedrock:InvokeModel, thenaws configure. - First-time Anthropic use on Bedrock requires a use-case form — open the model in the Bedrock console's model catalog and submit it.
- Confirm which models your account can actually call:
aws bedrock-runtime converse --region us-east-1 \
--model-id us.anthropic.claude-sonnet-4-6 \
--messages '[{"role":"user","content":[{"text":"hi"}]}]' \
--query 'output.message.content[0].text' --output text- Put the working id in
policy/models.yaml. That file is the only place it goes — no code change.
DEFAULT_MODEL=bedrock/claude-sonnet-4-6 make ask \
Q="What is the salary band for a Senior Engineer II?" USER=danaThat question retrieves confidential content, so the cloud model is denied by registry-ceiling and self-hosted inference answers instead. Both decisions appear in the response and in the ledger.
Bind for 0.0.0.0:5432 failed: port is already allocated — something else holds Postgres's port: a local install, or another checkout of this repo (Compose namespaces containers by directory, so a second clone tries to start its own). Stop the other one, or change the host port in docker-compose.yml and update DATABASE_URL and APP_DATABASE_URL to match.
ModuleNotFoundError: No module named 'provenance' — the editable install points at wherever it was installed from. Re-run pip install -e . from the repo root.
Ingest hangs or times out — OLLAMA_BASE_URL is wrong. Ollama is the only network call in that path.
model "nomic-embed-text" not found — run make pull-models, or pull it on whichever host OLLAMA_BASE_URL names.
Retrieval role 'rag' bypasses RLS — APP_DATABASE_URL is pointing at the owner role. It must be rag_app. The app refuses to start rather than serve unenforced.
Schema changes don't take — db/init.sql only runs on a fresh volume. Use make reset-db && make migrate && make ingest.
relation "documents" does not exist — make migrate hasn't run.
Demo mode adds the guardrails a publicly reachable instance needs: a model allowlist so an open URL can't spend money on a cloud provider, canned questions only, and a per-client rate limit.
# .env
DEMO_MODE=true
DEMO_ALLOWED_MODELS=ollama/llama3.1:8bA Cloudflare Tunnel exposes it without opening a port:
cloudflared tunnel create controlplane
cloudflared tunnel route dns controlplane controlplane.example.com
cloudflared tunnel run controlplaneKeeping self-hosted inference on the host rather than renting a GPU isn't incidental — the whole point of the demo is that confidential content stays on hardware you control.
make eval # retrieval tier: deterministic, no chat model
make eval-full # adds answer checks: facts, forbidden content, citations, grounding, refusals
make baseline # record current failures into evals/baseline.yamlLeaks are measured at retrieval, not in the answer. Once an unauthorized chunk reaches the model's context, it has already been exposed: to the model, to traces, and to any log that captures the prompt.
The scorer reads entitlements from the corpus files, not from retrieved metadata. Asking a retrieved chunk to report its own ACL would let a leak vouch for itself. A retrieved source absent from the corpus counts as a leak rather than defaulting to permitted.
Two checks catch different lies. citation_unsupported fires when a source tag names a document that was never retrieved. ungrounded_value fires when a currency or percentage figure appears nowhere in the context — the more dangerous case, where a real tag for a real retrieved document is attached to an invented number. Checking only the tag misses it entirely. Both are deliberately narrow and deterministic; claim-level attribution needs a judge.
Items assert a source only where the fact is unique within the persona's visible set. A fact stated in three documents fails citation_missing at random depending on which the model picks. Several items rely on permission-scoped uniqueness: the same fact exists in an internal document, but that persona cannot see it.
The gate is a ratchet. evals/baseline.yaml records each known failure with a reason and the phase that fixes it.
- A new failure is a regression, and the build goes red.
- A known failure that starts passing also turns the build red until the baseline is tightened.
The list of accepted failures can only shrink. A permanently red gate trains people to ignore it; a ratchet keeps it honest while being truthful about where the system is today. Catching a fix matters as much as catching a break — it is what stops an accepted-failure list from quietly becoming permanent.
In CI the retrieval tier blocks merges. The full tier runs on a small local model (llama3.2:3b) and is advisory, because a 3B model's misses are noise, not signal. Point it at Bedrock through GitHub OIDC when it should block.
Refusal detection is a phrase heuristic. Replace it with an LLM judge (faithfulness, answer relevance) when the corpus grows.
make compare # vector vs hybrid on the golden set, retrieval tier onlyHybrid, fused with reciprocal rank fusion. Vector search misses exact tokens — a dollar amount, a command name, a doc ID. Keyword search misses paraphrase. Both run, and RRF fuses the rankings: score = sum(1 / (k + rank)). RRF needs no score normalization between cosine distance and ts_rank, two scales that have no business being compared directly.
Superseded versions are demoted, not filtered. "What did the old PTO policy say?" is a legitimate question, so dropping retired documents would be wrong. The penalty applies only when an active version of the same doc_id is also in the candidate set, so a retired document that is the only version available is never pushed below unrelated noise. The context block also labels it SUPERSEDED and the prompt tells the model to answer from the active version.
Measured effect: superseded documents in context went from 10/10 to 0/10 with no retrieval misses introduced. Hybrid also fixed the one genuine quality failure in the baseline — a salary figure living in a table cell that vector-only retrieval surfaced but the model failed to state or cite.
No retrieval change ships without a comparison. make compare runs both modes over the golden set and prints leaks, retrieval misses, and superseded-in-context counts, plus which items changed. Retrieval tier only, so it is deterministic — no model in the loop.
Tunables in .env: RETRIEVAL_MODE, FETCH_K, RRF_K, VECTOR_WEIGHT, KEYWORD_WEIGHT, PREFER_ACTIVE, SUPERSEDED_PENALTY.
Not yet done in Phase 3: a cross-encoder reranker (deliberately deferred — it adds a heavy dependency for gains the fusion may already cover; measure first) and incremental sync driven by source-system change feeds.
make reset-db && make migrate && make ingest # fresh volume, owned schema
make rls-test # prove the database enforces
make eval # leak count should be 0Enforcement is in the database, not the application. Retrieval runs as rag_app, a NOBYPASSRLS, SELECT-only role. Row-level security policies on chunks and documents grant visibility only through doc_acl. Chunks the caller may not see are not filtered out of the results — they are never in the result set, so there is no post-filter to forget to apply, and a bug in the Python layer cannot widen what a query returns.
Entitlements are transaction-scoped. Each request opens a transaction and issues SET LOCAL app.groups. The scope dies with the transaction, so a pooled connection can never carry one caller's entitlements into the next caller's query.
Fail closed by construction. current_setting('app.groups', true) returns NULL when unset, which matches no rows. An unauthenticated caller sees nothing because of how the policy is written, not because of a Python check that could be skipped. An unknown principal raises rather than defaulting to a group.
The gate refuses to run unless enforcement is on. assert_rls_enforced() checks at startup and before every eval run that the retrieval role does not bypass RLS and sees nothing with no groups set. It guards the failure that would otherwise be silent — someone pointing APP_DATABASE_URL at the owner role, which is exempt from every policy. A green eval gate has to mean enforcement was actually applied.
make rls-test bypasses the application entirely and queries as the app role directly: per-persona visible counts for all five personas, specific forbidden documents (Sam must not reach HR-007, Riley must not reach ENG-012), and that entitlements do not survive the transaction that set them. Expected counts are computed from the corpus files rather than the database, so the test cannot agree with a bug in ingest by sharing its source of truth.
Refusals do not disclose. When nothing retrievable matches, the answer is that no available information covers it — not "you lack permission to see that," which confirms the document exists.
ACLs propagate at ingest. Grants are rewritten on every ingest rather than added to, so a revoked group actually disappears. Chunks above the current count are deleted, so a shortened document leaves no orphans. A document with an empty acl is rejected: there is no implicit default.
Known gaps. Group membership comes from evals/personas.yaml, a stand-in for an identity provider; Phase 5 replaces the source without touching retrieval. Deletes in the source system are not yet detected — a document removed upstream stays indexed until the next full ingest.
make migrate # adds model_calls and response_cache
make policy-test # pure policy evaluation: no database, no model, no network
make usage # spend and policy decisions from the call logThe registry is data. Adding, repricing, deprecating or retiring a model is an edit to policy/models.yaml. No enum, no code change, no release.
This is the direct lesson from a prior control-plane build where model selection ran off an enum in application code. Every new provider meant a pull request, a release and a rollback risk, and the list of what was actually approved lived in code, in a wiki, and in someone's head — three sources that disagreed. The registry carries a retired entry on purpose: a retired model must be refused, not quietly absent.
Policy is data too, and it fails closed. Rules in policy/routing.yaml are evaluated in order, first match decides, and the last rule is a deny — so a policy file that fails to anticipate a case refuses rather than falling through to allow. The shape is deliberately OPA-compatible (a decision, a reason and a rule id on every answer), so moving the rules to Rego is a transport change, not a redesign.
The registry ceiling outranks the rule file. A model's max_classification is checked before any rule runs. A permissive rule cannot grant a model access to data more sensitive than its own registry entry allows — the two have to agree, and the stricter one wins.
Classification comes from the retrieved context, and the most sensitive chunk governs. One confidential chunk constrains the whole call, because a prompt is indivisible once it is sent. In practice that means a question Dana asks about salary bands cannot be answered by an external provider, while the same question shape about public content can.
One choke point. gateway.complete() is the only code in the system that constructs a model client. chain.llm() now raises if anything tries to build one directly, because a policy layer that one direct call can bypass is decoration.
Refusals are logged before anything else runs. A denial that leaves no trace is indistinguishable from a policy that was never evaluated. model_calls records the refusal, the rule that produced it, and the fallback that actually answered.
Cache and cost. Responses are cached on a hash of model plus full prompt, with a TTL. Cost is attributed per call from the registry's rates. Token counts are estimates (~4 characters per token) and labelled as such — exact accounting needs provider usage fields, which Phase 6 records. An estimate everyone knows is an estimate beats a precise-looking number nobody checked.
make usage reports which rules are firing and how often calls are refused, not just the dollar total. Which rules fire is the governance question; the dollars are a side effect.
Two rules in the file never fire, and that is documented rather than hidden. With the current registry, the ceiling catches a cloud model plus sensitive data before the rule file is consulted, so confidential-stays-local and restricted-stays-local are unreachable. They are real defense in depth — they catch a future registry edit that mis-registers a cloud model as restricted-capable — but an untested rule is false assurance, so make policy-test exercises both against a synthetic registry entry.
The eval gate disables the cache. A cached run would report 44 passes without calling the model once. The gate measures the model; the cache is for serving.
Measured on the full golden set: 44 calls, 0 denied, all routed to self-hosted inference, $0.00 — the corpus contains confidential and restricted documents, so every call that touches them is constrained to local by policy rather than by configuration.
make migrate # adds the evidence table and its triggers
make ask Q="..." USER=sam # every question writes an entry
make verify # recompute the whole hash chain
make trace ID=sam # entries by principal, correlation id, or entry id
make append-only # confirm UPDATE and DELETE are refused
make tamper-demo # prove the chain detects a silent editWhat an answer has to prove, after the fact. Who asked, what they were entitled to, which chunks were retrieved, which model ran, which rule permitted it, and that the record has not been altered since. One entry per question, answered or refused.
Digests, not answers. The ledger stores a SHA-256 of the answer and chunk ids, never the answer text or the chunk text. Storing answers would recreate — in an append-only table that cannot be deleted from — a copy of exactly the confidential content the permission layer works to contain. The digest proves what was said to anyone who still has the text and discloses nothing to anyone who does not.
Refusals are entries. A question that returned nothing, and a question refused by routing policy, both write to the ledger. A ledger holding only successes proves nothing about enforcement: the refusals are how you show the controls fired rather than that they were merely configured.
Append-only in the database. Triggers refuse UPDATE and DELETE outright. Rewriting history requires ALTER TABLE ... DISABLE TRIGGER — deliberately awkward, so an accident cannot do it and a deliberate act leaves a schema change behind.
Tamper-evident, not tamper-proof. Each entry hashes its own content plus the previous entry's hash, so altering or removing a past row invalidates every hash after it. verify recomputes each digest from stored content rather than trusting the stored hash, which catches an edit even if the editor also updated that row's hash — the break just moves to the next link. Anyone who can rewrite the entire table in order can still forge a consistent chain. Detecting quiet edits is the goal; defeating an operator with database ownership is not, and claiming otherwise would be dishonest.
make tamper-demo edits a row behind the triggers, shows verification fail, then restores it. A hash chain nobody has watched break is a claim, not a control.
Correlation across layers. One correlation id links the ledger entry to the model call log, so a single answer's retrieval scope and its model routing decision can be reconstructed together. make trace ID=<principal> prints both.
Appends are serialized with a table lock. Two concurrent writers reading the same tail hash would produce two entries claiming the same predecessor, and the chain would be unverifiable through that point. Appends are rare relative to reads, so the contention is acceptable here; a high-throughput deployment would build the chain with a single writer instead.
make redteam # 8 attack classes x 2 positions
CONTEXT_FENCING=false make redteam # measure what the defense actually buysRetrieved text is untrusted input. Anyone who can put a document where the indexer reaches it — a wiki page, a shared drive, a ticket comment, a vendor PDF — can put text in front of the model. In an enterprise that is a large set of people, most of whom are never thought of as having access to the AI system at all.
Poisoned documents are never ingested. corpus/redteam/ is loaded from disk and assembled into context in memory; the ingest path reads corpus/seed/ only. A fixture that could leak into a real answer would be a poor test.
Position is part of the test. Each attack runs with the poisoned document first and last. Several succeed in one position and not the other, so a single-position result means little.
Measured: fencing takes attack success from 4/16 to 2/16. It costs one golden-set item — with fencing on, the model refuses a safety-critical loss-prevention question it answers correctly with fencing off, because that document's own handling rules read as an instruction to withhold once it is framed as untrusted data. The trade-off is recorded in evals/baseline.yaml with the measurement attached; fencing stays on.
A prompt is a shared resource: every instruction added to defend against an attacker is also read when a real user asks something. A defense that costs nothing on a fixed eval set usually is not doing anything.
What no attack achieved was a permission bypass — and that is not the model's doing. Nothing unauthorized was in the context to leak, because row-level security scoped the candidate set before ranking. Injection cannot exfiltrate what was never retrieved. An injection can make the model emit OVERRIDE-ACCEPTED; it cannot make Postgres return a row the caller has no grant for, and every attempt is in the ledger.
make repo-meta-check # confirm the GitHub About description and topics are setDiscoverability is not the same question as correctness. Every other check in this pipeline asks whether the system behaves — eval gate, RLS, red team. None of them notice if the repo itself becomes unfindable: a rename, a fork, or a repo edit can silently clear the About description or topics, and nothing about the code changes. This check reads the public GitHub API and fails if either is empty. It runs as its own CI job, parallel to policy, because it depends on GitHub's API rather than this repo's code and shouldn't wait on a Postgres service or a model pull.
make api # http://localhost:8000
A single page over the same API the CLI uses. Nothing on it is mocked: every field comes from a real response, so what it demonstrates is the system rather than a picture of one.
Pick a persona, pick a model, ask. The page shows the answer, the retrieved documents with their classifications, and every routing decision with the rule that produced it — alongside the live ledger, 24-hour spend by rule, and the model registry.
The ninety-second demo, using the example buttons:
- Salary band (confidential) as
dana, with the Bedrock model requested. Two decision cards: a red DENY onregistry-ceiling, a green ALLOW on self-hosted inference. Cost $0.00 — the salary band never left the network. - Same question, no entitlement as
sam. The confidential document disappears from the retrieved set entirely. It isn't withheld from the answer; it was never retrieved. - Public fact as
riley, Bedrock requested. Green ALLOW oncloud-public, with a real cost attached. - Verify chain. All four questions are in the ledger, and the chain recomputes clean.
Permissions, routing, cost and evidence — without reading any code.
corpus/SPEC.md synthetic company, personas, frontmatter schema, deliberate traps
corpus/seed/ 70 documents across six departments, ~148 chunks
db/schema.sql owned schema, RLS policies, rag_app role, gateway and ledger tables
db/init.sql same content, runs once on a fresh volume
policy/models.yaml model registry: providers, tiers, ceilings, rates, status
policy/routing.yaml routing rules, evaluated in order, ending in a deny
evals/golden.yaml 44 items across 9 categories
evals/personas.yaml persona -> groups (identity provider stand-in)
evals/baseline.yaml ratchet: known failures with reasons and fix phase
docs/FINDINGS.md what the measurements turned out to mean
docs/console.png console screenshot
docs/*.json preserved eval runs: naive baseline and enforced
.github/workflows/ policy job, repo-meta job, then eval-gate (retrieval blocks, full advisory)
src/provenance/ config, db, identity, store, ingest, retrieval, chain, gateway,
registry, evidence, audit, evals, compare, usage,
rls_test, policy_test, repo_meta_check, api