TruthLens AI

TruthLens AI is a small Generative AI project I built to explore a question I kept coming back to: how can we make an LLM answer useful without asking the user to simply trust it?

Instead of stopping at retrieval + generation, TruthLens checks the answer at the claim level. It retrieves evidence, generates an answer from that evidence, verifies individual claims, checks whether those claims are actually relevant to the question, and tracks whether every part of a multi-part question was answered.

The current demo uses German public health-insurance documents as its knowledge base, but the pipeline itself is designed around general retrieval and verification steps rather than hard-coded insurance rules.

What it does

A user can ask a normal question, including a multi-part one. TruthLens then:

1. breaks the question into separate information needs when needed,
2. retrieves relevant passages from the local knowledge base,
3. reranks the retrieved evidence,
4. generates an evidence-grounded answer,
5. verifies each factual claim against its source evidence,
6. checks whether each verified claim is relevant to what the user actually asked,
7. measures whether all parts of the question were answered,
8. returns a partial verified answer instead of filling unsupported gaps.

Claims are classified as:

• SUPPORTED — the evidence supports the claim
• CONTRADICTED — the evidence conflicts with the claim
• INSUFFICIENT EVIDENCE — the available evidence is not strong enough to establish the claim

Demo screenshots

Fully verified answer

For a question that is fully supported by the knowledge base, TruthLens returns a verified answer and marks the information need as answered.

Partial verified answer

For a multi-part question where only part of the requested information is supported, TruthLens returns the supported answer and explicitly shows the unsupported part under Could Not Answer.

Example

A useful test case for the current demo is:

What is the statutory health insurance contribution rate, is it 50 percent, and what is the population of Berlin?

TruthLens can verify the contribution-rate part from the supplied health-insurance documents and reject the proposed 50% value. It does not invent an answer for Berlin's population when that information is not supported by the retrieved source material.

That separation between what can be verified and what cannot is the main idea behind the project.

Pipeline

User Question
     |
     v
Question Analysis / Decomposition
     |
     v
Evidence Retrieval
     |
     v
Sentence Reranking + Grounding Threshold
     |
     v
LLM Answer + Claim Generation
     |
     v
Claim-Level NLI Verification
     |
     v
Relevance Check
     |
     v
Information-Need Coverage
     |
     v
Verified / Partial Final Answer

Main components

Question analysis

Multi-part questions are decomposed into standalone information needs so that the system can tell the difference between a fully answered question and a partially answered one.

Retrieval

The current retriever uses sentence embeddings with a FAISS index, followed by sentence-level reranking. Evidence below the grounding threshold is filtered before it reaches the generation stage.

Answer generation

The generation layer runs locally through Ollama using llama3.1:8b. The prompt restricts the model to the retrieved evidence and requires structured claims with source document, page, and evidence text.

Claim verification

Each generated claim is checked against its evidence using the NLI model:

cross-encoder/nli-deberta-v3-base

The verifier uses entailment, contradiction, and neutral probabilities to produce the final claim verdict.

Relevance and completeness

A claim can be factually supported and still be irrelevant to the user's question. TruthLens checks both.

It also evaluates each decomposed information need separately. If one part cannot be supported, the interface shows that part under Could Not Answer rather than hallucinating a response.

Tech stack

• Python
• Streamlit
• Ollama
• Llama 3.1 8B
• Sentence Transformers
• all-MiniLM-L6-v2
• cross-encoder/nli-deberta-v3-base
• FAISS
• PyTorch / Transformers
• Pydantic

Project structure

truthlens-ai/
|
|-- app/
|   |-- core/           # main pipeline orchestration
|   |-- planning/       # question decomposition
|   |-- ingestion/      # loading, chunking, sentence splitting, embeddings
|   |-- retrieval/      # vector retrieval and reranking
|   |-- llm/            # Ollama generation and final synthesis
|   |-- verification/   # claim-level NLI verification
|   |-- evaluation/     # relevance / evaluation components
|   |-- schemas/        # structured response models
|   `-- benchmarks/
|
|-- benchmarks/         # benchmark questions / test cases
|-- data/
|   `-- knowledge_base/
|       `-- sources.json
|
|-- streamlit_app.py
|-- styles.css
|-- requirements.txt
`-- README.md

Knowledge base

The current prototype uses public material from the German Federal Ministry of Health (Bundesministerium für Gesundheit), including resources such as Healthy in Germany and Health for All.

Only the source manifest is committed to this repository. Raw PDFs, processed files, embeddings, and the generated FAISS index are intentionally excluded from Git because they can be rebuilt locally and would unnecessarily increase the repository size.

Running locally

1. Clone the repository

git clone https://github.com/N-JrP/truthlens-ai.git
cd truthlens-ai

2. Create and activate a virtual environment

Windows PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

3. Install dependencies

pip install -r requirements.txt

4. Install Ollama and pull the model

Install Ollama, then run:

ollama pull llama3.1:8b

Make sure Ollama is running before starting TruthLens.

5. Prepare the local knowledge base

Place the source PDFs in:

data/knowledge_base/raw/

Then run the ingestion steps:

python -m app.ingestion.run_chunking
python -m app.ingestion.embedding_generator
python -m app.ingestion.faiss_indexer

This creates the processed documents, embeddings, metadata, and local FAISS index used by the retriever.

6. Start the app

streamlit run streamlit_app.py

Why I built it this way

A normal RAG pipeline can retrieve good documents and still produce an answer that overstates, mixes up, or adds information. I wanted the project to make those intermediate decisions visible instead of hiding everything behind one final paragraph.

That is why TruthLens separates:

• retrieval quality,
• factual support,
• claim relevance,
• question coverage,
• and final answer synthesis.

The goal is not to claim that an LLM can become perfectly reliable. The goal is to make it much clearer which parts of an answer have evidence behind them and which parts do not.

Current limitations

This is a portfolio prototype, not a production fact-checking service.

• The current knowledge base is small and domain-specific.
• Model thresholds are currently fixed rather than calibrated on a large benchmark.
• The local setup requires rebuilding the FAISS index because generated knowledge-base artifacts are not committed.
• Claim verification quality still depends on the NLI model and the evidence retrieved upstream.
• The current UI focuses on one local user workflow rather than authentication, persistence, or multi-user deployment.

Next steps

Some areas I would like to explore next:

• broader benchmark coverage across different domains,
• retrieval and verifier threshold calibration,
• conflict handling when multiple sources disagree,
• automated evaluation reports,
• source-level provenance views,
• deployment-friendly model and index configuration.

---

Built as a hands-on project around RAG, grounded generation, NLI verification, relevance evaluation, and explainable AI workflows.