The Truthfulness Spectrum Hypothesis

Zhuofan Josh Ying, Shauli Ravfogel, Nikolaus Kriegeskorte, Peter Hase

This repo contains code for the paper: The Truthfulness Spectrum Hypothesis

[Paper link]

---

Setup

Create and activate conda environment:

conda create -n truth_spec python=3.12
conda activate truth_spec

Install the dependencies with:

pip install -r requirements.txt

Usage

Datasets

To create the sycophancy dataset for a new model:

python scripts/sycophancy_dataset.py all --model llama-70b-3.3

Extract Activations

For bigger models:

python extract_feats.py --model llama-70b-3.3 --layers sparse

For smaller models:

python extract_feats.py --model llama-8b --layers all

Train & Test Probes

cd scripts
./train_test_probes_main.sh

This shell script calls train_test_probes.py to train and test probes.

Analysis

See the following scripts and notebooks for various analysis:

Concept erasure experiments:

• Stratified INLP: scripts/stratified_inlp.py
• LEACE: scripts/leace.py

Probe geometry analysis (Mahalanobis cosine similarity):

• scripts/geometric_analysis.ipynb

Causal experiments:

• scripts/causal.py

Acknowledgement

This codebase is based on this repo.

Citation

@misc{ying2026truth,
      title={The Truthfulness Spectrum Hypothesis}, 
      author={Zhuofan Ying and Shauli Ravfogel and Nikolaus Kriegeskorte and Peter Hase},
      year={2026},
      eprint={2602.20273},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2602.20273}, 
}