PLM-Caliper

PLM-Caliper is a model-agnostic framework for false discovery rate (FDR) control in protein search with protein language models (PLMs). It uses query-derived decoys together with score calibration to estimate false discoveries and identify score thresholds corresponding to user-specified FDR levels.

PLM-Caliper operates on the output scores of existing protein search methods and does not require retraining or fine-tuning the underlying search models.

This repository contains:

• the main PLM-Caliper implementation;
• scripts for generating protein-retrieval scores with different search methods;
• utilities for decoy generation and score processing;
• analysis code for reproducing the experiments reported in the paper.

---

Installation

Requirements

PLM-Caliper is implemented in Python. Additional software is required only for reproducing experiments involving specific protein search or structure-prediction pipelines.

Environment setup

Clone the repository:

git clone git@github.com:batmen-lab/PLMCaliper.git

Create and activate the environment:

conda create -n plmcaliper python=3.11 -y
conda activate plmcaliper

Install the required dependencies:

python -m pip install -r requirements.txt

Data

Download the required dataset (links below) and arrange the project as follows:

PLMCaliper/
├── data/
│   ├── astral.fa             # ASTRAL sequence database
│   ├── astral.tsv
│   ├── astral4f_query.fa     # 4-family query subset
│   ├── astral4f_query.tsv
│   ├── class_all.fa          # BAGEL4 classified bacteriocins
│   ├── class_all.tsv
│   ├── putative.fa           # BAGEL4 putative bacteriocins
│   └── putative.tsv
│
├── requirements/             # pip requirements per search env
│
├── libs/
│
├── results/
│
└── src/
    ├── analysis/
    ├── PLM_searching_cmds/
    ├── PLMCaliper/
    ├── utils/
    └── setup.sh

• Reference databases, library dependencies (PLM methods): — download from zenodo
• PDB structures: — download from pdbstyle-2.08

---

Quick Start

PLM-Caliper generates decoy queries and calibrates search scores; the target and decoy score tables are produced by an upstream retrieval method.

1. Make Decoy Queries

The input consists of query protein sequences, and the output is the corresponding extended-Markov decoy sequences.

python src/PLMCaliper/PLMCaliper.py make-decoy \
  --fasta data/astral.fa \
  --out data/astral_extended_mkv2.fa \
  --design extended_mkv \
  --markov-order 2

2. Run Protein Retrieval

Search the original queries and the decoy queries against the same target database using the same method. The modified libs used to retain complete retrieval hits are available at zenodo.

Then create the search environments (plmsearch, dhr, and tmvec) and load the paths:

bash src/setup.sh --create-envs
source src/PLM_searching_cmds/.env

Run retrieval for the original queries:

bash src/PLM_searching_cmds/run_search.sh \
  -m tmvec \
  -q data/astral.fa \
  -t data/astral.fa

Run retrieval for the decoy queries:

bash src/PLM_searching_cmds/run_search.sh \
  -m tmvec \
  -q data/astral_extended_mkv2.fa \
  -t data/astral.fa

Use -m plm, -m dhr, or -m blastp to run the other retrieval methods. Note that .fa files must be converted to .tsv using src/utils/fa2tsv.py before running DHR.

The parsed score tables are written as data/result_<method>_<query>_<target>.txt.

For custom score tables, provide a delimited file with a header and at least these columns:

qid	tid	score

3. Run Calibration

Important

For dhr and blastp, score preprocessing is required: DHR scores must be converted to a larger-is-better scale, and missing BLASTp hits must be filled in.

# DHR: flip distances into similarities
python src/utils/process_searching_score.py dhr-postprocess \
  --query astral \
  --db astral \
  --decoy extended_mkv2

# BLASTp: fill the query-target pairs BLASTp never reported
python src/utils/process_searching_score.py blastp-densify \
  --real data/result_blastp_astral_astral.txt \
  --decoy data/result_blastp_astral_extended_mkv2_astral.txt \
  --fasta data/astral.fa \
  --decoy-suffix _mkv \
  --out-real data/result_blastp_postprocessed_astral_astral.txt \
  --out-decoy data/result_blastp_postprocessed_astral_extended_mkv2_astral.txt

The calibration module requires the following inputs:

• Target scores: scores from the original query sequence.
• Decoy scores: scores from the decoy query sequence generated by make-decoy.
• Target FDR: the desired false discovery rate level, passed as --target-fdr.

mkdir -p results
python src/PLMCaliper/PLMCaliper.py calibrate \
  --real data/result_tmvec_astral_astral.txt \
  --decoy data/result_tmvec_astral_extended_mkv2_astral.txt \
  --out results/plmcaliper_cutoffs.tsv \
  --target-fdr 0.50 \
  -hits --hits-out results/plmcaliper_hits.tsv

Main output plmcaliper_cutoffs.tsv:

qid	rep_id	target_fdr	cutoff_score	est_fdp	n_selected

Optional hits output plmcaliper_hits.tsv:

qid	tid	rep_id	score	est_fdp

score is the retrieval score. est_fdp is PLM-Caliper's estimated false discovery proportion (FDP).

---

Reproducing the Paper Results

The experimental analysis code is organized under:

src/analysis/

Each subdirectory corresponds to an experiment reported in the paper. To save computation time, precomputed PLM-based search scores and calibration files are available for download from Hugging Face.

---

1. Decoy and Calibration Ablation

Directory:

src/analysis/ablation/

This experiment evaluates the effects of decoy design and score calibration on false discovery control. The analysis includes different decoy-generation strategies and compares FDR estimation before and after score calibration.

This experiment starts from search scores, so run the retrieval step above first if data/ does not already hold them. The commands below produce one method-decoy pair, TM-Vec searched against the shuf decoy design:

conda activate plmcaliper
source src/PLM_searching_cmds/.env

python src/PLMCaliper/PLMCaliper.py make-decoy \
  --fasta data/astral.fa \
  --out data/astral_shuf.fa \
  --design shuf

bash src/PLM_searching_cmds/run_search.sh \
  -m tmvec \
  -q data/astral.fa \
  -t data/astral.fa

bash src/PLM_searching_cmds/run_search.sh \
  -m tmvec \
  -q data/astral_shuf.fa \
  -t data/astral.fa

The ablation evaluates shuf, rev, mkv, and dplm decoys with plm, dhr, tmvec, and blastp. The pre-computed score tables are available from Hugging Face.

Run the experiment with:

bash src/analysis/ablation/run_ablation.sh -m tmvec -d shuf

---

2. BAGEL4 Bacteriocin Analysis

Directory:

src/analysis/bagel/

This experiment evaluates PLM-Caliper for FDR-guided annotation of putative bacteriocins using the BAGEL4 benchmark.

The analysis applies FDR-controlled protein retrieval to identify candidate bacteriocin homologs while controlling the expected proportion of false discoveries.

First, generate real and decoy similarity scores for the BAGEL dataset using each search method.

conda activate plmcaliper
source src/PLM_searching_cmds/.env

python src/PLMCaliper/PLMCaliper.py make-decoy \
  --fasta data/putative.fa \
  --out data/putative_extended_mkv2.fa \
  --design extended_mkv \
  --markov-order 2

bash src/PLM_searching_cmds/run_search.sh \
  -m tmvec \
  -q data/putative.fa \
  -t data/class_all.fa

bash src/PLM_searching_cmds/run_search.sh \
  -m tmvec \
  -q data/putative_extended_mkv2.fa \
  -t data/class_all.fa

The figures place every sequence on a shared t-SNE map, whose coordinates come from TM-Vec embeddings. Download the published t-SNE layout into data/bagel/:

python -m pip install -U "huggingface_hub[cli]"

mkdir -p data/bagel
hf download yifanyang993/PLMCaliper --repo-type dataset \
  --include "tsne/*" --local-dir data/bagel
mv data/bagel/tsne/*.npy data/bagel/ && rmdir data/bagel/tsne
rm -rf data/bagel/.cache

Run the experiment with:

bash src/analysis/bagel/run_bagel.sh -target_fdr 0.20 -m tmvec

---

3. FDR-Guided MSA Construction and Structure Prediction

Directory:

src/analysis/ColabFold/

This experiment evaluates whether FDR-guided sequence selection can reduce the number of retrieved sequences used for multiple-sequence alignment (MSA) construction while preserving downstream protein structure-prediction quality.

The pipeline includes data preparation, MSA construction, PLM-Caliper calibration, structure prediction, evaluation, and visualization.

The search database is UniRef50, available from the UniProt FTP site. The release used in the paper holds 38,794,121 sequences. Place it at data/uniref50/uniref50.fasta.

Install the dependencies for this experiment:

conda activate plmcaliper
source src/PLM_searching_cmds/.env
conda run -n tmvec python -m pip install --no-deps -e libs/tm-vec-master deepblast==1.0.2

Run the experiment with:

bash src/analysis/ColabFold/run_colabfold.sh -m dhr

---

4. Conformal Benchmark

Directory:

src/analysis/conformal_benchmark/

This experiment compares PLM-Caliper with the conformal risk control baseline for controlling false discoveries in protein retrieval.

Run the experiment with:

conda activate plmcaliper

bash src/analysis/conformal_benchmark/run_conformal.sh -m dhr -split_level fold