Code for ProbingRankLlama (https://dl.acm.org/doi/pdf/10.1145/3731120.3744603)

1. Install all requirements in requirements.txt.
2. Generate and save all desired Activations by running rankllama-activation.py.
3. Update Queries and Documents as desired in sequences.py.
4. Find context neurons by running context_neurons.py.

Fine-Grained Checkpoint Experiments for Ranking LLMs

This repository hosts the checkpoints for various fine-tuned LLMs used in probing and interpretability experiments on ranking tasks. T These models have been trained using the Tevatron ranking framework, as suggested in the RankLLaMA paper

---

Available Fine-Tuned Models

The following model checkpoints are publicly available on HuggingFace:

• RankLLaMA3

  • RankLLaMA3
  • RankLLaMA3 R2
  • RankLLaMA3 R8
  • RankLLaMA3 R8 LoRA MLP
  • RankLLaMA3 R32 LoRA MHA

• RankLLaMA2

  • RankLLaMA2-7B (via paper)
  • RankLLaMA2-13B (via paper)

• RankMistral

  • RankMistral
  • RankMistral R8
  • RankMistral R8 MLP Only

• RankPythia

  • RankPythia
  • RankPythia R8
  • RankPythia R8 MLP Only

---

Experiments

You can run all probing and interpretability experiments described in our paper using any of the above fine-tuned checkpoints. The experiments are designed to measure the alignment of IR feature representations with intermediate layers of these ranking LLMs.

🧪 Note:
The experiments presented in the paper were conducted on full LoRA fine-tuned Rank-8 and Rank-32 models of Pythia, LLaMA3, and LLaMA2 (7B and 13B). These configurations provide optimal probing fidelity for feature tracking and layerwise interpretability.

---

Citation

If you use these checkpoints or reproduce the experiments, please cite the appropriate HuggingFace models using the provided BibTeX entries in the bibtex.bib file.

---

📚 Citation

If you use this repository or build upon the experiments, please cite the following paper:

@article{chowdhury2024probing,
  title={Probing Ranking LLMs: Mechanistic Interpretability in Information Retrieval},
  author={Chowdhury, Tanya and Allan, James},
  journal={arXiv preprint arXiv:2410.18527},
  year={2024}
}