LLM Security: Thesis Code

Code for my bachelor's thesis on LLM security. Covers three attacks and two defenses.

Experiments

	Type	Model	Dataset	Result
MIA (RMIA)	Attack	GPT-2	AG News	AUC = 0.834
PII Leakage	Attack	GPT-2-small	ECHR	Entities extracted via NER
Adversarial GCG	Attack	OpenLLaMA-3B	AdvBench	100% jailbreak rate
Canary Insertion	Defense	LLaMA-3 (Ollama)	N/A	Blocked all injections
SHAP XAI	Defense	DistilBERT SST-2	N/A	Token-level attributions

Setup

pip install -r requirements.txt

For the canary defense, Ollama is also needed:

ollama pull llama3 && ollama serve

Running

python attacks/membership_inference/mia.py
python attacks/pii_leakage/pii_attack.py
python attacks/adversarial/adversarial_attack.py
python defenses/canary_insertion/canary.py
python defenses/xai/shap_explainer.py

Hardware

• GPU: NVIDIA L20 48GB
• Python 3.10, CUDA 11.8, Ubuntu 22.04

References

• Ye et al., "Enhanced Membership Inference Attacks against Machine Learning Models", CCS 2022
• Lukas et al., "Analyzing Leakage of Personally Identifiable Information in Language Models", IEEE S&P 2023
• Zou et al., "Universal and Transferable Adversarial Attacks on Aligned Language Models", arXiv 2023
• Lundberg & Lee, "A Unified Approach to Interpreting Model Predictions", NeurIPS 2017