Just-In-Time Instruction Retrieval

A proof-of-concept demonstrating that selectively retrieving relevant instructions at inference time outperforms loading the full instruction library into every prompt — reducing token usage while maintaining or improving code quality.

Rather than prepending all N instructions to every request, this system uses a FAISS vector index to retrieve only the k most relevant instructions for each incoming prompt, keeping context tight and costs low as instruction libraries scale.

Results

Tested on the full HumanEval benchmark (164 problems) against three conditions:

Condition	pass@1
Baseline (no instructions)	77.4%
Static (all instructions)	81.1%
JIT k=5	86.6%
JIT k=10	84.8%

JIT retrieval with k=5 improves pass@1 by 9.2 percentage points over the baseline, while using ~80% fewer tokens than the static condition.

See poc_summary.md for the full write-up, methodology, and analysis.

Setup

Requirements: Python 3.8+

pip install -r requirements.txt

Set your Anthropic API key:

export ANTHROPIC_API_KEY=your_key_here

Usage

Run the full experiment (all three conditions on HumanEval):

python poc_experiment.py

This will generate .jsonl result files and produce comparison plots.

Files

File	Description
poc_experiment.py	Main experiment script
functions.py	Embedding and retrieval utilities
manual_eval.py	Manual evaluation helper
sandbox.ipynb	Exploratory analysis notebook
poc_summary.md	Full results write-up

License

Apache 2.0 — see LICENSE.