Offline evaluator that scores generated text with sentence-level BLEU. The implementation is in this repo (evalkit/bleu.py) — modified n-gram precision, clipping, brevity penalty, geometric mean — so you can see why a score is high or low, not just the number.
Companion to sentence-transformer-eval. Same sample tickets, opposite metric. BLEU rewards overlapping words; sentence transformers reward overlapping meaning.
source (reference) ──► [structure 20%] ──► [BLEU 80%]
│
p1–p4 × brevity penalty
BLEU (Papineni et al., 2002) was built for machine translation: how many n-grams in the candidate also appear in a reference, clipped so repeating "the the the" cannot inflate the score, then penalised if the candidate is too short.
That is useful when the output is supposed to stay close to a source — extractive summaries, constrained generation, regression checks that a model did not drift off the golden wording.
It is the wrong tool when a good answer is a paraphrase. A rewritten ticket summary that a human would accept can score near 0 because the 4-grams changed.
python compare_ngrams.py on the bundled pairs:
| Pair | p1 | p2 | p3 | p4 | BP | BLEU |
|---|---|---|---|---|---|---|
| Ticket paraphrase | 0.50 | 0.11 | 0.00 | 0.00 | 0.82 | 0.16 |
| Near copy-paste | 0.83 | 0.82 | 0.80 | 0.78 | 1.00 | 0.82 |
| Too short (dropped facts) | 0.80 | 0.50 | 0.33 | 0.00 | 0.20 | 0.11 |
| Unrelated | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.10 |
- p1–p4 — modified precision for unigrams through 4-grams. Paraphrase still shares some words (p1=0.50) but almost no 3-grams or 4-grams.
- Raw BLEU is 0 as soon as any p_n is 0. That happens constantly on short LLM outputs. Smoothed BLEU (add-one on n-grams that exist) is what this repo uses for pass/fail.
- Brevity penalty (BP) < 1 when the output is shorter than the source. The truncated invoice ticket has decent unigram overlap (p1=0.80) and still fails because BP=0.20.
On the same paraphrase, sentence-transformer-eval scores cosine 0.86. That pair is the whole argument for using both metrics.
| Layer | Weight | What it catches |
|---|---|---|
| Structure | 20% | Empty or out-of-bounds output |
| BLEU | 80% | Weak n-gram overlap vs the source |
Pass: overall ≥ 0.50 and the output is non-empty.
Measured on data/samples.json (same texts as the sentence-transformer repo):
| Case | Result | BLEU | What it shows |
|---|---|---|---|
| Good paraphrase | FAIL | 0.17 | Human-ok rewrite; n-grams barely match |
| Near copy-paste | PASS | 0.82 | This is what BLEU is good at |
| Drops critical facts | FAIL | 0.11 | Brevity penalty on a short candidate |
| Invented refund | FAIL | 0.27 | Extra unmatched n-grams cut precision |
| Unrelated output | FAIL | 0.10 | No shared n-grams |
| Empty output | FAIL | 0.00 | Structure layer |
Python 3.9+. No model download, no third-party runtime deps.
python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt # pytest only
python run_eval.py
python compare_ngrams.py
pytest tests/ -qrun_eval.py prints per-item p1–p4, brevity penalty, raw vs smoothed BLEU, and writes report.json. Point it at your own data with --data path/to/items.json. Each item needs id, source_text, and generated_output.
evalkit/
bleu.py # tokenize, clip, BP, raw + smoothed sentence BLEU
structure.py # length check
evaluator.py # weights and pass/fail
data/samples.json # six labelled cases (shared with the ST repo)
run_eval.py
compare_ngrams.py
A lexical gate next to a semantic one. Use BLEU when you care that wording stayed close to a reference. Use sentence transformers when you care that meaning stayed close. The paraphrase sample is meant to fail here and pass there.