LangSmith evaluation of LLM outputs

How I use LangSmith as a QA engineer: a golden dataset, traced runs, custom code evaluators, and an experiment compare that fails the release if a new model version regresses.

This repo does not call a paid LLM. Two recorded summarizer versions (v1, v2) replay labelled ticket summaries so the experiment is deterministic. The LangSmith pieces are the dataset shape, @traceable spans, evaluate(), and the evaluators.

Companion metrics: sentence-transformer-eval (meaning) and bleu-eval (word overlap). Same support-ticket domain.

golden dataset ──► traced summarizer (v1 / v2)
                          │
                          ▼
              LangSmith evaluate()
                          │
         ┌────────────────┼────────────────┐
         ▼                ▼                ▼
    structure        coverage        faithfulness
     (empty?)      (must-include)   (invented claims)
                          │
                          ▼
              experiment compare → release gate

Why LangSmith

Sentence transformers and BLEU score a single output. LangSmith is the harness around that: store the cases, trace every run, attach several evaluators, and diff two versions.

The trap this repo is built to show: v1 and v2 both pass 40%. A pass-rate dashboard would call them equal. The experiment table does not:

Example	v1	v2	Delta
android-login	PASS	FAIL	regression — invented refund
ios-payments	PASS	PASS	unchanged
dashboard-slow	FAIL	PASS	improvement — coverage restored
two-factor	FAIL	FAIL	unchanged
empty-summary	FAIL	FAIL	unchanged

run_eval.py exits 1 when there is a regression, even if the headline pass rate did not drop. That is the release gate.

Evaluators

LangSmith SDK v0.2 signatures: inputs, outputs, reference_outputs. Each returns {key, score, comment}.

Evaluator	Score	Catches
structure	1 if the summary is non-empty	Silent empty generations
coverage	1 if every must_include term appears	Dropped facts (dashboard, Android 14)
faithfulness	1 unless a forbidden claim is invented	Hallucinated refund / charged

Pass = all three scores are 1. Coverage and faithfulness are the pair that BLEU and embeddings each miss on their own: embeddings let the refund through, BLEU punishes a valid paraphrase. Here the refund is an explicit faithfulness fail, and the paraphrase (android-login v1) still passes.

Run locally (no API key)

python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt

python run_eval.py
pytest tests/ -q

run_eval.py prints the v1 vs v2 table and writes report.json.

Run on LangSmith

cp .env.example .env   # set LANGSMITH_API_KEY
pip install -r requirements.txt
python run_langsmith.py

run_langsmith.py calls langsmith.evaluate() twice (ticket-summarizer-v1, ticket-summarizer-v2) with the same evaluators. @traceable wraps the summarizer so each ticket is a span. If the key is missing, it still runs with upload_results=False.

Then open smith.langchain.com and compare the two experiments.

Layout

data/dataset.json      # golden cases: inputs, reference labels, v1/v2 outputs
evalkit/
  app.py               # recorded summarizer versions
  evaluators.py        # structure, coverage, faithfulness
  experiment.py        # local evaluate() + version compare
run_eval.py            # no key — release gate
run_langsmith.py       # SDK evaluate() + traces

What this is for

A release check on an LLM feature: pin a dataset, score every example with code evaluators, and fail CI when the new version invents a fact — even if the overall pass rate looks unchanged.