eval: per-client pipeline validation workflow (onboarding step) #127

Why

Pipeline tuning is not a one-time activity — this is a multi-client product, and
each client brings their own contract corpus, their own ERPNext, their own
deployment. For every onboarding we need to arrive at knob values that work for
that client's data.

But we can't just re-run #113 per client:

• Hand-authoring ~150 grounded questions (the eval: build the reference tuning set (~150 Q, dev/test split) + candidate generator #112 approach) does not scale to N
  clients.
• We must not run the sweep against a client's live contract collection — the
  chunk_size / embedding sweeps rebuild the collection and would disrupt their
  users.
• We often can't hold a client's contract data on our own infra (sensitive).

The model: tune once, validate per client

Tune the pipeline once (#113) on the reference corpus (#112) → produces the
shipped defaults in config.py / pipeline/constants.py and the baseline
thresholds. Most knobs are corpus-independent, so this is the expensive
statistical work done a single time.

Validate per client (this issue) → a small, mostly-automated check that the
shipped defaults hold for the client's corpus, with a targeted re-tune only when
a slice is genuinely weak.

Knob taxonomy

Scope

1. scripts/generate_eval_set.py — corpus → candidate {question, ground_truth_answer, ground_truth_contexts, case_class, split} triples, in the same schema as evaluation/test_dataset.json. LLM-generated candidates grounded in real docname:chunk_index keys pulled from the client's Qdrant collection; emits a review file for a human to spot-check a sample before use. Same tool is used to draft the reference set in eval: build the reference tuning set (~150 Q, dev/test split) + candidate generator #112.
2. Per-client validation run — ~20–40 questions on the client's corpus, ingested into a separate {QDRANT_COLLECTION}_eval collection (populated from their ERPNext during the pre-cutover window), evaluate.py --split against it, compared to the reference baseline thresholds from eval: tune the pipeline on the reference corpus (docs/PIPELINE_TUNING.md pass) #113.
3. Targeted re-tune — if a case_class slice is below threshold, sweep only the implicated knob (usually RETRIEVAL_TOP_K or the system prompt), not the full eval: tune the pipeline on the reference corpus (docs/PIPELINE_TUNING.md pass) #113 pass. Record the per-client override in their .env.
4. Onboarding artifact — the per-client eval set + validation results + any knob overrides go into the client's deployment record; add the step to docs/CLIENT_DEPLOYMENT_RUNBOOK.md.

Where it runs

A separate eval collection on the client's infra during the pre-cutover window —
never the live contract collection. Our infra only if contractually
allowed to hold their contract data.

Depends on

• eval: build the reference tuning set (~150 Q, dev/test split) + candidate generator #112 — reference set format + the candidate generator (shared tooling)
• eval: tune the pipeline on the reference corpus (docs/PIPELINE_TUNING.md pass) #113 — the reference defaults and baseline thresholds this validates against

Related

• docs/PIPELINE_TUNING.md, docs/CLIENT_DEPLOYMENT_RUNBOOK.md
• feat(evaluation): act on RAGAS results — threshold gating + visible notification #49 (CI regression gate — same per-case_class numbers)
• eval: run the golden-dataset RAGAS eval as a Langfuse Dataset + Experiment #109 (Langfuse Datasets/Experiments — could host per-client sets)

Area: Evaluation