Skip to content

datasets.md

Latest commit

 

History

History
64 lines (47 loc) · 2.62 KB

datasets.md

File metadata and controls

64 lines (47 loc) · 2.62 KB
Raw

Datasets

RAGProof runs against a dataset of evaluation cases. You can hand-write one as JSONL or generate one from a corpus.

Case kinds

Kind Purpose Required fields
qa A question answerable from the corpus question, expected_source_ids
unanswerable A question the corpus cannot answer question
injection A question paired with a planted adversarial instruction question, adversarial_payload

qa cases carry an expected_answer used by completeness and the debugging exact-match metric. expected_source_ids are what retrieval metrics score against.

Generating a dataset

ragproof generate --corpus ./docs --out dataset.jsonl \
  --qa 40 --unanswerable 10 --injection 10 --seed 0

The corpus can be a folder or a single file. TXT and Markdown are read by the core install; PDF and DOCX need the ingest extra (pip install 'ragproof[ingest]'). Files over 5 MB and files that fail to parse are skipped with a reported reason, never silently.

Generation needs RAGPROOF_GEN_MODEL and its provider settings. It:

  1. Chunks the corpus deterministically for a given seed.
  2. Writes a question and answer for sampled chunks, then runs a second pass that confirms the question is answerable from that chunk. Failures are discarded and counted.
  3. Writes unanswerable questions on the corpus topic and verifies they cannot be answered from the retrieved context. Answerable candidates are discarded.
  4. Attaches an inert injection payload to sampled questions for robustness testing.

The output is a JSONL review file. Edit it freely before freezing.

Sampling and case ordering are deterministic for a fixed seed and corpus. The model's wording is not, so two runs may phrase questions differently even though they sample the same chunks.

Source ids for generated cases

Generated qa and injection cases use the source document path as expected_source_ids. If your pipeline returns chunk-level ids that do not match those paths, run with source_match: document and expose document_id in each retrieved chunk's metadata so retrieval metrics can line the two up.

Freezing

ragproof freeze dataset.jsonl

Freezing validates the cases (a qa case must have at least one expected_source_id, an injection case must have an adversarial_payload), computes a canonical sha256 over the cases, and writes dataset.frozen.json with the hash embedded. Loading a frozen dataset recomputes the hash and refuses the file if it was changed after freezing, so a run always evaluates the exact cases that were frozen.

Runs accept either a plain JSONL file or a frozen dataset.