feat(annotation): wire completeness into export (columns + meta summary) by henrycgbaker · Pull Request #268 · bertelsmannstift/pragmata

henrycgbaker · 2026-07-01T16:38:14Z

Stack (6 PRs): #246 → #247 → #268 → #248 → #269 → #249

This PR (3/6): Split out of #247 (was 1001 lines / XL). Wires the panel-completeness calculator (#247) into the retrieval export path. Stacked on #247.

Scope

Schema

core/schemas/annotation_export.py:
- RetrievalExportRow += panel_complete: bool, n_annotated_chunks: int, n_submitted_chunks: int, n_discarded_chunks: int, n_records_seen: int. All defaulted; appended at the tail so existing column order is preserved.
- New KBucketStat (per-K-bucket panel counts) and CompletenessSummary (aggregates + by_k_bucket cross-tab + full per-K histogram) Pydantic models.
- AnnotationExportMeta += completeness_summary: CompletenessSummary | None and completeness_status: Literal["ok","failed","not_requested"].

Single-walk retrieval pipeline

core/annotation/export_fetcher.py: new walk_retrieval_records (one Argilla scroll per dataset, no status filter — sees discards-only records too so the integrity check has the full record set) + RetrievalRecordSnapshot dataclass + fetch_retrieval_from_records (status filter applied in Python at typed-row construction).
core/annotation/export_runner.py: when Task.RETRIEVAL in tasks, walks once and feeds both the typed-row projection and the completeness aggregator. compute_completeness failures degrade the sidecar (sets completeness_status="failed") without losing the already-fetched export.

API

annotation/__init__.py: lazy re-exports for CompletenessSummary, KBucketStat.

Tests

test_export_runner.py: write_export_csv stamps completeness columns by record_uuid; sidecar carries completeness_summary; completeness_status ok/failed/not_requested set explicitly.
test_annotation_export.py: RetrievalExportRow field-order locked.

Why

The retriever is threshold-based, so dropping partial panels biases the metrics toward small K (MNAR). Surfacing both per-row predicate columns AND a bucketed sidecar aggregate lets the eval pipeline either filter complete panels per-K or apply condensed-list scoring without re-querying the data layer.

The single-walk shape (one retrieval scroll feeds both fetch_task and compute_completeness) costs one extra in-memory pass over already-fetched records but spares the Argilla server a second full scroll per export — meaningful on the in-flight batches (~hundreds of panels).

The STRICT panel_complete policy question (which shape to ship — this PR assumes (A) from #247) is discussed on #247, since that's where the predicate is actually computed; this PR just surfaces it as a column.

Test plan

uv run python -m pytest tests/unit/core/annotation/test_export_runner.py tests/unit/core/schemas/test_annotation_export.py

ddimmery · 2026-07-03T19:00:02Z

+        try:
+            completeness_report = compute_completeness_from_records(retrieval_snapshots)
+            completeness_status = "ok"
+        except Exception:


The generic exception should be made more specific (e.g. KeyboardInterrupt gets captured by the base Exception class).

ddimmery · 2026-07-03T19:05:01Z

+    # lost or duplicated upstream). Surfaced per-row so consumers can spot
+    # corrupted panels in dataframe pipelines without re-joining the sidecar.


If compute_completeness_from_records raises an error, this will be indistinguishable from an successful complete with empty data. Would there be a good way to surface this? e.g. a Sentinel value indicating an error?

ddimmery · 2026-07-03T19:07:48Z

+        dataset = client.datasets(ds_name, workspace=workspace_name)
+        if dataset is None:
+            continue
+        for record in dataset.records(with_responses=True):


This collects all responses and then filters here, rather than like with grounding/generation which filter with query = rg.Query(filter=rg.Filter([("response.status", "in", statuses)])). Shouldn't matter, but might as well be consistent.

intentionally different: walk_retrieval_records needs to see unfiltered so the completeness pass sees chunk-records w/ no terminal response (needed fo integrity check and K counting)

ddimmery · 2026-07-03T19:08:54Z

+# for metrics purposes — it's an explicit decision, not a hole. Shared by
+# fetch_task's status-filter callers, completeness, and panel_status so the
+# three never drift on the metric definition.
+TERMINAL_STATUSES = frozenset({"submitted", "discarded"})


Is this used? Maybe in later diffs?

ddimmery · 2026-07-03T19:11:10Z

    calibration_enabled: dict[Task, bool],
    constraint_summary: dict[str, int],
+    completeness_summary: CompletenessSummary | None = None,
+    completeness_status: str = "not_requested",


I think you can drop the type-checker ignore on line 209 if you type this as Literal["ok","failed","not_requested"]

**Stack (6 PRs):** #246 → #247 → #268 → #248 → #269 → #249 **Chain goal:** Persist K (number of retrieved chunks per query) on retrieval records, surface it as panel-completeness columns + sidecar at export time, ship a live `annotation status` CLI on top, and add an opt-in `--tag-partial-panels` advisory write so annotators can filter partially-done panels in the Argilla UI. **This PR (1/6):** Foundation. Stamps `n_retrieved_chunks` on retrieval record metadata at import and surfaces it on `RetrievalAnnotation`. Ships nothing user-visible on its own — the export columns (#247/#268) and live status CLI (#248/#269) consume it. --- ## Scope ### Record building - `core/annotation/record_builder.py`: add `"n_retrieved_chunks": len(pair.chunks)` to every chunk-record's metadata dict. ### Task definition - `core/annotation/argilla_task_definitions.py`: declare `IntegerMetadataProperty("n_retrieved_chunks", min=1, visible_for_annotators=False)` on retrieval task settings. ### Schema - `core/schemas/annotation_export.py`: `RetrievalAnnotation.n_retrieved_chunks: int` field (mirrors the existing `chunk_rank: int`). ### Export readback - `core/annotation/export_fetcher.py`: `_build_row` retrieval branch reads `metadata.get("n_retrieved_chunks", 0)` (defensive default for records imported before this PR). ### Tests - New `test_n_retrieved_chunks_stamped_on_every_chunk_record` in `test_import.py` locks the persist contract. - Metadata-name list in `test_argilla_task_definitions.py` updated. - Fixture refresh across `test_annotation_export.py`, `test_export_runner.py`, `test_export_constraint_checks.py`, `test_iaa_runner.py`, `test_export_api.py`. ## Why K varies per query (~15% K<5, 61% K=5, 24% K>5; the retriever is threshold-based, not top-K), so the export side can't infer K from `chunk_rank` or from record-counting alone — it needs a persisted SSOT to drive true top-K metrics in the consumer pipeline (precision@K, recall@K, NDCG@K, MRR@K). End-to-end mirror of the existing `chunk_rank` field: stamped at import, declared on the task, modelled on `RetrievalAnnotation`, read back by the fetcher. A one-off backfill script for records imported before this PR (in-flight batches) lives outside the supported CLI/API surface — retrospective ops migration, not packaged functionality. ## Base `feat/querygen-resumable-stack` — kitchen-sink integration branch carrying the full dep surface this stack assumes (`Locale`, `atomic_write_text`, `LogicalConstraint`, `WIDGET_FIELD_PLACEHOLDERS`, the `export_constraint_checks` rename). Rebases cleanly to `main` once those upstream stacks land. ## Test plan - [x] `uv run python -m pytest tests/unit` (863 passing) ---

**Stack (6 PRs):** #246 (merged) → #247 → #268 → #248 → #269 → #249 **Chain goal:** Persist K (number of retrieved chunks per query) on retrieval records, surface it as panel-completeness columns + sidecar at export time, ship a live `annotation status` CLI on top, and add an opt-in `--tag-partial-panels` advisory write so annotators can filter partially-done panels in the Argilla UI. **This PR (2/6):** The standalone, export-agnostic panel-completeness calculator, plus the shared retrieval-walk primitive it and #268 both consume. Split out of what was originally a single ~1000-line PR; wiring completeness into the retrieval export path ships separately in #268. --- ## Scope ### Completeness aggregator - New `core/annotation/completeness.py`: groups retrieval snapshots by `record_uuid`, distinct-by-`chunk_id`, derives per-panel facts, emits `CompletenessReport` (`by_uuid` + `summary`). Aggregation is split into a typed `_PanelAccumulator` plus `_aggregate_snapshots` (grouping) / `_summarize_groups` (transform) passes. ### Shared retrieval walk - `core/annotation/export_fetcher.py`: `RetrievalRecordSnapshot` + `walk_retrieval_records` — one Argilla scroll, no status filter (so records lacking terminal responses are still seen for the integrity check), feeding both this module's completeness pass and #268's row emission. Plus `resolve_task_purposes`, a topology lookup extracted from `fetch_task` (no behaviour change). ### Schema - `core/schemas/annotation_export.py`: `CompletenessSummary` + `KBucketStat` models for the export sidecar (populated by #268). ### Tests - New `test_completeness.py`: happy path, distinct-by-chunk_id, orphan exclusion, integrity warnings (records-vs-K + mixed-backfill within panel), all-unknown-K warning, K-bucket cross-tab + per-K histogram, STRICT `panel_complete` (discards do NOT count), `n_discarded_chunks` subset of `n_annotated_chunks`, calibration walked when topology declares it. ## Why STRICT `panel_complete` The computed `panel_complete: bool` is STRICT: True iff every K chunk in the panel has at least one **submitted** response. Discarded responses are abstentions, not judgements (pragmata's `DiscardReason` enum is refusal-only: `INVALID_OR_UNREALISTIC` / `UNCLEAR` / `OUTSIDE_REVIEWER_EXPERTISE`), so treating them as covered would feed unjudged chunks into NDCG@K / precision@K denominators as 0-relevance labels. The bool is derivable from the count columns, so a consumer wanting the permissive policy computes `n_annotated_chunks == n_retrieved_chunks` in one line. The retriever is threshold-based, so dropping partial panels biases metrics toward small K (MNAR); this module is the shared calculation both the export path (#268) and the live status path (#248) build on — one aggregator, two consumers. ## Test plan - [x] `uv run python -m pytest tests/unit` (1040 passing)

retrieval.csv gains panel_complete (STRICT: every K chunk has a submitted response) plus n_annotated_chunks (terminal), n_submitted_chunks (used by panel_complete), n_discarded_chunks, n_records_seen (integrity). annotation_export.meta.json gains a completeness_summary block plus a completeness_status discriminator (ok / failed / not_requested) that disambiguates 'not retrieval-exported' from 'compute_completeness raised'. Single-walk: walk_retrieval_records issues ONE Argilla scroll; both the typed-row projection (fetch_retrieval_from_records) and the completeness aggregator consume the same in-memory record set. Violations flow through as LogicalConstraint objects (matching the constraint_id SSOT on main), not the pre-split list[str].

…ignore Addresses review feedback on #268: - type assemble_export_meta's completeness_status param (and the run_export local) as Literal["ok","failed","not_requested"], removing the # type: ignore[arg-type] at the AnnotationExportMeta callsite. - document why the completeness compute uses a deliberately broad except: it guards an already-fetched export from a derived-sidecar failure, and catches Exception (not BaseException) so KeyboardInterrupt/SystemExit still propagate.

…249) **Stack (6 PRs):** #246 → #247 → #268 → #248 → #269 → #249 **This PR (6/6):** Adds the advisory `needs_completion` tag write onto `annotation status`. - `status --tag-partial-panels`: stamps `needs_completion` on the **unresolved chunks of PARTIAL panels** (`0 < submitted < K`) and clears stale tags, sharing the single status walk. Annotators filter `needs_completion=true` in the Argilla UI to focus on the records that will complete a partially-done panel. - **Partial-only gate** (not "any incomplete panel"): a fully-unstarted panel is never tagged, so the tag stays a selective mop-up signal instead of ≈ everything pending. - **Multi-domain + prod/cal-split aware**: a panel whose chunks are split across a workspace's production + calibration datasets (per-item calibration) is evaluated as one unit; writes route to each chunk's owning dataset. Reuses `metadata_ops` for the replace-safe upsert. Stacked on #269. Also re-exports `TagResult` from the `pragmata.annotation` facade: `StatusReport.tag_result` is a public field of this type, but the facade export was pre-declared two PRs ago (before `panel_status.py` existed), dropped once that PR's version of the module still didn't define the class, and never re-added once this PR added it for real.

github-actions Bot added feature Adds or expands user-facing functionality size: M 200-499 LOC labels Jul 1, 2026

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/02b-wire-export-completeness branch from fd0eb3d to 2d18117 Compare July 3, 2026 13:34

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/02-export-completeness branch from 2791caa to 25391ca Compare July 3, 2026 13:34

henrycgbaker requested a review from ddimmery July 3, 2026 13:46

ddimmery approved these changes Jul 3, 2026

View reviewed changes

saschagobel added contracts Changes schemas, type contracts, or validation logic for inputs, outputs, and persisted artifacts annotation Changes affecting the annotation tool labels Jul 4, 2026

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/02-export-completeness branch from 25391ca to 3cab000 Compare July 6, 2026 13:49

Base automatically changed from feat/annotation-retrieval-completeness/02-export-completeness to main July 6, 2026 13:55

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/02b-wire-export-completeness branch from 2d18117 to de3f8ff Compare July 7, 2026 07:46

henrycgbaker merged commit 9c35d7e into main Jul 7, 2026
4 checks passed

henrycgbaker deleted the feat/annotation-retrieval-completeness/02b-wire-export-completeness branch July 7, 2026 08:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(annotation): wire completeness into export (columns + meta summary)#268

feat(annotation): wire completeness into export (columns + meta summary)#268
henrycgbaker merged 2 commits into
mainfrom
feat/annotation-retrieval-completeness/02b-wire-export-completeness

henrycgbaker commented Jul 1, 2026 •

edited

Loading

Uh oh!

ddimmery Jul 3, 2026

Uh oh!

ddimmery Jul 3, 2026

Uh oh!

ddimmery Jul 3, 2026

Uh oh!

henrycgbaker Jul 7, 2026

Uh oh!

ddimmery Jul 3, 2026

Uh oh!

ddimmery Jul 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		# lost or duplicated upstream). Surfaced per-row so consumers can spot
		# corrupted panels in dataframe pipelines without re-joining the sidecar.

Uh oh!

Conversation

henrycgbaker commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scope

Schema

Single-walk retrieval pipeline

API

Tests

Why

Test plan

Uh oh!

ddimmery Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

ddimmery Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

ddimmery Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

henrycgbaker Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddimmery Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

ddimmery Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

henrycgbaker commented Jul 1, 2026 •

edited

Loading