feat(annotation): tag partial panels via status --tag-partial-panels by henrycgbaker · Pull Request #249 · bertelsmannstift/pragmata

henrycgbaker · 2026-06-01T11:40:47Z

Stack (6 PRs): #246 → #247 → #268 → #248 → #269 → #249

This PR (6/6): Adds the advisory needs_completion tag write onto annotation status.

status --tag-partial-panels: stamps needs_completion on the unresolved chunks of PARTIAL panels (0 < submitted < K) and clears stale tags, sharing the single status walk. Annotators filter needs_completion=true in the Argilla UI to focus on the records that will complete a partially-done panel.
Partial-only gate (not "any incomplete panel"): a fully-unstarted panel is never tagged, so the tag stays a selective mop-up signal instead of ≈ everything pending.
Multi-domain + prod/cal-split aware: a panel whose chunks are split across a workspace's production + calibration datasets (per-item calibration) is evaluated as one unit; writes route to each chunk's owning dataset.

Reuses metadata_ops for the replace-safe upsert. Stacked on #269.

Also re-exports TagResult from the pragmata.annotation facade: StatusReport.tag_result is a public field of this type, but the facade export was pre-declared two PRs ago (before panel_status.py existed), dropped once that PR's version of the module still didn't define the class, and never re-added once this PR added it for real.

**Stack (6 PRs):** #246 → #247 → #268 → #248 → #269 → #249 **Chain goal:** Persist K (number of retrieved chunks per query) on retrieval records, surface it as panel-completeness columns + sidecar at export time, ship a live `annotation status` CLI on top, and add an opt-in `--tag-partial-panels` advisory write so annotators can filter partially-done panels in the Argilla UI. **This PR (1/6):** Foundation. Stamps `n_retrieved_chunks` on retrieval record metadata at import and surfaces it on `RetrievalAnnotation`. Ships nothing user-visible on its own — the export columns (#247/#268) and live status CLI (#248/#269) consume it. --- ## Scope ### Record building - `core/annotation/record_builder.py`: add `"n_retrieved_chunks": len(pair.chunks)` to every chunk-record's metadata dict. ### Task definition - `core/annotation/argilla_task_definitions.py`: declare `IntegerMetadataProperty("n_retrieved_chunks", min=1, visible_for_annotators=False)` on retrieval task settings. ### Schema - `core/schemas/annotation_export.py`: `RetrievalAnnotation.n_retrieved_chunks: int` field (mirrors the existing `chunk_rank: int`). ### Export readback - `core/annotation/export_fetcher.py`: `_build_row` retrieval branch reads `metadata.get("n_retrieved_chunks", 0)` (defensive default for records imported before this PR). ### Tests - New `test_n_retrieved_chunks_stamped_on_every_chunk_record` in `test_import.py` locks the persist contract. - Metadata-name list in `test_argilla_task_definitions.py` updated. - Fixture refresh across `test_annotation_export.py`, `test_export_runner.py`, `test_export_constraint_checks.py`, `test_iaa_runner.py`, `test_export_api.py`. ## Why K varies per query (~15% K<5, 61% K=5, 24% K>5; the retriever is threshold-based, not top-K), so the export side can't infer K from `chunk_rank` or from record-counting alone — it needs a persisted SSOT to drive true top-K metrics in the consumer pipeline (precision@K, recall@K, NDCG@K, MRR@K). End-to-end mirror of the existing `chunk_rank` field: stamped at import, declared on the task, modelled on `RetrievalAnnotation`, read back by the fetcher. A one-off backfill script for records imported before this PR (in-flight batches) lives outside the supported CLI/API surface — retrospective ops migration, not packaged functionality. ## Base `feat/querygen-resumable-stack` — kitchen-sink integration branch carrying the full dep surface this stack assumes (`Locale`, `atomic_write_text`, `LogicalConstraint`, `WIDGET_FIELD_PLACEHOLDERS`, the `export_constraint_checks` rename). Rebases cleanly to `main` once those upstream stacks land. ## Test plan - [x] `uv run python -m pytest tests/unit` (863 passing) ---

**Stack (6 PRs):** #246 (merged) → #247 → #268 → #248 → #269 → #249 **Chain goal:** Persist K (number of retrieved chunks per query) on retrieval records, surface it as panel-completeness columns + sidecar at export time, ship a live `annotation status` CLI on top, and add an opt-in `--tag-partial-panels` advisory write so annotators can filter partially-done panels in the Argilla UI. **This PR (2/6):** The standalone, export-agnostic panel-completeness calculator, plus the shared retrieval-walk primitive it and #268 both consume. Split out of what was originally a single ~1000-line PR; wiring completeness into the retrieval export path ships separately in #268. --- ## Scope ### Completeness aggregator - New `core/annotation/completeness.py`: groups retrieval snapshots by `record_uuid`, distinct-by-`chunk_id`, derives per-panel facts, emits `CompletenessReport` (`by_uuid` + `summary`). Aggregation is split into a typed `_PanelAccumulator` plus `_aggregate_snapshots` (grouping) / `_summarize_groups` (transform) passes. ### Shared retrieval walk - `core/annotation/export_fetcher.py`: `RetrievalRecordSnapshot` + `walk_retrieval_records` — one Argilla scroll, no status filter (so records lacking terminal responses are still seen for the integrity check), feeding both this module's completeness pass and #268's row emission. Plus `resolve_task_purposes`, a topology lookup extracted from `fetch_task` (no behaviour change). ### Schema - `core/schemas/annotation_export.py`: `CompletenessSummary` + `KBucketStat` models for the export sidecar (populated by #268). ### Tests - New `test_completeness.py`: happy path, distinct-by-chunk_id, orphan exclusion, integrity warnings (records-vs-K + mixed-backfill within panel), all-unknown-K warning, K-bucket cross-tab + per-K histogram, STRICT `panel_complete` (discards do NOT count), `n_discarded_chunks` subset of `n_annotated_chunks`, calibration walked when topology declares it. ## Why STRICT `panel_complete` The computed `panel_complete: bool` is STRICT: True iff every K chunk in the panel has at least one **submitted** response. Discarded responses are abstentions, not judgements (pragmata's `DiscardReason` enum is refusal-only: `INVALID_OR_UNREALISTIC` / `UNCLEAR` / `OUTSIDE_REVIEWER_EXPERTISE`), so treating them as covered would feed unjudged chunks into NDCG@K / precision@K denominators as 0-relevance labels. The bool is derivable from the count columns, so a consumer wanting the permissive policy computes `n_annotated_chunks == n_retrieved_chunks` in one line. The retriever is threshold-based, so dropping partial panels biases metrics toward small K (MNAR); this module is the shared calculation both the export path (#268) and the live status path (#248) build on — one aggregator, two consumers. ## Test plan - [x] `uv run python -m pytest tests/unit` (1040 passing)

…ry) (#268) **Stack (6 PRs):** #246 → #247 → #268 → #248 → #269 → #249 **This PR (3/6):** Split out of #247 (was 1001 lines / XL). Wires the panel-completeness calculator (#247) into the retrieval export path. Stacked on #247. --- ## Scope ### Schema - `core/schemas/annotation_export.py`: - `RetrievalExportRow` += `panel_complete: bool`, `n_annotated_chunks: int`, `n_submitted_chunks: int`, `n_discarded_chunks: int`, `n_records_seen: int`. All defaulted; appended at the tail so existing column order is preserved. - New `KBucketStat` (per-K-bucket panel counts) and `CompletenessSummary` (aggregates + `by_k_bucket` cross-tab + full per-K histogram) Pydantic models. - `AnnotationExportMeta` += `completeness_summary: CompletenessSummary | None` and `completeness_status: Literal["ok","failed","not_requested"]`. ### Single-walk retrieval pipeline - `core/annotation/export_fetcher.py`: new `walk_retrieval_records` (one Argilla scroll per dataset, no status filter — sees discards-only records too so the integrity check has the full record set) + `RetrievalRecordSnapshot` dataclass + `fetch_retrieval_from_records` (status filter applied in Python at typed-row construction). - `core/annotation/export_runner.py`: when `Task.RETRIEVAL in tasks`, walks once and feeds both the typed-row projection and the completeness aggregator. `compute_completeness` failures degrade the sidecar (sets `completeness_status="failed"`) without losing the already-fetched export. ### API - `annotation/__init__.py`: lazy re-exports for `CompletenessSummary`, `KBucketStat`. ### Tests - `test_export_runner.py`: `write_export_csv` stamps completeness columns by `record_uuid`; sidecar carries `completeness_summary`; `completeness_status` ok/failed/not_requested set explicitly. - `test_annotation_export.py`: `RetrievalExportRow` field-order locked. ## Why The retriever is threshold-based, so dropping partial panels biases the metrics toward small K (MNAR). Surfacing both per-row predicate columns AND a bucketed sidecar aggregate lets the eval pipeline either filter complete panels per-K or apply condensed-list scoring without re-querying the data layer. The single-walk shape (one retrieval scroll feeds both fetch_task and compute_completeness) costs one extra in-memory pass over already-fetched records but spares the Argilla server a second full scroll per export — meaningful on the in-flight batches (~hundreds of panels). The STRICT `panel_complete` policy question (which shape to ship — this PR assumes (A) from #247) is discussed on #247, since that's where the predicate is actually computed; this PR just surfaces it as a column. ## Test plan - [ ] `uv run python -m pytest tests/unit/core/annotation/test_export_runner.py tests/unit/core/schemas/test_annotation_export.py`

Add the config-free partial-panel tag write onto 'annotation status': it stamps 'needs_completion' on the unresolved chunks of PARTIAL panels (0 < submitted < K) and clears stale tags, sharing the single status walk. Reuses metadata_ops for the safe upsert.

StatusReport.tag_result is a public field typed TagResult, but the facade never gained a lazy-export entry for it (it was pre-declared in an earlier PR before panel_status.py existed, dropped when that PR's export target still didn't define it, and never re-added once this PR added the class itself). pragmata.annotation.TagResult now resolves like its sibling panel_status exports.

ddimmery · 2026-07-07T12:52:44Z

+def build_metadata_upsert(
+    record: rg.Record,
+    updates: Mapping[str, object],
+    *,
+    remove_keys: Iterable[str] = (),
+) -> rg.Record | None:


Hm. This is all non-obvious behavior, but as best I can tell without testing, this is also recommended for v2.8.0. We're still just saying "argilla>=2.0,<3.0" -- maybe it would be better to pin to 2.8 and force intentional version changes to reduce the risk of something changing about upsert behavior and clobbering information we want to retain?

ddimmery · 2026-07-07T12:57:53Z

+
+    batched: dict[tuple[str, str], list[rg.Record]] = {}
+    n_tagged = n_cleared = n_already = 0
+    for (_ws, uuid), group in _group_by_panel(collected.records).items():


So we're completely separating calibration from production here, and "partial" means "within the context of this particular dataset"? This is ok, but it should be documented.

ddimmery · 2026-07-07T13:01:57Z

+"""Shared safe metadata operations for live Argilla mutations.
+
+Used by both the ``--tag-partial-panels`` write path in ``panel_status`` and the
+one-off backfill script under ``scripts/``. Centralises the two safety


I don't see this script, is it meant to be here?

ddimmery · 2026-07-07T13:07:48Z

+                upsert = build_metadata_upsert(rec.record, {}, remove_keys=[NEEDS_COMPLETION_KEY])
+            else:
+                continue
+            if upsert is None:


This should never be the case, should it?

ddimmery · 2026-07-07T13:17:28Z

+NEEDS_COMPLETION_KEY = "needs_completion"
+NEEDS_COMPLETION_VALUE = "true"


I think there should be a comment indicating that if these are changed, there may need to be manual modifications to the Argilla data to remove old, stale metadata (the tools here will not get rid of it afaict). I can't imagine there will ever be a reason to change this mid-run, though.

Clarify that PARTIAL tagging is dataset-local and ignores min_submitted overlap targets, drop the stale scripts/ backfill reference, and note the stale-metadata risk if the needs_completion tag constants change.

github-actions Bot added feature Adds or expands user-facing functionality size: M 200-499 LOC labels Jun 1, 2026

This was referenced Jun 1, 2026

feat(annotation): persist n_retrieved_chunks on retrieval records #246

Merged

feat(annotation): panel completeness calculator #247

Merged

feat(annotation): panel-status core engine #248

Merged

github-actions Bot added size: L 500-999 LOC and removed size: M 200-499 LOC labels Jun 1, 2026

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/04-tag-incomplete branch from 7c11d13 to bbe639d Compare July 1, 2026 10:08

github-actions Bot added size: M 200-499 LOC and removed size: L 500-999 LOC labels Jul 1, 2026

henrycgbaker changed the title ~~feat(annotation): add --tag-incomplete advisory write to annotation status~~ feat(annotation): tag partial panels via status --tag-partial-panels Jul 1, 2026

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/04-tag-incomplete branch 2 times, most recently from 17d1d81 to fc37f6d Compare July 1, 2026 13:58

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/03-status-command branch from 481a9fd to b8f430b Compare July 1, 2026 16:37

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/04-tag-incomplete branch from fc37f6d to 139d097 Compare July 1, 2026 16:37

github-actions Bot added size: L 500-999 LOC and removed size: M 200-499 LOC labels Jul 1, 2026

henrycgbaker mentioned this pull request Jul 1, 2026

feat(annotation): status API + CLI command #269

Merged

1 task

henrycgbaker changed the base branch from feat/annotation-retrieval-completeness/03-status-command to feat/annotation-retrieval-completeness/03b-status-api-cli July 1, 2026 16:38

henrycgbaker mentioned this pull request Jul 1, 2026

feat(annotation): wire completeness into export (columns + meta summary) #268

Merged

1 task

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/04-tag-incomplete branch from 139d097 to 9d28183 Compare July 3, 2026 13:34

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/03b-status-api-cli branch from 0f63545 to 4624141 Compare July 3, 2026 13:34

github-actions Bot added size: M 200-499 LOC and removed size: L 500-999 LOC labels Jul 3, 2026

saschagobel added api Changes affecting the public Python API or user-facing function signatures annotation Changes affecting the annotation tool cli Changes affecting the command-line interface, CLI options, or terminal behavior labels Jul 4, 2026

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/03b-status-api-cli branch 2 times, most recently from 461de42 to 15c0a59 Compare July 7, 2026 12:09

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/04-tag-incomplete branch from 9d28183 to de870d4 Compare July 7, 2026 12:11

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/03b-status-api-cli branch from 15c0a59 to fe70147 Compare July 7, 2026 12:38

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/04-tag-incomplete branch from de870d4 to 63acdac Compare July 7, 2026 12:44

henrycgbaker changed the base branch from feat/annotation-retrieval-completeness/03b-status-api-cli to main July 7, 2026 12:48

henrycgbaker added 2 commits July 7, 2026 14:48

henrycgbaker requested a review from ddimmery July 7, 2026 12:49

henrycgbaker force-pushed the feat/annotation-retrieval-completeness/04-tag-incomplete branch from 63acdac to 789bb9e Compare July 7, 2026 12:49

henrycgbaker marked this pull request as ready for review July 7, 2026 12:50

ddimmery approved these changes Jul 7, 2026

View reviewed changes

docs(annotation): address PR review comments on tagging semantics

016802d

Clarify that PARTIAL tagging is dataset-local and ignores min_submitted overlap targets, drop the stale scripts/ backfill reference, and note the stale-metadata risk if the needs_completion tag constants change.

github-actions Bot added size: L 500-999 LOC and removed size: M 200-499 LOC labels Jul 7, 2026

henrycgbaker merged commit 4fc2478 into main Jul 7, 2026
4 checks passed

henrycgbaker deleted the feat/annotation-retrieval-completeness/04-tag-incomplete branch July 7, 2026 14:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(annotation): tag partial panels via status --tag-partial-panels#249

feat(annotation): tag partial panels via status --tag-partial-panels#249
henrycgbaker merged 3 commits into
mainfrom
feat/annotation-retrieval-completeness/04-tag-incomplete

henrycgbaker commented Jun 1, 2026 •

edited

Loading

Uh oh!

ddimmery Jul 7, 2026

Uh oh!

ddimmery Jul 7, 2026

Uh oh!

ddimmery Jul 7, 2026

Uh oh!

ddimmery Jul 7, 2026

Uh oh!

ddimmery Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		NEEDS_COMPLETION_KEY = "needs_completion"
		NEEDS_COMPLETION_VALUE = "true"

Uh oh!

Conversation

henrycgbaker commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ddimmery Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddimmery Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddimmery Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddimmery Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

ddimmery Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

henrycgbaker commented Jun 1, 2026 •

edited

Loading