feat(annotation): panel completeness calculator#247
Merged
henrycgbaker merged 2 commits intoJul 6, 2026
Merged
Conversation
github-actions
Bot
added
feature
This was referenced Jun 1, 2026
Collaborator
Author
|
open question: the export ships I picked (i)
NB: annotator-facing needs_completion tag (live status path, PR #249) is unaffected by this — it's computed in-memory during the status walk, not from the CSV. |
henrycgbaker
force-pushed
the
feat/annotation-retrieval-completeness/02-export-completeness
branch
from
42bd0be to
2791caa
Compare
This was referenced Jul 1, 2026
henrycgbaker
changed the title
henrycgbaker
force-pushed
the
feat/annotation-retrieval-completeness/01-persist-k
branch
from
b232f54 to
30b5118
Compare
henrycgbaker
force-pushed
the
feat/annotation-retrieval-completeness/02-export-completeness
branch
from
2791caa to
25391ca
Compare
ddimmery
approved these changes
Jul 3, 2026
| Shared by ``compute_completeness`` (standalone) and ``run_export`` (which | ||
| walks once and feeds both the row-emission and the completeness passes). | ||
| """ | ||
| groups: dict[str, dict] = {} |
Collaborator
There was a problem hiding this comment.
Could use a dataclass or TypedDict for clearer type safety.
| _BUCKET_KEYS = ("k_lt_5", "k_eq_5", "k_gt_5") | ||
|
|
||
|
|
||
| def compute_completeness_from_records(snapshots: list[RetrievalRecordSnapshot]) -> CompletenessReport: |
Collaborator
There was a problem hiding this comment.
This function is pretty complicated. To simplify you could extract the aggregation part (iterating through snapshots to populate the dict) and the transformation part (iterating through groups to construct the summary). Not critical, but could be useful (could also allow testing individual components more cleanly.
henrycgbaker
added a commit
that referenced
this pull request
Jul 6, 2026
**Stack (6 PRs):** #246 → #247 → #268 → #248 → #269 → #249 **Chain goal:** Persist K (number of retrieved chunks per query) on retrieval records, surface it as panel-completeness columns + sidecar at export time, ship a live `annotation status` CLI on top, and add an opt-in `--tag-partial-panels` advisory write so annotators can filter partially-done panels in the Argilla UI. **This PR (1/6):** Foundation. Stamps `n_retrieved_chunks` on retrieval record metadata at import and surfaces it on `RetrievalAnnotation`. Ships nothing user-visible on its own — the export columns (#247/#268) and live status CLI (#248/#269) consume it. --- ## Scope ### Record building - `core/annotation/record_builder.py`: add `"n_retrieved_chunks": len(pair.chunks)` to every chunk-record's metadata dict. ### Task definition - `core/annotation/argilla_task_definitions.py`: declare `IntegerMetadataProperty("n_retrieved_chunks", min=1, visible_for_annotators=False)` on retrieval task settings. ### Schema - `core/schemas/annotation_export.py`: `RetrievalAnnotation.n_retrieved_chunks: int` field (mirrors the existing `chunk_rank: int`). ### Export readback - `core/annotation/export_fetcher.py`: `_build_row` retrieval branch reads `metadata.get("n_retrieved_chunks", 0)` (defensive default for records imported before this PR). ### Tests - New `test_n_retrieved_chunks_stamped_on_every_chunk_record` in `test_import.py` locks the persist contract. - Metadata-name list in `test_argilla_task_definitions.py` updated. - Fixture refresh across `test_annotation_export.py`, `test_export_runner.py`, `test_export_constraint_checks.py`, `test_iaa_runner.py`, `test_export_api.py`. ## Why K varies per query (~15% K<5, 61% K=5, 24% K>5; the retriever is threshold-based, not top-K), so the export side can't infer K from `chunk_rank` or from record-counting alone — it needs a persisted SSOT to drive true top-K metrics in the consumer pipeline (precision@K, recall@K, NDCG@K, MRR@K). End-to-end mirror of the existing `chunk_rank` field: stamped at import, declared on the task, modelled on `RetrievalAnnotation`, read back by the fetcher. A one-off backfill script for records imported before this PR (in-flight batches) lives outside the supported CLI/API surface — retrospective ops migration, not packaged functionality. ## Base `feat/querygen-resumable-stack` — kitchen-sink integration branch carrying the full dep surface this stack assumes (`Locale`, `atomic_write_text`, `LogicalConstraint`, `WIDGET_FIELD_PLACEHOLDERS`, the `export_constraint_checks` rename). Rebases cleanly to `main` once those upstream stacks land. ## Test plan - [x] `uv run python -m pytest tests/unit` (863 passing) ---
Base automatically changed from
feat/annotation-retrieval-completeness/01-persist-k
to
main
July 6, 2026 13:02
Groups retrieval snapshots by record_uuid, distinct-by-chunk_id, derives per-panel facts, emits CompletenessReport (by_uuid + summary). STRICT default for panel_complete: pragmata's DiscardReason enum is refusal-only (INVALID_OR_UNREALISTIC / UNCLEAR / OUTSIDE_REVIEWER_EXPERTISE), so a discarded chunk is an abstention, not a judgement -- treating it as covered would feed unjudged chunks into NDCG@K / precision@K denominators as if they carried a 0-relevance label.
Addresses review feedback on #247: replace the untyped dict accumulator with a _PanelAccumulator dataclass, and split the monolithic compute_completeness_from_records into _aggregate_snapshots (grouping pass) + _summarize_groups (transform pass) so each is independently testable. Behaviour-preserving.
henrycgbaker
force-pushed
the
feat/annotation-retrieval-completeness/02-export-completeness
branch
from
25391ca to
3cab000
Compare
henrycgbaker
deleted the
feat/annotation-retrieval-completeness/02-export-completeness
branch
henrycgbaker
added a commit
that referenced
this pull request
Jul 7, 2026
…ry) (#268) **Stack (6 PRs):** #246 → #247 → #268 → #248 → #269 → #249 **This PR (3/6):** Split out of #247 (was 1001 lines / XL). Wires the panel-completeness calculator (#247) into the retrieval export path. Stacked on #247. --- ## Scope ### Schema - `core/schemas/annotation_export.py`: - `RetrievalExportRow` += `panel_complete: bool`, `n_annotated_chunks: int`, `n_submitted_chunks: int`, `n_discarded_chunks: int`, `n_records_seen: int`. All defaulted; appended at the tail so existing column order is preserved. - New `KBucketStat` (per-K-bucket panel counts) and `CompletenessSummary` (aggregates + `by_k_bucket` cross-tab + full per-K histogram) Pydantic models. - `AnnotationExportMeta` += `completeness_summary: CompletenessSummary | None` and `completeness_status: Literal["ok","failed","not_requested"]`. ### Single-walk retrieval pipeline - `core/annotation/export_fetcher.py`: new `walk_retrieval_records` (one Argilla scroll per dataset, no status filter — sees discards-only records too so the integrity check has the full record set) + `RetrievalRecordSnapshot` dataclass + `fetch_retrieval_from_records` (status filter applied in Python at typed-row construction). - `core/annotation/export_runner.py`: when `Task.RETRIEVAL in tasks`, walks once and feeds both the typed-row projection and the completeness aggregator. `compute_completeness` failures degrade the sidecar (sets `completeness_status="failed"`) without losing the already-fetched export. ### API - `annotation/__init__.py`: lazy re-exports for `CompletenessSummary`, `KBucketStat`. ### Tests - `test_export_runner.py`: `write_export_csv` stamps completeness columns by `record_uuid`; sidecar carries `completeness_summary`; `completeness_status` ok/failed/not_requested set explicitly. - `test_annotation_export.py`: `RetrievalExportRow` field-order locked. ## Why The retriever is threshold-based, so dropping partial panels biases the metrics toward small K (MNAR). Surfacing both per-row predicate columns AND a bucketed sidecar aggregate lets the eval pipeline either filter complete panels per-K or apply condensed-list scoring without re-querying the data layer. The single-walk shape (one retrieval scroll feeds both fetch_task and compute_completeness) costs one extra in-memory pass over already-fetched records but spares the Argilla server a second full scroll per export — meaningful on the in-flight batches (~hundreds of panels). The STRICT `panel_complete` policy question (which shape to ship — this PR assumes (A) from #247) is discussed on #247, since that's where the predicate is actually computed; this PR just surfaces it as a column. ## Test plan - [ ] `uv run python -m pytest tests/unit/core/annotation/test_export_runner.py tests/unit/core/schemas/test_annotation_export.py`
henrycgbaker
added a commit
that referenced
this pull request
Jul 7, 2026
…249) **Stack (6 PRs):** #246 → #247 → #268 → #248 → #269 → #249 **This PR (6/6):** Adds the advisory `needs_completion` tag write onto `annotation status`. - `status --tag-partial-panels`: stamps `needs_completion` on the **unresolved chunks of PARTIAL panels** (`0 < submitted < K`) and clears stale tags, sharing the single status walk. Annotators filter `needs_completion=true` in the Argilla UI to focus on the records that will complete a partially-done panel. - **Partial-only gate** (not "any incomplete panel"): a fully-unstarted panel is never tagged, so the tag stays a selective mop-up signal instead of ≈ everything pending. - **Multi-domain + prod/cal-split aware**: a panel whose chunks are split across a workspace's production + calibration datasets (per-item calibration) is evaluated as one unit; writes route to each chunk's owning dataset. Reuses `metadata_ops` for the replace-safe upsert. Stacked on #269. Also re-exports `TagResult` from the `pragmata.annotation` facade: `StatusReport.tag_result` is a public field of this type, but the facade export was pre-declared two PRs ago (before `panel_status.py` existed), dropped once that PR's version of the module still didn't define the class, and never re-added once this PR added it for real.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (6 PRs): #246 (merged) → #247 → #268 → #248 → #269 → #249
Chain goal: Persist K (number of retrieved chunks per query) on retrieval records, surface it as panel-completeness columns + sidecar at export time, ship a live
annotation statusCLI on top, and add an opt-in--tag-partial-panelsadvisory write so annotators can filter partially-done panels in the Argilla UI.This PR (2/6): The standalone, export-agnostic panel-completeness calculator, plus the shared retrieval-walk primitive it and #268 both consume. Split out of what was originally a single ~1000-line PR; wiring completeness into the retrieval export path ships separately in #268.
Scope
Completeness aggregator
core/annotation/completeness.py: groups retrieval snapshots byrecord_uuid, distinct-by-chunk_id, derives per-panel facts, emitsCompletenessReport(by_uuid+summary). Aggregation is split into a typed_PanelAccumulatorplus_aggregate_snapshots(grouping) /_summarize_groups(transform) passes.Shared retrieval walk
core/annotation/export_fetcher.py:RetrievalRecordSnapshot+walk_retrieval_records— one Argilla scroll, no status filter (so records lacking terminal responses are still seen for the integrity check), feeding both this module's completeness pass and feat(annotation): wire completeness into export (columns + meta summary) #268's row emission. Plusresolve_task_purposes, a topology lookup extracted fromfetch_task(no behaviour change).Schema
core/schemas/annotation_export.py:CompletenessSummary+KBucketStatmodels for the export sidecar (populated by feat(annotation): wire completeness into export (columns + meta summary) #268).Tests
test_completeness.py: happy path, distinct-by-chunk_id, orphan exclusion, integrity warnings (records-vs-K + mixed-backfill within panel), all-unknown-K warning, K-bucket cross-tab + per-K histogram, STRICTpanel_complete(discards do NOT count),n_discarded_chunkssubset ofn_annotated_chunks, calibration walked when topology declares it.Why STRICT
panel_completeThe computed
panel_complete: boolis STRICT: True iff every K chunk in the panel has at least one submitted response. Discarded responses are abstentions, not judgements (pragmata'sDiscardReasonenum is refusal-only:INVALID_OR_UNREALISTIC/UNCLEAR/OUTSIDE_REVIEWER_EXPERTISE), so treating them as covered would feed unjudged chunks into NDCG@K / precision@K denominators as 0-relevance labels. The bool is derivable from the count columns, so a consumer wanting the permissive policy computesn_annotated_chunks == n_retrieved_chunksin one line.The retriever is threshold-based, so dropping partial panels biases metrics toward small K (MNAR); this module is the shared calculation both the export path (#268) and the live status path (#248) build on — one aggregator, two consumers.
Test plan
uv run python -m pytest tests/unit(1040 passing)