feat(annotation): per-item PartitionManifestEntry + per-item calibration assignment by henrycgbaker · Pull Request #227 · bertelsmannstift/pragmata

henrycgbaker · 2026-05-27T20:52:15Z

Stack (4 PRs): #226 → #227 → #228 → #229 (companion to standalone IAA fix #225)

Chain goal: Switch calibration vs production assignment from per-record_uuid to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap.

This PR (2/4): Per-item partition logic + schema reshape + per-task fraction cascade. Ships the user-visible feature without the absolute cap (#229) or the API-hardening pass (#228).

Scope

Schema

core/schemas/annotation_import.py: PartitionManifestEntry replaces calibration: bool with grounding_generation_calibration: dict[Task, bool] (keyed by task) and retrieval_chunk_calibration: dict[str, bool] (keyed by chunk_id). Per-task calibration_fraction_at_import provenance.

Partition logic

core/annotation/record_builder.py: assign_partitions buckets units via per-(task, unit) digest hash(seed || task || unit_id) against fraction * 2^32 so per-task draws are statistically independent. _enumerate_units resolves the per-task unit list, _write_unit routes the bool back into the right per-record dict.
Returns dict[str, PartitionManifestEntry] as before; the PartitionResult bundle is extracted in refactor(annotation): extract PartitionResult bundle + collapse count helpers #228.

Settings cascade

core/settings/annotation_settings.py: calibration_fraction is now inheritable across deployment / workspace / task scopes via the Inherit sentinel pattern established in refactor(annotation): introduce WorkspaceSettings + TaskSettings with inheritance propagation #206. _check_calibration_topology walks per-(ws, task) using resolved values. The _inherit walk is hoisted into a single helper so resolved_task no longer repeats the chain four times.

API / CLI

api/annotation_import.py: ImportResult per-task dicts (calibration_count, production_count, calibration_fraction, realised_calibration_fraction) replace the scalar fields.
cli/commands/annotation.py: CLI output prints one line per task.

Tests

Per-chunk routing independence, per-task seed independence, inheritance cascade, and per-(ws, task) topology validation. Manifest IO round-trip with the new shape.

Breaking change

Pre-v0 on-disk manifests with calibration: bool no longer load. Pragmata is pre-1.0; affected workspaces re-bootstrap by deleting partition.meta.json and re-importing.

Why

Per ADR-0012 (#226): per-item partition is the natural fit for the existing naive Krippendorff bootstrap and matches Argilla's per-item record creation. Per-task fraction cascade lets operators right-size each task independently (e.g. retrieval at 5%, grounding at 20%).

Test plan

uv run python -m pytest tests/unit (770 passing in this branch's worktree)

Document the decision to partition calibration vs production per annotation item (chunk for retrieval, record for grounding/generation) rather than per record_uuid. Implementation lands in follow-up PRs. - New ADR-0012 covering the option-set, statistical rationale (Kish design effect under naive bootstrap), strategic positioning, and out-of-scope future work. - README index entry for the new ADR. - annotation-import-pipeline.md gets a new section describing the per-record manifest schema (grounding_generation_calibration + retrieval_chunk_calibration) referenced by ADR-0012.

Remove PartitionManifestEntry schema block from Decision section - those details belong in the design doc, not the ADR. Trim Out-of-scope section to the one item directly implied by this decision (cluster-aware IAA); other roadmap items moved to backlog.

Merge verbose 'Context - design alternatives' and 'Strategic positioning' sections into a four-bullet 'Rationale' section, preserving the key statistical argument (Kish design effect) and the dependency point (per-record implies cluster-aware IAA work).

…tems Cap unit is the annotation item (chunks for retrieval, records for grounding/generation), not the record. Rename propagated through ADR-0012 and the design doc, including the at-import provenance field name.

Aligns with the existing naming convention (0010-annotation-..., 0011-annotation-...). References updated in decisions/README.md and design/annotation-import-pipeline.md.

…ading Remove reference to test_cap_under_split_imports_is_order_dependent_by_design which does not exist yet; order-dependence is still noted via assign_partitions. Rename 'Per-record manifest schema' heading to 'Partition manifest entry schema' to match the per-item framing throughout the section.

…ion assignment Switch calibration vs production partitioning from per-record_uuid to per annotation item, as documented in ADR-0012. The annotation item differs by task: grounding and generation keep one item per record_uuid; retrieval gets one item per (record_uuid, chunk_id), so different chunks of the same record can land in different buckets. - core/schemas: PartitionManifestEntry replaces `calibration: bool` with `grounding_generation_calibration: dict[Task, bool]` and `retrieval_chunk_calibration: dict[str, bool]`. Per-task `calibration_fraction_at_import` provenance. - core/annotation/record_builder: assign_partitions buckets units via per-(task, unit_id) digest hash(seed || task || unit_id) against fraction * 2^32 so per-task draws are statistically independent. `_enumerate_units` resolves the per-task unit list, `_write_unit` routes the bool back into the right per-record dict. - core/settings: `calibration_fraction` is now inheritable across deployment / workspace / task scopes via the Inherit sentinel pattern established in PR #206. `_check_calibration_topology` walks per-(ws, task) using resolved values. The `_inherit` walk is hoisted into a single helper so `resolved_task` no longer repeats the chain four times. - api/annotation_import: ImportResult per-task dicts (calibration_count, production_count, calibration_fraction, realised_calibration_fraction) replace the scalar fields. CLI output prints one line per task. - Tests cover per-chunk routing independence, per-task seed independence, inheritance cascade, and per-(ws, task) topology validation. Schema break: pre-v0 on-disk manifests with `calibration: bool` no longer load. Pragmata is pre-1.0; affected workspaces re-bootstrap by deleting `partition.meta.json` and re-importing.

Audit follow-ups on the per-item partition assignment introduced here: - core/schemas: replace the _check_fraction_range model_validator with a declarative dict[Task, Annotated[float, Field(ge=0, le=1)]] on PartitionManifestEntry, matching the Annotated-constraint idiom. - core/settings: type the _inherit helper (generic) and the resolved_task at() closure. - tests/integration: repair the calibration suite, which asserted the pre-per-task scalar ImportResult shape (calibration_count > 0, dict + dict) and the removed entry.calibration attribute. Deselected by default (-m 'not integration'), so the breakage was CI-invisible.

Retrieval calibration is keyed per chunk_id (retrieval_chunk_calibration, unit_id "{record_uuid}:{chunk_id}"). Two chunks sharing a chunk_id in one pair collapse to a single manifest flag, so fan-out emits one record per chunk but forces both into the same calibration/production bucket and the per-task counts under-count the records actually submitted. chunk_id is documented as a unique identifier, so enforce it at the schema edge with a QueryResponsePair validator rather than silently mis-routing malformed input.

**Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (1/4):** Lands the ADR and the matching design-doc section before the implementation, so subsequent PRs can reference the decision and the schema shape. --- ## Scope - `docs/decisions/0012-per-item-calibration-partition.md` (new): the decision, the option-set considered, Kish design-effect arithmetic justifying per-item over per-record under the existing naive Krippendorff bootstrap, strategic positioning vs LLM-judge mainstream + industry annotation services, consequences, out-of-scope future work, and references. - `docs/decisions/README.md`: index entry for the new ADR. - `docs/design/annotation-import-pipeline.md`: new "Per-record manifest schema" section describing the `PartitionManifestEntry` shape that #227 implements (`grounding_generation_calibration` + `retrieval_chunk_calibration`). ## Test plan - [x] Docs only - no test changes.

…ition

The at() helper collapsed every inheritable field to a single Locale | int | float | None union, so mypy could not match each resolved value to its ResolvedTaskSettings param type. Call _inherit directly per field so its generic T is inferred concretely.

saschagobel

Thanks for the substantial work here. I'm requesting changes for the API/core boundary issue and missing validation on inherited calibration_fraction, and I left the remaining comments as non-blocking questions/suggestions around partition summaries, retrieval ID stability, manifest reuse semantics, and test coverage.

Argilla record id now uses a hash of chunk_id instead of the chunk's list position, so it matches the (record_uuid, chunk_id) partition identity. Reimporting a pair with chunks in a different order keeps each chunk's record id stable, preventing stale records being stranded in the wrong calibration/production dataset.

When a record was first imported under a narrower topology (e.g. retrieval-only) and a task is later added to the workspaces config, assign_partitions now backfills the new task's assignment on the existing manifest entry using the same deterministic digest, instead of leaving it absent and silently defaulting to production. Existing assignments stay locked and provenance is preserved.

saschagobel

Thanks for addressing the review feedback. Approving this from a code-review perspective, with the remaining note that the current mypy failure in record_builder.py needs to be fixed before merge.

…ition

… helpers (#228) **Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (3/4):** Pure refactor with no behaviour change. Tightens the api / core seam introduced in #227 so per-task fraction reporting and per-pair `record_uuid` derivation are each computed exactly once. --- ## Scope - `core/annotation/record_builder.py`: introduce `PartitionResult(assignments, pairs_by_rid, calibration_fraction)`. `assign_partitions` now returns this bundle; `fan_out_records(client, settings, *, partition)` consumes it and `_build_batches(pairs_by_rid, assignments)` iterates the rid map instead of recomputing `derive_record_uuid(pair)` per pair. - `core/annotation/record_builder.py`: replace `count_calibration_per_task` (returning `dict[Task, int]`) with `count_units_per_task` returning a `TaskUnitCounts(calibration, total)` frozen dataclass so the api layer gets both tallies in a single pass. - `api/annotation_import.py`: drop `_total_units_per_task` and `_resolve_per_task_fraction`. The api now reads `partition.calibration_fraction` directly and uses the merged `count_units_per_task` helper. ## Why After #227 lands, the api layer recomputes per-task fraction via its own workspace walk (`_resolve_per_task_fraction`), and `_build_batches` recomputes `record_uuid` for every pair even though `assign_partitions` already derived it. Both are wasted work; the natural fix is to surface the precomputed values via a bundle return type. This also unblocks the cap-related fields in #229 - `PartitionResult` is the obvious place to carry per-task cap resolution. Reviewer sanity check: assertions in `test_partition.py` move from `result[rid]` to `partition.assignments[rid]`; the values they check are unchanged. ## Test plan - [x] `uv run python -m pytest tests/unit` (770 passing)

henrycgbaker mentioned this pull request May 27, 2026

refactor(annotation): extract PartitionResult bundle + collapse count helpers #228

Merged

1 task

github-actions Bot added feature Adds or expands user-facing functionality size: L 500-999 LOC labels May 27, 2026

henrycgbaker marked this pull request as draft May 28, 2026 07:32

henrycgbaker requested a review from saschagobel June 4, 2026 15:09

henrycgbaker marked this pull request as ready for review June 4, 2026 15:09

henrycgbaker force-pushed the feat/annotation-per-item-calibration/03-partition branch from 9e37d80 to fb3d9d3 Compare June 4, 2026 19:35

henrycgbaker added 6 commits June 17, 2026 09:14

docs(annotation): rename ADR-0012 to include annotation- prefix

141cf84

Aligns with the existing naming convention (0010-annotation-..., 0011-annotation-...). References updated in decisions/README.md and design/annotation-import-pipeline.md.

henrycgbaker force-pushed the feat/annotation-per-item-calibration/02-adr branch from 32e5be7 to 77294a7 Compare June 17, 2026 09:23

henrycgbaker added 3 commits June 17, 2026 09:29

henrycgbaker force-pushed the feat/annotation-per-item-calibration/03-partition branch from fb3d9d3 to bc07cba Compare June 17, 2026 09:30

Base automatically changed from feat/annotation-per-item-calibration/02-adr to main June 17, 2026 10:45

henrycgbaker and others added 2 commits June 17, 2026 12:12

Merge branch 'main' into feat/annotation-per-item-calibration/03-part…

c7a06b1

…ition

saschagobel requested changes Jun 17, 2026

View reviewed changes

henrycgbaker added 4 commits June 18, 2026 11:37

refactor(annotation): move partition summary logic into core

c64f8bf

fix(annotation): range-validate inherited calibration_fraction

5dd78c0

test(annotation): cover multi-chunk retrieval totals in calibration

8e61ef1

test(annotation): assert per-chunk retrieval bucket routing

f35c3f0

github-actions Bot added size: XL 1000+ LOC and removed size: L 500-999 LOC labels Jun 18, 2026

henrycgbaker added 2 commits June 18, 2026 12:02

saschagobel self-requested a review June 19, 2026 12:03

saschagobel approved these changes Jun 19, 2026

View reviewed changes

henrycgbaker and others added 2 commits June 19, 2026 12:25

fix(annotation): narrow chunk_id in backfill retrieval branch for mypy

6cdf915

Merge branch 'main' into feat/annotation-per-item-calibration/03-part…

5b0b98f

…ition

henrycgbaker requested a review from saschagobel June 19, 2026 12:28

saschagobel approved these changes Jun 19, 2026

View reviewed changes

saschagobel merged commit f589b05 into main Jun 19, 2026
4 checks passed

saschagobel deleted the feat/annotation-per-item-calibration/03-partition branch June 19, 2026 12:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(annotation): per-item PartitionManifestEntry + per-item calibration assignment#227

feat(annotation): per-item PartitionManifestEntry + per-item calibration assignment#227
saschagobel merged 19 commits into
mainfrom
feat/annotation-per-item-calibration/03-partition

henrycgbaker commented May 27, 2026 •

edited

Loading

Uh oh!

saschagobel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saschagobel left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

henrycgbaker commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scope

Schema

Partition logic

Settings cascade

API / CLI

Tests

Breaking change

Why

Test plan

Uh oh!

saschagobel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saschagobel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

henrycgbaker commented May 27, 2026 •

edited

Loading