docs(annotation): ADR-0012 per-item calibration partition#226
Merged
Conversation
github-actions
Bot
added
feature
This was referenced May 27, 2026
saschagobel
approved these changes
Jun 16, 2026
saschagobel
left a comment
saschagobel
left a comment
Collaborator
There was a problem hiding this comment.
Approved, thanks. The ADR and design-doc additions capture a real decision clearly; I left only non-blocking comments around keeping the ADR leaner, avoiding duplication with the design doc, and tightening a few naming/scope details.
saschagobel
added
documentation
Document the decision to partition calibration vs production per annotation item (chunk for retrieval, record for grounding/generation) rather than per record_uuid. Implementation lands in follow-up PRs. - New ADR-0012 covering the option-set, statistical rationale (Kish design effect under naive bootstrap), strategic positioning, and out-of-scope future work. - README index entry for the new ADR. - annotation-import-pipeline.md gets a new section describing the per-record manifest schema (grounding_generation_calibration + retrieval_chunk_calibration) referenced by ADR-0012.
Remove PartitionManifestEntry schema block from Decision section - those details belong in the design doc, not the ADR. Trim Out-of-scope section to the one item directly implied by this decision (cluster-aware IAA); other roadmap items moved to backlog.
Merge verbose 'Context - design alternatives' and 'Strategic positioning' sections into a four-bullet 'Rationale' section, preserving the key statistical argument (Kish design effect) and the dependency point (per-record implies cluster-aware IAA work).
…tems Cap unit is the annotation item (chunks for retrieval, records for grounding/generation), not the record. Rename propagated through ADR-0012 and the design doc, including the at-import provenance field name.
Aligns with the existing naming convention (0010-annotation-..., 0011-annotation-...). References updated in decisions/README.md and design/annotation-import-pipeline.md.
…ading Remove reference to test_cap_under_split_imports_is_order_dependent_by_design which does not exist yet; order-dependence is still noted via assign_partitions. Rename 'Per-record manifest schema' heading to 'Partition manifest entry schema' to match the per-item framing throughout the section.
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/02-adr
branch
from
32e5be7 to
77294a7
Compare
saschagobel
approved these changes
Jun 17, 2026
saschagobel
pushed a commit
that referenced
this pull request
Jun 19, 2026
…ion assignment (#227) **Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (2/4):** Per-item partition logic + schema reshape + per-task fraction cascade. Ships the user-visible feature without the absolute cap (#229) or the API-hardening pass (#228). --- ## Scope ### Schema - `core/schemas/annotation_import.py`: `PartitionManifestEntry` replaces `calibration: bool` with `grounding_generation_calibration: dict[Task, bool]` (keyed by task) and `retrieval_chunk_calibration: dict[str, bool]` (keyed by `chunk_id`). Per-task `calibration_fraction_at_import` provenance. ### Partition logic - `core/annotation/record_builder.py`: `assign_partitions` buckets units via per-(task, unit) digest `hash(seed || task || unit_id)` against `fraction * 2^32` so per-task draws are statistically independent. `_enumerate_units` resolves the per-task unit list, `_write_unit` routes the bool back into the right per-record dict. - Returns `dict[str, PartitionManifestEntry]` as before; the `PartitionResult` bundle is extracted in #228. ### Settings cascade - `core/settings/annotation_settings.py`: `calibration_fraction` is now inheritable across deployment / workspace / task scopes via the `Inherit` sentinel pattern established in #206. `_check_calibration_topology` walks per-(ws, task) using resolved values. The `_inherit` walk is hoisted into a single helper so `resolved_task` no longer repeats the chain four times. ### API / CLI - `api/annotation_import.py`: `ImportResult` per-task dicts (`calibration_count`, `production_count`, `calibration_fraction`, `realised_calibration_fraction`) replace the scalar fields. - `cli/commands/annotation.py`: CLI output prints one line per task. ### Tests - Per-chunk routing independence, per-task seed independence, inheritance cascade, and per-(ws, task) topology validation. Manifest IO round-trip with the new shape. ## Breaking change Pre-v0 on-disk manifests with `calibration: bool` no longer load. Pragmata is pre-1.0; affected workspaces re-bootstrap by deleting `partition.meta.json` and re-importing. ## Why Per ADR-0012 (#226): per-item partition is the natural fit for the existing naive Krippendorff bootstrap and matches Argilla's per-item record creation. Per-task fraction cascade lets operators right-size each task independently (e.g. retrieval at 5%, grounding at 20%). ## Test plan - [x] `uv run python -m pytest tests/unit` (770 passing in this branch's worktree)
saschagobel
pushed a commit
that referenced
this pull request
Jun 30, 2026
… helpers (#228) **Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (3/4):** Pure refactor with no behaviour change. Tightens the api / core seam introduced in #227 so per-task fraction reporting and per-pair `record_uuid` derivation are each computed exactly once. --- ## Scope - `core/annotation/record_builder.py`: introduce `PartitionResult(assignments, pairs_by_rid, calibration_fraction)`. `assign_partitions` now returns this bundle; `fan_out_records(client, settings, *, partition)` consumes it and `_build_batches(pairs_by_rid, assignments)` iterates the rid map instead of recomputing `derive_record_uuid(pair)` per pair. - `core/annotation/record_builder.py`: replace `count_calibration_per_task` (returning `dict[Task, int]`) with `count_units_per_task` returning a `TaskUnitCounts(calibration, total)` frozen dataclass so the api layer gets both tallies in a single pass. - `api/annotation_import.py`: drop `_total_units_per_task` and `_resolve_per_task_fraction`. The api now reads `partition.calibration_fraction` directly and uses the merged `count_units_per_task` helper. ## Why After #227 lands, the api layer recomputes per-task fraction via its own workspace walk (`_resolve_per_task_fraction`), and `_build_batches` recomputes `record_uuid` for every pair even though `assign_partitions` already derived it. Both are wasted work; the natural fix is to surface the precomputed values via a bundle return type. This also unblocks the cap-related fields in #229 - `PartitionResult` is the obvious place to carry per-task cap resolution. Reviewer sanity check: assertions in `test_partition.py` move from `result[rid]` to `partition.assignments[rid]`; the values they check are unchanged. ## Test plan - [x] `uv run python -m pytest tests/unit` (770 passing)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (4 PRs): #226 → #227 → #228 → #229 (companion to standalone IAA fix #225)
Chain goal: Switch calibration vs production assignment from per-
record_uuidto per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap.This PR (1/4): Lands the ADR and the matching design-doc section before the implementation, so subsequent PRs can reference the decision and the schema shape.
Scope
docs/decisions/0012-per-item-calibration-partition.md(new): the decision, the option-set considered, Kish design-effect arithmetic justifying per-item over per-record under the existing naive Krippendorff bootstrap, strategic positioning vs LLM-judge mainstream + industry annotation services, consequences, out-of-scope future work, and references.docs/decisions/README.md: index entry for the new ADR.docs/design/annotation-import-pipeline.md: new "Per-record manifest schema" section describing thePartitionManifestEntryshape that feat(annotation): per-item PartitionManifestEntry + per-item calibration assignment #227 implements (grounding_generation_calibration+retrieval_chunk_calibration).Test plan