refactor(annotation): extract PartitionResult bundle + collapse count helpers#228
Merged
saschagobel merged 3 commits intoJun 30, 2026
Merged
Conversation
github-actions
Bot
added
feature
This was referenced May 27, 2026
henrycgbaker
added
annotation
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/03-partition
branch
from
9e37d80 to
fb3d9d3
Compare
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/04-hardening
branch
from
4c2967c to
8ea2b98
Compare
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/03-partition
branch
from
fb3d9d3 to
bc07cba
Compare
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/04-hardening
branch
from
8ea2b98 to
df99374
Compare
saschagobel
pushed a commit
that referenced
this pull request
Jun 17, 2026
**Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (1/4):** Lands the ADR and the matching design-doc section before the implementation, so subsequent PRs can reference the decision and the schema shape. --- ## Scope - `docs/decisions/0012-per-item-calibration-partition.md` (new): the decision, the option-set considered, Kish design-effect arithmetic justifying per-item over per-record under the existing naive Krippendorff bootstrap, strategic positioning vs LLM-judge mainstream + industry annotation services, consequences, out-of-scope future work, and references. - `docs/decisions/README.md`: index entry for the new ADR. - `docs/design/annotation-import-pipeline.md`: new "Per-record manifest schema" section describing the `PartitionManifestEntry` shape that #227 implements (`grounding_generation_calibration` + `retrieval_chunk_calibration`). ## Test plan - [x] Docs only - no test changes.
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/04-hardening
branch
from
df99374 to
c2795c2
Compare
saschagobel
pushed a commit
that referenced
this pull request
Jun 19, 2026
…ion assignment (#227) **Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (2/4):** Per-item partition logic + schema reshape + per-task fraction cascade. Ships the user-visible feature without the absolute cap (#229) or the API-hardening pass (#228). --- ## Scope ### Schema - `core/schemas/annotation_import.py`: `PartitionManifestEntry` replaces `calibration: bool` with `grounding_generation_calibration: dict[Task, bool]` (keyed by task) and `retrieval_chunk_calibration: dict[str, bool]` (keyed by `chunk_id`). Per-task `calibration_fraction_at_import` provenance. ### Partition logic - `core/annotation/record_builder.py`: `assign_partitions` buckets units via per-(task, unit) digest `hash(seed || task || unit_id)` against `fraction * 2^32` so per-task draws are statistically independent. `_enumerate_units` resolves the per-task unit list, `_write_unit` routes the bool back into the right per-record dict. - Returns `dict[str, PartitionManifestEntry]` as before; the `PartitionResult` bundle is extracted in #228. ### Settings cascade - `core/settings/annotation_settings.py`: `calibration_fraction` is now inheritable across deployment / workspace / task scopes via the `Inherit` sentinel pattern established in #206. `_check_calibration_topology` walks per-(ws, task) using resolved values. The `_inherit` walk is hoisted into a single helper so `resolved_task` no longer repeats the chain four times. ### API / CLI - `api/annotation_import.py`: `ImportResult` per-task dicts (`calibration_count`, `production_count`, `calibration_fraction`, `realised_calibration_fraction`) replace the scalar fields. - `cli/commands/annotation.py`: CLI output prints one line per task. ### Tests - Per-chunk routing independence, per-task seed independence, inheritance cascade, and per-(ws, task) topology validation. Manifest IO round-trip with the new shape. ## Breaking change Pre-v0 on-disk manifests with `calibration: bool` no longer load. Pragmata is pre-1.0; affected workspaces re-bootstrap by deleting `partition.meta.json` and re-importing. ## Why Per ADR-0012 (#226): per-item partition is the natural fit for the existing naive Krippendorff bootstrap and matches Argilla's per-item record creation. Per-task fraction cascade lets operators right-size each task independently (e.g. retrieval at 5%, grounding at 20%). ## Test plan - [x] `uv run python -m pytest tests/unit` (770 passing in this branch's worktree)
Base automatically changed from
feat/annotation-per-item-calibration/03-partition
to
main
June 19, 2026 12:29
… helpers Pure refactor with no behaviour change. Tightens the api/core seam introduced in #3 so per-task fraction reporting and per-pair record_uuid derivation are each computed exactly once. - core/annotation/record_builder: introduce `PartitionResult(assignments, pairs_by_rid, calibration_fraction)`. `assign_partitions` now returns this bundle; `fan_out_records(client, settings, *, partition)` consumes it and `_build_batches(pairs_by_rid, assignments)` iterates the rid map instead of recomputing `derive_record_uuid(pair)` per pair. - core/annotation/record_builder: replace `count_calibration_per_task` (returning `dict[Task, int]`) with `count_units_per_task` returning a `TaskUnitCounts(calibration, total)` frozen dataclass so the api layer gets both tallies in a single pass. - api/annotation_import: drop `_total_units_per_task` and `_resolve_per_task_fraction`. The api now reads `partition.calibration_fraction` directly and uses the merged `count_units_per_task` helper. Reviewer can sanity-check the refactor by diffing the test_partition.py test bodies pre and post: assertions move from `result[rid]` to `partition.assignments[rid]` but the values they check are unchanged.
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/04-hardening
branch
from
c2795c2 to
f1aebdd
Compare
saschagobel
approved these changes
Jun 29, 2026
saschagobel
left a comment
saschagobel
left a comment
Collaborator
There was a problem hiding this comment.
The refactor looks behavior-preserving overall; I left one minor question about PartitionResult.calibration_fraction being populated while the API still appears to use summary.configured_fraction.
… summary summarize_partitions took settings and re-resolved the per-task configured fraction via _configured_fraction_per_task, duplicating the resolution assign_partitions already performs. Pass the precomputed PartitionResult.calibration_fraction through instead, making the bundled value a single source of truth and dropping the redundant helper.
saschagobel
approved these changes
Jun 30, 2026
saschagobel
deleted the
feat/annotation-per-item-calibration/04-hardening
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (4 PRs): #226 → #227 → #228 → #229 (companion to standalone IAA fix #225)
Chain goal: Switch calibration vs production assignment from per-
record_uuidto per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap.This PR (3/4): Pure refactor with no behaviour change. Tightens the api / core seam introduced in #227 so per-task fraction reporting and per-pair
record_uuidderivation are each computed exactly once.Scope
core/annotation/record_builder.py: introducePartitionResult(assignments, pairs_by_rid, calibration_fraction).assign_partitionsnow returns this bundle;fan_out_records(client, settings, *, partition)consumes it and_build_batches(pairs_by_rid, assignments)iterates the rid map instead of recomputingderive_record_uuid(pair)per pair.core/annotation/record_builder.py: replacecount_calibration_per_task(returningdict[Task, int]) withcount_units_per_taskreturning aTaskUnitCounts(calibration, total)frozen dataclass so the api layer gets both tallies in a single pass.api/annotation_import.py: drop_total_units_per_taskand_resolve_per_task_fraction. The api now readspartition.calibration_fractiondirectly and uses the mergedcount_units_per_taskhelper.Why
After #227 lands, the api layer recomputes per-task fraction via its own workspace walk (
_resolve_per_task_fraction), and_build_batchesrecomputesrecord_uuidfor every pair even thoughassign_partitionsalready derived it. Both are wasted work; the natural fix is to surface the precomputed values via a bundle return type. This also unblocks the cap-related fields in #229 -PartitionResultis the obvious place to carry per-task cap resolution.Reviewer sanity check: assertions in
test_partition.pymove fromresult[rid]topartition.assignments[rid]; the values they check are unchanged.Test plan
uv run python -m pytest tests/unit(770 passing)