feat(annotation): calibration_max_records absolute cap + inheritance cascade#229
Draft
henrycgbaker wants to merge 6 commits into
Draft
feat(annotation): calibration_max_records absolute cap + inheritance cascade#229henrycgbaker wants to merge 6 commits into
henrycgbaker wants to merge 6 commits into
Conversation
github-actions
Bot
added
feature
This was referenced May 27, 2026
henrycgbaker
added
annotation
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/04-hardening
branch
from
4c2967c to
8ea2b98
Compare
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/05-cap
branch
from
6582de7 to
56b4439
Compare
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/04-hardening
branch
from
8ea2b98 to
df99374
Compare
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/05-cap
branch
from
56b4439 to
4a4f5fd
Compare
saschagobel
pushed a commit
that referenced
this pull request
Jun 17, 2026
**Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (1/4):** Lands the ADR and the matching design-doc section before the implementation, so subsequent PRs can reference the decision and the schema shape. --- ## Scope - `docs/decisions/0012-per-item-calibration-partition.md` (new): the decision, the option-set considered, Kish design-effect arithmetic justifying per-item over per-record under the existing naive Krippendorff bootstrap, strategic positioning vs LLM-judge mainstream + industry annotation services, consequences, out-of-scope future work, and references. - `docs/decisions/README.md`: index entry for the new ADR. - `docs/design/annotation-import-pipeline.md`: new "Per-record manifest schema" section describing the `PartitionManifestEntry` shape that #227 implements (`grounding_generation_calibration` + `retrieval_chunk_calibration`). ## Test plan - [x] Docs only - no test changes.
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/04-hardening
branch
from
df99374 to
c2795c2
Compare
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/05-cap
branch
from
4a4f5fd to
8161e15
Compare
saschagobel
pushed a commit
that referenced
this pull request
Jun 19, 2026
…ion assignment (#227) **Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (2/4):** Per-item partition logic + schema reshape + per-task fraction cascade. Ships the user-visible feature without the absolute cap (#229) or the API-hardening pass (#228). --- ## Scope ### Schema - `core/schemas/annotation_import.py`: `PartitionManifestEntry` replaces `calibration: bool` with `grounding_generation_calibration: dict[Task, bool]` (keyed by task) and `retrieval_chunk_calibration: dict[str, bool]` (keyed by `chunk_id`). Per-task `calibration_fraction_at_import` provenance. ### Partition logic - `core/annotation/record_builder.py`: `assign_partitions` buckets units via per-(task, unit) digest `hash(seed || task || unit_id)` against `fraction * 2^32` so per-task draws are statistically independent. `_enumerate_units` resolves the per-task unit list, `_write_unit` routes the bool back into the right per-record dict. - Returns `dict[str, PartitionManifestEntry]` as before; the `PartitionResult` bundle is extracted in #228. ### Settings cascade - `core/settings/annotation_settings.py`: `calibration_fraction` is now inheritable across deployment / workspace / task scopes via the `Inherit` sentinel pattern established in #206. `_check_calibration_topology` walks per-(ws, task) using resolved values. The `_inherit` walk is hoisted into a single helper so `resolved_task` no longer repeats the chain four times. ### API / CLI - `api/annotation_import.py`: `ImportResult` per-task dicts (`calibration_count`, `production_count`, `calibration_fraction`, `realised_calibration_fraction`) replace the scalar fields. - `cli/commands/annotation.py`: CLI output prints one line per task. ### Tests - Per-chunk routing independence, per-task seed independence, inheritance cascade, and per-(ws, task) topology validation. Manifest IO round-trip with the new shape. ## Breaking change Pre-v0 on-disk manifests with `calibration: bool` no longer load. Pragmata is pre-1.0; affected workspaces re-bootstrap by deleting `partition.meta.json` and re-importing. ## Why Per ADR-0012 (#226): per-item partition is the natural fit for the existing naive Krippendorff bootstrap and matches Argilla's per-item record creation. Per-task fraction cascade lets operators right-size each task independently (e.g. retrieval at 5%, grounding at 20%). ## Test plan - [x] `uv run python -m pytest tests/unit` (770 passing in this branch's worktree)
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/04-hardening
branch
from
c2795c2 to
f1aebdd
Compare
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/05-cap
branch
from
8161e15 to
be8fa2e
Compare
saschagobel
pushed a commit
that referenced
this pull request
Jun 30, 2026
… helpers (#228) **Stack (4 PRs):** #226 → #227 → #228 → #229 (companion to standalone IAA fix #225) **Chain goal:** Switch calibration vs production assignment from per-`record_uuid` to per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap. **This PR (3/4):** Pure refactor with no behaviour change. Tightens the api / core seam introduced in #227 so per-task fraction reporting and per-pair `record_uuid` derivation are each computed exactly once. --- ## Scope - `core/annotation/record_builder.py`: introduce `PartitionResult(assignments, pairs_by_rid, calibration_fraction)`. `assign_partitions` now returns this bundle; `fan_out_records(client, settings, *, partition)` consumes it and `_build_batches(pairs_by_rid, assignments)` iterates the rid map instead of recomputing `derive_record_uuid(pair)` per pair. - `core/annotation/record_builder.py`: replace `count_calibration_per_task` (returning `dict[Task, int]`) with `count_units_per_task` returning a `TaskUnitCounts(calibration, total)` frozen dataclass so the api layer gets both tallies in a single pass. - `api/annotation_import.py`: drop `_total_units_per_task` and `_resolve_per_task_fraction`. The api now reads `partition.calibration_fraction` directly and uses the merged `count_units_per_task` helper. ## Why After #227 lands, the api layer recomputes per-task fraction via its own workspace walk (`_resolve_per_task_fraction`), and `_build_batches` recomputes `record_uuid` for every pair even though `assign_partitions` already derived it. Both are wasted work; the natural fix is to surface the precomputed values via a bundle return type. This also unblocks the cap-related fields in #229 - `PartitionResult` is the obvious place to carry per-task cap resolution. Reviewer sanity check: assertions in `test_partition.py` move from `result[rid]` to `partition.assignments[rid]`; the values they check are unchanged. ## Test plan - [x] `uv run python -m pytest tests/unit` (770 passing)
Base automatically changed from
feat/annotation-per-item-calibration/04-hardening
to
main
June 30, 2026 11:28
…cascade Adds a per-task absolute cap on calibration items, inheritable across deployment / workspace / task scopes alongside the existing ``calibration_fraction``. Smaller of (fraction × N_items, cap) wins. Existing manifest entries are never demoted (the cap binds only on new items at the current import). - core/settings: ``calibration_max_records: PositiveInt | None | Inherit`` on TaskSettings + WorkspaceSettings, ``PositiveInt | None = None`` at deployment scope. ResolvedTaskSettings gets the resolved value. - core/schemas: ``calibration_max_records_at_import: dict[Task, int | None]`` on PartitionManifestEntry as per-task provenance, with range validation. - core/annotation/record_builder: assign_partitions now collects eligible candidates per task, sorts by digest, and truncates to ``remaining = max(0, cap - existing_cal)`` so cap-bound runs pick the lowest-digest units deterministically. PartitionResult carries the resolved cap dict. - api/annotation_import: ``calibration_max_records`` kwarg + ImportResult field. CLI prints ``cap=N`` per task when set. - cli/commands/annotation: ``--calibration-max-records`` flag with ``--no-calibration`` conflict check. - Unit tests: cap binds, cap=0 effective via existing-over-cap warn-path, per-task cap provenance, order-dependence under binding cap pinned as a documented property. Settings cascade tests for the new field. - Integration test: cap binds against live Argilla (records actually land in the correct buckets at the configured cap).
Replace the hand-rolled cap-range loop in _check_fraction_range with a declarative dict[Task, Annotated[int, Field(ge=1)] | None] on calibration_max_records_at_import, matching the Annotated-constraint idiom. The fraction-range check is left in place; it belongs to the per-item partition PR lower in the stack.
…ax_items Cap unit is the annotation item (chunks for retrieval, records for grounding/generation), not the record. Rename propagated through all Python sources: settings models, record_builder, API, CLI flag (--calibration-max-items), and all tests.
henrycgbaker
force-pushed
the
feat/annotation-per-item-calibration/05-cap
branch
from
be8fa2e to
7b1572b
Compare
… scopes Two small hardening fixes uncovered by audit on the per-item partition feature stack: - core/annotation/record_builder: ``fan_out_records`` now raises ``RuntimeError`` when the partition manifest assigns records to a task absent from the current workspaces topology. Previously it warned and continued, silently dropping data on disk if an operator reshaped their workspaces between imports. - core/settings/annotation_settings: workspace- and task-scoped ``calibration_fraction`` now constrained to [0.0, 1.0] via a shared ``CalibrationFraction = Annotated[float, Field(ge=0.0, le=1.0)]`` alias. The deployment-scope guard already had this range check; the new alias closes the gap so a YAML setting of `workspaces.x.calibration_fraction: 1.5` is rejected at load. - api/annotation_import: clarify the manifest-write-ordering comment - a retry under a binding cap is order-dependent (the cap-vs-existing logic is documented in ``test_cap_under_split_imports_is_order_dependent_by_design``). - tests: ``test_records_for_task_absent_from_topology_raises`` replaces the prior warn-and-skip test. Locale-mismatch test gets a complete 3-task topology so the topology guard isn't tripped incidentally. - Integration: ``test_per_chunk_retrieval_routing_to_argilla`` verifies per-chunk independence end-to-end against a live Argilla dataset.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (4 PRs): #226 → #227 → #228 → #229 (companion to standalone IAA fix #225)
Chain goal: Switch calibration vs production assignment from per-
record_uuidto per annotation item (chunk for retrieval, record for grounding / generation), with inheritable per-task fraction and absolute cap.This PR (4/4): Adds the absolute cap (
calibration_max_records) as a sibling knob tocalibration_fraction, inheritable across the same scopes. Plus the matching CLI flag, api kwarg,ImportResultfield, and integration coverage.Scope
Settings cascade
core/settings/annotation_settings.py:calibration_max_records: PositiveInt | None | InheritonTaskSettings+WorkspaceSettings,PositiveInt | None = Noneat deployment scope.ResolvedTaskSettingsgets the resolved value.at("calibration_max_records")line added toresolved_task.Schema provenance
core/schemas/annotation_import.py:calibration_max_records_at_import: dict[Task, int | None]field onPartitionManifestEntrywith range validation.Partition logic
core/annotation/record_builder.py:assign_partitionscollects eligible candidates per task, sorts by digest, and truncates toremaining = max(0, cap - existing_cal)so cap-bound runs pick the lowest-digest units deterministically.PartitionResultgainscalibration_max_records: dict[Task, int | None]. Smaller of (fraction × N_items, cap) wins.API / CLI
api/annotation_import.py:calibration_max_recordskwarg +ImportResult.calibration_max_recordsfield. CLI summary printscap=Nper task when set.cli/commands/annotation.py:--calibration-max-recordsflag with a--no-calibrationconflict check.Tests
test_calibration_max_records_caps_groundingverifies the cap binds against live Argilla (records actually land in the correct buckets at the configured cap).Why
calibration_fractionalone is awkward at scale: 5% of a 100k-pair corpus is 5000 calibration items, which is far more than any reasonable IAA cohort can submit. An absolute cap lets operators say "at most 200 calibration chunks for retrieval" without having to recompute the fraction every time the corpus grows. Inheritability matches the existing pattern forcalibration_fraction; the cap unit is the annotation item, mirroring the partition granularity from #227.Order-dependence under binding cap
When the cap is binding across multiple imports, the final calibration set is a function of
(corpus, seed, import_order)rather than(corpus, seed)alone. This is a direct consequence of the manifest-lock invariant: once an entry is in calibration, a tightened cap on a later import cannot demote it. Documented in theassign_partitionsdocstring and pinned bytest_cap_under_split_imports_is_order_dependent_by_design.Test plan
uv run python -m pytest tests/unit(770 passing)uv run python -m pytest tests/integration/test_annotation_calibration.py::test_calibration_max_records_caps_grounding -m "integration and annotation"(verified against live Argilla)