Skip to content

fix(annotation): decouple partition_scope from dataset_id#234

Draft
henrycgbaker wants to merge 2 commits into
demo-2026-05-26from
fix/partition-scope-decoupling
Draft

fix(annotation): decouple partition_scope from dataset_id#234
henrycgbaker wants to merge 2 commits into
demo-2026-05-26from
fix/partition-scope-decoupling

Conversation

@henrycgbaker

@henrycgbaker henrycgbaker commented May 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Introduces partition_scope: SafePathSegment on AnnotationSettings, defaulting to dataset_id (for backwards compatibility, but also as a sensible default)
  • resolve_import_paths() and load_partition_manifest() now key off partition_scope; dataset_name() (Argilla naming) continues using dataset_id
  • PartitionManifest.dataset_id renamed to partition_scope in Python, serialised as "dataset_id" via Pydantic alias — existing on-disk manifests load without migration
  • import_records() API gains an optional partition_scope kwarg

Problem fixed

Two annotation configs sharing dataset_id: "" would share one partition manifest, so the first config to hit calibration_max_records would exhaust the cap for all subsequent imports in the same run — even across unrelated domains with different workspaces.

Backwards compat / future palns

  • No changes required for existing callers. When partition_scope is not set, it defaults to dataset_id, preserving the exact previous behaviour. Existing partition.meta.json files on disk are unaffected (JSON key stays "dataset_id").
  • Plan to refactor how datasets can be named with dataset_id (currently, has to be a suffix, but this is needless constraint now we have partition_scope)

Test plan

  • 807 unit tests pass (uv run pytest tests/unit/)
  • New integration regression test TestPartitionScopeDecoupling::test_independent_scopes_each_get_calibration — two configs with dataset_id: "", distinct partition_scope, calibration_max_records: 30; asserts scope B gets > 0 calibration records after scope A hits the cap
  • Existing integration tests (tests/integration/test_annotation_calibration.py) unchanged in behaviour

Introduce partition_scope on AnnotationSettings (defaults to dataset_id
for backwards compat). resolve_import_paths and load_partition_manifest
key off partition_scope; dataset_name() continues using dataset_id.

Fixes calibration cap leaking across domains that share an empty
dataset_id. Users can now set dataset_id: "" with distinct partition_scope
values to get clean Argilla names and independent calibration budgets.

PartitionManifest.dataset_id is renamed to partition_scope in Python but
serialised as "dataset_id" via Pydantic alias so existing manifests
load without migration.
@github-actions github-actions Bot added fix Corrects broken or unintended behavior size: S 50-199 LOC labels May 27, 2026
- Move _default_partition_scope validator out of field block into the
  validators section, consistent with all other validators in the class
- Construct PartitionManifest via canonical partition_scope= kwarg rather
  than the dataset_id alias (completes the rename)
- Round-trip test uses model_dump_json(by_alias=True) to verify the
  actual on-disk wire format, not just model symmetry
- Regression test checks both RETRIEVAL and GROUNDING calibration counts
@henrycgbaker

Copy link
Copy Markdown
Collaborator Author

Three things deliberately not fixed; capturing for future reference.

  1. partition_scope: "" is indistinguishable from "not set" if not self.partition_scope treats an explicit empty string the same as an omitted field. A user who writes partition_scope: "" in YAML while having dataset_id: "DEM" will silently get partition_scope="DEM". Fixing this properly requires a None sentinel (SafePathSegment | None = None), which is a wider type change. Left as-is since an explicitly empty partition_scope with a non-empty dataset_id has no meaningful use case today.

  2. Defaulting is one-way: partition_scope ← dataset_id, not vice versa:
    If a caller sets partition_scope="proj" without setting dataset_id, Argilla dataset names are unscoped (retrieval_production, no suffix). Intentional — the whole point of the decoupling — but a sharp edge. API docstring on import_records warns about this.

  3. export_annotations() and teardown() don't expose partition_scope:
    Neither touches the partition manifest today, so no operational need. If either ever needs to correlate with manifest state, the kwarg will need to be added at that point.

@henrycgbaker henrycgbaker marked this pull request as draft May 28, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Corrects broken or unintended behavior size: S 50-199 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant