fix(annotation): decouple partition_scope from dataset_id#234
fix(annotation): decouple partition_scope from dataset_id#234henrycgbaker wants to merge 2 commits into
Conversation
Introduce partition_scope on AnnotationSettings (defaults to dataset_id for backwards compat). resolve_import_paths and load_partition_manifest key off partition_scope; dataset_name() continues using dataset_id. Fixes calibration cap leaking across domains that share an empty dataset_id. Users can now set dataset_id: "" with distinct partition_scope values to get clean Argilla names and independent calibration budgets. PartitionManifest.dataset_id is renamed to partition_scope in Python but serialised as "dataset_id" via Pydantic alias so existing manifests load without migration.
- Move _default_partition_scope validator out of field block into the validators section, consistent with all other validators in the class - Construct PartitionManifest via canonical partition_scope= kwarg rather than the dataset_id alias (completes the rename) - Round-trip test uses model_dump_json(by_alias=True) to verify the actual on-disk wire format, not just model symmetry - Regression test checks both RETRIEVAL and GROUNDING calibration counts
|
Three things deliberately not fixed; capturing for future reference.
|
Summary
partition_scope: SafePathSegmentonAnnotationSettings, defaulting todataset_id(for backwards compatibility, but also as a sensible default)resolve_import_paths()andload_partition_manifest()now key offpartition_scope;dataset_name()(Argilla naming) continues usingdataset_idPartitionManifest.dataset_idrenamed topartition_scopein Python, serialised as"dataset_id"via Pydantic alias — existing on-disk manifests load without migrationimport_records()API gains an optionalpartition_scopekwargProblem fixed
Two annotation configs sharing
dataset_id: ""would share one partition manifest, so the first config to hitcalibration_max_recordswould exhaust the cap for all subsequent imports in the same run — even across unrelated domains with different workspaces.Backwards compat / future palns
partition_scopeis not set, it defaults todataset_id, preserving the exact previous behaviour. Existingpartition.meta.jsonfiles on disk are unaffected (JSON key stays"dataset_id").Test plan
uv run pytest tests/unit/)TestPartitionScopeDecoupling::test_independent_scopes_each_get_calibration— two configs withdataset_id: "", distinctpartition_scope,calibration_max_records: 30; asserts scope B gets > 0 calibration records after scope A hits the captests/integration/test_annotation_calibration.py) unchanged in behaviour