diff --git a/CHANGELOG.d/dichotomous-measurement-policy.md b/CHANGELOG.d/dichotomous-measurement-policy.md new file mode 100644 index 000000000..0743e9a09 --- /dev/null +++ b/CHANGELOG.d/dichotomous-measurement-policy.md @@ -0,0 +1,3 @@ +## Added + +- Added a LineageWeave-owned dichotomous measurement-policy contract that keeps missing/abstain/adjudication states outside 0/1 responses, keeps Rasch distinct from generic one-parameter logistic IRT, exposes only `rasch`, `irt_2plm`, `irt_3plm`, and `irt_4plm` as normal production model families, does not infer a family from domain labels or hand-authored flags, and keeps draft/pilot observations scoreless until activation evidence supports publication. diff --git a/docs/adr/0301-dichotomous-measurement-policy.md b/docs/adr/0301-dichotomous-measurement-policy.md new file mode 100644 index 000000000..705421dcd --- /dev/null +++ b/docs/adr/0301-dichotomous-measurement-policy.md @@ -0,0 +1,49 @@ +# ADR 0301: Govern dichotomous measurement policy at the LineageWeave boundary + +- Status: Proposed +- Date: 2026-09-01 +- Depends on: ADR 0300 (`contextual-orchestrator` ownership boundary) + +## Context + +LineageWeave owns instrument, rubric, evidence-binding, pilot lifecycle, and buyer-facing interpretation policy. It does not own reusable psychometric numerical estimation or model-provider orchestration. New importance/significance/actionability/evidence-style instruments need a versioned observation contract before an administrator can pilot or publish them. + +Collapsing missingness or adjudication into a numeric response corrupts the measurement channel. Collapsing Rasch into a generic one-parameter logistic label also erases the model's measurement-theoretic requirements. Conversely, choosing 3PLM or 4PLM merely because they fit better statistically over-parameterizes the instrument without a substantive asymptote mechanism or recovery evidence. + +## Decision + +1. New evaluative instruments use dichotomous observations by default. `0` means the versioned not-supported criterion is met; `1` means the versioned supported criterion is met. Missing, not-observable, abstain, invalid-evidence, and adjudication-required states remain outside the binary response. +2. The normal production model-family identifiers are `rasch`, `irt_2plm`, `irt_3plm`, and `irt_4plm`. Rasch is not an alias for generic 1PL logistic IRT. Generic `irt_1pl_logistic` is not exposed without a future superseding scientific ADR. +3. LineageWeave does not infer a model family from a product domain, item label, or hand-authored mechanism flags. Model selection remains unavailable until an administrator explicitly binds a family to evidence from the intended use and the owning psychometric runtime validates the corresponding recovery contract. +4. Draft and pilot instruments may preserve observations without a latent scoring model. A published instrument must bind a model family and an activation-evidence reference. Insufficient evidence preserves observations without issuing a latent score. +5. This repository's policy contract performs no numerical estimation. fast-mlsirm owns reusable psychometric kernels and recovery diagnostics; TEPP owns temporal/event/multilevel measurement semantics; contextual-orchestrator owns all LLM model/provider routing and judge orchestration. + +## Activation evidence + +Operational scoring is fail-closed. The administrator must bind evidence appropriate to the intended use, including known-truth or controlled-data recovery, bias and MAE/RMSE, interval coverage, convergence, dimensionality/local dependence, linking/anchors, DIF/invariance, and judge-facet recovery when LLM raters are used. 3PLM additionally requires lower-asymptote recovery/identification evidence. 4PLM additionally requires lower- and upper-asymptote recovery/identification and boundary-behavior evidence. + +## Consequences + +- Existing ordinal observations are not mechanically dichotomized; they retain their existing versioned contract. +- Paired-comparison/ranking outcomes remain a separate Bradley-Terry/Thurstone-style dichotomous channel. +- LLM-as-a-Judge observations remain rater/method observations with provenance, never ground truth. +- The administrator workbench can build on a stable policy vocabulary without embedding estimation code or provider logic in LineageWeave. +- A future model-family change or activation rule change requires an explicit versioned policy/ADR update rather than a silent enum or UI relabel. + +## Verification + +`tests/test_measurement_policy.py` protects binary response semantics, missing/abstain separation, Rasch/2PLM/3PLM/4PLM identifiers, and fail-closed publication activation. `tests/test_ddd_architecture_fitness.py` separately rejects Rasch↔1PL shorthand in production runtime vocabulary. + +## References (APA 7th) + +American Educational Research Association, American Psychological +Association, & National Council on Measurement in Education. (2014). +*Standards for educational and psychological testing*. American +Educational Research Association. + +Birnbaum, A. (1968). Some latent trait models and their use in inferring an +examinee's ability. In F. M. Lord & M. R. Novick, *Statistical theories of +mental test scores* (pp. 397–479). Addison-Wesley. + +Lord, F. M. (1980). *Applications of item response theory to practical +testing problems*. Lawrence Erlbaum Associates. diff --git a/docs/ubiquitous-language.md b/docs/ubiquitous-language.md index cdc136b7d..ef300ab5a 100644 --- a/docs/ubiquitous-language.md +++ b/docs/ubiquitous-language.md @@ -43,7 +43,7 @@ The governed transition from pilot to operational scoring after the instrument's A Rasch-family measurement model with its own measurement-theoretic requirements, including the intended invariance/specific-objectivity interpretation, common discrimination as part of the model, targeting, and Rasch-specific fit expectations. `rasch` is never an alias, label, or shorthand for generic one-parameter logistic IRT. **2PLM (`irt_2plm`)** -A logistic IRT family allowing item discrimination to vary. It is the default logistic IRT candidate for psychology/SEM-lineage measurement when varying discrimination is substantively allowed. +A logistic IRT family allowing item discrimination to vary. It is selected only through explicit intended-use evidence and the owning psychometric runtime's recovery contract; LineageWeave does not infer it from a domain label. **3PLM (`irt_3plm`)** A dichotomous IRT family with a lower asymptote. It is considered only when a substantive lower-asymptote/guessing mechanism is justified and identifiable; better likelihood alone is insufficient. diff --git a/lineageweave/measurement_policy.py b/lineageweave/measurement_policy.py new file mode 100644 index 000000000..6c722a4b6 --- /dev/null +++ b/lineageweave/measurement_policy.py @@ -0,0 +1,169 @@ +"""Version-stable LineageWeave measurement-policy vocabulary. + +This module owns product policy only. It deliberately contains no IRT fitting, +parameter estimation, judge routing, provider selection, or temporal analysis. +Those responsibilities stay with fast-mlsirm, contextual-orchestrator, and TEPP. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import StrEnum + + +class MeasurementModelFamily(StrEnum): + """Production model families allowed by the current measurement policy. + + Rasch is intentionally a distinct family identifier. Generic one-parameter + logistic IRT is not exposed as a normal production choice. + """ + + RASCH = "rasch" + IRT_2PLM = "irt_2plm" + IRT_3PLM = "irt_3plm" + IRT_4PLM = "irt_4plm" + + +class InstrumentLifecycle(StrEnum): + """Governed lifecycle of a versioned measurement instrument.""" + + DRAFT = "draft" + PILOT = "pilot" + PUBLISHED = "published" + RETIRED = "retired" + + +class DichotomousObservationState(StrEnum): + """State of one rubric-governed dichotomous observation.""" + + OBSERVED = "observed" + MISSING = "missing" + NOT_OBSERVABLE = "not_observable" + ABSTAIN = "abstain" + INVALID_EVIDENCE = "invalid_evidence" + ADJUDICATION_REQUIRED = "adjudication_required" + + +@dataclass(frozen=True, slots=True) +class DichotomousItemPolicy: + """Versioned rubric that gives the binary response an explicit meaning.""" + + item_id: str + rubric_version: str + not_supported_criterion: str + supported_criterion: str + + def __post_init__(self) -> None: + values = ( + self.item_id, + self.rubric_version, + self.not_supported_criterion, + self.supported_criterion, + ) + if any(not isinstance(value, str) for value in values): + raise TypeError("dichotomous item policy fields must be strings") + if any(not value.strip() for value in values): + raise ValueError("dichotomous item policy fields must be non-empty") + for field_name, value in ( + ("item_id", self.item_id), + ("rubric_version", self.rubric_version), + ): + if value != value.strip(): + raise ValueError(f"{field_name} must not contain surrounding whitespace") + if self.not_supported_criterion.strip() == self.supported_criterion.strip(): + raise ValueError("0 and 1 rubric criteria must be distinct") + + +@dataclass(frozen=True, slots=True) +class InstrumentMeasurementPolicy: + """Scoring activation contract for one immutable instrument revision. + + Draft and pilot instruments may preserve observations without a latent + model. A published instrument must bind both a governed model family and an + activation-evidence reference; otherwise operational latent scoring remains + unavailable. Adapters must parse lifecycle and model-family strings into the + governed enums before constructing this domain object; raw strings cannot + bypass lifecycle-specific activation rules. Identity and evidence references + likewise must be parsed to text before construction instead of relying on + incidental string-method failures. + """ + + instrument_id: str + revision: int + lifecycle: InstrumentLifecycle + model_family: MeasurementModelFamily | None + activation_evidence_ref: str | None + + def __post_init__(self) -> None: + if not isinstance(self.lifecycle, InstrumentLifecycle): + raise TypeError("lifecycle must be an InstrumentLifecycle") + if self.model_family is not None and not isinstance( + self.model_family, MeasurementModelFamily + ): + raise TypeError("model_family must be a MeasurementModelFamily when supplied") + if not isinstance(self.instrument_id, str): + raise TypeError("instrument_id must be a string") + if not self.instrument_id.strip(): + raise ValueError("instrument_id must be non-empty") + if self.instrument_id != self.instrument_id.strip(): + raise ValueError("instrument_id must not contain surrounding whitespace") + if type(self.revision) is not int: + raise TypeError("instrument revision must be an integer") + if self.revision < 1: + raise ValueError("instrument revision must be a positive integer") + if self.activation_evidence_ref is not None: + if not isinstance(self.activation_evidence_ref, str): + raise TypeError("activation_evidence_ref must be a string when supplied") + if not self.activation_evidence_ref.strip(): + raise ValueError("activation evidence reference must be non-empty when supplied") + if self.activation_evidence_ref != self.activation_evidence_ref.strip(): + raise ValueError( + "activation_evidence_ref must not contain surrounding whitespace" + ) + if self.lifecycle is InstrumentLifecycle.PUBLISHED: + if self.model_family is None: + raise ValueError("published instrument requires a measurement model family") + if self.activation_evidence_ref is None: + raise ValueError("published instrument requires activation evidence") + + +@dataclass(frozen=True, slots=True) +class DichotomousObservation: + """One 0/1 criterion observation or an explicitly unscored state. + + ``response=0`` means the versioned support criterion was not satisfied and + ``response=1`` means it was satisfied. A response is never used to encode + missingness, abstention, invalid evidence, or an unresolved adjudication. + Adapters must parse the observation-state enum before construction so a raw + string cannot silently enter the governed binary-response domain. + """ + + state: DichotomousObservationState + response: int | None + + def __post_init__(self) -> None: + if not isinstance(self.state, DichotomousObservationState): + raise TypeError("state must be a DichotomousObservationState") + if self.response is not None and type(self.response) is not int: + raise TypeError("response must be an integer or None") + if self.state is DichotomousObservationState.OBSERVED: + if self.response not in (0, 1): + raise ValueError("observed dichotomous response must be integer 0 or 1") + return + if self.response is not None: + raise ValueError("unscored dichotomous states must not carry a 0/1 response") + + @classmethod + def observed(cls, response: int) -> "DichotomousObservation": + """Create an observed 0/1 response under the instrument's rubric.""" + return cls(DichotomousObservationState.OBSERVED, response) + + @classmethod + def unscored( + cls, + state: DichotomousObservationState, + ) -> "DichotomousObservation": + """Create a non-response state that remains outside the binary channel.""" + if state is DichotomousObservationState.OBSERVED: + raise ValueError("observed state requires an explicit 0 or 1 response") + return cls(state, None) diff --git a/tests/test_measurement_policy.py b/tests/test_measurement_policy.py new file mode 100644 index 000000000..1feea99fd --- /dev/null +++ b/tests/test_measurement_policy.py @@ -0,0 +1,146 @@ +"""Contracts for LineageWeave-owned measurement policy, not numerical estimation.""" + +from __future__ import annotations + +import pytest + +from lineageweave.measurement_policy import ( + DichotomousItemPolicy, + DichotomousObservation, + DichotomousObservationState, + InstrumentLifecycle, + InstrumentMeasurementPolicy, + MeasurementModelFamily, +) + + +def test_production_model_families_keep_rasch_distinct_from_generic_1pl() -> None: + """Public model identifiers expose the governed families and no generic 1PL.""" + assert {family.value for family in MeasurementModelFamily} == { + "rasch", + "irt_2plm", + "irt_3plm", + "irt_4plm", + } + + +def test_observed_dichotomous_response_accepts_only_zero_or_one() -> None: + """An observed criterion result is binary; missing-like states are not scores.""" + assert DichotomousObservation.observed(0).response == 0 + assert DichotomousObservation.observed(1).response == 1 + + for invalid in (-1, 2): + with pytest.raises(ValueError, match="0 or 1"): + DichotomousObservation.observed(invalid) + + +def test_nonobserved_states_never_carry_a_binary_response() -> None: + """Missing, abstain and adjudication states remain outside the 0/1 channel.""" + for state in ( + DichotomousObservationState.MISSING, + DichotomousObservationState.NOT_OBSERVABLE, + DichotomousObservationState.ABSTAIN, + DichotomousObservationState.INVALID_EVIDENCE, + DichotomousObservationState.ADJUDICATION_REQUIRED, + ): + observation = DichotomousObservation.unscored(state) + assert observation.response is None + assert observation.state is state + + with pytest.raises(ValueError, match="observed state"): + DichotomousObservation.unscored(DichotomousObservationState.OBSERVED) + + +def test_item_policy_names_both_sides_of_the_binary_rubric() -> None: + """Zero and one encode explicit rubric criteria instead of score compression.""" + policy = DichotomousItemPolicy( + item_id="importance-evidence-1", + rubric_version="2026-09-01", + not_supported_criterion="Evidence does not establish the stated criterion.", + supported_criterion="Evidence establishes the stated criterion.", + ) + assert policy.not_supported_criterion != policy.supported_criterion + + with pytest.raises(ValueError, match="distinct"): + DichotomousItemPolicy( + item_id="importance-evidence-1", + rubric_version="2026-09-01", + not_supported_criterion="same", + supported_criterion="same", + ) + + with pytest.raises(ValueError, match="non-empty"): + DichotomousItemPolicy(" ", "v1", "not supported", "supported") + + +def test_pilot_instrument_may_preserve_observations_without_a_latent_model() -> None: + """Pilot data are valid evidence even when no scoring model is defensible yet.""" + policy = InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=1, + lifecycle=InstrumentLifecycle.PILOT, + model_family=None, + activation_evidence_ref=None, + ) + assert policy.model_family is None + + +def test_published_instrument_requires_model_and_activation_evidence() -> None: + """Operational scoring remains unavailable until model and evidence are bound.""" + with pytest.raises(ValueError, match="activation evidence"): + InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=1, + lifecycle=InstrumentLifecycle.PUBLISHED, + model_family=MeasurementModelFamily.IRT_2PLM, + activation_evidence_ref=None, + ) + + with pytest.raises(ValueError, match="model family"): + InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=1, + lifecycle=InstrumentLifecycle.PUBLISHED, + model_family=None, + activation_evidence_ref="evidence://pilot/2026-09", + ) + + policy = InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=1, + lifecycle=InstrumentLifecycle.PUBLISHED, + model_family=MeasurementModelFamily.IRT_2PLM, + activation_evidence_ref="evidence://pilot/2026-09", + ) + assert policy.lifecycle is InstrumentLifecycle.PUBLISHED + + +@pytest.mark.parametrize( + ("instrument_id", "revision", "activation_evidence_ref", "message"), + [ + (" ", 1, None, "instrument_id"), + ("instrument", 0, None, "positive integer"), + ("instrument", 1, " ", "non-empty"), + ], +) +def test_instrument_identity_revision_and_evidence_are_bounded( + instrument_id: str, + revision: int, + activation_evidence_ref: str | None, + message: str, +) -> None: + """Invalid identity, revision, and evidence values fail before activation.""" + with pytest.raises(ValueError, match=message): + InstrumentMeasurementPolicy( + instrument_id=instrument_id, + revision=revision, + lifecycle=InstrumentLifecycle.PILOT, + model_family=None, + activation_evidence_ref=activation_evidence_ref, + ) + + +def test_unscored_state_rejects_a_binary_payload_at_the_constructor_boundary() -> None: + """Direct construction cannot bypass the unscored-state invariant.""" + with pytest.raises(ValueError, match="must not carry"): + DichotomousObservation(DichotomousObservationState.MISSING, 0) diff --git a/tests/test_measurement_policy_canonical_text.py b/tests/test_measurement_policy_canonical_text.py new file mode 100644 index 000000000..9c5970f17 --- /dev/null +++ b/tests/test_measurement_policy_canonical_text.py @@ -0,0 +1,50 @@ +"""Canonical-text boundaries for LineageWeave-owned measurement policy.""" + +from __future__ import annotations + +import pytest + +from lineageweave.measurement_policy import ( + DichotomousItemPolicy, + InstrumentLifecycle, + InstrumentMeasurementPolicy, +) + + +@pytest.mark.parametrize("field_name", ["item_id", "rubric_version"]) +def test_item_policy_rejects_surrounding_whitespace_in_identity_fields( + field_name: str, +) -> None: + """Opaque item and rubric identities must not acquire whitespace aliases.""" + values = { + "item_id": "item-1", + "rubric_version": "2026-09-01", + "not_supported_criterion": "Evidence does not support the criterion.", + "supported_criterion": "Evidence supports the criterion.", + } + values[field_name] = f" {values[field_name]} " + + with pytest.raises(ValueError, match="surrounding whitespace"): + DichotomousItemPolicy(**values) + + +@pytest.mark.parametrize( + ("instrument_id", "activation_evidence_ref"), + [ + (" instrument-1 ", None), + ("instrument-1", " evidence://pilot/2026-09 "), + ], +) +def test_instrument_policy_rejects_surrounding_whitespace_in_canonical_refs( + instrument_id: str, + activation_evidence_ref: str | None, +) -> None: + """Instrument and evidence references fail closed instead of forming aliases.""" + with pytest.raises(ValueError, match="surrounding whitespace"): + InstrumentMeasurementPolicy( + instrument_id=instrument_id, + revision=1, + lifecycle=InstrumentLifecycle.PILOT, + model_family=None, + activation_evidence_ref=activation_evidence_ref, + ) diff --git a/tests/test_measurement_policy_runtime_types.py b/tests/test_measurement_policy_runtime_types.py new file mode 100644 index 000000000..0816843b6 --- /dev/null +++ b/tests/test_measurement_policy_runtime_types.py @@ -0,0 +1,115 @@ +"""Fail-closed runtime type checks for the measurement policy boundary.""" + +from __future__ import annotations + +import pytest + +from lineageweave.measurement_policy import ( + DichotomousItemPolicy, + DichotomousObservation, + DichotomousObservationState, + InstrumentLifecycle, + InstrumentMeasurementPolicy, + MeasurementModelFamily, +) + + +def test_string_lifecycle_cannot_bypass_published_activation_requirements() -> None: + """A deserialized-looking string must not evade the published-state invariant.""" + with pytest.raises(TypeError, match="lifecycle"): + InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=1, + lifecycle="published", # type: ignore[arg-type] + model_family=None, + activation_evidence_ref=None, + ) + + +def test_string_model_family_is_not_accepted_as_a_governed_family() -> None: + """The domain object accepts only the versioned enum, not an unvalidated string.""" + with pytest.raises(TypeError, match="model_family"): + InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=1, + lifecycle=InstrumentLifecycle.PUBLISHED, + model_family="irt_2plm", # type: ignore[arg-type] + activation_evidence_ref="evidence://pilot/2026-09", + ) + + +def test_string_observation_state_cannot_enter_the_binary_policy_domain() -> None: + """Adapters must parse the governed state before constructing a domain observation.""" + with pytest.raises(TypeError, match="state"): + DichotomousObservation("missing", None) # type: ignore[arg-type] + + +def test_item_policy_rejects_non_string_transport_fields_explicitly() -> None: + """Malformed transport values must fail at the item-policy boundary, not via ``.strip``.""" + with pytest.raises(TypeError, match="item policy fields must be strings"): + DichotomousItemPolicy( + item_id=17, # type: ignore[arg-type] + rubric_version="v1", + not_supported_criterion="No supporting evidence", + supported_criterion="Supporting evidence present", + ) + + +def test_instrument_policy_rejects_non_string_identity_explicitly() -> None: + """Instrument identity is a governed string and must not rely on incidental ``.strip`` errors.""" + with pytest.raises(TypeError, match="instrument_id"): + InstrumentMeasurementPolicy( + instrument_id=17, # type: ignore[arg-type] + revision=1, + lifecycle=InstrumentLifecycle.DRAFT, + model_family=None, + activation_evidence_ref=None, + ) + + +def test_instrument_policy_rejects_non_string_activation_reference_explicitly() -> None: + """Activation evidence references must be parsed to text before entering product policy.""" + with pytest.raises(TypeError, match="activation_evidence_ref"): + InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=1, + lifecycle=InstrumentLifecycle.PUBLISHED, + model_family=MeasurementModelFamily.IRT_2PLM, + activation_evidence_ref=17, # type: ignore[arg-type] + ) + + +def test_instrument_revision_rejects_boolean_transport_value_as_wrong_type() -> None: + """JSON booleans are Python integers; they must not become instrument revisions.""" + with pytest.raises(TypeError, match="revision"): + InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=True, # type: ignore[arg-type] + lifecycle=InstrumentLifecycle.DRAFT, + model_family=None, + activation_evidence_ref=None, + ) + + +def test_observed_response_rejects_boolean_transport_value_as_wrong_type() -> None: + """A JSON boolean must not be accepted or classified as an ordinary invalid 0/1 score.""" + with pytest.raises(TypeError, match="response"): + DichotomousObservation(DichotomousObservationState.OBSERVED, True) # type: ignore[arg-type] + + +def test_unscored_response_rejects_non_integer_transport_value_as_wrong_type() -> None: + """Malformed non-null response payloads fail at the type boundary before state semantics.""" + with pytest.raises(TypeError, match="response"): + DichotomousObservation(DichotomousObservationState.MISSING, "0") # type: ignore[arg-type] + + +def test_governed_enum_instances_still_construct_normally() -> None: + """Runtime hardening must preserve the supported typed contract.""" + policy = InstrumentMeasurementPolicy( + instrument_id="importance-evidence", + revision=1, + lifecycle=InstrumentLifecycle.PUBLISHED, + model_family=MeasurementModelFamily.IRT_2PLM, + activation_evidence_ref="evidence://pilot/2026-09", + ) + assert policy.model_family is MeasurementModelFamily.IRT_2PLM