Skip to content

[CEFR] Add leakage-safe longitudinal language-profile and drift analysis after invariance gates #275

Description

@seonghobae

Temporal measurement gap

A single CEFR-aligned assessment result can support placement, but repeated results cannot be interpreted as language growth until form, task, rater, scoring, mode, population, and language-profile comparability have been evaluated. TEPP is the correct owner for temporal, multilevel, relational, and multiple-membership analysis of repeated language profiles.

This issue owns longitudinal and contextual language-profile analysis artifacts. It does not own assessment sessions, raw responses, task content, CEFR descriptor prose, cross-sectional scoring, cut scores, or LMS decisions.

Dependencies

Blocked by:

  1. release of cwl_cefr_language_assessment/v1 from ContextualWisdomLab/learning-interoperability-contracts PR feat(temporal): add typed six-clock values and uncertain intervals #5;
  2. immutable CEFR profile results from Psychometrics Commons issue feat(api): enumerate project histories via loopback collection GET #424;
  3. Rust scoring/linking/DIF/classification evidence from fast-mlsirm issue #1484;
  4. TEPP temporal/event foundation and persistence contracts being implemented on protected main.

Do not build this as a Python post-hoc regression over copied result JSON.

Analysis boundary

Consume only versioned, immutable result references and approved contextual membership references. Preserve at least:

result_reference
participant_reference
instrument_release_reference
assessment_blueprint_reference
scoring_profile_reference
cut_score_revision_reference
claim_status_code
target_language
domain_result_reference
observed_at
published_at
available_at
recorded_at
correction_or_supersession_reference

Raw responses, audio, transcripts, task content, rater payloads, and PII remain in their owning systems.

Multi-clock and leakage contract

For every analysis cutoff T, require:

available_at(result) <= T

Separate:

  • assessment event/administration time;
  • response observation time;
  • result publication time;
  • system ingestion/recorded time;
  • correction/supersession time;
  • analysis knowledge cutoff.

A later correction, rescoring run, cut-score revision, or RLD update must not leak into an earlier historical analysis.

Measurement structure

Evaluate, rather than assume:

  • longitudinal configural/metric/scalar or appropriate approximate invariance;
  • form, item/task, rubric, rater, AI-rater family/version, mode, and accommodation drift;
  • cut-score and linking drift;
  • population and language-profile drift;
  • within-person versus between-person separation;
  • irregular assessment intervals;
  • cross-classified context: cohort, instructor, sponsor organization, course offering, content release, project, and learning environment;
  • multiple membership when one learner/occasion belongs to multiple relevant contexts;
  • missing-by-design domains and selective retesting;
  • correction/supersession without historical mutation.

Do not treat ordinal CEFR labels as equally spaced numeric observations. Use the underlying versioned domain-score/posterior evidence or appropriate latent-state measurement model.

Rust arithmetic ownership

All result-affecting temporal and multilevel arithmetic is Rust-owned:

  • longitudinal state transition;
  • random intercept/slope and context effects;
  • continuous-time or interval-aware transition where adopted;
  • invariance/drift tests and uncertainty;
  • change-point/onset detection where scientifically supported;
  • trajectory estimation and prediction intervals;
  • true-state simulation and recovery;
  • CPU multithreading and GPU execution where benchmark evidence justifies it.

Python may validate contracts, orchestrate jobs, and render reports only.

First bounded vertical

Implement a research-only English A1–B2 repeated-placement analysis with:

  • four domains: reading, listening, written production, spoken production;
  • at least three occasions per simulated participant;
  • irregular time gaps;
  • two form releases;
  • human and AI-rater-version drift in productive domains;
  • course/cohort/instructor cross-classification;
  • one multiple-membership sponsor/context dimension;
  • delayed result availability and one later correction;
  • explicit no-change verdict when invariance or linking evidence fails.

Outputs:

language_trajectory_artifact
language_invariance_artifact
language_drift_artifact
language_context_effect_artifact
language_change_interpretation_artifact

Every artifact pins exact source result identities, model/engine versions, data cutoff, membership revision, failure denominator, uncertainty, and limitations.

Recovery evidence

The simulator must inject known truth for:

  • stable person differences and within-person change;
  • domain correlations;
  • form/linking shifts;
  • item/task/rater drift;
  • occasion effects;
  • context random effects;
  • multiple-membership weights;
  • irregular lags;
  • missingness and delayed availability;
  • correction events.

Report bias/RMSE, interval coverage, state/transition recovery, drift-detection precision/recall, false change declarations, backend parity, worker-count determinism, failure denominator, and Monte Carlo uncertainty.

Acceptance

  • No trajectory or growth claim is emitted when required invariance/linking evidence fails.
  • No future-available result or correction enters an earlier cutoff.
  • Ordinal CEFR labels are never averaged or treated as an interval scale.
  • Within-person change is separated from stable between-person/context differences.
  • Cross-classified and multiple-membership design is explicit; no primary_group shortcut.
  • Historical results and trajectories are immutable; correction creates superseding artifacts.
  • Every number is reproducible from exact input and model artifacts.
  • Production statement/branch coverage and public API docstrings are 100%.
  • True-state recovery, CPU/GPU parity, memory/throughput evidence, and no skipped required GPU/recovery tests.
  • Visual outputs include exact-value accessible tables, uncertainty, provenance, data cutoff, and no causal language unless a separate causal design supports it.
  • No Council of Europe endorsement, validation, certification, or logo-use claim.

Standards and research baseline

  • CEFR Companion Volume (2020).
  • Council of Europe examination-linking and test-development manuals.
  • AERA, APA, & NCME Standards (2014).
  • Longitudinal measurement-invariance, DSEM/continuous-time, multilevel, cross-classified, and multiple-membership primary literature documented in APA 7th form.

TEPP may measure change in a governed language-profile system; it must not turn repeated labels into growth by assumption.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions