feat(detect): confirmatory DETECT dimensionality analysis (Zhang & Stout, 1999)

[seonghobae]

Summary

Iteration 5 of the autonomous paper-implementation loop. Adds confirmatory DETECT dimensionality analysis (Zhang & Stout, 1999) to mlsirm-core with a thin Python wrapper, stacked on #221 (Haberman subscores).

DETECT estimates pairwise conditional covariances of binary items given the latent composite (proxied by sum scores) and scores a hypothesized item clustering: positive within-cluster / negative between-cluster conditional covariances indicate multidimensional simple structure.

What's implemented

• mlsirm_core::detect::detect_analysis — sum-score-conditioned, bias-corrected conditional covariances and the DETECT / ASSI / RATIO / MADCOV100 / MCOV100 indices against a known clustering.
  • ccov_ij = (cov | total + cov | rest)/2, rest score S − X_i − X_j; per-group ML covariance (divide by n); group weights freq/N recomputed per conditioning vector; singleton groups contribute 0.
  • DETECT = 100·mean(ccov·δ), ASSI = mean(sign(ccov)·δ), RATIO = Σ(ccov·δ)/Σ|ccov|, MADCOV100 = 100·mean|ccov|, MCOV100 = 100·mean(ccov); δ = +1 same-cluster else −1; labels opaque, equality-only.
• PyO3 binding detect_analysis + Python wrapper fast_mlsirm.detect_analysis → DetectResult (validation and marshaling only; all numerics in Rust).

Source verification (paper-first discipline)

Zhang & Stout (1999a/b) are paywalled; the read oracle is the CRAN sirt R source, transcribed line-by-line: detect.index.R, ccov.np.R, ccov_np_compute_ccov_sum_score.R, conf.detect.R (Robitzsch, 2024). Zhang & Stout (1999), Stout et al. (1996), Zhang (2007), Jang & Roussos (2007) are cited as cited in Robitzsch (2024). The adversarial spec-verify pass returned NO-GO on the first spec revision (it misread sirt's default scale_score=TRUE z-standardize+round path); the scope was reduced to the explicit scale_score=FALSE raw-sum-score path and all 8 required fixes were applied before implementation.

Documented scope exclusions: kernel-smoothed default conditioning, missing data (sirt pairwise-deletes with a rowSums na.rm quirk), sqrt(N)-weighted variants (identical to unweighted under complete data), exploratory cluster search, polytomous DETECT. RATIO 0/0 (NaN in R) is rejected with an error.

Evidence

Check	Result
Rust suite (cargo test -p mlsirm-core)	464 passed
500-rep Monte Carlo (#[ignore]): 2D simple structure vs unidimensional separation	passed
Fixture parity: all 5 indices + all 10 per-pair ccovs vs independent NumPy transcription literals	exact to 1e-12
Mutation spot-checks	4/4 killed (δ sign swap → 4 fails; dropped bias correction → 5; n−1 divisor → 5; rest = S−X_i only → 5)
Python tests (pytest -k detect_analysis)	2 passed
Adversarial spec-verify	NO-GO → reduced scope, 8/8 fixes applied
Adversarial impl-review	1 blocking defect found (float→i64 label cast could silently wrap out-of-range labels and collapse the partition) → fixed with i64 range guard + regression test

Disclosed test limitations: the fixture cannot discriminate scale_score (unique-value grouping is invariant to monotone transforms unless rounding merges groups) — the scope statement pins that contract; weighted ≡ unweighted under complete data is an identity (variants cut rather than shipped untestable); DETECT == MCOV100 for a single cluster is an identity anchored by the multi-cluster fixtures.

Trust boundary

checked_mul overflow guards in both core and binding; responses restricted to exact 0/1; cluster labels never used as indices (hostile i64::MIN/i64::MAX labels tested); Python wrapper rejects float labels outside i64 range before casting.

Stacked on #221 → base seonghobae-haberman-subscores.

[Copilot]

Pull request overview

Adds confirmatory DETECT dimensionality analysis to the Rust core (mlsirm-core) with a thin PyO3/Python wrapper, exposing results as a DetectResult and pinning correctness against independent reference literals (plus guard/hostile-input tests).

Changes:

• Implement mlsirm_core::detect::detect_analysis (conditional covariance estimation + DETECT/ASSI/RATIO/MADCOV100/MCOV100 indices) and export the new module.
• Expose detect_analysis via PyO3 and a Python wrapper (fast_mlsirm.detect_analysis) returning a typed DetectResult.
• Add Rust and Python tests that pin index outputs and per-pair ccovs to independent NumPy transcription literals, plus rejection/hostile-label coverage.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/mlsirm-core/src/lib.rs	Exposes the new detect module from the Rust core.
crates/mlsirm-core/src/detect.rs	Implements DETECT analysis logic and rustdoc describing scope/source parity.
crates/fast-mlsirm-py/src/lib.rs	Adds PyO3 binding for detect_analysis returning dict outputs to Python.
python/fast_mlsirm/detect.py	Adds Python wrapper + DetectResult dataclass with input validation/marshaling.
python/fast_mlsirm/__init__.py	Re-exports detect_analysis / DetectResult as part of the public API.
tests/unit/detect_tests.rs	Adds Rust-side fixture and behavioral tests (including ignored Monte Carlo).
tests/test_paper_features.py	Adds Python-side parity + trust-boundary rejection tests for DETECT.
CHANGELOG.md	Documents the new DETECT feature, scope, and test evidence.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[seonghobae]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

[Copilot]

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested ch...

All three review comments addressed in commits fix(detect) and the follow-up style rename:

1. Integer dtype i64 range check (detect.py): The else branch now enforces bounds for all integer inputs before astype(int64). Unsigned types (e.g. uint64) get an upper-bound-only check to avoid NumPy mixed-signedness promotion issues when comparing to I64_MIN; signed types narrower than int64 are always in range. A regression test for uint64 > int64.max was also added.

2. Mismatched backticks in doc comment (detect.rs:63-64): Reformatted the RATIO/NaN sentence so 0/0 = NaN sits on a single line as one inline-code span.

3. Precompute total-score ordering (detect.rs:247-248): Added ccov_sum_score_presorted which accepts a caller-supplied sorted order. detect_analysis now sorts total once before the pair loop and calls ccov_sum_score_presorted for c1; c2 (rest-score) still re-sorts per pair as required. All 464 Rust tests and both Python detect tests pass.

[seonghobae]

@copilot resolve the merge conflicts on this branch.

[Copilot]

@copilot resolve the merge conflicts on this branch.

Merge conflicts resolved in commit 8e477ff. Merged origin/seonghobae-ksirt-kernel-smoothing into this branch, keeping the DETECT additions alongside the ksirt changes from the base branch. Four files had conflicts: CHANGELOG.md, crates/fast-mlsirm-py/src/lib.rs, python/fast_mlsirm/__init__.py, and tests/test_paper_features.py. Rust core compiles cleanly after the merge.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

crates/mlsirm-core/src/detect.rs:157

• ccov_sum_score allocates and sorts a full order vector every call. Since it is invoked for the rest-score pass for every item pair, this makes DETECT scale as O(n_pairs * n_persons log n_persons) and can become a major bottleneck for realistic item counts. Consider replacing sort-based grouping with a bucketed aggregation over the integer score range (raw sums in 0..n_items, rest scores in 0..n_items-2), or at least reusing a scratch order buffer to avoid per-pair allocations.

fn ccov_sum_score(score: &[i64], xi: &[f64], xj: &[f64]) -> f64 {
    let n = score.len() as f64;
    let mut order: Vec<usize> = (0..score.len()).collect();
    order.sort_unstable_by_key(|&p| score[p]);
    ccov_sum_score_presorted(score, xi, xj, n, &order)