Skip to content

Add Livingston-Lewis (1995) classification accuracy and consistency - #228

Merged
seonghobae merged 1 commit into
mainfrom
seonghobae-livingston-lewis
Jul 25, 2026
Merged

seonghobae merged 1 commit into
mainfrom
seonghobae-livingston-lewis

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Implements Livingston & Lewis (1995) single-administration classification accuracy and consistency, as implemented in CRAN betafunctions 1.9.0 LL.CA (read line by line; the original article was not consulted directly). Stacked on #227.

What

  • mlsirm_core::classification::livingston_lewis: effective test length -> true-score raw moments (factorial-moment identity, unrounded ETL scale) -> Hanson (1991, as cited in Haakstad, 2022) four-parameter beta moment fit with two-parameter [0,1] fail-safe -> binomial observed-score model with N = round_ties_even(ETL) -> accuracy cells (tp/fp/tf/ff), sensitivity/specificity, consistency cells, Cohen's kappa.
  • Singularity-safe composite Gauss-Legendre quadrature: power substitution when a beta shape < 1, endpoint-graded panels otherwise (a plain fixed rule loses ~2e-5 mass on high-shape cases; caught during development).
  • PyO3 binding + thin Python wrapper fast_mlsirm.livingston_lewis returning a dataclass.

Documented divergences from the oracle

  • Single k = round_ties_even(N*c) threshold in BOTH accuracy and consistency blocks. betafunctions mixes round (accuracy) with floor (consistency), making its consistency cells asymmetric (measured p.ij-p.ji = 0.0345 on a test fixture); here p_ij == p_ji by construction.
  • Pass = observed score >= cut is the positive class (the oracle labels fail as positive; its sensitivity is our specificity).
  • Fail-safe also engages on numerically invalid 4P fits (oracle only checks out-of-bounds support).
  • Conditional ratios (sensitivity/specificity/kappa) are explicit NaN when their margin or chance denominator vanishes (found by adversarial impl-review; oracle propagates NaN silently).

Evidence

Check Result
Spec-verify (adversarial, pre-implementation) GO-WITH-FIXES; all 5 fixes applied (singular-safe quadrature, asymmetry wording, failsafe divergence, independent-replication literals, non-integer-ETL fixture)
Independent replication scipy.integrate.quad pipeline, 4 fixtures (2P fail-safe, 2P skewed, genuine 4P path, extreme-cut round/floor discriminator), literals pinned at 1e-7/1e-9
Mutation spot-checks 5/5 killed: integrand swap, unrounded-ETL k (required the extreme-cut fixture; base fixtures could not discriminate), k-1 -> k shift, ddof n-1 -> n, fail-safe drop
cargo test -p mlsirm-core 498 passed, 0 failed
500-rep Monte Carlo (#[ignore]) consistency recovers empirical two-form agreement
Python tests 2 passed (pinned literals + trust-boundary rejections)
Adversarial impl-review 2 confirmed NaN defects found -> fixed with explicit ratio guard + documented semantics + regression test

Known limitation (disclosed in-test): at exactly-zero denominators IEEE 0/0 already yields NaN, so a dropped ratio guard is not killed by the NaN test; the guard's value is for tiny nonzero denominators.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Livingston–Lewis (1995) single-administration classification accuracy/consistency to the Rust core and exposes it through the PyO3 extension and Python API, with pinned-literal tests validating the numerical pipeline against an independent SciPy quadrature replication.

Changes:

  • Implement mlsirm_core::classification::livingston_lewis, including effective test length → beta moment fit → binomial observed-score integrals (accuracy/consistency/kappa).
  • Expose the new functionality via PyO3 (fast_mlsirm._core) and a Python wrapper returning a LivingstonLewisResult dataclass.
  • Add Rust and Python test fixtures covering 2P fail-safe, 4P path, threshold rounding behavior, and trust-boundary input validation; update the changelog.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
crates/mlsirm-core/src/classification.rs Core Livingston–Lewis implementation, quadrature/integral machinery, and result struct.
crates/mlsirm-core/src/reliability.rs Makes inc_beta pub(crate) so classification can reuse it for binomial tail computation.
crates/fast-mlsirm-py/src/lib.rs Adds PyO3 binding that returns a dict of Livingston–Lewis outputs.
python/fast_mlsirm/classification.py Adds LivingstonLewisResult dataclass + livingston_lewis() wrapper calling the compiled core.
python/fast_mlsirm/__init__.py Re-exports livingston_lewis at the package top level.
tests/unit/classification_tests.rs Adds Rust-side fixtures and invariants for Livingston–Lewis outputs.
tests/test_paper_features.py Adds Python-side pinned-literal and rejection tests for the wrapper/core boundary.
CHANGELOG.md Documents the newly added Livingston–Lewis feature and its documented divergences from the oracle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/mlsirm-core/src/classification.rs Outdated
Comment thread crates/mlsirm-core/src/classification.rs
Comment thread python/fast_mlsirm/classification.py Outdated
Comment thread crates/mlsirm-core/src/classification.rs Outdated
@seonghobae

Copy link
Copy Markdown
Contributor Author

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

Copilot AI commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested ch...

All four review items addressed in commit Fix review comments: rename p_tf/p_ff→p_tn/p_fn, cache GL64, explicit docstring, standardize citation year:

  1. Field naming (p_tf/p_ff → p_tn/p_fn): Renamed to unambiguous p_tn (true negative: P(true fail, observed fail)) and p_fn (false negative: P(true pass, observed fail)) in the Rust struct, PyO3 binding, Python dataclass, and all Rust/Python tests. Derived field formulas (accuracy, sensitivity, specificity) updated accordingly.
  2. GL64 caching: Added static GL64: OnceLock<(Vec<f64>, Vec<f64>)> with an inline fn gl64() accessor. beta_weighted_integral now uses the cached rule instead of recomputing Newton iterations on every call.
  3. Docstring: LivingstonLewisResult now lists each accuracy cell with its full definition (true positive, false positive, true negative, false negative), the four-cell sum invariant, and derived-field formulas.
  4. Citation year: Standardized to Haakstad (2022) in classification.rs (was 2023), matching the Python docstring and CHANGELOG.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comment thread python/fast_mlsirm/classification.py
Comment thread crates/mlsirm-core/src/classification.rs
Comment thread CHANGELOG.md
@seonghobae
seonghobae force-pushed the seonghobae-feldt-alpha-ci branch from 40a260a to afb84cf Compare July 25, 2026 09:29
@seonghobae
seonghobae force-pushed the seonghobae-livingston-lewis branch from 9ff6ca2 to c0ec7b2 Compare July 25, 2026 09:30
@seonghobae
seonghobae force-pushed the seonghobae-feldt-alpha-ci branch from afb84cf to 8d59b0f Compare July 25, 2026 10:04
@seonghobae
seonghobae force-pushed the seonghobae-livingston-lewis branch from c0ec7b2 to 51fb72f Compare July 25, 2026 10:05
@seonghobae
seonghobae force-pushed the seonghobae-feldt-alpha-ci branch from 8d59b0f to 64418ec Compare July 25, 2026 10:14
@seonghobae
seonghobae force-pushed the seonghobae-livingston-lewis branch from 51fb72f to 940a26c Compare July 25, 2026 10:14
@seonghobae
seonghobae force-pushed the seonghobae-feldt-alpha-ci branch 2 times, most recently from 67b034e to 863f33e Compare July 25, 2026 11:44
Base automatically changed from seonghobae-feldt-alpha-ci to main July 25, 2026 11:53
Rebased onto main after #227 squash merge.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@seonghobae
seonghobae force-pushed the seonghobae-livingston-lewis branch from 940a26c to 0abfd60 Compare July 25, 2026 11:59
@seonghobae
seonghobae merged commit 0d197b4 into main Jul 25, 2026
32 checks passed
@seonghobae
seonghobae deleted the seonghobae-livingston-lewis branch July 25, 2026 12:08
seonghobae added a commit that referenced this pull request Jul 25, 2026
Rebased onto main after #228 squash merge.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
seonghobae added a commit that referenced this pull request Jul 25, 2026
…229)

Rebased onto main after #228 squash merge.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants