Skip to content

fix: correct DS-np candidate evaluation in dsnp_limits#22

Merged
flaviobarros merged 1 commit into
masterfrom
fix/dsnp-limits-ucl2-bug
Jul 1, 2026
Merged

fix: correct DS-np candidate evaluation in dsnp_limits#22
flaviobarros merged 1 commit into
masterfrom
fix/dsnp-limits-ucl2-bug

Conversation

@flaviobarros

Copy link
Copy Markdown
Owner

The bug

In dsnp_limits(), the inner function eval_candidate() hardcoded the second-stage upper control limit:

ucl2 <- a + 1.5    # always wl_accept + 1.5, ignoring the loop variable c

Meanwhile the outer enumeration loop iterates over c (the integer ucl2_accept threshold) and stores ucl2 = c + 0.5 in the results table. The actual calls to dsnp_prob_accept(), dsnp_arl(), and dsnp_ass() inside eval_candidate() always evaluated with the fixed value a + 1.5, never with the candidate's own ucl2.

Consequence: all candidates sharing the same (wl_accept, ucl1_reject) pair but differing only in ucl2_accept received identical performance metrics. The limit search did not truly explore the second-stage threshold dimension.

Why existing tests didn't catch it

The consistency test re-evaluates the best candidate using its table values (c + 0.5). The ranking algorithm places c = a + 1 (the smallest valid second-stage threshold) at the top for typical parameters, and a + 1.5 == (a + 1) + 0.5, so the re-evaluation accidentally matched.

The fix

Add c as a parameter to eval_candidate() and compute ucl2 <- c + 0.5 instead of ucl2 <- a + 1.5.

Statistical reasoning

The DS-np chart has three integer thresholds:

  • wl_accept = a: accept at first stage if x1 <= a
  • ucl1_reject = b: signal at first stage if x1 >= b
  • ucl2_accept = c: accept at second stage if x1 + x2 <= c

The fractional limits are wl = a + 0.5, ucl1 = b - 0.5, ucl2 = c + 0.5. The value c controls how permissive the second stage is: a larger c means more combined counts are accepted, reducing p_signal. The search must evaluate each c independently to find the optimal tradeoff between false alarm rate (p_signal0) and out-of-control detection (arl1).

New tests

Four regression tests that would have caught the original bug:

  1. Candidates with same (wl, ucl1) but different ucl2 must produce different p_signal0
  2. Every candidate's stored metrics match a direct dsnp_prob_accept/dsnp_arl call
  3. ucl2 == ucl2_accept + 0.5 for every candidate
  4. p_signal0 is non-increasing in ucl2 (larger limit = more accepting)

Verification

  • All 362 tests pass
  • Published example (Joekes et al. 2015): ARL0 = 803.41, ARL1 = 193.22, ASS0 = 35.94 — all match
  • API unchanged

The inner eval_candidate() function hardcoded ucl2 = a + 1.5 instead of
using the loop variable c. This meant every candidate with the same
(wl_accept, ucl1_reject) pair was evaluated with the same ucl2 regardless
of its ucl2_accept value. The results table stored ucl2 = c + 0.5 but the
actual dsnp_prob_accept/dsnp_arl calls always used a + 1.5.

The bug was invisible in existing tests because the consistency test
re-evaluates using the table's ucl2 (c + 0.5), and the ranking tends to
place c = a + 1 at the top, where c + 0.5 == a + 1.5.

Fix: add c parameter to eval_candidate and compute ucl2 = c + 0.5.
Add four regression tests that catch this class of bug:
- Candidates with same (wl,ucl1) but different ucl2 must differ in p_signal
- Every candidate's metrics must match a direct numeric core call
- ucl2 column must equal ucl2_accept + 0.5
- p_signal must be non-increasing in ucl2 (larger ucl2 = more accepting)
@flaviobarros flaviobarros merged commit bb4e712 into master Jul 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant