fix: correct DS-np candidate evaluation in dsnp_limits#22
Merged
Conversation
The inner eval_candidate() function hardcoded ucl2 = a + 1.5 instead of using the loop variable c. This meant every candidate with the same (wl_accept, ucl1_reject) pair was evaluated with the same ucl2 regardless of its ucl2_accept value. The results table stored ucl2 = c + 0.5 but the actual dsnp_prob_accept/dsnp_arl calls always used a + 1.5. The bug was invisible in existing tests because the consistency test re-evaluates using the table's ucl2 (c + 0.5), and the ranking tends to place c = a + 1 at the top, where c + 0.5 == a + 1.5. Fix: add c parameter to eval_candidate and compute ucl2 = c + 0.5. Add four regression tests that catch this class of bug: - Candidates with same (wl,ucl1) but different ucl2 must differ in p_signal - Every candidate's metrics must match a direct numeric core call - ucl2 column must equal ucl2_accept + 0.5 - p_signal must be non-increasing in ucl2 (larger ucl2 = more accepting)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
In
dsnp_limits(), the inner functioneval_candidate()hardcoded the second-stage upper control limit:Meanwhile the outer enumeration loop iterates over
c(the integerucl2_acceptthreshold) and storesucl2 = c + 0.5in the results table. The actual calls todsnp_prob_accept(),dsnp_arl(), anddsnp_ass()insideeval_candidate()always evaluated with the fixed valuea + 1.5, never with the candidate's ownucl2.Consequence: all candidates sharing the same
(wl_accept, ucl1_reject)pair but differing only inucl2_acceptreceived identical performance metrics. The limit search did not truly explore the second-stage threshold dimension.Why existing tests didn't catch it
The consistency test re-evaluates the best candidate using its table values (
c + 0.5). The ranking algorithm placesc = a + 1(the smallest valid second-stage threshold) at the top for typical parameters, anda + 1.5 == (a + 1) + 0.5, so the re-evaluation accidentally matched.The fix
Add
cas a parameter toeval_candidate()and computeucl2 <- c + 0.5instead ofucl2 <- a + 1.5.Statistical reasoning
The DS-np chart has three integer thresholds:
wl_accept = a: accept at first stage ifx1 <= aucl1_reject = b: signal at first stage ifx1 >= bucl2_accept = c: accept at second stage ifx1 + x2 <= cThe fractional limits are
wl = a + 0.5,ucl1 = b - 0.5,ucl2 = c + 0.5. The valueccontrols how permissive the second stage is: a largercmeans more combined counts are accepted, reducingp_signal. The search must evaluate eachcindependently to find the optimal tradeoff between false alarm rate (p_signal0) and out-of-control detection (arl1).New tests
Four regression tests that would have caught the original bug:
(wl, ucl1)but differentucl2must produce differentp_signal0dsnp_prob_accept/dsnp_arlcallucl2 == ucl2_accept + 0.5for every candidatep_signal0is non-increasing inucl2(larger limit = more accepting)Verification