I've been chasing down an intermittent crash in permuteMeasEq.
Here's the minimal reproducible example from my Apple Silicon MBP running Tahoe 26.4
library('lavaan')
library('semTools')
library('parallel')
group='whichdata'
testformula_r='anxious_worry =~ hitop20 + hitop34 + hitop89 + hitop203 + hitop248 + hitop265'
rdata = read.csv('./anxworry_gp_cn.csv', header=TRUE)
fit_config = cfa(testformula_r, data = rdata, group = group, estimator = "WLSMV", ordered = c("hitop20", "hitop34", "hitop89", "hitop203", "hitop248", "hitop265"))
fit_metric = cfa(testformula_r, data = rdata, group = group, group.equal = "loadings", estimator = "WLSMV", ordered = c("hitop20", "hitop34", "hitop89", "hitop203", "hitop248", "hitop265"))
RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(12345)
out_matrix <- permuteMeasEq(nPermute=1000, uncon=fit_config, con=fit_metric, param="loadings", parallelType="multicore", ncpus=14)
anxworry_gp_cn.csv
The above crashes with the output:
This is lavaan 0.6-21
lavaan is FREE software! Please report any bugs.
###############################################################################
This is semTools 0.5-8
All users of R (or SEM) are invited to submit functions or ideas for functions.
###############################################################################
No AFIs were selected, so only chi-squared will be permuted.
Error in x$AFI : $ operator is invalid for atomic vectors
In addition: Warning messages:
1: lavaan->lavTestScore():
se is not `standard'; not implemented yet; falling back to ordinary score
test
2: In parallel::mclapply(...) :
scheduled core 13 encountered error in user code, all values of the job will be affected
It runs fine sequentially, and doesn't crash if ncpus is set to less than 13. With 13 or more, it crashes on
the 20th iteration given to the 13th worker. Debugging further with help from Claude, I found that the 20th iteration given to the 13th worker fails at do.call(getMIs, ...) call at permuteOnce.mgcfa line 1420 with error code 1 from Lapack routine 'dgesdd' during La.svd. The aggregation at line 990 of permuteMeasEq.R (sapply(permuDist, function(x) x$AFI)) then accesses $AFI on the atomic-class try-error and crashes. Claude helped me capture and compare the matrix that's causing dgesdd to fail in both the multicore and sequential cases and found that these matrices differ. The intermediate from multicore is a 52x52 full rank non-negative matrix. When running sequentially, the intermediate can be decomposed. Something about the forked context is producing a different intermediate that dgesdd is choking on for that specific permutation of this input data. dgesdd failing on full-rank non-negative matrices is documented in Reference-LAPACK/lapack#672 and depends on the BLAS implementation.
Given that dgesdd may occasionally fail, I'd propose wrapping getAFIs/getMIs in tryCatch so any failures just contribute NA to the null-distribution as opposed to killing the worker.
I've been chasing down an intermittent crash in
permuteMeasEq.Here's the minimal reproducible example from my Apple Silicon MBP running Tahoe 26.4
anxworry_gp_cn.csv
The above crashes with the output:
It runs fine sequentially, and doesn't crash if
ncpusis set to less than 13. With 13 or more, it crashes onthe 20th iteration given to the 13th worker. Debugging further with help from Claude, I found that the 20th iteration given to the 13th worker fails at
do.call(getMIs, ...)call atpermuteOnce.mgcfaline 1420 with error code 1 from Lapack routine'dgesdd'duringLa.svd. The aggregation at line 990 ofpermuteMeasEq.R(sapply(permuDist, function(x) x$AFI))then accesses$AFIon the atomic-classtry-errorand crashes. Claude helped me capture and compare the matrix that's causingdgesddto fail in both the multicore and sequential cases and found that these matrices differ. The intermediate from multicore is a 52x52 full rank non-negative matrix. When running sequentially, the intermediate can be decomposed. Something about the forked context is producing a different intermediate thatdgesddis choking on for that specific permutation of this input data.dgesddfailing on full-rank non-negative matrices is documented in Reference-LAPACK/lapack#672 and depends on the BLAS implementation.Given that
dgesddmay occasionally fail, I'd propose wrappinggetAFIs/getMIsintryCatchso any failures just contribute NA to the null-distribution as opposed to killing the worker.