nested parallel_for fails with stack overflow when using `getNumThreads` rather than hard-coding the number of threads via `max_allowed_parallelism`

[paciorek]

The last test in test-threads.R produces:

   Cnc1 <- Cnc$new()
    ## expect_silent(out <- Cnc1$go(100, 100, 10))
    ## Error: C stack usage  14983114845556 is too close to the limit
    ## Error: no more error handlers available (recursive errors?); invoking 'abort' restart
    ## *** longjmp causes uninitialized stack frame ***: terminated

However, if I manually hard-code the number of threads like this: tbb::global_control gc(tbb::global_control::max_allowed_parallelism,4); instead of using getNumThreads(4.0), it is fine. And from earlier versions of the codebase that didn't control the number of threads, nested parallel_for is also fine.

So once we address issue #151, I will experiment with trying to figure out why use of getNumThreads is a problem.

[paciorek]

So it seems to be caused by calling getNumThreads as the error occurs even if the result of the call is not used to set max_allowed_parallelism.

Also noting that in some cases the error message also says:

Error: No such namespace: nCompiler.

This is getNumThreads:

inline int getNumThreads(double value_) {
  int value = (int) value_;
  Rcpp::Environment nc = Rcpp::Environment::namespace_env("nCompiler");
  Rcpp::Function get_nOption = nc["get_nOption"];
  int option_value = Rcpp::as<int>(get_nOption("nThreads"));
  if (option_value > 0)
    value = option_value;
  if(value == 0)
    value = 100000;
   return value; 
}

[paciorek]

So the error only occurs when getNumThreads is called in the inner loop. Claude indicates this is because R is not thread safe and the R API (which Rcpp wraps around) can't be called from TBB workers.

Ideally we'd have some overall control where we determine there is nested looping and decide on the number of threads for inner and outer and pass the inner number in as a plain C++ variable. As a simple next step perhaps we figure out how to determine we are in an inner loop and simply set the number of threads to 1. (Though I can imagine cases where the number of outer iterations is small and we also want parallelism at the inner stage.)