BLAS threads

CHANGELOG.md

@@ -14,6 +14,9 @@
 - On an Int32 PETSc build (the `PetscInt = "Int32"` preference), `LibPETSc.DMStagStencil`, `MatStencil` and the other C structs had `Int64` fields, so PETSc read them wrong. They now take the integer width of the loaded library, and the two stencil constructors convert their indices to it.
 - `LibPETSc.DMStagRestoreProductCoordinateArraysRead` restored the arrays through the writable restore, which does not match the read-only get.
 - The `KSP` and `SNES` docstrings did not say when the options given to the constructor are applied: once at construction for `KSP`, at every `solve!` for `SNES` (and `TS`).
+- `-blas_num_threads` had no effect. Every PETSc_jll build calls BLAS through libblastrampoline, so PETSc's BLAS runs in Julia's OpenBLAS pool, which PETSc cannot size. `initialize` now forwards the option to `LinearAlgebra.BLAS.set_num_threads`.
+- `initialize(petsclib; options)` appended `-no_signal_handler` to the caller's `options` vector.
+- On Windows, the `options` and `log_view` given to `initialize` never reached PETSc: they went through `ENV["PETSC_OPTIONS"]`, which PETSc there reads from its own copy of the environment. They are now PETSc's command line (`LibPETSc.PetscInitialize` takes a `Vector{String}`), and still override a `PETSC_OPTIONS` set before Julia starts.
 
 ### Added
 
@@ -35,6 +38,7 @@
 
 ### Changed
 
+- When several MPI ranks share a node, `initialize` sets Julia's BLAS pool to one thread, unless `-blas_num_threads`, `OPENBLAS_NUM_THREADS` or `OMP_NUM_THREADS` says otherwise. Each rank otherwise ran a pool of busy-waiting threads, which made a 4-rank DMStag Stokes solve 25× slower. Serial runs and one rank per node are unaffected.
 - `solution(ksp)`, `solution(ts)`, `local_coordinates(dm)` and the `vatol`/`vrtol` of `tolerances(ts)` return a borrowed `PetscVec`, as `solution(snes)` already did, instead of a `VecPtr`. Both are `AbstractPetscVec`s with the same ownership, so only code that checks for `VecPtr` by type notices.
 - `LibPETSc` release functions that free a C array of handles (`VecDestroyVecs`, `MatDestroyMatrices`, `DMPlexRestoreConeRecursive`, …) no longer accept a Julia array of raw handles, which PETSc would have freed as its own memory. They take the pointer PETSc returned; `MatDestroySubMatrices`, `MatDestroyMatrices`, `VecNestRestoreSubVecsRead` and the two subdomain destroyers also take the vector the matching wrapper returned.
 

scripts/renames.jl

@@ -402,6 +402,9 @@ const INTERNAL = Set{Symbol}([
     :call_with_views,
     :field_views,
     :TSStepHookFn,
+    :apply_blas_num_threads,
+    :BLAS_THREAD_VARIABLES,
+    :ranks_on_node,
     :_unsafe_local_array,
     :_local_arrays,
     :_restore_local_arrays!,

src/autowrapped/Sys_wrappers.jl

@@ -5981,130 +5981,50 @@ end
 	return nothing
 end 
 
+# override for PetscInitialize; C signature: PetscInitialize(int* argc, char*** args, const char file[], const char help[])
 """
-	argc::Cint = PetscInitialize(petsclib::PetscLibType, args::Cchar, file::String, help::String) 
-Initializes the PETSc database and MPI.
-`PetscInitialize()` calls MPI_Init() if that has yet to be called,
-so this routine should always be called near the beginning of
-your program -- usually the very first line!
+	PetscInitialize(petsclib::PetscLibType, args::Vector{String}, file::Union{Nothing, String} = nothing, help::Union{Nothing, String} = nothing)
+Initializes the PETSc database and MPI, with `args` as the command line: `args[1]` is the program name and the rest are
+options, one token per entry, such as `["prog", "-ksp_type", "cg"]`. Options given here override those in the
+`PETSC_OPTIONS` environment variable.
+
+PETSc keeps pointers to `args` until `PetscFinalize()`, so the strings are held here until the next call.
 
 Collective on `MPI_COMM_WORLD` or `PETSC_COMM_WORLD` if it has been set
 
 Input Parameters:
-- `argc` - count of number of command line arguments
-- `args` - the command line arguments
-- `file` - [optional] PETSc database file, append ":yaml" to filename to specify YAML options format.
-Use `NULL` or empty string to not check for code specific file.
-Also checks `~/.petscrc`, `.petscrc` and `petscrc`.
-Use `-skip_petscrc` in the code specific file (or command line) to skip `~/.petscrc`, `.petscrc` and `petscrc` files.
-- `help` - [optional] Help message to print, use `NULL` for no message
-
-If you wish PETSc code to run ONLY on a subcommunicator of `MPI_COMM_WORLD`, create that
-communicator first and assign it to `PETSC_COMM_WORLD` BEFORE calling `PetscInitialize()`.
-then do this. If ALL processes in the job are using `PetscInitialize()` and `PetscFinalize()` then you don't need to do this, even
-if different subcommunicators of the job are doing different things with PETSc.
-
-Options Database Keys:
-- `-help [intro]`                                          - prints help method for each option; if `intro` is given the program stops after printing the introductory help message
-- `-start_in_debugger [(noxterm)],[(gdb|lldb|...)]`        - Starts program in debugger
-- `-on_error_attach_debugger [(noxterm)],[(gdb|lldb|...)]` - Starts debugger when error detected
-- `-on_error_emacs machinename`                            - causes `emacsclient` to jump to error file if an error is detected
-- `-on_error_abort`                                        - calls `abort()` when error detected (no traceback)
-- `-on_error_mpiabort`                                     - calls `MPI_abort()` when error detected
-- `-error_output_stdout`                                   - prints PETSc error messages to `stdout` instead of the default `stderr`
-- `-error_output_none`                                     - does not print the error messages (but handles errors in the same way as if this was not called)
-- `-debugger_ranks rank1,rank2,...`                        - Indicates MPI ranks to start in debugger
-- `-debugger_pause secs`                                   - Pauses debugger, use if it takes a long time for the debugger to start up on your system, `sleeptime` is number of seconds to sleep
-- `-stop_for_debugger`                                     - Print message on how to attach debugger manually to
-process and wait (`-debugger_pause`) seconds for attachment
-- `-malloc_dump`                                           - prints a list of all unfreed memory at the end of the run
-- `-malloc_test`                                           - like `-malloc_dump` `-malloc_debug`, only active for debugging build, ignored in optimized build. Often set in `PETSC_OPTIONS` environmental variable
-- `-malloc_view [filename]`                                - show a list of all allocated memory during `PetscFinalize()`
-- `-malloc_view_threshold t`                               - only list memory allocations of size greater than t with `-malloc_view`
-- `-malloc_requested_size`                                 - malloc logging will record the requested size rather than (possibly large) size after alignment
-- `-fp_trap`                                               - Stops on floating point exceptions
-- `-no_signal_handler`                                     - Indicates not to trap error signals
-- `-python exe`                                            - Initializes Python, and optionally takes a Python executable name
-- `-mpiuni-allow-multiprocess-launch`                      - allow `mpiexec` to launch multiple independent MPI-Uni jobs, otherwise a sanity check error is invoked to prevent misuse of MPI-Uni
-
-Options Database Keys for Option Database:
-- `-skip_petscrc`           - skip the default option files `~/.petscrc`, `.petscrc`, `petscrc`
-- `-options_monitor`        - monitor all set options to standard output for the whole program run
-- `-options_monitor_cancel` - cancel options monitoring hard-wired using `PetscOptionsMonitorSet()`
-
-Options -options_monitor_{all,cancel} are
-position-independent and apply to all options set since the PETSc start.
-They can be used also in option files.
-
-See `PetscOptionsMonitorSet()` to do monitoring programmatically.
-
-Options Database Keys for Profiling:
-See Users-Manual: ch_profiling for details.
-- `-info [filename][:[~]c1,c2,...[:[~]self]]`            - Prints verbose information for classes c1, c2, etc. See `PetscInfo()`.
-- `-log_sync`                                            - Enable barrier synchronization for all events. This option is useful to debug imbalance within each event,
-however it slows things down and gives a distorted view of the overall runtime.
-- `-log_trace [filename]`                                - Print traces of all PETSc calls to the screen (useful to determine where a program
-hangs without running in the debugger).  See `PetscLogTraceBegin()`.
-- `-log_view [:filename:format][,[:filename:format]...]` - Prints summary of flop and timing information to screen or file, see `PetscLogView()` (up to 4 viewers)
-- `-log_view_memory`                                     - Includes in the summary from -log_view the memory used in each event, see `PetscLogView()`.
-- `-log_view_gpu_time`                                   - Includes in the summary from -log_view the time used in each GPU kernel, see `PetscLogView().
-- `-log_view_gpu_energy`                                 - Includes in the summary from -log_view the energy (estimated with power*gtime) consumed in each GPU kernel, see `PetscLogView()`.
-- `-log_view_gpu_energy_meter`                           - Includes in the summary from -log_view the energy (readings from meters) consumed in each GPU kernel, see `PetscLogView()`.
-- `-log_exclude: c1,c2,...`                              - excludes subset of object classes from logging, for example vec,ksp would exclude the `Vec` and `KSP` classes
-- `-log [filename]`                                      - Logs profiling information in a dump file, see `PetscLogDump()`.
-- `-log_all [filename]`                                  - Same as `-log`.
-- `-log_mpe [filename]`                                  - Creates a logfile viewable by the utility Jumpshot (in MPICH distribution)
-- `-log_perfstubs`                                       - Starts a log handler with the perfstubs interface (which is used by TAU)
-- `-log_nvtx`                                            - Starts an nvtx log handler for use with Nsight
-- `-log_roctx`                                           - Starts an roctx log handler for use with rocprof on AMD GPUs
-- `-viewfromoptions on,off`                              - Enable or disable `XXXSetFromOptions()` calls, for applications with many small solves turn this off
-- `-get_total_flops`                                     - Returns total flops done by all processors
-- `-memory_view`                                         - Print memory usage at end of run
-- `-check_pointer_intensity 0,1,2`                       - if pointers are checked for validity (debug version only), using 0 will result in faster code
-
-Options Database Keys for SAWs:
-- `-saws_port portnumber`        - port number to publish SAWs data, default is 8080
-- `-saws_port_auto_select`       - have SAWs select a new unique port number where it publishes the data, the URL is printed to the screen
-this is useful when you are running many jobs that utilize SAWs at the same time
-- `-saws_log filename`           - save a log of all SAWs communication
-- `-saws_https certificate_file` - have SAWs use HTTPS instead of HTTP
-- `-saws_root directory`         - allow SAWs to have access to the given directory to search for requested resources and files
-
-Environmental Variables:
-- `PETSC_TMP`                     - alternative directory to use instead of `/tmp`
-- `PETSC_SHARED_TMP`              - `/tmp` is shared by all processes
-- `PETSC_NOT_SHARED_TMP`          - each process has its own private `/tmp`
-- `PETSC_OPTIONS`                 - a string containing additional options for PETSc in the form of command line "-key value" pairs
-- `PETSC_OPTIONS_YAML`            - (requires configuring PETSc to use libyaml with `--download-yaml`) a string containing additional options for PETSc in the form of a YAML document
-- `PETSC_VIEWER_SOCKET_PORT`      - socket number to use for socket viewer
-- `PETSC_VIEWER_SOCKET_MACHINE`   - machine to use for socket viewer to connect to
+- `args` - the command line, program name first
+- `file` - an optional options file, or `nothing`
+- `help` - an optional help message printed by `-help`, or `nothing`
 
 Level: beginner
 
-See also: `PetscFinalize()`, `PetscInitializeFortran()`, `PetscGetArgs()`, `PetscInitializeNoArguments()`, `PetscLogGpuTime()`
+See also: `PetscFinalize()`, `PetscInitializeNoArguments()`, `PetscGetArgs()`
 
 # External Links
 $(_doc_external("Sys/PetscInitialize"))
 """
-function PetscInitialize(petsclib::PetscLibType, args::Cchar, file::String, help::String)
-    error("PetscInitialize: no generated method for these argument types")
-end
+function PetscInitialize(petsclib::PetscLibType, args::Vector{String}, file::Union{Nothing, String} = nothing, help::Union{Nothing, String} = nothing) end
 
-@for_petsc function PetscInitialize(petsclib::$UnionPetscLib, args::Cchar, file::String, help::String )
-	argc_ = Ref{Cint}()
-
-    @chk ccall(
-               (:PetscInitialize, $petsc_library),
-               PetscErrorCode,
-               (Ptr{Cint}, Ptr{Ptr{Cchar}}, Ptr{Cchar}, Ptr{Cchar}),
-               argc_, args, file, help,
-              )
-
-	argc = argc_[]
-
-	return argc
-end 
+# the command line each library was last initialized with, kept alive for PETSc
+const _petsc_initialize_args = IdDict{Any, Any}()
 
+@for_petsc function PetscInitialize(petsclib::$UnionPetscLib, args::Vector{String}, file::Union{Nothing, String} = nothing, help::Union{Nothing, String} = nothing)
+	isempty(args) && throw(ArgumentError("args must start with a program name"))
+	args = copy(args)
+	argv = Ptr{Cchar}[pointer(a) for a in args]
+	push!(argv, C_NULL)
+	_petsc_initialize_args[petsclib] = (args, argv)
+	argc_ = Ref{Cint}(length(args))
+	args_ = Ref{Ptr{Ptr{Cchar}}}(pointer(argv))
+	@chk ccall(
+		(:PetscInitialize, $petsc_library),
+		PetscErrorCode,
+		(Ptr{Cint}, Ptr{Ptr{Ptr{Cchar}}}, Ptr{Cchar}, Ptr{Cchar}),
+		argc_, args_, something(file, C_NULL), something(help, C_NULL),
+	)
+	return nothing
+end
 """
 	PetscInitializeFortran(petsclib::PetscLibType) 
 Routine that should be called soon AFTER