Out of memory runtime

[aulwes]

I'm running a large code using 220 ranks across 4 nodes of AMD MI300A. Running into out of memory error, even when using -o 'tools:nm=false;'. My colleague, Sriram, had submitted an MR, #84, that had fixed this, but I don't see his changes in latest code. Does the latest code still address the out of memory issue his MR had fixed?

[svalat]

Ah, thanks for the report, I totally rewrote this part of the code.

Now it is parallel and the code less spaghetti than it became.

There is still a limit now configured via, I suppose 350 is too large for you ?

malt -o stack:addr2lineBucket=350

You can also control the number of threads :

malt -o stack:addr2lineThreads=8

Is it solving the crash on your side ?
Do you have long paths so it eat a large part of the buffer ? (350 looked to me already low).

[svalat]

Maybe I need to pass by a file here to definitely avoid this issue.

[svalat]

Hum, is it not because you have very large binaries so maybe the stack:addr2lineThreads=8 is too large also because you multiply by the number of local mpi ranks ?

[aulwes]

ok, let me play around with those parameters. I also tried to profile only one rank using a wrapper script

#!/bin/bash

rank=${SLURM_PROCID}
if [[ "${rank}" = "" ]]; then
rank=${FLUX_TASK_RANK}
fi

TOOL=malt

exe=$1
shift

args="$@"

pargs="--mpi -o 'tools:nm=false;'"

if [[ "${rank}" = "" ]]; then
echo "Unable to get rank from env. Exiting..."
exit -1
else
if [[ ${rank} -le 0 ]]; then
echo "Running ${TOOL} ${pargs} ${exe} ${args}"
${TOOL} ${pargs} ${exe} $args
else
$exe $args
fi
fi

But when I ran, it seg faults, but I don't see any errors.

[svalat]

If you want to filter on ranks, there is now an option in MALT :

mpirun -np 200 malt --mpi -o filter:ranks=10,5-8,128 ./my_program

If ever it segfault, the fast way to get stack trace easily in 1.5.0 for MPI is :

mpirun -np 200 malt --mpi --gdb-bt ./my_program

Now I'm tracking mmap there can be an issue with that part also because it has not yet been extensively tested.

You can also try to disable in case :

malt -o c:mmap=false .......

[aulwes]

This is great!! Let me try the filter. is disabling mmap only in 1.5.0?

[svalat]

Tracking of mmap has been added in 1.4.0 and I added the option to skip in 1.5.0.

[svalat]

I made a patch : for large ELF files (larger than 50MB => -o stack:addr2lineHuge=52428800) I don't split down in buckets + use a temp file to pass all the addresses, and don't do it in parallel.

This is in currently in master branch ready for next release.

[svalat]

All the fixes are normally in 1.5.0 and 1.6.0 I just released (huge file not in parallel + in single call + passing addresses not via args but file).

Note : I renamed the option to get better consistency in 1.6.0.

malt -o addr2line:huge=......