Memory Leakage problems for GPGPU-Sim 4.x

[topcuburak]

Hello,

I am trying to carry out a CUDA application that implements matrix multiplication of 2 of 1024 * 1024 matrices. However, the process is killed because of my RAM fullness. I have searched memory leakage problems of the simulator, and I observed that there are lots of memory leakage points as mentioned below:

==30102== LEAK SUMMARY:
==30102== definitely lost: 18,642 bytes in 2,187 blocks
==30102== indirectly lost: 59,510 bytes in 1,188 blocks
==30102== possibly lost: 288 bytes in 1 blocks
==30102== still reachable: 345,513,712 bytes in 3,556,543 blocks ( I think those are the uninitialized objects)
==30102== of which reachable via heuristic:
==30102== newarray : 11,416 bytes in 47 blocks
==30102== suppressed: 0 bytes in 0 blocks

Furthermore, Valgrind stops searching for possible mem leakages after it reaches some threshold (It displays that More than 10000000 total errors detected. I'm not reporting any more.). I also tried one of the tested configurations (implemented with turning architecture) to see if the problem related to the GPU architecture I configured. Still, I got similar memory leakage results.

I am attaching both configurations and leakage outputs obtained via Valgrind. Specified flags for Valgrind is included in the output files.

The definite leakage is related to the destructor of __libc_csu_init function. Also, there are cuda_runtime_api.cc linked memory leakage points. I hope you can fix it.

turing_config.zip

ampere_config.zip

[FJShen]

Hi,
Thanks for raising this issue to our attention.
This issue is currently being fixed at accel-sim/gpgpu-sim_distribution#34. This is work in progress.

A lot of the "still reachable" blocks belong to simulated GPU parts that live throughout the entirety of the simulation, i.e. a pointer is malloc'd at the beginning of the program and will be used (i.e. cannot be freed) till the very end of the simulation. In such case there is no performance benefit in deleting those pointers before return 0 since the memory space will be reclaimed by the OS anyway when the program exits.

In your case the "definitely lost" + "indirectly lost" + "possibly lost" sum is relatively small, which means leaked memory is probably not the major reason for your program's crash - maybe it is just that the simulator is making too big a memory footprint.

Valgrind's leak memory analysis for SASS (trace driven) mode is a lot more cryptic than what you posted (assuming you were running in PTX mode) because there are lots of (~300MB) "possibly lost" blocks when simulating a relatively tiny workload and it takes effort to identify which are "still reachable" and which are "really lost". Please hold on for our updates, but in the meanwhile a faster remedy would be to upgrade your RAM.

[topcuburak]

@FJShen Well, thanks for your interest.