dynamic mesh calculations on GPU?

[jasonleong84]

Hi, I want to understand the progress of the OpenFOAM 2.3.1 libraries ported over to RapidCFD. I understand that in earlier versions, the dynamic mesh calculations were done on CPU. Is this still the case now? Thanks. -Jason

[TonkomoLLC]

I think this is true that dynamic mesh calculations are done on the CPU. You can confirm by checking the code for src/dynamicMesh. I did not see references to thrust or host/device attributes when I checked just now

[jasonleong84]

Thanks for the quick reply. I am trying to compile RapidCFD now with CUDA 11.1 as you seem to have the least problems with it. I am facing the same issue as what you have reported in CUDA 11.3 (‘const_iterator’ is ambiguous). Did you ever figure out the solution to this error on 11.3?

[TonkomoLLC]

I do not have a solution for the const_iterator’ is ambiguous issue with dynamicMesh compilation - sorry! The only hints are in #92. I did not see this problem in CUDA 11.3, if I recall. I think CUDA 11.1 worked perfectly, though.
Good luck!

[jasonleong84]

Apologies... You said you saw this in 11.5 but I'm seeing this in 11.1 and 11.2.

[TonkomoLLC]

Hmmm. I am not sure what to say... I have an installation of RapidCFD with CUDA 11.2 installed and interDyMFoam works... I just double checked. Just to be sure, please make sure that if you are switching CUDA versions that there is no issue with parts of a newer version of CUDA interfering with an old version of CUDA. One way to confirm this is to ensure that the old version of CUDA is completely uninstalled (including CUDA installed, for example, from an Ubuntu PPA). Perhaps this is already your case, butI have no other ideas right now

[jasonleong84]

Thanks for the help, I will continue trying to get it installed. In CUDA 11.1 and 11.2, do you have $CUDA_HOME/extras/CUPTI/lib64?

[TonkomoLLC]

Here is what I set for cuda:
'''
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
export PATH=$PATH:$CUDA_HOME/bin
'''

[jasonleong84]

Thanks, I partially solved the problem by using CUDA 11.0 in the Nvidia HPC SDK 22.1 distribution. The CUDA 11.1 I used earlier did not have the CUPTI/lib64 folder. Unfortunately, it only compiles the libraries in the src folder but fails to compile any of the solvers in the applications folder with this error:

libOpenFOAM.so: undefined reference to void Foam::AINVPreconditioner::preconditionImpl(Foam::gpuField&, Foam::gpuField const&, unsigned char) const'`

I've seen in a previous post that you have encountered this problem before and I've tried a fresh install from a new clone of RapidCFD but I'm still seeing this error. Sorry for going off-topic and thanks for bearing with me, but if you have any suggestions here I'm all ears. I hope to test the dynamic mesh portion of RapidCFD once I can compile the solvers.

[TonkomoLLC]

I do not have first-hand experience with the issue you are reporting, as I mentioned in #72. I am glad to see that you already found #72 and are trying to contact the op who reported the same compiler issue.

You seem to be switching around CUDA versions quite a bit. Based on my experience fielding questions here, for CUDA <= 11.1, the three most common issues with RapidCFD compilation are 1) not setting the sm_# for c and c++ in wmake/rules, and 2) having a strange CUDA installation, particularly if one changes the CUDA version on the same machine, or 3) not doing a fresh install of RapidCFD when changing CUDA versions. You confirmed that you are indeed doing a fresh install, so this is not the problem.

I apologize for not having more specific advice. If you are already setting sm_# correctly in c & c++ wmake rules for your particular GPU, then I would focus on ensuring your CUDA installation is correct and not "contaminated" by other versions of CUDA that you may have previously installed.

Good luck!

[jasonleong84]

Thanks for your inputs. Unfortunately even a clean install of Nvidia drivers and CUDA compilers, the same problems still persists. However, there is some progress by following one of the suggestions in #72. I will continue our discussion there until I am able to fix this issue. Thanks again.