startup time scales with large number of Exodus labeled/side sets

[Chuyang-Liu]

I am opening this issue to document a startup-scaling problem we are seeing when using Exodus meshes with a large number of labeled sets/side sets.

In a previous test, an Exodus mesh with roughly 20,000 total side sets and labeled sets took about 2–3 hours on NERSC before the simulation actually started. The run appeared to spend most of this time around mesh creation/partitioning, for example, near:

MESH_PartitionWithZoltan: Using partitioning method RCB for ZOLTAN

In my current mesh, the number of diagnostic regions is much larger, with approximately 7.4 times more total side sets and labeled sets. Based on the previous timing, the expected startup time could be:

3 hours × 7.4 ≈ 22.3 hours

This is a major practical issue because the maximum wall-time limit on NERSC is 48 hours. Spending about 22 hours just on startup / mesh-region loading would leave much less time for the actual simulation.

An important detail is that, for the steady-state and cyclic steady-state runs, most of these diagnostic regions are not referenced in the XML input file under either:

ParameterList name="regions" type="ParameterList"

or:

ParameterList name="observations" type="ParameterList"

However, because the labeled sets and side sets are present in the Exodus mesh itself, ATS/Amanzi still appears to spend substantial time during mesh creation/partitioning.

Would it be possible to improve this part of the mesh/region handling? For example:

1. Only process Exodus labeled/side sets that are actually referenced in the XML input file.
2. Speed up the loading or construction of regions from the Exodus mesh.

Any suggestions on best practices, possible input options, or future code improvements would be greatly appreciated.

Thank you very much.

[jd-moulton]

It's possible that this part of the mesh handling could be improved, but as you can tell from the scaling it was not intended to have so many face sets/side sets. There may be better ways to do what you need to do, and yet having options to ignore regions that are not used in the current input file might useful. We'll need to think through any downstream issues that might cause.

Shorter term though you can pre-partition your mesh and use that partitioned mesh for each run. At least in that case you are only paying for the cost of partitioning once, and it could be done on another machine. The catch is the partitioned mesh is locked into the number of cores you've used to partition it, and so much like the checkpoint files that part of future simulations cannot change.

An example is given on Pin Shuai's page

https://pinshuai.github.io/Compile-ATS-1.0-on-NERSC/

It's a little out of date, so don't pay much attention to the details but the key command is meshconvert (which is an MSTK utility that is built and installed when the TPLs are built and installed). So it will be in the bin directory of your TPLs installation.

srun -n 128 meshconvert --partition-method=2 ../CoalCreek_mesh-100m-frac1pct-landcover.exo ./CoalCreek_final_mesh.par

Once you have partitioned the mesh you will see the same number of files as MPI ranks you used (in this case 128). That is the mesh, and you just need to change the extension for references to it from exo to par.

[Chuyang-Liu]

Thank you so much for the explanation and suggestion, David! This makes sense. I will try pre-partitioning the mesh using meshconvert and then use the resulting .par mesh for the ATS runs.