Generating Gauss mesh with cubedsphere partition causes crash on non-Rubiks MPI

### What happened?

It is currently only possible to use the structured mesh generator to construct a mesh from a Gauss grid partitioned like a cubedsphere, for example O12 partitioned like CS12, if the number of MPI ranks is set to either 1 or 6 x (odd number)**2 (i.e. the sphere is partitioned like a Rubik's cube and the poles fall entirely within a single rank). In the case where the generation fails, a single MPI rank around the south pole throws an exception and calls MPI_ABORT.

This has been tested in an atlas fork using the following ctests
 - [cc file for test]( https://github.com/lewisn-met/atlas/blob/bugfix/gauss-with-cubedsphere-partition/src/tests/mesh/test_gauss_with_cs_partition.cc)
 - [CMakeLists for different MPI configurations](https://github.com/lewisn-met/atlas/blob/77731e330d12b844fc742e2a5b268d69129886ce/src/tests/mesh/CMakeLists.txt#L98-L128t)

The issue has been isolated to [this line](https://github.com/lewisn-met/atlas/blob/77731e330d12b844fc742e2a5b268d69129886ce/src/atlas/meshgenerator/detail/StructuredMeshGenerator.cc#L1593) in the StructuredMeshGenerator.

To add more detail to the example of O12 partitioned like CS12 running on 12 MPI ranks:
The call `node_numbering.at()` fails on rank 10 with 
`vector::_M_range_check: __n (which is 119) >= this->size() (which is 106)` 
and printing out variables in this line yields:
```
node_numbering.size(): 106
ilat :6
offset_loc.at(ilat): 100
ip1: 19
```

An ideal resolution to this would be for the structured mesh generation to work regardless of the number of MPI ranks.




### What are the steps to reproduce the bug?

 - Clone [this fork](https://github.com/lewisn-met/atlas) and checkout the branch bugfix/gauss-with-cubedsphere-partition. 
 - Build Atlas as you would develop
 - Run the test `atlas_test_gauss_with_cs_partition_MPI_12`
 - This will throw the exception and print some additional debug output around where the issue occurs.

### Version

0.46.0

### Platform (OS and architecture)

x86_64 GNU/Linux

### Relevant log output

<details>

```shell
Running 6 tests:
Running case 0: CS12 O12
Test "CS12 O12" failed with unhandled exception: vector::_M_range_check: __n (which is 119) >= this->size() (which is 106)
[cazldf00003T:00000] An error occurred in MPI_Allgather
[cazldf00003T:00000] reported by process [1074855937,10]
[cazldf00003T:00000] on communicator MPI_COMM_WORLD
[cazldf00003T:00000] MPI_ERR_TRUNCATE: message truncated
[cazldf00003T:00000] MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[cazldf00003T:00000] and MPI will try to terminate your MPI job as well)

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 11 in communicator MPI_COMM_WORLD
  Proc: [[16401,1],11]
  Errorcode: -1

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

[9] =========================================
[9] Terminated (signal intercepted by atlas)
[9] -----------------------------------------
[9] BACKTRACE
[9] -----------------------------------------
[9] backtrace [1] stack has 23 addresses
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[9] (/lib64/libc.so.6+)0x3fc30 
[9] (/lib64/libc.so.6+clock_nanosleep)0x65 
[9] (/lib64/libc.so.6+__nanosleep)0x17 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12+void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&))0x9b 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[9] (/lib64/libc.so.6+)0x2a610 
[9] (/lib64/libc.so.6+__libc_start_main)0x80 
[9] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[9] end of backtrace dump ...
[9] =========================================

[9] Calling MPI_Abort

[3] =========================================
[3] Terminated (signal intercepted by atlas)
[3] -----------------------------------------
[3] BACKTRACE
[3] -----------------------------------------
[3] backtrace [1] stack has 23 addresses
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[3] (/lib64/libc.so.6+)0x3fc30 
[3] (/lib64/libc.so.6+clock_nanosleep)0x65 
[3] (/lib64/libc.so.6+__nanosleep)0x17 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12+void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&))0x9b 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[3] (/lib64/libc.so.6+)0x2a610 
[3] (/lib64/libc.so.6+__libc_start_main)0x80 
[3] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[3] end of backtrace dump ...
[3] =========================================

[3] Calling MPI_Abort

[7] =========================================
[7] Terminated (signal intercepted by atlas)
[7] -----------------------------------------
[7] BACKTRACE
[7] -----------------------------------------
[7] backtrace [1] stack has 23 addresses
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[7] (/lib64/libc.so.6+)0x3fc30 
[7] (/lib64/libc.so.6+clock_nanosleep)0x65 
[7] (/lib64/libc.so.6+__nanosleep)0x17 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12+void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&))0x9b 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[7] (/lib64/libc.so.6+)0x2a610 
[7] (/lib64/libc.so.6+__libc_start_main)0x80 
[7] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[7] end of backtrace dump ...
[7] =========================================

[7] Calling MPI_Abort

[11] =========================================
[11] Terminated (signal intercepted by atlas)
[11] -----------------------------------------
[11] BACKTRACE
[11] -----------------------------------------
[11] backtrace [1] stack has 23 addresses
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[11] (/lib64/libc.so.6+)0x3fc30 
[11] (/lib64/libc.so.6+clock_nanosleep)0x65 
[11] (/lib64/libc.so.6+__nanosleep)0x17 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12+void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&))0x9b 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[11] (/lib64/libc.so.6+)0x2a610 
[11] (/lib64/libc.so.6+__libc_start_main)0x80 
[11] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[11] end of backtrace dump ...
[11] =========================================

[11] Calling MPI_Abort

[2] =========================================
[2] Terminated (signal intercepted by atlas)
[2] -----------------------------------------
[2] BACKTRACE
[2] -----------------------------------------
[2] backtrace [1] stack has 35 addresses
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[2] (/lib64/libc.so.6+)0x3fc30 
[2] (/lib64/libc.so.6+__sched_yield)0xb 
[2] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libopen-pal.so.80+opal_progress)0xb9 
[2] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_request_default_wait)0x120 
[2] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_sendrecv_actual)0xb4 
[2] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_allgather_intra_bruck)0x114 
[2] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_tuned_allgather_intra_dec_fixed)0x4a 
[2] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+PMPI_Allgather)0x14a 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit_mpi.so+eckit::mpi::Parallel::allGather(void const*, unsigned long, void*, unsigned long, eckit::mpi::Data::Code) const)0x11d 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+void eckit::mpi::Comm::allGather<long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > > >(long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >) const)0x14a 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generateGlobalElementNumbering(atlas::Mesh&) const)0x196 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate_mesh(atlas::StructuredGrid const&, atlas::grid::Distribution const&, atlas::meshgenerator::Region const&, atlas::Mesh&) const)0x4dd2 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&, atlas::Mesh&) const)0x915 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x49 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::MeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x3c 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src
[1] =========================================
[1] Terminated (signal intercepted by atlas)
[1] -----------------------------------------
[1] BACKTRACE
[1] -----------------------------------------
[1] backtrace [1] stack has 23 addresses
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[1] (/lib64/libc.so.6+)0x3fc30 
[1] (/lib64/libc.so.6+clock_nanosleep)0x65 
[1] (/lib64/libc.so.6+__nanosleep)0x17 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12+void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&))0x9b 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[1] (/lib64/libc.so.6+)0x2a610 
[1] (/lib64/libc.so.6+__libc_start_main)0x80 
[1] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[1] end of backtrace dump ...
[1] =========================================

[1] Calling MPI_Abort

[0] =========================================
[0] Terminated (signal intercepted by atlas)
[0] -----------------------------------------
[0] BACKTRACE
[0] -----------------------------------------
[0] backtrace [1] stack has 35 addresses
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[0] (/lib64/libc.so.6+)0x3fc30 
[0] (/lib64/libc.so.6+__sched_yield)0xb 
[0] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libopen-pal.so.80+opal_progress)0xb9 
[0] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_request_default_wait)0x120 
[0] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_sendrecv_actual)0xb4 
[0] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_allgather_intra_bruck)0x114 
[0] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_tuned_allgather_intra_dec_fixed)0x4a 
[0] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+PMPI_Allgather)0x14a 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit_mpi.so+eckit::mpi::Parallel::allGather(void const*, unsigned long, void*, unsigned long, eckit::mpi::Data::Code) const)0x11d 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+void eckit::mpi::Comm::allGather<long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > > >(long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >) const)0x14a 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generateGlobalElementNumbering(atlas::Mesh&) const)0x196 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate_mesh(atlas::StructuredGrid const&, atlas::grid::Distribution const&, atlas::meshgenerator::Region const&, atlas::Mesh&) const)0x4dd2 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&, atlas::Mesh&) const)0x915 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x49 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::MeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x3c 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src
[5] =========================================
[5] Terminated (signal intercepted by atlas)
[5] -----------------------------------------
[5] BACKTRACE
[5] -----------------------------------------
[5] backtrace [1] stack has 23 addresses
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[5] (/lib64/libc.so.6+)0x3fc30 
[5] (/lib64/libc.so.6+clock_nanosleep)0x65 
[5] (/lib64/libc.so.6+__nanosleep)0x17 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12+void std::this_thread::sleep_for<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&))0x9b 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[5] (/lib64/libc.so.6+)0x2a610 
[5] (/lib64/libc.so.6+__libc_start_main)0x80 
[5] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[5] end of backtrace dump ...
[5] =========================================

[5] Calling MPI_Abort

[8] =========================================
[8] Terminated (signal intercepted by atlas)
[8] -----------------------------------------
[8] BACKTRACE
[8] -----------------------------------------
[8] backtrace [1] stack has 35 addresses
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[8] (/lib64/libc.so.6+)0x3fc30 
[8] (/lib64/libc.so.6+__sched_yield)0xb 
[8] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libopen-pal.so.80+opal_progress)0xb9 
[8] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_request_default_wait)0x120 
[8] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_sendrecv_actual)0xb4 
[8] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_allgather_intra_bruck)0x114 
[8] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_tuned_allgather_intra_dec_fixed)0x4a 
[8] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+PMPI_Allgather)0x14a 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit_mpi.so+eckit::mpi::Parallel::allGather(void const*, unsigned long, void*, unsigned long, eckit::mpi::Data::Code) const)0x11d 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+void eckit::mpi::Comm::allGather<long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > > >(long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >) const)0x14a 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generateGlobalElementNumbering(atlas::Mesh&) const)0x196 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate_mesh(atlas::StructuredGrid const&, atlas::grid::Distribution const&, atlas::meshgenerator::Region const&, atlas::Mesh&) const)0x4dd2 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&, atlas::Mesh&) const)0x915 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x49 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::MeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x3c 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src
[4] =========================================
[4] Terminated (signal intercepted by atlas)
[4] -----------------------------------------
[4] BACKTRACE
[4] -----------------------------------------
[4] backtrace [1] stack has 35 addresses
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[4] (/lib64/libc.so.6+)0x3fc30 
[4] (/lib64/libc.so.6+__sched_yield)0xb 
[4] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libopen-pal.so.80+opal_progress)0xb9 
[4] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_request_default_wait)0x120 
[4] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_sendrecv_actual)0xb4 
[4] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_allgather_intra_bruck)0x114 
[4] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_tuned_allgather_intra_dec_fixed)0x4a 
[4] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+PMPI_Allgather)0x14a 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit_mpi.so+eckit::mpi::Parallel::allGather(void const*, unsigned long, void*, unsigned long, eckit::mpi::Data::Code) const)0x11d 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+void eckit::mpi::Comm::allGather<long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > > >(long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >) const)0x14a 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generateGlobalElementNumbering(atlas::Mesh&) const)0x196 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate_mesh(atlas::StructuredGrid const&, atlas::grid::Distribution const&, atlas::meshgenerator::Region const&, atlas::Mesh&) const)0x4dd2 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&, atlas::Mesh&) const)0x915 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x49 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::MeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x3c 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src
[6] =========================================
[6] Terminated (signal intercepted by atlas)
[6] -----------------------------------------
[6] BACKTRACE
[6] -----------------------------------------
[6] backtrace [1] stack has 35 addresses
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x48 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::backtrace[abi:cxx11]())0x18 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::library::atlas_signal_handler(int, siginfo_t*, void*))0x1e9 
[6] (/lib64/libc.so.6+)0x3fc30 
[6] (/lib64/libc.so.6+__sched_yield)0xb 
[6] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libopen-pal.so.80+opal_progress)0xb9 
[6] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_request_default_wait)0x120 
[6] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_sendrecv_actual)0xb4 
[6] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_base_allgather_intra_bruck)0x114 
[6] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+ompi_coll_tuned_allgather_intra_dec_fixed)0x4a 
[6] (/home/users/darth/opt/mobbs-63/util/spice_gnu/lib/libmpi.so.40+PMPI_Allgather)0x14a 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libeckit_mpi.so+eckit::mpi::Parallel::allGather(void const*, unsigned long, void*, unsigned long, eckit::mpi::Data::Code) const)0x11d 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+void eckit::mpi::Comm::allGather<long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > > >(long, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >, __gnu_cxx::__normal_iterator<long*, std::vector<long, std::allocator<long> > >) const)0x14a 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generateGlobalElementNumbering(atlas::Mesh&) const)0x196 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate_mesh(atlas::StructuredGrid const&, atlas::grid::Distribution const&, atlas::meshgenerator::Region const&, atlas::Mesh&) const)0x4dd2 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::StructuredMeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&, atlas::Mesh&) const)0x915 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::meshgenerator::MeshGeneratorImpl::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x49 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/lib/libatlas.so.0.46+atlas::MeshGenerator::generate(atlas::Grid const&, atlas::grid::Distribution const&) const)0x3c 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[2] (/lib64/libc.so.6+)0x2a610 
[2] (/lib64/libc.so.6+__libc_start_main)0x80 
[2] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[2] end of backtrace dump ...
[2] =========================================

[2] Calling MPI_Abort
/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[0] (/lib64/libc.so.6+)0x2a610 
[0] (/lib64/libc.so.6+__libc_start_main)0x80 
[0] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[0] end of backtrace dump ...
[0] =========================================

[0] Calling MPI_Abort
/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[8] (/lib64/libc.so.6+)0x2a610 
[8] (/lib64/libc.so.6+__libc_start_main)0x80 
[8] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[8] end of backtrace dump ...
[8] =========================================

[8] Calling MPI_Abort
/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[4] (/lib64/libc.so.6+)0x2a610 
[4] (/lib64/libc.so.6+__libc_start_main)0x80 
[4] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[4] end of backtrace dump ...
[4] =========================================

[4] Calling MPI_Abort
/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 
[6] (/lib64/libc.so.6+)0x2a610 
[6] (/lib64/libc.so.6+__libc_start_main)0x80 
[6] (/data/users/lewis.napper/Documents/atlasMOBundle/build-mo-bundle-dev-Debug/atlas/src/tests/mesh/atlas_test_gauss_with_cs_partition_MPI_12) 

[6] end of backtrace dump ...
[6] =========================================

[6] Calling MPI_Abort
--------------------------------------------------------------------------
prterun has exited due to process rank 10 with PID 0 on node cazldf00003T calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
--------------------------------------------------------------------------
```

</details>

### Accompanying data

_No response_

### Organisation

UKMO

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generating Gauss mesh with cubedsphere partition causes crash on non-Rubiks MPI #367

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Generating Gauss mesh with cubedsphere partition causes crash on non-Rubiks MPI #367

Description

What happened?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

Organisation

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions