Interest in using Zoltan instead of ParMETIS

[andrewdnolan]

Sharing @vijaysm's question from #470 here:

Is there any interest in moving from Metis/Parmetis to using Zoltan for your partitioning needs? Zoltan also has graph partitioners if that is what you are interested in, and the parallel performance is comparable to ParMetis [1]. In addition, Zoltan exposes other types of partitioning algorithms that are geometry- and topology-based. This is what MOAB relies on for optimal decompositions in coupled model settings.

[1] https://www.osti.gov/servlets/purl/1111609

Wanted to move this over here so it didn't get lost. I think this something we should consider, but it will be a little while till we could get around to this. We should return to this prior to Omega being merged back in E3SM.