Parallel Cgns reader/writer

[a-jp]

Based on a rebased branch of develop this adds the capability to read in parallel or serial and write in parallel or serial cgns mesh files (badly named branch, sorry, bit of scope creep...).

These new readers and writers are based heavily on code from an existing code base I wrote. Given the usage of apf/pumi by other libraries this PR should also provide this read/write capability I hope to Omega_h and MFEM.

I have developed this exclusively using cgns meshes constructed via Salome. I don't believe there will be a problem coming from this, but worth noting.

The PR is controlled by ENABLE_CGNS. When off, the build behaves as normal. This PR adds several new requirements c++14 being one of them when ENABLE_CGNS=ON. Additionally, hdf5, and cgns are all dependencies once ENABLE_CGNS is set to ON.

If it's of any use I can provide scripts to build hdf5/cgns to evaluate this capability. I've pushed some test meshes to pumi-meshes also which will need a separate PR and provides testing of the new cgns capability.

I've followed the format of the other file format converters ugrid/gmsh etc, so there are comparative features like from_cgns etc.

I would add that much of my work related to testing the resulting mesh and therefore some additional tests are off by default, but can be enabled on the command_line if required. These additional tests are quite verbose.

This PR also required my previous PR which has now been merged to construct hybrid meshes in parallel using the new assemble/finalize calls.

[cwsmith]

A few initial cmake changes were added to a draft PR here:

a-jp#1

[cwsmith]

@a-jp The PR is ready for review: a-jp#1

[cwsmith]

All CGNS tests (ctest -R cgns) pass with address sanitizer enabled (-fsanitize=address, GCC 7.4).

[jedbrown]

Happy holidays, all. Does this need further work before it can merge? We'll be interested in working with CGNS meshes from your tools soon.

[cwsmith]

@jedbrown Happy Holidays!
Great. Assuming the CGNS version it used isn't too old now (need to check this) then it will only need to be synced with develop again and tested. I'll put this on the todo list for the first week of January.

[a-jp]

Hi @cwsmith long time no speak. I've always felt really guilty I didn't finish my work here. I've kept a close eye on cgns and there are a few relevant changes I'd need to check. Otherwise I'd want to check it works with latest hdf too. When I contributed this I also sent you a build script for cgns and hdf (pre spack). Probably worth getting the newer build tools working to replicate the script. Happy to contribute time to finally finish this after the holidays if that would be of use? Feel free to prod. Best place to start would to optionally enable latest cgns (parallel enabled) and hdf in your tool chain build and then let me know. 👍

[cwsmith]

@a-jp Hello! Happy Holidays!
No worries at all. Thanks for offering to help. I'll try the latest CGNS & HDF to see if there are breaking API changes. If there are, and I'm stuck, I'll post the info here.

[a-jp]

@cwsmith and to you! Ok, I think I submitted some unit tests too so we need to check those also in serial and parallel. Keep me posted.

[cwsmith]

@jedbrown @a-jp My work on this is delayed for a bit while we finish up another PR. During the break it seems I was being optimistic on the depth of the post-break todo list :\

[jedbrown]

Thanks, no worries. Please just let us know when you have an ETA. Gmsh workflows and on-the-fly meshing is working for us at the moment, but that may change quickly as we move to larger models with complex geometry.

[cwsmith]

@jedbrown @KennethEJansen I will start working on the merge this week and will post updates here as things progress.

[KennethEJansen]

@jedbrown Here I am sharing the questions that @cwsmith and I surfaced after some preliminary discussions on this last Friday. Apologies in advance for my ignorance of CGNS capabilities/formats and nascent knowledge of the PETSc aspects of the fluid application CEED-PHASTA but, exposing ignorance is the best route to eliminating it so I am going to draft my understanding of how we would like to modify SCOREC/core + CGNS to make an effective problem description input for CEED-PHASTA. I am going to put it in "phases" which might end up corresponding to a development plan.

1. The most basic need would be a globally numbered coordinates and connectivity pushed into a SERIAL CGNS data "container".

2. Next up would be unique boundary surface descriptions. I see two options here.
   a) mesh sets (a la ITAPS that Tim Tautges used). I think this will bring some challenges but keeping this brief so not elaborating.
   b) follow the gmsh approach and assign Physical Groups to surface "entities". As I understand CEED-PHASTA, boundary elements are faces so I guess this is an additional connectivity written for all the elements on the boundary (except periodic boundaries) and that 2D surface connectivity has (together or in a separate array) an extra integer identifying the model surface each face is "classified"/present on. I presume that PETSc already processes gmsh inputs with this "amount" of information into Dirichlet (nodal) boundary conditions and surface element groups so this seems like the cleanest interface. The only unknown I can think of is how/whether a parallel PETSc job would have an easy job segmenting this boundary connectivity (which presumably has global numbering) across the ranks. In the first pass we would assume that the user would "hard code" BC values into yaml and q functions and not use SCOREC/core+PHASTA notions of preparing BC fields for the problem description. Further, no actual BC types here, just surface idea that yaml +qfunctions convert to specfic BCs (pointing this out to @cameron since PHASTA processes that information out of attributes but not doing that here).

3. Last field-wise and least critical are initial conditions/solution transfer. I presume CGNS would just hold a linearly numbered (matching coordinates) field of nVars*nNodes and we just have to figure out which index to run on first. Here to we could skip that and continue to process this also from yaml and qfunctions BUT I think longer term this might be pretty easy to add and valuable of continuing cases already run in PHASTA or other.

4. Pretty much as soon as we get at least 1. and 2. working in SERIAL I hope we are able to get SCOREC/core able to write a parallel CGNS file that PETSC can read. Here is where I get super-ignorant on about every aspect of this so I will list ideas that may not contain the easy/logical solution. I am starting from the assumption that PETSc uses global numbering and thus "likes" or maybe even requires that the partition is linear in global numbering. Since SCOREC/core follows a different notion where entities have the concept of a partID and an on-rank 0..#entities on part entityID and then a further identifier that tracks which part uniquely is the owner of that entity there is a bit of a disconnect. Thinking out loud, I guess SCOREC/core can add a tag to its current parallel description that is the global node number. Since it has knowledge of the number of owned vertices on each part, it would seem a fairly easy exercise to generate that globalID to be unique and follow the on-part SCOREC/core numbering and just skip writing the non-owned nodes that are replicas of nodes owned by other parts. Where I get uncertain is how that gets written into a CGNS file so options I can think of are:
   a) serial file where a baton-passing-style write has ranks write one at a time in increasing rank order for each of the global arrays (and obviously write only their owned nodal fields). Connectivities could do the same but of course they are unique to a given part. This is a non-scalable but, only at the file write stage which might not be an issue until really big problems.

b) Does CGNS support a more parallel write where each part writes the same but CGNS tracks each "parts" portion of th write in a way that a CGNS reader in PETSc will get the data in the way PETSc requires?

[cwsmith]

develop merged into cmake_cgns_reader @ a496cc9 (this branch has everything from a-jp/core + cmake improvements) with only a few minor conflicts. Without CGNS, or Simmetrix, enabled all tests are passing. My next step is to build CGNS and enable it in the build.

[jedbrown]

I can give more detail when I'm back at a keyboard, but CGNS maps cleanly to MPI-IO primitives. There is no difference between a "parallel file". Any file can be read or written in parallel.

• There is a contiguous numbering of all nodes.
• each element topology will have one contiguous block
• boundary conditions (like gmsh physical surfaces) will refer to a list of faces. The list need not be contiguous, but it can be a simple range.
• periodic conditions should create faces for both the donor and recipient. My code currently needs these to be oriented equivalently, but does not need any supplementary info.
• Pointwise uses a convention of making separate face element blocks for each BC. The HLPW meshes I've looked at all seem to use this convention.
• CGNS has a bunch of BC types specific to fluids. I didn't have in mind to use this info in the near term, but we could revisit if we get external users. At most it would become defaults that could be overridden at run-time (yaml, etc).

[KennethEJansen]

• There is a contiguous numbering of all nodes.

I think I understand this for nodes.

• each element topology will have one contiguous block

Keeping it simple for now, I think I follow that if I have an all hex mesh then all my hex elements are in one contiguous block.
But, for that mesh, I assume I would have a second connectivity of all the quad elements that appear on the boundary, right? Gmsh takes this a step further with all the 1D (edge) elements that appear on model edges (intersections of faces) and then even another step of all model vertices as 0D "elements" I presume for now we can/will stop at a volume and face connectivity (and when we get CEED-PHASTA ready for multi-topology then that would become volume connectiviites and boundary face connectivities).

• boundary conditions (like gmsh physical surfaces) will refer to a list of faces. The list need not be contiguous, but it can be a simple range.

You say "list of faces". This sounds more like mesh sets than gmsh (at least as far as I understand gmsh). That is, I think gmsh thinks of each of its BOUNDARY mesh faces as (potentially) having a unique physical group/surface. This is how SCOREC thinks of things as well-- classification (though they take it much farther and have interior mesh faces that are classified on the model region and so on). Mesh sets is inverse classification (for each model face, these are the mesh faces classified on that model face). Clearly once you have one, getting the other is not hard but I am trying t be sure I understand what is needed (and how to write it into the CGNS file...I assume you have NOT put this into the CGNS writer that CEED-PHASTA uses since so far we only write for ParaView, right?) .

Stopping here as what is below is not critical in the first pass (I think I understand and agree with that).

• periodic conditions should create faces for both the donor and recipient. My code currently needs these to be oriented equivalently, but does not need any supplementary info.
• Pointwise uses a convention of making separate face element blocks for each BC. The HLPW meshes I've looked at all seem to use this convention.
• CGNS has a bunch of BC types specific to fluids. I didn't have in mind to use this info in the near term, but we could revisit if we get external users. At most it would become defaults that could be overridden at run-time (yaml, etc).

[jedbrown]

I think this screenshot from cgnsview will give a better picture of the organization. This file was exported by Pointwise, so follows their conventions (which seem sensible to me).

• One contiguous GridCoordinates, which gives a unique global ID to each node
• TetElements followed by PyramidElements followed by PrismElements cover the volume.
• Each semantic surface (like Nacelle here) has a Elements_t node at the same level (under blk-1) with the element connectivity. This is a Pointwise convention, and it would also be possible to lump all the triangle faces into a single file node.
• Under ZoneBC, there is a BC_t node for each semantic surface, pointing to the element range (which is contiguous because of the Pointwise convention, but need not be)
• For each BC, there is also a Family_t node under Base that can offer more semantic information about that boundary type. The screenshot here doesn't use that in a meaningful way.
• Note that Symmetry contains both triangles and quads, so there are two Elements_t nodes in the file, tri_Symmetry and quad_Symmetry, with element numbering contiguous across the two. The corresponding BC_t Symmetry node refers to the contiguous range over both face topologies.

Note that you generally don't manage writing these file nodes manually, but rather through the Boundary Condition interfaces.

To answer your question, yes this is more of the "mesh sets" model. In PETSc, we'll create the volume elements and interpolate the mesh (create the faces and edges), then broker identification of the faces through a vertex (to keep communication structured and scalable) so we can add them to the appropriate Face Sets.

[cwsmith]

@jedbrown @KennethEJansen cmake_cgns_reader was merged into develop @ 0472e54. With CGNS enabled all tests are passing. The build used CGNS develop @ fc85f2f and HDF5 1.14.0.

One of the CI test configs failed (https://github.com/SCOREC/core/actions/runs/4140959986/jobs/7160099941), but if the nightly tests pass this will be merged into master tomorrow morning.

[jedbrown]

Great! Will Ken know how to create a small sample CGNS mesh I can look at? Or can you do that or give me instructions?

[a-jp]

All when I committed the CGNS capabilities into core I also, as a parallel commit, contributed CGNS test meshes into the "pumi-meshes" repo to aid the unit tests I wrote. Possibly these will help?

[cwsmith]

@jedbrown I haven't looked into the cgns mesh generation workflow. As folks figure things out it would be nice to have a README or wiki page on the process.

The repo/directory that @a-jp is referring to is here: https://github.com/SCOREC/pumi-meshes/tree/b8973d0bd907d73218a611f8b6f7efbe580acd09/cgns
It is a submodule of scorec/core.

[jedbrown]

Thanks! Just to clarify, can scorec/core now write equivalent files to those in the repo? (I can work with those files.)

[cwsmith]

You're welcome. Possibly; there is a new API apf:WriteCGNS(...) here

core/apf/apfCGNS.cc

Line 1010 in e904a50

void WriteCGNS(const char *prefix, apf::Mesh *m, const apf::CGNSBCMap &cgnsBCMap)

@KennethEJansen and I were just skimming through its inputs.

One of the calls to it from the cgns test driver is here:

core/test/cgns.cc

Line 250 in e904a50

apf::writeCGNS(cgnsOutputName.c_str(), m, cgnsBCMap);

[jrwrigh]

Not to revive a dead branch, but just to clarify the answer:

Possibly; there is a new API apf:WriteCGNS(...) here

I've used the test/cgns.cc with the additional keyword and verified that the written-out CGNS file is valid, at least according to CGNSView.