Update Discover site configs, fix CI buillds, install GEOS-enabled spack-stack on Discover / AWS ParallelCluster

[climbfuji]

Summary

This PR unfortunately bundles several changes that became necessary as I worked through the issues.

• Container CI tests:
  • Correct mapl-mpich incompatibility
  • Update versions in jedi-ci specs
  • Build mpich@4.2.0 externally for clang container(due to problems with mpich in spack-stack - issue created in spack develop)
  • Correct Intel Compiler version (don't use oneAPI version, but the actual Intel compiler version)
  • Update OpenMPI version and associated OpenMPI oversubscribe env variables/config settings
• Ubuntu Intel CI tests: correct Intel Compiler version (don't use oneAPI version, but the actual Intel compiler version) and don't find external curl
• configs/common/modules*.yaml: set openmpi@5: oversubscribe env variables
• configs/common/packages.yaml: Use require syntax for boost
• Site config update for aws-pcluster
• Site config update for (existing) discover, rename to discover-scu16
• Add discover-scu17 site config (Milan)
• For Hercules, make current "alternative" GNU+OpenMPI stack the only option for GNU (i.e. remove mvapich2)
• Make mysql an optional dependency (variant) for ewok-env, off by default
• Documentation updates for all of the above

Testing

• Discover SCU16 GNU
  • Build jedi-bundle / skylab and run experiments qg-fullDA, l95-fullDA, skylab-aero-weather (1 cycle)
  • Run jedi-bundle ctests (jedi-bundle branch feature/no_oops_trapfpe)
  • Build GEOS-GCM and run a c12 test
• Discover SCU16 Intel
  • Note I had to add LDFLAGS="-L/usr/local/other/gcc/11.2.0/lib64" to the cmake command when building jedi-bundle.
  • Build jedi-bundle / skylab and run experiments qg-fullDA, l95-fullDA, skylab-aero-weather (1 cycle)
  • Run jedi-bundle ctests (jedi-bundle branch feature/no_oops_trapfpe)
  • Build GEOS-GCM and run a c12 test - deferred to Discover issues (follow-up to PR https://github.com/JCSDA/spack-stack/pull/993) #1011
• Discover SCU17 GNU
  • Note I had to load ecflow_ui like this: LD_PRELOAD="/usr/local/other/gcc/12.3.0/lib64/libstdc++.so" ecflow_ui
  • Build jedi-bundle / skylab and run experiments qg-fullDA, l95-fullDA, skylab-aero-weather (1 cycle)
  • Run jedi-bundle ctests (jedi-bundle branch feature/no_oops_trapfpe)
  • Build GEOS-GCM and run a c12 test
• Discover SCU17 Intel
  • Build jedi-bundle / skylab and run experiments qg-fullDA, l95-fullDA- deferred to Discover issues (follow-up to PR https://github.com/JCSDA/spack-stack/pull/993) #1011
  • Note Skylab experiment skylab-aero-weather hangs in variational task somewhere in bump?)
  • Run jedi-bundle ctests (jedi-bundle branch feature/no_oops_trapfpe)- deferred to Discover issues (follow-up to PR https://github.com/JCSDA/spack-stack/pull/993) #1011
  • try Build GEOS-GCM and run a c12 test - not successful, see known issue Segfault in GEOS-GCM with Intel on AWS ParallelCluster #988
• AWS ParallelCluster GNU
  • Build jedi-bundle / skylab and run experiments qg-fullDA, l95-fullDA, skylab-atm-land-small, skylab-trace-gas (1 cycle), skylab-aero-weather (1 cycle)
  • Build GEOS-GCM and run a c12 test
• AWS ParallelCluster Intel
  • Build jedi-bundle / skylab and run experiments qg-fullDA, l95-fullDA, skylab-atm-land-small, skylab-trace-gas (1 cycle), skylab-aero-weather (1 cycle)
  • try Build GEOS-GCM and run a c12 test - not successful, see known issue Segfault in GEOS-GCM with Intel on AWS ParallelCluster #988
• @climbfuji's macOS (M1 / Monterey) Clang (13.1.6)
  • Build jedi-bundle / skylab and run experiments qg-fullDA, l95-fullDA, skylab-atm-land-small
  • Build GEOS-GCM and run a c12 test
    • Used: cmake -DUSE_F2PY=OFF -DCMAKE_SHARED_LINKER_FLAGS="-Wl,-flat_namespace" -DCMAKE_EXE_LINKER_FLAGS="-Wl,-flat_namespace" -DCMAKE_INSTALL_PREFIX=/Users/heinzell/scratch/geos-gcm-spack-stack-20210118/GEOS_20240119/GEOSgcm/install-20240212 .. 2>&1 | tee log.cmake

Applications affected

List all known applications (UFS WM, JEDI, SRW, etc.) intentionally or unintentionally affected by this PR.

Systems affected

List all systems intentionally or unintentionally affected by this PR.

Dependencies

• waiting on Add patch for building ecflow@5.11.4 with Intel oneAPI classic compilers spack#402
• waiting on For ecflow@5.11.4, apply ctsapi_cassert.path for all compilers, not just Intel Classic spack#404

Issue(s) addressed

Resolves #1001
Resolves #996
Resolves #987

Checklist

• This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.

[climbfuji]

@mathomp4 FYI - here are the site config updates for Discover SCU16 and SCU17. I am going to add you as reviewer, but I don't expect you to look at all of this - if you could take a peek at the Discover stuff, that would be appreciated though. Thanks!

[mathomp4]

The discover stuff looks good to me!

[srherbener]

The changes for openmi 5.x and hdf5 1.14.3 look good. Thanks!