-------------------------------------------------------------------------- By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: c403-114 Local adapter: hfi1_0 Local port: 1 -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: c403-114 Local device: hfi1_0 -------------------------------------------------------------------------- [c403-114.stampede2.tacc.utexas.edu:122141] mca_base_component_repository_open: unable to open mca_mtl_psm: /usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/mca_mtl_psm.so: failed to map segment from shared object (ignored) Traceback (most recent call last): File "../../td/EquilibriumCrystalStructure__TD_457028483760_000/runner", line 181, in comparison=aflow.compare_materials_dir(COMPARE_DIR) File "/work2/08773/bwaters/stampede2/bwaters/job-1797b2e1-5846-4ebc-b848-57b86ddd69b3-007-de5b879f-1e5a-4932-9ae1-0653da532b8f/TE_619429874221_000-and-SM_662785656123_000-1682975887/staged_job_files/repository/td/EquilibriumCrystalStructure__TD_457028483760_000/util/aflow_util.py", line 309, in compare_materials_dir output=self.aflow_command([ File "/work2/08773/bwaters/stampede2/bwaters/job-1797b2e1-5846-4ebc-b848-57b86ddd69b3-007-de5b879f-1e5a-4932-9ae1-0653da532b8f/TE_619429874221_000-and-SM_662785656123_000-1682975887/staged_job_files/repository/td/EquilibriumCrystalStructure__TD_457028483760_000/util/aflow_util.py", line 267, in aflow_command return subprocess.check_output(cmd_str, shell=True, stderr=subprocess.PIPE,encoding="utf-8") File "/usr/lib/python3.8/subprocess.py", line 415, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.8/subprocess.py", line 493, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.8/subprocess.py", line 858, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/lib/python3.8/subprocess.py", line 1639, in _execute_child self.pid = _posixsubprocess.fork_exec( OSError: [Errno 12] Cannot allocate memory Command exited with non-zero status 1 {"realtime":19.42,"usertime":27.70,"systime":43.32,"memmax":130184,"memavg":0}