-------------------------------------------------------------------------- By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: acn100 Local adapter: mlx5_0 Local port: 1 -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: acn100 Local device: mlx5_0 -------------------------------------------------------------------------- [acn100:1102815] *** Process received signal *** [acn100:1102815] Signal: Segmentation fault (11) [acn100:1102815] Signal code: (128) [acn100:1102815] Failing at address: (nil) [acn100:1102815] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f0a20759090] [acn100:1102815] [ 1] /usr/local/lib/libtorch_cpu.so(+0x46b846e)[0x7f09f742346e] [acn100:1102815] [ 2] /usr/local/lib/libtorch_cpu.so(+0x46b9b70)[0x7f09f7424b70] [acn100:1102815] [ 3] /usr/local/lib/libtorch_cpu.so(+0x46c71fb)[0x7f09f74321fb] [acn100:1102815] [ 4] /usr/local/lib/libtorch_cpu.so(_ZN5torch3jit16InterpreterState3runERSt6vectorIN3c106IValueESaIS4_EE+0x52)[0x7f09f741f212] [acn100:1102815] [ 5] /usr/local/lib/libtorch_cpu.so(+0x46a71a6)[0x7f09f74121a6] [acn100:1102815] [ 6] /usr/local/lib/libtorch_cpu.so(_ZNK5torch3jit6MethodclESt6vectorIN3c106IValueESaIS4_EERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4hashISD_ESt8equal_toISD_ESaISt4pairIKSD_S4_EEE+0x180)[0x7f09f7095170] [acn100:1102815] [ 7] /scratch.global/bwaters/jobs/bwaters/job-f0ae3902-527e-42ff-8273-f1c18e0998df-007-1d1cd6c0-7d31-40bd-a50c-b51c8e757427/TE_911154113667_003-and-MO_781946209112_000-1711746630/staged_job_files/repository/md/TorchML__MD_173118614730_000/libkim-api-model-driver.so(_ZN12PytorchModel3RunERN3c106IValueE+0x381)[0x7f0a1c0dbfb1] [acn100:1102815] [ 8] /scratch.global/bwaters/jobs/bwaters/job-f0ae3902-527e-42ff-8273-f1c18e0998df-007-1d1cd6c0-7d31-40bd-a50c-b51c8e757427/TE_911154113667_003-and-MO_781946209112_000-1711746630/staged_job_files/repository/md/TorchML__MD_173118614730_000/libkim-api-model-driver.so(_ZN32TorchMLModelDriverImplementation3RunEPKN3KIM21ModelComputeArgumentsE+0x5c)[0x7f0a1c0d9d7c] [acn100:1102815] [ 9] /scratch.global/bwaters/jobs/bwaters/job-f0ae3902-527e-42ff-8273-f1c18e0998df-007-1d1cd6c0-7d31-40bd-a50c-b51c8e757427/TE_911154113667_003-and-MO_781946209112_000-1711746630/staged_job_files/repository/md/TorchML__MD_173118614730_000/libkim-api-model-driver.so(_ZN32TorchMLModelDriverImplementation7ComputeEPKN3KIM21ModelComputeArgumentsE+0xd)[0x7f0a1c0d9e3d] [acn100:1102815] [10] /scratch.global/bwaters/jobs/bwaters/job-f0ae3902-527e-42ff-8273-f1c18e0998df-007-1d1cd6c0-7d31-40bd-a50c-b51c8e757427/TE_911154113667_003-and-MO_781946209112_000-1711746630/staged_job_files/repository/md/TorchML__MD_173118614730_000/libkim-api-model-driver.so(_ZN18TorchMLModelDriver7ComputeEPKN3KIM12ModelComputeEPKNS0_21ModelComputeArgumentsE+0x33)[0x7f0a1c0d18e3] [acn100:1102815] [11] /usr/local/lib/libkim-api.so.2(_ZNK3KIM19ModelImplementation12ModelComputeEPKNS_16ComputeArgumentsE+0x4a0)[0x7f0a2009d900] [acn100:1102815] [12] /usr/local/lib/libkim-api.so.2(_ZNK3KIM19ModelImplementation7ComputeEPKNS_16ComputeArgumentsE+0x667)[0x7f0a200abe67] [acn100:1102815] [13] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS7PairKIM7computeEii+0x20d)[0x7f0a213e2c2d] [acn100:1102815] [14] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS6Verlet5setupEi+0x3a2)[0x7f0a21328982] [acn100:1102815] [15] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS3Run7commandEiPPc+0xc7e)[0x7f0a212bdb8e] [acn100:1102815] [16] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS5Input15execute_commandEv+0xc0f)[0x7f0a2113596f] [acn100:1102815] [17] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS5Input4fileEv+0x175)[0x7f0a21135c25] [acn100:1102815] [18] lammps(+0x13fe)[0x55ea58c6d3fe] [acn100:1102815] [19] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f0a2073a083] [acn100:1102815] [20] lammps(+0x148e)[0x55ea58c6d48e] [acn100:1102815] *** End of error message *** Traceback (most recent call last): File "../../td/TriclinicPBCEnergyAndForces__TD_892847239811_003/runner", line 45, in run_lammps lammps_process = subprocess.check_call( File "/usr/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['lammps', '-in', 'isolated_atom.lammps.Si.in']' died with . During handling of the above exception, another exception occurred: Traceback (most recent call last): File "../../td/TriclinicPBCEnergyAndForces__TD_892847239811_003/runner", line 207, in isolated_atom_energies[symbol] = get_isolated_atom_energy( File "../../td/TriclinicPBCEnergyAndForces__TD_892847239811_003/runner", line 71, in get_isolated_atom_energy run_lammps(templated_input, lammps_output) File "../../td/TriclinicPBCEnergyAndForces__TD_892847239811_003/runner", line 55, in run_lammps raise Exception("LAMMPS did not exit properly:\n" + extrainfo) Exception: LAMMPS did not exit properly: LAMMPS (2 Aug 2023 - Update 1) Command exited with non-zero status 1 {"realtime":4.50,"usertime":2.35,"systime":0.95,"memmax":273480,"memavg":0}