-------------------------------------------------------------------------- By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: acn100 Local adapter: mlx5_0 Local port: 1 -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There is at least non-excluded one OpenFabrics device found, but there are no active ports detected (or Open MPI was unable to use them). This is most certainly not what you wanted. Check your cables, subnet manager configuration, etc. The openib BTL will be ignored for this job. Local host: acn100 -------------------------------------------------------------------------- [acn100:1213849] *** Process received signal *** [acn100:1213849] Signal: Segmentation fault (11) [acn100:1213849] Signal code: (128) [acn100:1213849] Failing at address: (nil) [acn100:1213849] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7feff0685090] [acn100:1213849] [ 1] /usr/local/lib/libtorch_cpu.so(+0x46b846e)[0x7fefc742346e] [acn100:1213849] [ 2] /usr/local/lib/libtorch_cpu.so(+0x46b9b70)[0x7fefc7424b70] [acn100:1213849] [ 3] /usr/local/lib/libtorch_cpu.so(+0x46c71fb)[0x7fefc74321fb] [acn100:1213849] [ 4] /usr/local/lib/libtorch_cpu.so(_ZN5torch3jit16InterpreterState3runERSt6vectorIN3c106IValueESaIS4_EE+0x52)[0x7fefc741f212] [acn100:1213849] [ 5] /usr/local/lib/libtorch_cpu.so(+0x46a71a6)[0x7fefc74121a6] [acn100:1213849] [ 6] /usr/local/lib/libtorch_cpu.so(_ZNK5torch3jit6MethodclESt6vectorIN3c106IValueESaIS4_EERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4hashISD_ESt8equal_toISD_ESaISt4pairIKSD_S4_EEE+0x180)[0x7fefc7095170] [acn100:1213849] [ 7] /scratch.global/bwaters/jobs/bwaters/job-591f9799-ed27-46c9-9285-e93e228e9acf-007-5b7841d7-532a-437e-84b4-a0075615e780/TE_429062973441_003-and-MO_781946209112_000-1711746991/staged_job_files/repository/md/TorchML__MD_173118614730_000/libkim-api-model-driver.so(_ZN12PytorchModel3RunERN3c106IValueE+0x381)[0x7fefe6532fb1] [acn100:1213849] [ 8] /scratch.global/bwaters/jobs/bwaters/job-591f9799-ed27-46c9-9285-e93e228e9acf-007-5b7841d7-532a-437e-84b4-a0075615e780/TE_429062973441_003-and-MO_781946209112_000-1711746991/staged_job_files/repository/md/TorchML__MD_173118614730_000/libkim-api-model-driver.so(_ZN32TorchMLModelDriverImplementation3RunEPKN3KIM21ModelComputeArgumentsE+0x5c)[0x7fefe6530d7c] [acn100:1213849] [ 9] /scratch.global/bwaters/jobs/bwaters/job-591f9799-ed27-46c9-9285-e93e228e9acf-007-5b7841d7-532a-437e-84b4-a0075615e780/TE_429062973441_003-and-MO_781946209112_000-1711746991/staged_job_files/repository/md/TorchML__MD_173118614730_000/libkim-api-model-driver.so(_ZN32TorchMLModelDriverImplementation7ComputeEPKN3KIM21ModelComputeArgumentsE+0xd)[0x7fefe6530e3d] [acn100:1213849] [10] /scratch.global/bwaters/jobs/bwaters/job-591f9799-ed27-46c9-9285-e93e228e9acf-007-5b7841d7-532a-437e-84b4-a0075615e780/TE_429062973441_003-and-MO_781946209112_000-1711746991/staged_job_files/repository/md/TorchML__MD_173118614730_000/libkim-api-model-driver.so(_ZN18TorchMLModelDriver7ComputeEPKN3KIM12ModelComputeEPKNS0_21ModelComputeArgumentsE+0x33)[0x7fefe65288e3] [acn100:1213849] [11] /usr/local/lib/libkim-api.so.2(_ZNK3KIM19ModelImplementation12ModelComputeEPKNS_16ComputeArgumentsE+0x4a0)[0x7fefeffc9900] [acn100:1213849] [12] /usr/local/lib/libkim-api.so.2(_ZNK3KIM19ModelImplementation7ComputeEPKNS_16ComputeArgumentsE+0x667)[0x7fefeffd7e67] [acn100:1213849] [13] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS7PairKIM7computeEii+0x20d)[0x7feff130ec2d] [acn100:1213849] [14] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS6Verlet5setupEi+0x3a2)[0x7feff1254982] [acn100:1213849] [15] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS3Run7commandEiPPc+0xc7e)[0x7feff11e9b8e] [acn100:1213849] [16] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS5Input15execute_commandEv+0xc0f)[0x7feff106196f] [acn100:1213849] [17] /usr/local/lib/liblammps.so.0(_ZN9LAMMPS_NS5Input4fileEv+0x175)[0x7feff1061c25] [acn100:1213849] [18] lammps(+0x13fe)[0x55d6715173fe] [acn100:1213849] [19] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7feff0666083] [acn100:1213849] [20] lammps(+0x148e)[0x55d67151748e] [acn100:1213849] *** End of error message *** Traceback (most recent call last): File "../../td/ClusterEnergyAndForces__TD_000043093022_003/runner", line 46, in run_lammps lammps_process = subprocess.check_call( File "/usr/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['lammps', '-in', 'isolated_atom.lammps.Si.in']' died with . During handling of the above exception, another exception occurred: Traceback (most recent call last): File "../../td/ClusterEnergyAndForces__TD_000043093022_003/runner", line 217, in isolated_atom_energies[symbol] = get_isolated_atom_energy( File "../../td/ClusterEnergyAndForces__TD_000043093022_003/runner", line 72, in get_isolated_atom_energy run_lammps(templated_input, lammps_output) File "../../td/ClusterEnergyAndForces__TD_000043093022_003/runner", line 56, in run_lammps raise Exception("LAMMPS did not exit properly:\n" + extrainfo) Exception: LAMMPS did not exit properly: LAMMPS (2 Aug 2023 - Update 1) Command exited with non-zero status 1 {"realtime":5.11,"usertime":2.02,"systime":0.64,"memmax":273784,"memavg":0}