-------------------------------------------------------------------------- By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: c401-041 Local adapter: hfi1_0 Local port: 1 -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: c401-041 Local device: hfi1_0 -------------------------------------------------------------------------- OpenBLAS blas_thread_init: pthread_create failed for thread 23 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 24 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 25 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 26 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 27 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 28 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 29 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 30 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 31 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 32 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 33 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 34 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 35 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 36 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 37 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 38 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 39 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 40 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 41 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 42 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 43 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 44 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 45 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 46 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 47 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 48 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 49 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 50 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 51 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 52 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 53 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max OpenBLAS blas_thread_init: pthread_create failed for thread 54 of 64: Resource temporarily unavailable OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 16384 max [c401-041:137516] *** Process received signal *** [c401-041:137516] Signal: Segmentation fault (11) [c401-041:137516] Signal code: Address not mapped (1) [c401-041:137516] Failing at address: 0x2b683ad909d0 [c401-041:137516] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x2b682a175090] [c401-041:137516] [ 1] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9aab)[0x2b682a32daab] [c401-041:137516] [ 2] /usr/local/lib/python3.8/dist-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-15028c96.3.21.so(blas_thread_shutdown_+0xb7)[0x2b682bb5bef7] [c401-041:137516] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x94ad0)[0x2b682a1c6ad0] [c401-041:137516] [ 4] /lib/x86_64-linux-gnu/libc.so.6(__libc_fork+0x24)[0x2b682a214f14] [c401-041:137516] [ 5] python3[0x65f0d0] [c401-041:137516] [ 6] python3(PyCFunction_Call+0xfa)[0x5f652a] [c401-041:137516] [ 7] python3(_PyObject_MakeTpCall+0x296)[0x5f7056] [c401-041:137516] [ 8] python3(_PyEval_EvalFrameDefault+0x5dae)[0x57107e] [c401-041:137516] [ 9] python3(_PyEval_EvalCodeWithName+0x26a)[0x569cea] [c401-041:137516] [10] python3(_PyFunction_Vectorcall+0x393)[0x5f6a13] [c401-041:137516] [11] python3(_PyEval_EvalFrameDefault+0x90f)[0x56bbdf] [c401-041:137516] [12] python3(_PyEval_EvalCodeWithName+0x26a)[0x569cea] [c401-041:137516] [13] python3(_PyFunction_Vectorcall+0x393)[0x5f6a13] [c401-041:137516] [14] python3[0x59c757] [c401-041:137516] [15] python3[0x5a7747] [c401-041:137516] [16] python3(PyObject_Call+0x25e)[0x5f5dfe] [c401-041:137516] [17] python3(_PyEval_EvalFrameDefault+0x1f2c)[0x56d1fc] [c401-041:137516] [18] python3(_PyEval_EvalCodeWithName+0x26a)[0x569cea] [c401-041:137516] [19] python3(_PyFunction_Vectorcall+0x393)[0x5f6a13] [c401-041:137516] [20] python3(PyObject_Call+0x62)[0x5f5c02] [c401-041:137516] [21] python3(_PyEval_EvalFrameDefault+0x1f2c)[0x56d1fc] [c401-041:137516] [22] python3(_PyEval_EvalCodeWithName+0x26a)[0x569cea] [c401-041:137516] [23] python3(_PyFunction_Vectorcall+0x393)[0x5f6a13] [c401-041:137516] [24] python3(_PyEval_EvalFrameDefault+0x1901)[0x56cbd1] [c401-041:137516] [25] python3(_PyEval_EvalCodeWithName+0x26a)[0x569cea] [c401-041:137516] [26] python3(_PyFunction_Vectorcall+0x393)[0x5f6a13] [c401-041:137516] [27] python3(_PyEval_EvalFrameDefault+0x90f)[0x56bbdf] [c401-041:137516] [28] python3(_PyFunction_Vectorcall+0x1b6)[0x5f6836] [c401-041:137516] [29] python3(_PyEval_EvalFrameDefault+0x90f)[0x56bbdf] [c401-041:137516] *** End of error message *** Segmentation fault Command exited with non-zero status 139 {"realtime":22.00,"usertime":27.22,"systime":37.24,"memmax":144500,"memavg":0}