RuntimeError: Job process returned error code 1

Traceback (most recent call last):
  File "/usr/local/bin/run_openkim_job.py", line 546, in <module>
    execute_in_place(runner, runner_driver, subject, subject_driver, env)
  File "/usr/local/bin/run_openkim_job.py", line 396, in execute_in_place
    raise RuntimeError(f"Job process returned error code {retcode}")
RuntimeError: Job process returned error code 1

output/pipeline.stdout:
-----------------------
BFGS:  511 12:15:37    -1203.944157         0.041029BFGS:  512 12:15:38    -1203.944513         0.040843BFGS:  513 12:15:39    -1203.944868         0.040657BFGS:  514 12:15:40    -1203.945220         0.040472BFGS:  515 12:15:41    -1203.945571         0.040287BFGS:  516 12:15:42    -1203.945920         0.040104BFGS:  517 12:15:43    -1203.946267         0.039920BFGS:  518 12:15:44    -1203.946612         0.039738BFGS:  519 12:15:44    -1203.946955         0.039556BFGS:  520 12:15:45    -1203.947297         0.039375BFGS:  521 12:15:46    -1203.947636         0.039194BFGS:  522 12:15:47    -1203.947974         0.039014BFGS:  523 12:15:48    -1203.948310         0.038835BFGS:  524 12:15:49    -1203.948644         0.038656BFGS:  525 12:15:50    -1203.948977         0.038478BFGS:  526 12:15:50    -1203.949307         0.038300BFGS:  527 12:15:51    -1203.949636         0.038123BFGS:  528 12:15:52    -1203.949963         0.037947BFGS:  529 12:15:53    -1203.950288         0.037771BFGS:  530 12:15:54    -1203.950611         0.037596

output/pipeline.stderr:
-----------------------
By default, for Open MPI 4.0 and later, infiniband ports on a deviceare not used by default.  The intent is to use UCX for these devices.You can override this policy by setting the btl_openib_allow_ib MCA parameterto true.  Local host:              c417-102  Local adapter:           hfi1_0  Local port:              1----------------------------------------------------------------------------------------------------------------------------------------------------WARNING: There was an error initializing an OpenFabrics device.  Local host:   c417-102  Local device: hfi1_0--------------------------------------------------------------------------OpenBLAS: malloc failed in gemm_driverCommand exited with non-zero status 1{"realtime":502.91,"usertime":4435.99,"systime":16130.73,"memmax":280680,"memavg":0}

output/kim.log:
---------------
2023-05-08:12:08:15CDT * 0 * information * 0x277b490 * KIM_LogImplementation.cpp:110 * Log object created.  Default verbosity level is 'information'.2023-05-08:12:08:15CDT * 1 * information * 0x277b490 * KIM_LogImplementation.cpp:242 * Log object renamed.  ID changed to '0_Collections'.2023-05-08:12:08:15CDT * 2 * information * 0_Collections * KIM_LogImplementation.cpp:246 * Log object renamed.  ID changed from '0x277b490'.2023-05-08:12:08:15CDT * 3 * information * 0_Collections * KIM_LogImplementation.cpp:120 * Log object destroyed.2023-05-08:12:08:15CDT * 0 * information * 0x277b490 * KIM_LogImplementation.cpp:110 * Log object created.  Default verbosity level is 'information'.2023-05-08:12:08:15CDT * 0 * information * 0x277ce90 * KIM_LogImplementation.cpp:110 * Log object created.  Default verbosity level is 'information'.2023-05-08:12:08:15CDT * 1 * information * 0x277ce90 * KIM_LogImplementation.cpp:242 * Log object renamed.  ID changed to '0x277b490_Collections'.2023-05-08:12:08:15CDT * 2 * information * 0x277b490_Collections * KIM_LogImplementation.cpp:246 * Log object renamed.  ID changed from '0x277ce90'.2023-05-08:12:08:15CDT * 3 * information * 0x277b490_Collections * KIM_LogImplementation.cpp:120 * Log object destroyed.2023-05-08:12:08:15CDT * 1 * information * 0x277b490 * KIM_LogImplementation.cpp:242 * Log object renamed.  ID changed to '0_SimulatorModel'.2023-05-08:12:08:15CDT * 2 * information * 0_SimulatorModel * KIM_LogImplementation.cpp:246 * Log object renamed.  ID changed from '0x277b490'.2023-05-08:12:08:15CDT * 3 * information * 0_SimulatorModel * KIM_LogImplementation.cpp:120 * Log object destroyed.2023-05-08:12:08:15CDT * 0 * information * 0x277b490 * KIM_LogImplementation.cpp:110 * Log object created.  Default verbosity level is 'information'.2023-05-08:12:08:15CDT * 0 * information * 0x277bd70 * KIM_LogImplementation.cpp:110 * Log object created.  Default verbosity level is 'information'.2023-05-08:12:08:15CDT * 1 * information * 0x277bd70 * KIM_LogImplementation.cpp:242 * Log object renamed.  ID changed to '0x277b490_Collections'.2023-05-08:12:08:15CDT * 2 * information * 0x277b490_Collections * KIM_LogImplementation.cpp:246 * Log object renamed.  ID changed from '0x277bd70'.2023-05-08:12:08:15CDT * 3 * information * 0x277b490_Collections * KIM_LogImplementation.cpp:120 * Log object destroyed.2023-05-08:12:08:15CDT * 1 * information * 0x277b490 * KIM_LogImplementation.cpp:242 * Log object renamed.  ID changed to '0_SimulatorModel'.2023-05-08:12:08:15CDT * 2 * information * 0_SimulatorModel * KIM_LogImplementation.cpp:246 * Log object renamed.  ID changed from '0x277b490'.