Visible to Intel only — GUID: GUID-5A4CD2B3-3C27-4190-B0B1-2BDF9A97B5AE
Error Message: Bad Termination
Error Message: No such file or Directory
Error Message: Permission Denied
Error Message: Fatal Error
Error Message: Bad File Descriptor
Error Message: Too Many Open Files
Problem: High Memory Consumption Readings
Problem: MPI Application Hangs
Problem: Password Required
Problem: Cannot Execute Binary File
Problem: MPI limitation for Docker*
Visible to Intel only — GUID: GUID-5A4CD2B3-3C27-4190-B0B1-2BDF9A97B5AE
Problem: MPI limitation for Docker*
Problem
The command fails with the following message:
[root@n1 /]# I_MPI_DEBUG=12 mpirun -n 2 -ppn 1 -env I_MPI_PIN_DOMAIN socket IMB-MPI1 bcast impi_shm_heap_init(): mbind failed (p=0x7f3078b0e000, size=536870912) impi_shm_heap_init(): mbind failed (p=0x7f9b808bc000, size=536870912)
Cause
MPI has a limitation on the dev/shm area. It should be not less than 4GB for a node with 2 sockets. By default, the Docker* container set 64MB, which is not enough.
Solution
- Make sure the problem is the small size of shm area:
$df -h /dev/shm root@n1 /]# df -h /dev/shm Filesystem Size Used Avail Use% Mounted on shm 4.0G 0 4.0G 0% /dev/shm
- If it is true, restart Docker using the following command:
docker run --shm-size=4gb …
Parent topic: Troubleshooting