Intel® MPI Library Developer Guide for Linux* OS

Download PDF
ID 768728
Date 6/24/2024
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
Intel® MPI Library Developer Guide for Linux* OS
Intel® MPI Library Developer Guide for Linux* OS x
Introduction Installation and Prerequisites Compiling and Linking Running Applications Debugging Applications Analysis and Tuning Troubleshooting Additional Supported Features Examples Notices and Disclaimers
Introduction x
Introducing Intel® MPI Library Conventions and Symbols Related Information
Installation and Prerequisites x
Installation Prerequisite Steps
Compiling and Linking x
Compiling an MPI Program Compilers Support ILP64 Support
Running Applications x
Running Intel® MPI Library in Containers Selecting a Library Configuration Running an MPI Program Running an MPI/OpenMP* Program MPMD Launch Mode Fabrics Control Job Schedulers Support Controlling Process Placement Java* MPI Applications Support
Running Intel® MPI Library in Containers x
Build a Singularity* Container for an MPI Application Run the Application with a Container
Fabrics Control x
Selecting Fabrics Libfabric* Support OFI* Providers Support
Debugging Applications x
Debugging Using -gtool for Debugging
Analysis and Tuning x
Displaying MPI Debug Information Tracing Applications Interoperability with Other Tools Through -gtool MPI Tuning
Troubleshooting x
Error Message: Bad Termination Error Message: No such file or Directory Error Message: Permission Denied Error Message: Fatal Error Error Message: Bad File Descriptor Error Message: Too Many Open Files Problem: High Memory Consumption Readings Problem: MPI Application Hangs Problem: Password Required Problem: Cannot Execute Binary File Problem: MPI limitation for Docker*
Additional Supported Features x
Asynchronous Progress Control Multiple Endpoints Support
Multiple Endpoints Support x
MPI_THREAD_SPLIT Programming Model Threading Runtimes Support Program Examples Code Change Guide
Examples x
async_progress_sample.c thread_split.cpp thread_split_omp_for.c thread_split_omp_task.c thread_split_pthreads.c
Intel® MPI Library Developer Guide for Linux* OS

Problem: MPI limitation for Docker*

Problem

The command fails with the following message: 

[root@n1 /]# I_MPI_DEBUG=12 mpirun -n 2 -ppn 1 -env I_MPI_PIN_DOMAIN socket IMB-MPI1 bcast
impi_shm_heap_init(): mbind failed (p=0x7f3078b0e000, size=536870912)
impi_shm_heap_init(): mbind failed (p=0x7f9b808bc000, size=536870912)

Cause

MPI has a limitation on the dev/shm area. It should be not less than 4GB for a node with 2 sockets. By default, the Docker* container set 64MB, which is not enough.

Solution

  1. Make sure the problem is the small size of shm area: 
    $df -h /dev/shm
root@n1 /]# df -h /dev/shm
Filesystem      Size Used Avail Use% Mounted on
shm             4.0G   0  4.0G   0%  /dev/shm
  2. If it is true, restart Docker using the following command: 
    docker run --shm-size=4gb …
Parent topic: Troubleshooting
Level Two Title