Intel® MPI Library

Developer Guide for Linux* OS

Download PDF
ID 768728
Date 10/31/2024
Public
Document Table of Contents
Document Table of Contents x
Developer Guide for Linux* OS
Developer Guide for Linux* OS x
Introduction Installation and Prerequisites Compiling and Linking Running Applications Debugging Applications Analysis and Tuning Troubleshooting Additional Supported Features Examples Notices and Disclaimers
Introduction x
Introducing Intel® MPI Library Conventions and Symbols Related Information
Installation and Prerequisites x
Installation Prerequisite Steps
Compiling and Linking x
Compiling an MPI Program Compilers Support ILP64 Support
Running Applications x
Running Intel® MPI Library in Containers Selecting a Library Configuration Running an MPI Program Running an MPI/OpenMP* Program MPMD Launch Mode Fabrics Control Job Schedulers Support Controlling Process Placement Java* MPI Applications Support
Running Intel® MPI Library in Containers x
Build a Singularity* Container for an MPI Application Run the Application with a Container
Fabrics Control x
Selecting Fabrics Libfabric* Support OFI* Providers Support
Debugging Applications x
Debugging Using -gtool for Debugging
Analysis and Tuning x
Displaying MPI Debug Information See Also Tracing Applications Interoperability with Other Tools Through -gtool MPI Tuning
Troubleshooting x
Error Message: Bad Termination Error Message: No such file or Directory Error Message: Permission Denied Error Message: Fatal Error Error Message: Bad File Descriptor Error Message: Too Many Open Files Problem: High Memory Consumption Readings Problem: MPI Application Hangs Problem: Password Required Problem: Cannot Execute Binary File Problem: MPI limitation for Docker*
Additional Supported Features x
Asynchronous Progress Control Device-Initiated Communications Notified One-Sided Communications Multiple Endpoints Support
Multiple Endpoints Support x
MPI_THREAD_SPLIT Programming Model Threading Runtimes Support Program Examples Code Change Guide
Examples x
async_progress_sample.c thread_split.cpp thread_split_omp_for.c thread_split_omp_task.c thread_split_pthreads.c
Developer Guide for Linux* OS

Displaying MPI Debug Information

The I_MPI_DEBUG environment variable provides a convenient way to get detailed information about an MPI application at runtime. You can set the variable value from 0 (the default value) to 1000. The higher the value, the more debug information you get. For example: 

$ mpirun  -genv I_MPI_DEBUG=2 -n 2 ./testc 
-genv I_MPI_DEBUG=2 -n 2 testc[1] MPI startup(): Internal info: pinning initialization was done[0] MPI startup(): Internal info: pinning initialization was done...
NOTE:
High values of I_MPI_DEBUG can output a lot of information and significantly reduce performance of your application. A value of I_MPI_DEBUG=5 is generally a good starting point, which provides sufficient information to find common errors.

By default, each printed line contains the MPI rank number and the message. You can also print additional information in front of each message, like process ID, time, host name and other information, or exclude some information printed by default. You can do this in two ways:

  • Add the '+' sign in front of the debug level number. In this case, each line is prefixed by the string <rank>#<pid>@<hostname>. For example: 
    $ mpirun -genv I_MPI_DEBUG=+2 -n 2 ./testc
[0#3520@clusternode1] MPI startup(): Multi-threaded optimized library
...

    To exclude any information printed in front of the message, add the '-' sign in a similar manner.

  • Add the appropriate flag after the debug level number to include or exclude some information. For example, to include time but exclude the rank number: 
    $ mpirun -genv I_MPI_DEBUG=2,time,norank -n 2 ./testc
11:59:59 MPI startup(): Multi-threaded optimized library
...

    For the list of all available flags, see the description of I_MPI_DEBUG in the Developer Reference.

To redirect the debug information output from stdout to stderr or a text file, use the I_MPI_DEBUG_OUTPUT environment variable: 

$ mpirun -genv I_MPI_DEBUG=2 -genv I_MPI_DEBUG_OUTPUT=/tmp/debug_output.txt -n 2 ./testc

Note that the output file name should not be longer than 256 symbols.

See Also

Intel® MPI Library Developer Reference, topic Other Environment Variables.

Parent topic: Analysis and Tuning
Level Two Title