Intel® MPI Library

Developer Guide for Linux* OS

Download PDF
ID 768728
Date 10/31/2024
Public
Document Table of Contents
Document Table of Contents x
Developer Guide for Linux* OS
Developer Guide for Linux* OS x
Introduction Installation and Prerequisites Compiling and Linking Running Applications Debugging Applications Analysis and Tuning Troubleshooting Additional Supported Features Examples Notices and Disclaimers
Introduction x
Introducing Intel® MPI Library Conventions and Symbols Related Information
Installation and Prerequisites x
Installation Prerequisite Steps
Compiling and Linking x
Compiling an MPI Program Compilers Support ILP64 Support
Running Applications x
Running Intel® MPI Library in Containers Selecting a Library Configuration Running an MPI Program Running an MPI/OpenMP* Program MPMD Launch Mode Fabrics Control Job Schedulers Support Controlling Process Placement Java* MPI Applications Support
Running Intel® MPI Library in Containers x
Build a Singularity* Container for an MPI Application Run the Application with a Container
Fabrics Control x
Selecting Fabrics Libfabric* Support OFI* Providers Support
Debugging Applications x
Debugging Using -gtool for Debugging
Analysis and Tuning x
Displaying MPI Debug Information Tracing Applications Interoperability with Other Tools Through -gtool MPI Tuning
Troubleshooting x
Error Message: Bad Termination Error Message: No such file or Directory Error Message: Permission Denied Error Message: Fatal Error Error Message: Bad File Descriptor Error Message: Too Many Open Files Problem: High Memory Consumption Readings Problem: MPI Application Hangs Problem: Password Required Problem: Cannot Execute Binary File Problem: MPI limitation for Docker*
Additional Supported Features x
Asynchronous Progress Control Device-Initiated Communications Notified One-Sided Communications Multiple Endpoints Support
Multiple Endpoints Support x
MPI_THREAD_SPLIT Programming Model Threading Runtimes Support Program Examples Code Change Guide Changes Required to Use the OpenMP* Threading Model Changes Required to Use the POSIX Threading Model See Also
Examples x
async_progress_sample.c thread_split.cpp thread_split_omp_for.c thread_split_omp_task.c thread_split_pthreads.c
Developer Guide for Linux* OS

Code Change Guide

The example in this section shows you one of the ways to change a legacy program to effectively use the advantages of the MPI_THREAD_SPLIT threading model.

In the original code (thread_split.cpp), the functions work_portion_1(), work_portion_2(), and work_portion_3() represent a CPU load that modifies the content of the memory pointed to by the in and out pointers. In this particular example, these functions perform correctness checking of the MPI_Allreduce() function.

Changes Required to Use the OpenMP* Threading Model

  1. To run MPI functions in a multithreaded environment, MPI_Init_thread() with the argument equal to MPI_THREAD_MULTIPLE must be called instead of MPI_Init().
  2. According to the MPI_THREAD_SPLIT model, in each thread you must execute MPI operations over the communicator specific to this thread only. So, in this example, the MPI_COMM_WORLD communicator must be duplicated several times so that each thread has its own copy of MPI_COMM_WORLD.

    NOTE: The limitation is that communicators must be used in such a way that the thread with thread_id n on one node communicates only with the thread with thread_id m on the other. Communications between different threads (thread_id n on one node, thread_id m on the other) are not supported.

  3. The data to transfer must be split so that each thread handles its own portion of the input and output data.
  4. The barrier becomes a two-stage one: the barriers on the MPI level and the OpenMP level must be combined.
  5. Check that the runtime sets up a reasonable affinity for OpenMP threads. Typically, the OpenMP runtime does this out of the box, but sometimes, setting up the OMP_PLACES=cores environment variable might be necessary for optimal multi-threaded MPI performance.

Changes Required to Use the POSIX Threading Model

  1. To run MPI functions in a multithreaded environment, MPI_Init_thread() with the argument equal to MPI_THREAD_MULTIPLE must be called instead of MPI_Init().
  2. You must execute MPI collective operation over a specific communicator in each thread. So the duplication of MPI_COMM_WORLD should be made, creating a specific communicator for each thread.
  3. The info key thread_id must be properly set for each of the duplicated communicators.

    NOTE: The limitation is that communicators must be used in such a way that the thread with thread_idn on one node communicates only with the thread with thread_idm on the other. Communications between different threads (thread_idn on one node, thread_idm on the other) are not supported.

  4. The data to transfer must be split so that each thread handles its own portion of the input and output data.
  5. The barrier becomes a two-stage one: the barriers on the MPI level and the POSIX level must be combined.
  6. The affinity of POSIX threads can be set up explicitly to reach optimal multithreaded MPI performance.

See Also

thread_split.cpp

Parent topic: Multiple Endpoints Support
Level Two Title