Intel® MPI Library Developer Guide for Linux* OS

Download PDF
ID 768728
Date 11/07/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
Intel® MPI Library Developer Guide for Linux* OS
Intel® MPI Library Developer Guide for Linux* OS x
Introduction Installation and Prerequisites Compiling and Linking Running Applications Debugging Applications Analysis and Tuning Troubleshooting Additional Supported Features Examples Notices and Disclaimers
Introduction x
Introducing Intel® MPI Library Conventions and Symbols Related Information
Installation and Prerequisites x
Installation Prerequisite Steps
Compiling and Linking x
Compiling an MPI Program Compilers Support ILP64 Support
Running Applications x
Running Intel® MPI Library in Containers Selecting a Library Configuration Running an MPI Program Running an MPI/OpenMP* Program MPMD Launch Mode Fabrics Control Job Schedulers Support Controlling Process Placement Java* MPI Applications Support
Running Intel® MPI Library in Containers x
Build a Singularity* Container for an MPI Application Run the Application with a Container
Fabrics Control x
Selecting Fabrics Libfabric* Support OFI* Providers Support
Debugging Applications x
Debugging Using -gtool for Debugging
Analysis and Tuning x
Displaying MPI Debug Information Tracing Applications Interoperability with Other Tools Through -gtool MPI Tuning
Troubleshooting x
Error Message: Bad Termination Error Message: No such file or Directory Error Message: Permission Denied Error Message: Fatal Error Error Message: Bad File Descriptor Error Message: Too Many Open Files Problem: High Memory Consumption Readings Problem: MPI Application Hangs Problem: Password Required Problem: Cannot Execute Binary File Problem: MPI limitation for Docker*
Additional Supported Features x
Asynchronous Progress Control Multiple Endpoints Support
Multiple Endpoints Support x
MPI_THREAD_SPLIT Programming Model MPI_THREAD_SPLIT Model Description Threading Runtimes Support Program Examples Code Change Guide
Examples x
async_progress_sample.c thread_split.cpp thread_split_omp_for.c thread_split_omp_task.c thread_split_pthreads.c
Intel® MPI Library Developer Guide for Linux* OS

MPI_THREAD_SPLIT Programming Model

The communication patterns that comply with the thread-split model must not allow cross-thread access to MPI objects to avoid thread synchronization and must disambiguate message matching, so that threads could be separately addressed and not more than one thread could match the message at the same time. Provided that, the user must notify the Intel MPI Library that the program complies with thread-split model, that is, it is safe to apply the high performance optimization.

Each MPI_THREAD_SPLIT-compliant program can be executed correctly with a thread-compliant MPI implementation under MPI_THREAD_MULTIPLE, but not every MPI_THREAD_MULTIPLE-compliant program follows the MPI_THREAD_SPLIT model.

This model allows MPI to apply optimizations that would not be possible otherwise, such as binding specific hardware resources to concurrently communicating threads and providing lockless access to MPI objects.

Since MPI_THREAD_SPLIT is a non-standard programming model, it is disabled by default and can be enabled by setting the environment variable I_MPI_THREAD_SPLIT. If enabled, the threading runtime control must also be enabled to enable the programming model optimizations (see Threading Runtimes Support).

Setting the I_MPI_THREAD_SPLIT variable does not affect behavior at other threading levels such as SINGLE and FUNNELED. To make this extension effective, request the MPI_THREAD_MULTIPLE level of support at MPI_Init_thread().

NOTE: Thread-split model has support for MPI point-to-point operations and blocking collectives.

MPI_THREAD_SPLIT Model Description

As mentioned above, an MPI_THREAD_SPLIT-compliant program must be at least a thread-compliant MPI program (supporting the MPI_THREAD_MULTIPLE threading level). In addition to that, the following rules apply:

  1. Different threads of a process must not use the same communicator concurrently.
  2. Any request created in a thread must not be accessed by other threads, that is, any non-blocking operation must be completed, checked for completion, or probed in the same thread.
  3. Communication completion calls that imply operation progress such as MPI_Wait(), MPI_Test() being called from a thread don’t guarantee progress in other threads.

The model implies that each process thread has a distinct logical thread number thread_id. thread_id must be set to a number in the range 0 to NT-1, where NT is the number of threads that can be run concurrently. thread_id can be set implicitly, or your application can assign it to a thread. Depending on the assignment method, there are two usage submodels:

  1. Implicit model: both you and the MPI implementation know the logical thread number in advance via a deterministic thread number query routine of the threading runtime. The implicit model is only supported for OpenMP* runtimes via omp_get_thread_num().
  2. Explicit model: you pass thread_id as an integer value converted to a string to MPI by setting an MPI Info object (referred to as info key in this document) to a communicator. The key thread_id must be used. This model fits task-based parallelism, where a task can be scheduled on any process thread.

The I_MPI_THREAD_ID_KEY variable sets the MPI info object key that is used to explicitly define the thread_id for a thread (thread_id by default).

Within the model, only threads with the same thread_id can communicate. To illustrate it, the following communication pattern complies to the MPI_THREAD_SPLIT model: Suppose Comm A and Comm B are two distinct communicators, aggregating the same ranks. The system of these two communicators will fit the MPI_THREAD_SPLIT model only if all threads with thread_id #0 use Comm A, while all threads with thread_id #1 use Comm B.

Parent topic: Multiple Endpoints Support
Level Two Title