Intel® MPI Library Developer Reference for Linux* OS

ID 768732
Date 11/07/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents

Environment Variables for Memory Policy Control

Intel® MPI Library supports non-uniform memory access (NUMA) nodes with high-bandwidth (HBW) memory (MCDRAM) on Intel® Xeon Phi™ processors (codenamed Knights Landing). Intel® MPI Library can attach memory of MPI processes to the memory of specific NUMA nodes. This section describes the environment variables for such memory placement control.

I_MPI_HBW_POLICY

Set the policy for MPI process memory placement for using HBW memory.

Syntax

I_MPI_HBW_POLICY=<user memory policy>[,<mpi memory policy>][,<win_allocate policy>]

In the syntax:

  • <user memory policy> - memory policy used to allocate the memory for user applications (required)
  • <mpi memory policy> - memory policy used to allocate the internal MPI memory (optional)
  • <win_allocate policy> - memory policy used to allocate memory for window segments for RMA operations (optional)

Each of the listed policies may have the values below:

Arguments

<value> The memory allocation policy used.
hbw_preferred Allocate the local HBW memory for each process. If the HBW memory is not available, allocate the local dynamic random access memory.
hbw_bind Allocate only the local HBW memory for each process.
hbw_interleave Allocate the HBW memory and dynamic random access memory on the local node in the round-robin manner.

Description

Use this environment variable to specify the policy for MPI process memory placement on a machine with HBW memory.

By default, Intel MPI Library allocates memory for a process in local DDR. The use of HBW memory becomes available only when you specify the I_MPI_HBW_POLICY variable.

Examples

The following examples demonstrate different configurations of memory placement:

  • I_MPI_HBW_POLICY=hbw_bind,hbw_preferred,hbw_bind

    Only use the local HBW memory allocated in user applications and window segments for RMA operations. Use the local HBW memory internally allocated in Intel® MPI Library first. If the HBW memory is not available, use the local DDR internally allocated in Intel MPI Library.

  • I_MPI_HBW_POLICY=hbw_bind,,hbw_bind

    Only use the local HBW memory allocated in user applications and window segments for RMA operations. Use the local DDR internally allocated in Intel MPI Library.

  • I_MPI_HBW_POLICY=hbw_bind,hbw_preferred

    Only use the local HBW memory allocated in user applications. Use the local HBW memory internally allocated in Intel MPI Library first. If the HBW memory is not available, use the local DDR internally allocated in Intel MPI Library. Use the local DDR allocated in window segments for RMA operations.

I_MPI_BIND_NUMA

Set the NUMA nodes for memory allocation.

Syntax

I_MPI_BIND_NUMA=<value>

Arguments

<value> Specify the NUMA nodes for memory allocation.
localalloc Allocate memory on the local node. This is the default value.
Node_1,…,Node_k Allocate memory according to I_MPI_BIND_ORDER on the specified NUMA nodes.

Description

Set this environment variable to specify the NUMA node set that is involved in the memory allocation procedure.

I_MPI_BIND_ORDER

Set this environment variable to define the memory allocation manner.

Syntax

I_MPI_BIND_ORDER=<value>

Arguments

<value> Specify the allocation manner.
compact Allocate memory for processes as close as possible (in terms of NUMA nodes), among the NUMA nodes specified in I_MPI_BIND_NUMA. This is the default value.
scatter Allocate memory among the NUMA nodes specified in I_MPI_BIND_NUMA using the round-robin manner.

Description

Set this environment variable to define the memory allocation manner among the NUMA nodes specified in I_MPI_BIND_NUMA. The variable has no effect without I_MPI_BIND_NUMA set.

I_MPI_BIND_WIN_ALLOCATE

Set this environment variable to control memory allocation for window segments.

Syntax

I_MPI_BIND_WIN_ALLOCATE=<value>

Arguments

<value> Specify the memory allocation behavior for window segments.
localalloc Allocate memory on the local node. This is the default value.
hbw_preferred Allocate the local HBW memory for each process. If the HBW memory is not available, allocate the local dynamic random access memory.
hbw_bind Allocate only the local HBW memory for each process.
hbw_interleave Allocate the HBW memory and dynamic random access memory on a local node in the round-robin manner.
<NUMA node id> Allocate memory on the given NUMA node.

Description

Set this environment variable to create window segments allocated in HBW memory with the help of the MPI_Win_allocate_shared or MPI_Win_allocate functions.

MPI_Info

You can control memory allocation for window segments with the help of an MPI_Info object, which is passed as a parameter to the MPI_Win_allocate or MPI_Win_allocate_shared function. In an application, if you specify such an object with the numa_bind_policy key, window segments are allocated in accordance with the value for numa_bind_policy. Possible values are the same as for I_MPI_BIND_WIN_ALLOCATE.

A code fragment demonstrating the use of MPI_Info:

MPI_Info info;
...
MPI_Info_create( &info );
MPI_Info_set( info, "numa_bind_policy", "hbw_preferred" );
...
MPI_Win_allocate_shared( size, disp_unit, info, comm, &baseptr, &win );
NOTE:

When you specify the memory placement policy for window segments, Intel MPI Library recognizes the configurations according to the following priority:

  1. Setting of MPI_Info.
  2. Setting of I_MPI_HBW_POLICY, if you specified <win_allocate policy>.
  3. Setting of I_MPI_BIND_WIN_ALLOCATE.