OFI* Providers Support
Intel® MPI Library supports tcp, netdir, psm2, sockets, verbs, and RxM OFI* providers. Each OFI provider is built as a separate dynamic library to ensure that a single libfabric* library can be run on top of different network adapters.
NOTE:
Use the environment variable FI_PROVIDER to select a provider. Set the FI_PROVIDER_PATH environment variable to specify the path to provider libraries.
To get a full list of environment variables available for configuring OFI, run the following command:
> fi_info -e
tcp
The TCP provider is a general purpose provider for the Intel MPI Library that can be used on any system that supports TCP sockets to implement the libfabric API. The provider lets you run the Intel MPI Library application over regular Ethernet, in a cloud environment that has no specific fast interconnect (e.g., GCP, Ethernet empowered Azure*, and AWS* instances) or using IPoIB.
The following runtime parameters can be used:
| Name | Description |
|---|---|
| FI_TCP_IFACE | Specifies a particular network interface. |
| FI_TCP_PORT_LOW_RANGE FI_TCP_PORT_HIGH_RANGE |
Sets the range of ports to be used by the TCP provider for its passive endpoint creation. This is useful when only a range of ports are allowed by the firewall for TCP connections. |
netdir
The NETDIR provider runs over NetworkDirect* interface for the compatible hardware. The following runtime parameters can be used:
| Name | Description |
|---|---|
| FI_NETDIR_MR_CACHE | Enables memory registration cache (default: 0) |
| FI_NETDIR_MR_CACHE_MAX_SIZE | Sets the maximum memory registration cache size (default: mem/core_count/2) |
psm2
The PSM2 provider runs over the PSM 2.x interface supported by the Intel® Omni-Path Fabric. PSM 2.x has all the PSM 1.x features, plus a set of new functions with enhanced capabilities. Since PSM 1.x and PSM 2.x are not application binary interface (ABI) compatible, the PSM2 provider works with PSM 2.x only and does not support Intel® True Scale Fabric.
The following runtime parameters can be used:
| Name | Description |
|---|---|
| FI_PSM2_INJECT_SIZE | Define the maximum message size allowed for fi_inject and fi_tinject calls. The default value is 64. |
| FI_PSM2_LAZY_CONN | Control the connection mode established between PSM2 endpoints that OFI endpoints are built on top of. When set to 0 (eager connection mode), connections are established when addresses are inserted into the address vector. When set to 1 (lazy connection mode), connections are established when addresses are used the first time in communication.
NOTE:
Lazy connection mode may reduce the start-up time on large systems at the expense of higher data path overhead.
|
sockets
The sockets provider is a general purpose provider that can be used on any system that supports TCP sockets. The provider is not intended to provide performance improvements over regular TCP sockets, but rather to allow developers to write, test, and debug application code even on platforms that do not have high-performance fabric hardware. The sockets provider supports all libfabric provider requirements and interfaces.
The following runtime parameter can be used:
| Name | Description |
|---|---|
| FI_SOCKETS_IFACE | Define the prefix or the name of the network interface. By default, it uses any. |
verbs
The verbs provider enables applications using OFI to be run over any verbs hardware (InfiniBand*, iWarp*, and so on). It uses the Linux Verbs API for network transport and provides a translation of OFI calls to appropriate verbs API calls. It uses librdmacm for communication management and libibverbs for other control and data transfer operations.
The verbs provider uses RxM utility provider to emulate FI_EP_RDM endpoint over verbs FI_EP_MSG endpoint by default. The verbs provider with FI_EP_RDM endpoint can be used instead of RxM by setting the FI_PROVIDER=^ofi_rxm runtime parameter.
The following runtime parameters can be used:
| Name | Description |
|---|---|
| FI_VERBS_INLINE_SIZE | Define the maximum message size allowed for fi_inject and fi_tinject calls. The default value is 64. |
| FI_VERBS_IFACE | Define the prefix or the full name of the network interface associated with the verbs device. The default value is ib. |
| FI_VERBS_MR_CACHE_ENABLE | Enable Memory Registration caching. The default value is 0. Set this environment variable to enable the memory registration cache.
NOTE:
Cache usage substantially improves performance, but may lead to correctness issues.
|
Dependencies
The verbs provider requires libibverbs (v1.1.8 or newer) and librdmacm (v1.0.16 or newer). If you are compiling libfabric from source and want to enable verbs support, it is essential to have the matching header files for the above two libraries. If the libraries and header files are not in default paths, specify them in the CFLAGS, LDFLAGS, and LD_LIBRARY_PATH environment variables.
RxM
The RxM (RDM over MSG) provider (ofi_rxm) is a utility provider that supports FI_EP_RDM endpoint emulated over FI_EP_MSG endpoint of the core provider.
The RxM provider requires the core provider to support the following features:
- MSG endpoints (FI_EP_MSG)
- FI_MSG transport (to support data transfers)
- FI_RMA transport (to support rendezvous protocol for large messages and RMA transfers)
- FI_OPT_CM_DATA_SIZE of at least 24 bytes
The following runtime parameters can be used:
| Name | Description |
|---|---|
| FI_OFI_RXM_BUFFER_SIZE | Define the transmit buffer size/inject size. Messages of smaller size are transmitted via an eager protocol and those above would be transmitted via a rendezvous protocol. Transmitted data is copied up to the specified size. By default, the size is 16k. |
| FI_OFI_RXM_SAR_LIMIT | Сontrol the RxM SAR (Segmentation аnd Reassembly) protocol. Messages of greater size are transmitted via rendezvous protocol. |
| FI_OFI_RXM_USE_SRX | Control the RxM receive path. If the variable is set to 1, the RxM uses Shared Receive Context of the core provider. The default value is 0.
NOTE:
Setting this variable to 1 improves memory consumption, but may increase small message latency as a side-effect.
|
Parent topic: Fabrics Control