Introduction
This article describes a method to compile and run a distributed memory coarray program using Intel® oneAPI HPC Toolkit. An example using Linux* is presented.
Version
Coarray Fortran was first available in Intel® Parallel Studio XE 2015. Today it is available in the Intel® Fortran Compiler (ifx) and the Intel® Fortran Compiler Classic (ifort) which are included in the oneAPI HPC Toolkit for Linux and Windows.
Application Notes
To compile for distributed memory coarrays, use compiler option -coarrays=distributed (Linux*) or /Qcoarrays:distributed (Windows*).
To compile for shared memory coarrays, use compiler option -coarrays=shared (Linux*) or /Qcoarrays:shared (Windows*).
Prerequisites
The Intel oneAPI HPC Toolkit or one of the Intel Fortran compilers is required for compilation and installed on the cluster nodes for execution.
Configuration Set Up
A key for running a distributed memory coarray program with process pinning on specific nodes is to build with the compiler option -coarray-config-file=<filename> (Linux*) or /Qcoarray-config-file:<filename> (Windows*). This enables you to take full advantage of Intel® MPI Library features in the coarrays environment, in the same way that mpiexec -config <filename> allows mpiexec to take its commands from <filename>.
Here is an example of the coarray config file:
-host host1 -env I_MPI_PIN_PROCESSOR_LIST 0,2,4 -n 3 <path to executable>/coarry_dist_host.x : -host host2 -env I_MPI_PIN_PROCESSOR_LIST 1,3,5 -n 3 <path to executable>/coarry_dist_host.x
This says to execute six coarray images of executable 'coarry_dist_host.x' on nodes host1 and host2, using processors 0, 2, 4 on host1, and processors 1, 3, 5 on host2. The I_MPI_PIN_PROCESSOR_LIST environment variable is used to achieve the process pinning on the indicated nodes.
Source Code Changes
See Verifying Correctness
Building the Application
Compile for distributed coarrays, create one coarray image, and specify the coarray configuration file:
ifort -coarray=distributed -coarray-num-images=1 -coarray-config-file=coarray_config.txt coarry_dist_host.f90 -o coarry_dist_host.x
Running the Application
Simply specify the name of the executable
> <path to executable>/coarry_dist_host.x
Hello from image 1 out of 6
total images, and running on host: host1
Hello from image 2 out of 6
total images, and running on host: host1
Hello from image 3 out of 6
total images, and running on host: host1
Hello from image 5 out of 6
total images, and running on host: host2
Hello from image 4 out of 6
total images, and running on host: host2
Hello from image 6 out of 6
total images, and running on host: host2
>
Verifying Correctness
Embed call hostnm(hostname) in your coarray program, then print 'hostname' to verify that the images are executed on the correct nodes/processors.
Benefits
This method enables coarray image pinning on specific nodes/node processors. Better load balance across cluster nodes might be obtained, or a subset of nodes easily partitioned.
Known Issues or Limitations
Distributed memory coarrays are only supported with Intel® MPI. See this article on how to use MPICH; this is unsupported.
See Also
Essential Guide to Distributed Memory Coarray Fortran with the Intel Fortran Compiler for Linux