Summary
This document describes how to build the Portable Extensible Toolkit for Scientific Computation (PETSc) with Intel® oneAPI Math Kernel Library (oneMKL) BLAS and LAPACK.
It also introduces how to enable Sparse Linear operation include Sparse BLAS and oneMKL PARDISO and Cluster PARDISO as direct solver in PETSc applications.
PETSc (http://www.mcs.anl.gov/petsc/index.html) is a set of libraries that provides functions for building high-performance large-scale applications. PETSc library includes routines for vector manipulation, sparse matrix computations, distributed arrays, linear and non-linear solvers, and extensible PDE solvers.
This application note focuses on building PETSc for Intel® Architecture Processors including IA-32 and Intel® 64 architecture, running Linux*.
Version Information
This document applies to oneMKL 2018 update 2 for Linux* and PETSc 3.8.4
Step 1 – Download
PETSc releases are available for download from the PETSc web site at http://www.mcs.anl.gov/petsc/download/index.html
To get oneMKL go to /en-us/intel-mkl/.
Step 2 – Configuration
- Use the following command to extract the PETSc files. A new folder petsc will be created:
$git clone -b maint https://bitbucket.org/petsc/petsc petsc
or
$ tar –xvzf petsc-3.8.4.tar.gz - Change to the PETSc folder:
$ cd petsc-3.8.4
- Set the PETSC_DIR environment variable to point to the location of the PETSc installation folder. For bash shell use:
$ export PETSC_DIR=$PWD
Step 3 – Build PETSc
PETSc includes a set of python configuration files which support the use of various compilers, MPI implementations and math libraries. The examples below show options for configuring PETSc to link to oneMKL BLAS and LAPACK functions. Developers need to ensure that other options are configured appropriately for their system. See the PETSc installation documentation for details: http://www.mcs.anl.gov/petsc/documentation/installation.html#blaslapack
Intel provides blas/lapack via oneMKL library. It usually works from GNU/Intel compilers on linux and MS/Intel compilers on Windows. One can specify it to PETSc configure with for eg: --with-blaslapack-dir=/opt/intel/mkl
If the above option does not work - one could determine the correct library list for your compilers using oneMKL Link Line Advisor and specify with the configure option --with-blaslapack-lib
- Invoke the configuration script with the following options to build PETSc with oneMKL (installed to the default location /opt/intel/mkl).
- For Intel processors with Intel 64 use the following option:
$ ./config/configure.py
...
--with-blas-lapack-dir=/opt/intel/mkl/lib/intel64
- For Intel 32-bit processors use the following options:
$ ./config/configure.py
...
--with-blas-lapack-dir==/opt/intel/mkl/lib/ia32
*P.S. If you get error message about lack of linking with some other MKL library while you execute python source file, please try to set blas library instead of directory, --with-blas-lapack-lib=\"$MKLROOT/lib/<intel64|ia32>/libmkl_rt.so\"
- Use the make file to build PETSc:
$ make all
Step 4 - Run PETSc
Run the PETSc tests to verify the build worked correctly:
$ make test
Enabling oneMKL Sparse Linear operation in PETSc applications
PETSc users now can also benefit from enabling oneMKL sparse linear operations inside their application. Development version of PETSc distributed via bitbucket.org now supports analogue for AIJ matrix format that calls oneMKL kernels for matrix vector multiplication. With this update, PETSc users can easily switch to oneMKL for sparse linear algebra operations and get performance benefit for most of PETSc solvers.
The following oneMKL functionality is currently supported in PETSc:
oneMKL BLAS/LAPACK as basic linear algebra operations
oneKL PARDISO and Cluster PARDISO as direct solver
oneMKL Sparse BLAS IE for AIJ matrix operations
In progress are the following extensions:
Support for BAIJ, SBAIJ formats
Support for MatMatMul() operation via oneMKL Sparse BLAS IE calls
Support for triple product operation via oneMKL Sparse BLAS IE calls
For any specific requests, please contact Intel Online Service Center or forum
How to use PETSc development copy with enabled oneMKL Sparse BLAS IE:
- Download last version of PETSc. Instructions for download can be found on PETSc website.
- Configure PETSc with MKL by adding --with-blas-lapack-dir=/path/to/mkl to configuration line.
Example: ./configure --with-blas-lapack-dir=/path/to/mkl --with-mpi-dir=/usr/local/mpich
More information on PETSc installation can be found here.
- When running PETSc application pass “-mat_type aijmkl” to executable or set matrix type using MatSetType(A,MATAIJMKL) call in source code. For more information, see PETSc examples.
Appendix A – System Configuration
PETSc build and testing was completed on a system with an Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz running Ubuntu* 16.04.4 LTS
Appendix B - References
- oneMKL User’s Guide and Reference Manual
- oneMKL User Forum
- PETSc web site