Intel® VTune™ Profiler

Cookbook

Download PDF
ID 766316
Date 6/03/2024
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
Cookbook
Cookbook x
Methodologies Configuration Recipes Tuning Recipes Notices and Disclaimers
Methodologies x
Analyzing Uncore Perfmon Events for Use of Intel® Data Direct I/O Technology in Intel® Xeon® Processors (NEW) Top-down Microarchitecture Analysis Method OpenMP* Code Analysis Method Custom Data Collection for Performance Analysis (NEW) Software Optimization for Intel® GPUs (NEW) Core Utilization in DPDK Apps PCIe Traffic in DPDK Apps DPDK Event Device Profiling Effective Utilization of Intel® Data Direct I/O Technology Compile a Portable Optimized Binary with the Latest Instruction Set
Configuration Recipes x
Profiling OpenVINO™ Applications (NEW) Profiling High Bandwidth Memory Performance on Intel® Xeon® CPU Max Series (NEW) Profiling Windows* Applications for Hybrid CPU Platforms (NEW) Viewing Analysis Results on a Web Browser (NEW) Profiling Machine Learning Applications (NEW) Profiling Single-Node Kubernetes* Applications (NEW) Analyzing Hot Code Paths Using Flame Graphs (NEW) Improving Hotspot Observability in a C++ Application Using Flame Graphs Measuring Performance Impact of NUMA in Multi-Processor Systems Profiling Games built with Unity* (NEW) Profiling Games built with Unreal Engine* (NEW) Profiling Java Applications as a Remote User (NEW) Profiling JavaScript* Code in Node.js* Analyzing CPU and FPGA (Intel® Arria® 10 GX) Interaction Profiling a .NET* Core Application Profiling Applications in Amazon Web Services* (AWS) EC2 Instances Enabling Performance Profiling in GitLab* CI Configuring a Hyper-V* Virtual Machine for Hardware-Based Hotspots Analysis Profiling an Application for Performance Anomalies (NEW) Profiling an OpenMP* Offload Application running on a GPU Profiling a SYCL* Application running on a GPU Profiling an FPGA-driven SYCL* Application Profiling Hardware Without Intel Sampling Drivers Profiling MPI Applications Profiling Docker* Containers Profiling a Remote Target Through a Proxy Server (NEW) Profiling in an Apptainer* Container Profiling Linux*, Android*, and QNX* System Boot Time Using Intel® VTune™ Profiler Server with Visual Studio Code and Intel® DevCloud for oneAPI (NEW) Using Intel® VTune™ Profiler Server in HPC Clusters Using the Command-Line Interface to Analyze the Performance of a SYCL* Application running on a GPU (NEW)
Tuning Recipes x
Cache-Related Latency Issues in Segmented Cache Environment False Sharing Frequent DRAM Accesses Poor Port Utilization Page Faults Instruction Cache Misses OS Thread Migration OpenMP* Imbalance and Scheduling Overhead Processor Cores Underutilization: OpenMP* Serial Time Ingredients Create a Baseline Run HPC Performance Characterization Analysis Identify OpenMP Serial Time Parallelize the Code Inspect Threading Errors See Also Scheduling Overhead in an Intel® oneAPI Threading Building Blocks Application PMDK Application Overhead
Cookbook

Processor Cores Underutilization: OpenMP* Serial Time

Use this recipe to identify a fraction of serial execution in an application that was parallelized with OpenMP. Discover additional opportunities for parallelization, and improve the scalability of the application.

Content Expert: Rupak Roy

The presence of a fraction of serial time in a parallel application can limit the scalability of the application. Scalability is the capacity of the application to fully utilize available hardware resources like cores for code execution.

According to Amdahl's law, the maximum speed-up for a parallel application is given by:

where:

  • P is a parallel portion of the application execution
  • N is a number of processor elements

If the serial part (1-P) of the application execution is greater, this decreases the possibility of a linear speed-up. The serial portion limits the performance scalability.

When your application is parallelized with OpenMP, the sequential code execution may be a result of one of these code executions:

  • The code executed out of the OpenMP regions.
  • The code executed inside the #pragma omp master or #pragma omp single constructs.

This recipe focuses on the code executed out of OpenMP regions. In this recipe, use Intel® VTune™ Profiler to:

  • Detect the serial time of the code executed outside of OpenMP regions
  • Analyze the distribution of serial hotspot functions/loops
  • Understand opportunities for code parallelization

Ingredients

This section lists hardware and software tools used for the performance analysis scenario.

  • Application: A miniFE Finite Element Mini-Application that is available for download from https://github.com/Mantevo/miniFE (OpenMP version)

  • Compiler: Intel® oneAPI DPC++/C++ Compiler 2024.0.0 or newer. This recipe relies on this compiler version to have necessary instrumentation inside Intel OpenMP runtime library used by VTune Profiler for analysis.

  • Performance analysis tools:

    • VTune Profiler version 2024.0 or newer - HPC Performance Characterization analysis

    • Intel® Inspector 2022.1: Threading Error analysis

      NOTE:

      Get the latest version of Intel® Inspector from this download page.

  • Operating system: Linux*, Ubuntu* 20.04.6

  • CPU: 11th Gen Intel® Core™ i9-11900KB @ 3.30GHz

Create a Baseline

  1. Use the openmp/src/Makefile.intel.openmp make file to build the application.

    • To enable debug information, add the -g option.
    • For simpler identification, you can see source file information in the names of the OpenMP regions by including the -parallel-source-info=2 compiler option.
  2. Run the compiled application with these parameters:
    • nx=200
    • ny=200
    • nz=200

The number of OpenMP threads corresponds to the number of physical cores. With one thread running per core (OMP_NUM_THREADS=16, OMP_PLACES=cores), the application takes about 50 seconds.

This is a performance baseline. You use this baseline for further optimizations.

Run HPC Performance Characterization Analysis

To understand potential performance bottlenecks in the sample, run the HPC Performance Characterization analysis in VTune Profiler.

  1. Click the New Project button on the toolbar and specify a name for the new project, for example: miniFE.

  2. In the Configure Analysis window, set these options:

    • In the WHERE pane, select the Local Host target system type.

    • In the WHAT pane, select the Launch Application target type.

    • Specify an application for analysis and use these parameters: nx 200 ny 200 nz 200.

    • In the HOW pane, from the Parallelism group, select HPC Performance Characterization .

  3. Click the Start button to run the analysis.

To run the analysis from the command line, use this command: 

vtune -collect hpc-performance -data-limit=0 ./miniFE.x nx 200 ny 200 nz 200

Once data collection is complete, VTune Profiler processes the result and resolves symbol information. This is necessary for source analysis.

Identify OpenMP Serial Time

Using the HPC Performance Characterization analysis, you can see HPC metrics that help you to understand these performance bottlenecks:

  • CPU utilization (parallelism)
  • Memory access efficiency
  • Vectorization

For applications which use the Intel OpenMP runtime, you can benefit from special OpenMP efficiency metrics that help identify issues with threading parallelism.

Start your analysis with the Summary view where you see application-level statistics. The Effective Physical Core Utilization metric has been flagged. This signals a performance problem that requires investigation:

When you dive deeper into the metric hierarchy, you see that the Serial Time (outside parallel regions) of the application occupies ~24% of its elapsed time.

Let us look at the serial hotspot in the matrix initialization code, which is the loop at line 133.

Consider running the HPC Performance Characterization analysis with call stacks to explore available optimization opportunities. Using call stacks can help you find a candidate for parallelism at a proper level of granularity. Since the call stack collection is not compatible with memory bandwidth analysis, make sure to disable the Analyze memory bandwidth configuration option first:

To run this analysis from the command line, use this command: 

vtune -collect hpc-performance -data-limit=0 -knob enable-stack-collection=true -knob collect-memory-bandwidth=false ./miniFE.x nx 200 ny 200 nz 200

NOTE:

While you can also use the Threading analysis to analyze OpenMP Serial Time with stacks, the HPC Performance Characterization analysis is a better starting point for a high level understanding of performance bottlenecks.

Once the data collection finishes, VTune Profiler displays results starting with the Summary view. To identify top hot spots and see their call stacks, switch to the Bottom-up view.

SparseMatrix_functions.hpp has a loop with iterations by matrix elements. This is a good location to insert parallelism.

Double click this row to open the source file at that location:

Parallelize the Code

To make the matrix initialization parallelized by OpenMP, insert the omp parallel for pragma : 

#pragma omp parallel for
for(int i=0; i<mat_init.n; ++i) {
    mat_init(i);

Re-compile the application and compare the execution time versus the original performance baseline to verify your optimization.

In this recipe, the Elapsed time of the application after optimization is approximately 42 seconds, which is ~21% speed-up of the application execution.

Re-run the HPC Performance Characterization analysis for the optimized version of the application:

Overall,

  • The Effective Physical Core Utilization has improved by 16%.
  • The fraction of OpenMP Serial Time has reduced to 3.3%.
  • VTune Profiler does not flag the OpenMP Serial Time metric since the threshold is 15%.

To further improve parallel efficiency, you can analyze the most imbalanced barriers. For more information, see OpenMP Imbalance and Scheduling Overhead.

Inspect Threading Errors

To complete your analysis for parallelism, check your code for threading errors like data races or deadlocks. Use Intel® Inspector to find potential data races and deadlocks that may not happen in particular hardware but can hurt in a different environment or even when the same environment uses different settings.

To speed up the check, use the command line interface and reduce the workload size: 

inspxe-cl -collect ti3 ./miniFE.x nx 40 ny 40 nz 40

You see that the Intel Inspector does not report any issues for the parallelized code.

Discuss this recipe in the Analyzer forum.

Parent topic: Tuning Recipes
Level Two Title