Prerequisites

If you intend to debug on a GPU, first check whether your device is supported for debugging kernels offloaded to GPU by checking the list of supported accelerators. If your GPU device is not listed above, then a breakpoint inside the kernel won’t be hit. In that case, you can still debug the offload to CPU.

For more information, go to the GPU Driver Page.

NOTE:

GPU devices listed in the first table with XeHPC or XeHPG Architectures are currently supported by the GDB debugger. GPU devices with the Xe Architecture are NOT supported by the GDB debugger, as shown in Figure 1 below:

Figure 1: Supported GPU Devices

Install GPU drivers

1. Install and configure the latest GPU drivers by following this link: <https://dgpu-docs.intel.com/installation-guides/index.html>

2. Refer to the Intel® oneAPI Toolkits Installation Guide for Linux* OS to install the oneAPI Toolkit. We recommend you install the Intel® oneAPI Base Toolkit (Base Kit).

You can also install an extension for Visual Studio Code* for debugging GPU with Intel Distribution for GDB. Refer to the Using Visual Studio Code with Intel® oneAPI Toolkits Guide.

Set up the GPU debugger

To set up the GPU debugger, you must have root access.

1. If you intend to debug on a GPU, a Linux Kernel that supports GPU debugging is needed.

   1. Follow the instructions in Install GPU drivers to download and install the necessary drivers.

   2. Enable i915 debug support in the Kernel:

      1. Open a terminal.

      2. For each GPU, set the value of the prelim_enable_eu_debug flag to 1. You can use the following shell command for this:

         for f in /sys/class/drm/card*/prelim_enable_eu_debug; do echo 1 > "$f"; done

         NOTE:

         System admins may have to give write permissions to the prelim_enable_eu_debug entry in the sysfs for regular users to be able to enable/disable debug.

   3. If you want to enable i915 debug support in kernel persistently, follow the instructions below (requires admin permissions):

      1. Open a terminal.

      2. Open the grub file in /etc/default.

      3. In the grub file, find the line GRUB_CMDLINE_LINUX_DEFAULT="".

      4. Enter the following text between the quotes (""):

         i915.debug_eu=1

      5. Update grub for these changes to take effect and reboot:

         sudo update-grub
         sudo reboot now

         NOTE:

         If you don’t have access to the graphics card, the driver is probably installed incorrectly. Go to GPU Debugging and try the suggestions. After you get your driver working, return here and continue the installation and prerequisites instructions.

   4. By default, the GPU driver does not allow workloads to run on a GPU longer than a certain amount of time. The driver kills such long-running workloads by resetting the GPU to prevent hangs. The hangcheck mechanism of the driver is disabled if the application is running under the debugger. If you plan to run long compute workloads also without a debugger being attached, consider applying GPU: Disable Hangcheck by adding the following to the grub file:

      1. Open a terminal.

      2. Open the grub file in /etc/default

      3. In the grub file, find the line GRUB_CMDLINE_LINUX_DEFAULT=""

      4. Enter the following text between the quotes ("").

         i915.enable_hangcheck=0

      5. Update GRUB for these changes to take effect and reboot:

         sudo update-grub
         sudo reboot now

2. Setup your oneAPI development environment. The environment setup examples shown below assume you are using the “component” directory layout. If you wish to use the “unified” directory layout, see the oneAPI Development Environmental Setup for details.

   Linux (sudo):

   source /opt/intel/oneapi/setvars.sh

   Linux (user):

   source ~/intel/oneapi/setvars.sh
   
   :: initializing oneAPI environment ...
      -bash: BASH_VERSION = 5.1.16(1)-release
      args: Using "$@" for setvars.sh arguments:
   :: advisor -- latest
   :: ccl -- latest
   :: compiler -- latest
   :: dal -- latest
   :: debugger -- latest
   :: dev-utilities -- latest
   :: dnnl -- latest
   :: dpcpp-ct -- latest
   :: dpl -- latest
   :: ipp -- latest
   :: ippcp -- latest
   :: mkl -- latest
   :: mpi -- latest
   :: tbb -- latest
   :: vtune -- latest
   :: oneAPI environment initialized ::

   If you can’t find setvars.sh file, you may need to check your Intel® oneAPI Base Toolkit installation. Confirm that it was installed in one of the expected locations, /opt/intel/oneapi/setvars.sh or ~/intel/oneapi/setvars.sh.

3. Set up debug environment variables. Use the following environment variable to enable debugger support for Intel® oneAPI Level Zero:

   export ZET_ENABLE_PROGRAM_DEBUGGING=1

4. Perform a system check.

   When everything is ready, confirm that the system configuration is reliable by running the following command:

   sycl-ls

   If there is no compatible GPU installed, you will see output similar to the following:

   [opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2  [2023.16.10.0.17_160000]
   [opencl:cpu:1] Intel(R) OpenCL, Genuine Intel(R) CPU 0000%@ OpenCL 3.0 (Build 0) [2023.16.10.0.17_160000]

   If there is an Intel GPU present and the appropriate GPU driver is installed, you will see output similar to the following:

   [opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2 [2023.16.9.0.01_160000]
   [opencl:cpu:1] Intel(R) OpenCL, Intel(R) Xeon(R) Gold 6248 CPU @2.50GHz OpenCL 3.0 (Build 0) [2023.16.9.0.01_160000]
   [ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Data Center GPU Max 1550 1.3 [1.3.26690]

   Notice the reference to a level_zero:gpu device in the very last line of the second set of output from the sycl-ls command; this represents a GPU that can be used for running and debugging Intel GPU SYCL code. You might see a GPU device listed in the sycl-ls output as opencl:gpu:0. An application that runs on an OpenCL GPU device cannot be debugged using gdb-oneapi. However, an OpenCL application that runs on the FPGA emulator or on an OpenCL CPU device can be debugged with gdb-oneapi.

   For more help understanding the output of the sycl-ls command see the SYCL Device Selection in the Host Code article in the Intel oneAPI Programming Guide.

Compile the program with debug information

You can use the sample project, Array Transform, to quickly get started with the application debugger.

Prerequisites

1. Install cmake version 3.15 or later using your preferred method.

2. Install the sample project by either of the following ways:

   • Use the oneAPI CLI Samples Browser to select Array Transform from the Getting Started category.

   • Download from GitHub*.

Compile the sample project

1. Navigate to the src of the sample project:

   cd array-transform/src

2. Compile the application using cmake:

   mkdir build
   cd build
   cmake ..
   make all
   ./array-transform
   
   user@ubuntu-nuc: ~/samples/array-transform/build$ make all
   [ 50%] Building CXX object CMakeFiles/array-transform.dir/src/array-transform.cpp.o
   [100%] Linking CXX executable array-transform
   [100%] Built target array-transform
   user@ubuntu-nuc: ~/samples/array-transform/build$ ./array-transform
   [SYCL] Using device: [Intel(R) Core(TM) i7-5557U CPU @ c.10GHz] from [Intel(R) OpenCL]
   success; result is correct.

   NOTE:

   You can still compile the program with optimization enabled (-O2 flag), which can be helpful if you intend to debug the GPU assembly.

   You can compile the program in several ways:

   • JIT compilation, which we recommend for debugging the sample; or

   • AOT compilation, which is not being used here.

   Refer to the CMakeLists.txt file and the README for information on configuring and building the application.

   See these following two videos for a quick overview on how to apply the JIT and AOT compilation options:

   • Debug Just-in-Time and Ahead-of-Time GPU Code with Intel® Distribution for GDB*

   • Compilation Options and Debugging: Just-in-Time and Ahead-of-Time GPU Code with Intel® Distribution for GDB*

   NOTE:

   The CMake file provided with the sample already passes the -g -O0 flags.

   • To compile the array-transform.cpp sample application without using CMake, issue the following commands:

     icpx -fsycl -g -O0 array-transform.cpp -o array-transform

     If compilation and linking is done separately, reapply the -g -O0 flags at the link step (the second line shown below). The link step translates these flags to be passed to the device compiler at runtime. Here is an example:

     icpx -fsycl -g -O0 -c array-transform.cpp
     icpx -fsycl -g -O0 array-transform.o -o array-transform

   • You can use AOT compilation to avoid longer JIT compilation times at runtime. JIT compilation can take significantly longer for large kernels when debugging. To use AOT compilation, refer to the Intel® oneAPI DPC++ Compiler Developer Guide and Reference.

Start a debug session

Start the debug session:

1. Start Intel Distribution for GDB as follows:

   gdb-oneapi array-transform

   You should see the (gdb) prompt.

2. You can control the device that is used to offload the kernel by defining the environment variable ONEAPI_DEVICE_SELECTOR. From the debugging session you can modify it with set env command.

   • For debugging on a CPU device:

     set env ONEAPI_DEVICE_SELECTOR=opencl:cpu
     run

     Example output:

     [SYCL] Using device: [Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz] from [Intel(R) OpenCL]

   • For debugging on the Level Zero GPU device:

     set env ONEAPI_DEVICE_SELECTOR=level_zero:gpu
     run

     NOTE:

     The Level Zero GPU device is the default choice when it is available and the variable is not set.

     Example output:

     [SYCL] Using device: [Intel(R) Data Center GPU Flex Series 140 [0x56c1]] from [Intel(R) Level-Zero]

   • For debugging on the FPGA-emulator:

     set env ONEAPI_DEVICE_SELECTOR=opencl:acc
     run

     Example output:

     [SYCL] Using device: [Intel(R) FPGA Emulation Device] from [Intel(R) FPGA Emulation Platform for OpenCL(TM) software]

3. To quit the Intel Distribution for GDB:

   quit

For your convenience, common Intel Distribution for GDB commands are provided in the Reference Sheet.

To debug the Array Transform sample and learn more about Intel Distribution for GDB, walk through basic debugging scenarios using the Tutorial.

Learn more

Notices and Disclaimers

Intel technologies may require enabled hardware, software or service activation.

No product or component can be absolutely secure.

Your costs and results may vary.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos.