Intel® oneAPI DPC++/C++ Compiler Developer Guide and Reference

ID 767253
Date 7/13/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents

Intel® oneAPI Level Zero Switch

Data Parallel C++ (DPC++) is just one of the many components of the oneAPI project. The Intel® oneAPI Level Zero (Level Zero) API provides low-level direct-to-metal interfaces that are tailored to the devices on a oneAPI project. While heavily influenced by other low-level APIs, such as OpenCL™ API, Level Zero is designed to evolve independently.

More information on Level Zero is available in the oneAPI Specification.

Packages to Install

The packages you must install are intel-level-zero-gpu and level-zero.

Level Zero Loader

Level Zero is supportable across different oneAPI compute device architectures. The Level Zero loader discovers all Level Zero drivers in the system. In addition, the Level Zero loader is also the Level Zero software development kit: It carries the Level Zero headers and libraries where you build Level Zero programs.

Level Zero GPU Driver

The driver is open-source and regular public releases are maintained. It does not come with DPC++ and must be installed independently. The Level Zero driver and OpenCL™ driver come in the same package. More info about the Level Zero driver is available at GitHub.

DPC++ Plugins

SYCL targets a variety of devices: CPU, GPU, and Field Programmable Gate Array (FPGA). Different devices can be operated through different low-level drivers, such as OpenCL for FPGA. The Plugin Interface (PI) is a unified SYCL API for working with different devices in a unified way. SYCL plugins implement specific translations of the PI API into low-level runtime. The Level Zero PI Plugin was created to enable devices supported through the Level Zero system.

Scenario Information

SYCL Device Selection

The PI performs device discovery of all available devices through all available PI plugins. The same physical hardware device can be seen as multiple different SYCL devices if multiple plugins support it (for example, OpenCL Gen90 and Level Zero Gen90). The SYCL runtime performs device selection from the available devices based on device selectors. The device selectors can be user-defined or built in (for example, gpu_selector).

Discovery of Multiple PI Plugins

The implication of support for the discovery of multiple plugins is that the same GPU card can be seen as multiple different GPU devices available under different PI plugins.

NOTE:
Corresponding runtimes (OpenCL and/or Level Zero) must be installed correctly and independently for PI to see their devices. The SYCL specification does not define which device will be used if there are multiple devices that match criteria (for example, is_gpu()).

Default Preference is Given to a Level Zero GPU on Linux

By default, if no special action is taken and the Level Zero runtime reports support for the installed GPU, then the SYCL runtime uses the installed GPU. This is true for standard built-in device selectors and custom device selectors, where no action is taken to change the default behavior.

Currently, on Windows, the preference is given to an OpenCL GPU.

Devices that are not supported with the Level Zero runtime (CPU/FPGA) continue to run with OpenCL.

How to Change the Default Preference

Use the SYCL_BE environment variable to change the default preference. The valid values are PI_OPENCL and PI_LEVEL0.

For example, if you specify SYCL_BE=PI_OPENCL and the PI OpenCL plugin reports the availability of the device of the required type, then that device is used. It overrides the default preference that is given to the Level Zero GPU, if the GPU is supported by the installed version of OpenCL.

NOTE:
The SYCL_BE setting only works when there are multiple choices.
Recommendation:
If your code does not work, try running it with SYCL_BE=PI_OPENCL to see if the problem is related to Level Zero.

How to See Where the Code is Running

Use the SYCL_PI_TRACE=1 environment variable to see where your code is running. It reports the choice made by the built-in device selectors, if they are used.

Use SYCL_PI_TRACE=-1 to enable verbose tracing of the PI and show all the devices detected by the PI discovery process.

How to Find all DPC++ Plugins and Supported Devices Discovered in the System

Use the sycl-ls utility to find all the plugins on your system. sycl-ls queries all the platforms and devices available through the plugins, and prints useful information about SYCL devices and their ID numbers. This information is useful when you want to designate a specific device to run a SYCL program. The SYCL_DEVICE_FILTER string is printed at each line to show three information pieces:

  • The backend that the plugin supports
  • The device_type
  • The device_id

Verbose output is available with $ sycl-ls --verbose, which gives you the same choices that are made by standard built-in device selectors and other custom device selectors.

SYCL_DEVICE_FILTER

The SYCL_DEVICE_FILTER environment variable limits the SYCL runtime to use a subset of the system's devices. Setting this environment variable affects all of the device query functions (platform::get_devices() and platform::get_platforms()) and all of the device selectors.

The value of this environment variable is a comma separated list of filters, where each filter is a triple of the form backend:device_type:device_num. Each element of the triple is optional, but each filter must have at least one value. The possible values of the backend are:

  • host
  • level_zero
  • opencl
  • cuda
  • hip
  • *

The possible values of the device_type are:

  • host
  • cpu
  • gpu
  • acc
  • *

The device_num is an integer that indexes the enumeration of devices from the sycl-ls utility tool, where the first device in that enumeration has index zero in each backend. For example, SYCL_DEVICE_FILTER=2 returns all devices with index '2' from all different backends. If multiple devices satisfy this device number (GPU and CPU devices can be assigned device number '2'), then the default_selector chooses the device with the highest heuristic point. When SYCL_DEVICE_ALLOWLIST is set, it is applied before counting devices and affects device_num values.

If a filter has all three elements of the triple, it selects only those devices that come from the given backend, have the specified device type, and have the given device index. If more than one filter is specified, the runtime is restricted to the union of devices selected by all filters. The runtime does not include the host backend and the host device automatically, unless one of the filters explicitly specifies the host device type. SYCL_DEVICE_FILTER=host should be set your program uses the host device only.

NOTE:

All device selectors throw an exception if the filtered list of devices does not include a device that satisfies the selector. For instance, SYCL_DEVICE_FILTER=cpu,level_zero causes host_selector() to throw an exception. SYCL_DEVICE_FILTER also limits loading only specified plugins into the SYCL runtime.

SYCL_DEVICE_FILTER=level_zero causes the cpu_selector to throw an exception since the SYCL runtime only loads the level_zero backend, which does not support any CPU devices at this time. When multiple devices satisfy the filter (example: SYCL_DEVICE_FILTER=gpu), only one of them is selected.