Reorder between CPU and GPU engines

Intel® oneAPI Deep Neural Network Developer Guide and Reference

Download PDF

ID 768875

Date 2/28/2024

Version

Public

A newer version of this document is available. Customers should click here to go to the newest version.

Visible to Intel only — GUID: GUID-EC336923-78FE-4A7C-A326-EA4942B9A253

View Details

Reorder between CPU and GPU engines

This C++ API example demonstrates programming flow when reordering memory between CPU and GPU engines.

Example code: cross_engine_reorder.cpp

Public headers

To start using oneDNN, we must first include the dnnl.hpp header file in the application. We also include dnnl_debug.h, which contains some debugging facilities such as returning a string representation for common oneDNN C types.

All C++ API types and functions reside in the dnnl namespace. For simplicity of the example we import this namespace.

cross_engine_reorder_tutorial() function

Engine and stream

All oneDNN primitives and memory objects are attached to a particular dnnl::engine, which is an abstraction of a computational device (see also Basic Concepts). The primitives are created and optimized for the device they are attached to, and the memory objects refer to memory residing on the corresponding device. In particular, that means neither memory objects nor primitives that were created for one engine can be used on another.

To create engines, we must specify the dnnl::engine::kind and the index of the device of the given kind. There is only one CPU engine and one GPU engine, so the index for both engines must be 0.

auto cpu_engine = engine(validate_engine_kind(engine::kind::cpu), 0);
auto gpu_engine = engine(validate_engine_kind(engine::kind::gpu), 0);

In addition to an engine, all primitives require a dnnl::stream for the execution. The stream encapsulates an execution context and is tied to a particular engine.

In this example, a GPU stream is created.

auto stream_gpu = stream(gpu_engine, stream::flags::in_order);

Wrapping data into oneDNN GPU memory object

Fill the data in CPU memory first, and then move data from CPU to GPU memory by reorder.

const auto tz = memory::dims {2, 16, 1, 1};
auto m_cpu
        = memory({{tz}, memory::data_type::f32, memory::format_tag::nchw},
                cpu_engine);
auto m_gpu
        = memory({{tz}, memory::data_type::f32, memory::format_tag::nchw},
                gpu_engine);
fill(m_cpu, tz);
auto r1 = reorder(m_cpu, m_gpu);

Creating a ReLU primitive

Let’s now create a ReLU primitive for GPU.

The library implements the ReLU primitive as a particular algorithm of a more general Eltwise primitive, which applies a specified function to each element of the source tensor.

Just as in the case of dnnl::memory, a user should always go through (at least) three creation steps (which, however, can sometimes be combined thanks to C++11):

Create an operation primitive descriptor (here dnnl::eltwise_forward::primitive_desc) that defines the operation parameters including a GPU memory descriptor, and GPU engine. Primitive descriptor is a lightweight descriptor of the actual algorithm that implements the given operation.
Create a primitive (here dnnl::eltwise_forward) that can be executed on GPU memory objects to compute the operation by a GPU engine.

NOTE:

Primitive creation might be a very expensive operation, so consider creating primitive objects once and executing them multiple times.

The code:

// ReLU primitive descriptor, which corresponds to a particular
// implementation in the library. Specify engine type for the ReLU
// primitive. Use a GPU engine here.
auto relu_pd = eltwise_forward::primitive_desc(gpu_engine,
        prop_kind::forward, algorithm::eltwise_relu, m_gpu.get_desc(),
        m_gpu.get_desc(), 0.0f);
// ReLU primitive
auto relu = eltwise_forward(relu_pd);

Getting results from a oneDNN GPU memory object

After the ReLU operation, users need to get data from GPU to CPU memory by reorder.

auto r2 = reorder(m_gpu, m_cpu);

Executing all primitives

Finally, let’s execute all primitives and wait for their completion via the following sequence:

Reorder(CPU,GPU) -> ReLU -> Reorder(GPU,CPU).

After execution of the first Reorder, ReLU has source data in GPU.
The input and output memory objects are passed to the ReLU execute() method using a <tag, memory> map. Each tag specifies what kind of tensor each memory object represents. All Eltwise primitives require the map to have two elements: a source memory object (input) and a destination memory (output). For executing on GPU engine, both source and destination memory object must use GPU memory.
After the execution of the ReLU on GPU, the second Reorder moves the results from GPU to CPU.

NOTE:

All primitives are executed in the SAME GPU stream (the first parameter of the execute() method).

Execution is asynchronous on GPU. This means that we need to call dnnl::stream::wait before accessing the results.

// wrap source data from CPU to GPU
r1.execute(stream_gpu, m_cpu, m_gpu);
// Execute ReLU on a GPU stream
relu.execute(stream_gpu, {{DNNL_ARG_SRC, m_gpu}, {DNNL_ARG_DST, m_gpu}});
// Get result data from GPU to CPU
r2.execute(stream_gpu, m_gpu, m_cpu);

stream_gpu.wait();

Validate the result

Now that we have the computed the result on CPU memory, let’s validate that it is actually correct.

if (find_negative(m_cpu, tz) != 0)
    throw std::logic_error(
            "Unexpected output, find a negative value after the ReLU "
            "execution.");

Upon compiling and running the example, the output should be just:

Example passed.

Select Your Language

Using Intel.com Search

Quick Links

Recent Searches

Advanced Search

Only search in

Intel® oneAPI Deep Neural Network Developer Guide and Reference

Reorder between CPU and GPU engines

Public headers

cross_engine_reorder_tutorial() function