Primitive Cache

Intel® oneAPI Deep Neural Network Developer Guide and Reference

Download PDF

ID 768875

Date 9/17/2024

Version

Public

Visible to Intel only — GUID: GUID-F158068A-ED10-4E06-A742-92F6A74AB1B0

View Details

Primitive Cache

Primitive creation time largely depends on the underlying implementation, for instance, oneDNN uses just-in-time compilation (JIT) to generate optimal code for some CPU and GPU implementations, which introduces overhead.

To mitigate primitive creation overhead, oneDNN provides the primitive cache which automatically caches created primitives to avoid repeating JIT compilation for the primitives with identical primitive parameters, attributes, underlying primitive implementations, etc. It can significantly reduce primitive creation overhead, especially when an application or a framework creates primitives for every instance of inference or iteration of training process.

The primitive cache is global hence a user does not have to maintain any persistent oneDNN resources to benefit from the primitive cache.

Managing Memory Consumption

The primitive cache has an upper limit for the number of primitives stored. Once capacity is exceeded, a primitive that was least recently used will be evicted from the cache. See the Run-time Controls section below for information on changing the cache capacity.

Profiling

Information about primitive cache hits and misses can be used for debug purposes. That information is part of the verbose output when any of profile_create, profile, or all values are used (Verbose Mode).

Build-time Controls

At build-time, support for this feature is controlled via cmake option ONEDNN_ENABLE_PRIMITIVE_CACHE.

CMake Option	Supported values (defaults in bold)	Description
ONEDNN_ENABLE_PRIMITIVE_CACHE	ON , OFF	Enables primitive cache

Run-time Controls

When the feature is enabled at build-time, the ONEDNN_PRIMITIVE_CACHE_CAPACITY environment variable can be used to change cache capacity or disable the cache.

Environment variable	Value	Description
ONEDNN_PRIMITIVE_CACHE_CAPACITY	<number>	Set cache capacity to <number> (default 1024 )
	0	Disable primitive cache

This feature can also be managed at run-time with the following functions:

dnnl_set_primitive_cache_capacity

The function setting takes precedence over the environment variable.

Select Your Language

Using Intel.com Search

Quick Links

Recent Searches

Advanced Search

Only search in

Intel® oneAPI Deep Neural Network Developer Guide and Reference

Primitive Cache

Managing Memory Consumption

Profiling

Build-time Controls

Run-time Controls