enum dnnl_normalization_flags

Intel® oneAPI Deep Neural Network Developer Guide and Reference

Download PDF

ID 768875

Date 6/20/2024

Version

Public

A newer version of this document is available. Customers should click here to go to the newest version.

Visible to Intel only — GUID: GUID-1EE9B1CB-0BCE-4A89-8F48-E16FA87ECD07

View Details

enum dnnl_normalization_flags_t

Overview

Flags for normalization primitives. More…

#include <dnnl_types.h>

enum dnnl_normalization_flags_t
{
    dnnl_normalization_flags_none = 0x0U,
    dnnl_use_global_stats         = 0x1U,
    dnnl_use_scale                = 0x2U,
    dnnl_use_shift                = 0x4U,
    dnnl_fuse_norm_relu           = 0x8U,
    dnnl_fuse_norm_add_relu       = 0x10U,
};

Detailed Documentation

Flags for normalization primitives.

Enum Values

dnnl_normalization_flags_none

Use no normalization flags.

If specified

on forward training propagation mean and variance are computed and stored as output
on backward propagation compute full derivative wrt data
on backward propagation prop_kind == dnnl_backward_data has the same behavior as prop_kind == dnnl_backward

dnnl_use_global_stats

Use global statistics.

If specified

on forward propagation use mean and variance provided by user (input)
on backward propagation reduces the amount of computations, since mean and variance are considered as constants

If not specified:

on forward propagation mean and variance are computed and stored as output
on backward propagation compute full derivative wrt data

dnnl_use_scale

Use scale parameter.

If specified:

on forward propagation use scale for the normalization results
on backward propagation (for prop_kind == dnnl_backward) compute diff wrt scale (hence one extra output used)

dnnl_use_shift

Use shift parameter.

If specified:

on forward propagation use shift (aka bias) for the normalization results
on backward propagation (for prop_kind == dnnl_backward) compute diff wrt shift (hence one extra output used)

dnnl_fuse_norm_relu

Fuse with ReLU.

The flag implies negative slope being 0. On training this is the only configuration supported. For inference, to use non-zero negative slope consider using Primitive Attributes: Post-ops.

If specified:

on inference this option behaves the same as if the primitive were fused with ReLU using post ops API with zero negative slope.
on training primitive requires workspace (required to be able to perform backward pass)

dnnl_fuse_norm_add_relu

Fuse with Add and then fuse with ReLU.

If specified:

on forward propagation apply element-wise binary Add operation to to the normalization results with an additional input tensor and then apply ReLU with negative slope being 0.
on training primitive requires workspace (required to be able to perform backward pass).
on backward propagation save the result of backward ReLU operation with input tensor and workspace from forward pass to extra output tensor and then perform backward normalization.

Select Your Language

Using Intel.com Search

Quick Links

Recent Searches

Advanced Search

Only search in

Intel® oneAPI Deep Neural Network Developer Guide and Reference

enum dnnl_normalization_flags_t

Overview

Detailed Documentation