Layer Normalization

Intel® oneAPI Deep Neural Network Developer Guide and Reference

Download PDF

ID 768875

Date 6/20/2024

Version

Public

A newer version of this document is available. Customers should click here to go to the newest version.

Visible to Intel only — GUID: GUID-99953D18-C78A-47C5-8A27-8809A4BD65BB

View Details

Layer Normalization

Overview

A primitive to perform layer normalization. More…

// structs

struct dnnl::layer_normalization_backward;
struct dnnl::layer_normalization_forward;

// global functions

dnnl_status_t DNNL_API dnnl_layer_normalization_forward_primitive_desc_create(
    dnnl_primitive_desc_t* primitive_desc,
    dnnl_engine_t engine,
    dnnl_prop_kind_t prop_kind,
    const_dnnl_memory_desc_t src_desc,
    const_dnnl_memory_desc_t dst_desc,
    const_dnnl_memory_desc_t stat_desc,
    float epsilon,
    unsigned flags,
    const_dnnl_primitive_attr_t attr
    );

dnnl_status_t DNNL_API dnnl_layer_normalization_backward_primitive_desc_create(
    dnnl_primitive_desc_t* primitive_desc,
    dnnl_engine_t engine,
    dnnl_prop_kind_t prop_kind,
    const_dnnl_memory_desc_t diff_src_desc,
    const_dnnl_memory_desc_t diff_dst_desc,
    const_dnnl_memory_desc_t src_desc,
    const_dnnl_memory_desc_t stat_desc,
    float epsilon,
    unsigned flags,
    const_dnnl_primitive_desc_t hint_fwd_pd,
    const_dnnl_primitive_attr_t attr
    );

dnnl_status_t DNNL_API dnnl_layer_normalization_forward_primitive_desc_create_v2(
    dnnl_primitive_desc_t* primitive_desc,
    dnnl_engine_t engine,
    dnnl_prop_kind_t prop_kind,
    const_dnnl_memory_desc_t src_desc,
    const_dnnl_memory_desc_t dst_desc,
    const_dnnl_memory_desc_t stat_desc,
    dnnl_data_type_t scale_shift_data_type,
    float epsilon,
    unsigned flags,
    const_dnnl_primitive_attr_t attr
    );

dnnl_status_t DNNL_API dnnl_layer_normalization_backward_primitive_desc_create_v2(
    dnnl_primitive_desc_t* primitive_desc,
    dnnl_engine_t engine,
    dnnl_prop_kind_t prop_kind,
    const_dnnl_memory_desc_t diff_src_desc,
    const_dnnl_memory_desc_t diff_dst_desc,
    const_dnnl_memory_desc_t src_desc,
    const_dnnl_memory_desc_t stat_desc,
    dnnl_data_type_t diff_scale_shift_data_type,
    dnnl_data_type_t scale_shift_data_type,
    float epsilon,
    unsigned flags,
    const_dnnl_primitive_desc_t hint_fwd_pd,
    const_dnnl_primitive_attr_t attr
    );

Detailed Documentation

A primitive to perform layer normalization.

Normalization is performed within the last logical dimension of data tensor.

Both forward and backward propagation primitives support in-place operation; that is, src and dst can refer to the same memory for forward propagation, and diff_dst and diff_src can refer to the same memory for backward propagation.

The layer normalization primitives computations can be controlled by specifying different dnnl::normalization_flags values. For example, layer normalization forward propagation can be configured to either compute the mean and variance or take them as arguments. It can either perform scaling and shifting using gamma and beta parameters or not.

See also:

Layer Normalization in developer guide

Global Functions

dnnl_status_t DNNL_API dnnl_layer_normalization_forward_primitive_desc_create(
    dnnl_primitive_desc_t* primitive_desc,
    dnnl_engine_t engine,
    dnnl_prop_kind_t prop_kind,
    const_dnnl_memory_desc_t src_desc,
    const_dnnl_memory_desc_t dst_desc,
    const_dnnl_memory_desc_t stat_desc,
    float epsilon,
    unsigned flags,
    const_dnnl_primitive_attr_t attr
    )

Creates a primitive descriptor for a layer normalization forward propagation primitive.

NOTE:

In-place operation is supported: the dst can refer to the same memory as the src.

Parameters:

primitive_desc	Output primitive_descriptor.
engine	Engine to use.
prop_kind	Propagation kind. Possible values are dnnl_forward_training and dnnl_forward_inference.
src_desc	Source memory descriptor.
dst_desc	Destination memory descriptor.
stat_desc	Memory descriptor for mean and variance. If this parameter is NULL, a zero memory descriptor, or a memory descriptor with format_kind set to dnnl_format_kind_undef, then the memory descriptor for stats is derived from `src_desc` by removing the last dimension.
epsilon	Layer normalization epsilon parameter.
flags	Layer normalization flags (dnnl_normalization_flags_t).
attr	Primitive attributes (can be NULL).

Returns:

dnnl_success on success and a status describing the error otherwise.

dnnl_status_t DNNL_API dnnl_layer_normalization_backward_primitive_desc_create(
    dnnl_primitive_desc_t* primitive_desc,
    dnnl_engine_t engine,
    dnnl_prop_kind_t prop_kind,
    const_dnnl_memory_desc_t diff_src_desc,
    const_dnnl_memory_desc_t diff_dst_desc,
    const_dnnl_memory_desc_t src_desc,
    const_dnnl_memory_desc_t stat_desc,
    float epsilon,
    unsigned flags,
    const_dnnl_primitive_desc_t hint_fwd_pd,
    const_dnnl_primitive_attr_t attr
    )

Creates a primitive descriptor for a layer normalization backward propagation primitive.

NOTE:

In-place operation is supported: the diff_dst can refer to the same memory as the diff_src.

Parameters:

primitive_desc	Output primitive_descriptor.
engine	Engine to use.
prop_kind	Propagation kind. Possible values are dnnl_backward_data and dnnl_backward (diffs for all parameters are computed in this case).
diff_src_desc	Diff source memory descriptor.
diff_dst_desc	Diff destination memory descriptor.
src_desc	Source memory descriptor.
stat_desc	Memory descriptor for mean and variance. If this parameter is NULL, a zero memory descriptor, or a memory descriptor with format_kind set to dnnl_format_kind_undef, then the memory descriptor for stats is derived from `src_desc` by removing the last dimension.
epsilon	Layer normalization epsilon parameter.
flags	Layer normalization flags (dnnl_normalization_flags_t).
hint_fwd_pd	Primitive descriptor for a respective forward propagation primitive.
attr	Primitive attributes (can be NULL).

Returns:

dnnl_success on success and a status describing the error otherwise.

dnnl_status_t DNNL_API dnnl_layer_normalization_forward_primitive_desc_create_v2(
    dnnl_primitive_desc_t* primitive_desc,
    dnnl_engine_t engine,
    dnnl_prop_kind_t prop_kind,
    const_dnnl_memory_desc_t src_desc,
    const_dnnl_memory_desc_t dst_desc,
    const_dnnl_memory_desc_t stat_desc,
    dnnl_data_type_t scale_shift_data_type,
    float epsilon,
    unsigned flags,
    const_dnnl_primitive_attr_t attr
    )

Creates a primitive descriptor for a layer normalization forward propagation primitive with a user-provided data type for the scale and shift memory objects.

NOTE:

In-place operation is supported: the dst can refer to the same memory as the src.

Parameters:

primitive_desc	Output primitive_descriptor.
engine	Engine to use.
prop_kind	Propagation kind. Possible values are dnnl_forward_training and dnnl_forward_inference.
src_desc	Source memory descriptor.
dst_desc	Destination memory descriptor.
stat_desc	Memory descriptor for mean and variance. If this parameter is NULL, a zero memory descriptor, or a memory descriptor with format_kind set to dnnl_format_kind_undef, then the memory descriptor for stats is derived from `src_desc` by removing the last dimension.
scale_shift_data_type	Data type of scale and shift memory. If neither scale nor shift flag are specified the parameter is ignored.
epsilon	Layer normalization epsilon parameter.
flags	Layer normalization flags (dnnl_normalization_flags_t).
attr	Primitive attributes (can be NULL).

Returns:

dnnl_success on success and a status describing the error otherwise.

dnnl_status_t DNNL_API dnnl_layer_normalization_backward_primitive_desc_create_v2(
    dnnl_primitive_desc_t* primitive_desc,
    dnnl_engine_t engine,
    dnnl_prop_kind_t prop_kind,
    const_dnnl_memory_desc_t diff_src_desc,
    const_dnnl_memory_desc_t diff_dst_desc,
    const_dnnl_memory_desc_t src_desc,
    const_dnnl_memory_desc_t stat_desc,
    dnnl_data_type_t diff_scale_shift_data_type,
    dnnl_data_type_t scale_shift_data_type,
    float epsilon,
    unsigned flags,
    const_dnnl_primitive_desc_t hint_fwd_pd,
    const_dnnl_primitive_attr_t attr
    )

Creates a primitive descriptor for a layer normalization backward propagation primitive with a user-provided data type for the scale and shift memory objects.

NOTE:

In-place operation is supported: the diff_dst can refer to the same memory as the diff_src.

Parameters:

primitive_desc	Output primitive_descriptor.
engine	Engine to use.
prop_kind	Propagation kind. Possible values are dnnl_backward_data and dnnl_backward (diffs for all parameters are computed in this case).
diff_src_desc	Diff source memory descriptor.
diff_dst_desc	Diff destination memory descriptor.
src_desc	Source memory descriptor.
stat_desc	Memory descriptor for mean and variance. If this parameter is NULL, a zero memory descriptor, or a memory descriptor with format_kind set to dnnl_format_kind_undef, then the memory descriptor for stats is derived from `src_desc` by removing the last dimension.
diff_scale_shift_data_type	Data type of diff scale and shift memory. If neither scale nor shift flag are specified the parameter is ignored.
scale_shift_data_type	Data type of scale and shift memory. If neither scale nor shift flag are specified the parameter is ignored.
epsilon	Layer normalization epsilon parameter.
flags	Layer normalization flags (dnnl_normalization_flags_t).
hint_fwd_pd	Primitive descriptor for a respective forward propagation primitive.
attr	Primitive attributes (can be NULL).

Returns:

dnnl_success on success and a status describing the error otherwise.

Select Your Language

Using Intel.com Search

Quick Links

Recent Searches

Advanced Search

Only search in

Intel® oneAPI Deep Neural Network Developer Guide and Reference

Layer Normalization

Overview

Detailed Documentation