Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics

Download PDF
ID 772991
Date 12/04/2020
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics
Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics x
Legal Information About This Document About Summary Statistics Algorithms and Interfaces in Summary Statistics Common Usage Model of Summary Statistics Algorithms Processing Data in Blocks Detecting Outliers in Datasets Dealing with Missing Observations Computing Quantiles for Streaming Data Bibliography
About This Document x
Conventions and Symbols
Algorithms and Interfaces in Summary Statistics x
Estimating Raw and Central Moments and Sums, Skewness, Excess Kurtosis, Variation, and Variance-Covariance/Correlation/Cross-Product Matrix Computing Median Absolute Deviation Computing Mean Absolute Deviation Computing Minimum/Maximum Values Calculating Order Statistics Estimating Quantiles Estimating a Pooled/Group Variance-Covariance Matrices/Means Estimating a Partial Variance-Covariance Matrix Performing Robust Estimation of a Variance-Covariance Matrix Detecting Multivariate Outliers Handling Missing Values in Matrices of Observations Parameterizing a Correlation Matrix Sorting an Observation Matrix
Estimating Raw and Central Moments and Sums, Skewness, Excess Kurtosis, Variation, and Variance-Covariance/Correlation/Cross-Product Matrix x
Computing Estimations for Large Datasets Calculating Multiple Estimates
Estimating Quantiles x
Computing Quantiles for Streaming Data with VSL_SS_METHOD_SQUANTS_ZW
Handling Missing Values in Matrices of Observations x
Basic Assumptions for the MI Method Basic Components of the MI Method
Detecting Outliers in Datasets x
Using the BACON Algorithm for Outlier Detection Using Robust Methods
Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics

Using the BACON Algorithm for Outlier Detection

The BACON algorithm is a tool for outlier detection that finds "suspicious" observations and provides unbiased statistical estimates for contaminated datasets.

Consider a dataset generated from a multivariate Gaussian distribution with the help of a corresponding generator available in Intel® oneAPI Math Kernel Library (oneMKL). Some of the observations are then replaced with the outliers from the multivariate Gaussian distribution that has a significantly bigger mathematical expectation. The number of outliers is approximately 20%.

To detect the outliers using the BACON algorithm, do the following:

  1. Initialize the algorithm parameters:

    1. Define the initialization scheme of the algorithm. You can choose between Median- and Mahalanobis distance-based schemes.

    2. Define rejection level as alpha and stopping criteria level as beta.

    The parameters are initialized as follows: 

    init_method = VSL_SS_METHOD_BACON_MEDIAN_INIT;
	beta  = 0.005;
	BaconN = VSL_SS_BACON_PARAMS_N;
	BaconParams[0] = init_method;
	BaconParams[1] = alpha;
	BaconParams[2] = beta;

    For details on the parameters, see Table Structure of the Array of BACON Parameters of the Summary Statistics section in [MKLMan].

  2. Pass the parameters into the library using a suitable editor: 

    errcode=vsldSSEditOutliersDetection( task, &BaconN, BaconParams, BaconWeights );

    The BaconWeights parameter is an array of weights that holds the output of the algorithm and points at suspicious observations. The size of the array equals the number of observations. The 0 value in the i-th position of the array indicates that the i-th observation requires special attention. The 1 value indicates that the observation is unbiased.

  3. Call the Compute  routine: 

    errcode = vsldSSCompute( task, VSL_SS_OUTLIERS, VSL_SS_METHOD_BACON );

When the computation completes, the BaconWeights array contains weights of the observations that have to be analyzed. You can use this array in further data processing. Register this array as an array of observation weights and use it in the usual manner. Expectedly, after all outliers are removed, the statistical estimates for the contaminated dataset are not biased.

See Also:

Detection of Multivariate Outliers

Parent topic: Detecting Outliers in Datasets
Level Two Title