Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics

Download PDF
ID 772991
Date 12/04/2020
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics
Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics x
Legal Information About This Document About Summary Statistics Algorithms and Interfaces in Summary Statistics Common Usage Model of Summary Statistics Algorithms Processing Data in Blocks Detecting Outliers in Datasets Dealing with Missing Observations Computing Quantiles for Streaming Data Bibliography
About This Document x
Conventions and Symbols
Algorithms and Interfaces in Summary Statistics x
Estimating Raw and Central Moments and Sums, Skewness, Excess Kurtosis, Variation, and Variance-Covariance/Correlation/Cross-Product Matrix Computing Median Absolute Deviation Computing Mean Absolute Deviation Computing Minimum/Maximum Values Calculating Order Statistics Estimating Quantiles Estimating a Pooled/Group Variance-Covariance Matrices/Means Estimating a Partial Variance-Covariance Matrix Performing Robust Estimation of a Variance-Covariance Matrix Detecting Multivariate Outliers Handling Missing Values in Matrices of Observations Parameterizing a Correlation Matrix Sorting an Observation Matrix
Estimating Raw and Central Moments and Sums, Skewness, Excess Kurtosis, Variation, and Variance-Covariance/Correlation/Cross-Product Matrix x
Computing Estimations for Large Datasets Calculating Multiple Estimates
Estimating Quantiles x
Computing Quantiles for Streaming Data with VSL_SS_METHOD_SQUANTS_ZW
Handling Missing Values in Matrices of Observations x
Basic Assumptions for the MI Method Basic Components of the MI Method
Detecting Outliers in Datasets x
Using the BACON Algorithm for Outlier Detection Using Robust Methods
Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics

Processing Data in Blocks

Summary Statistics enables block-based data analysis that can help you:

  1. compute statistical estimates for out-of-memory datasets, splitting them into blocks

  2. analyze in-memory data arrays that become available block by block

  3. tune your applications for out-of-memory data support

To compute statistical estimates for out-of-memory datasets, do the following:

  1. Set the estimates of your interest to zero, or to any other meaningful value: 

    for( i = 0; i < p; i++ )
{
     Xmean[i] = 0.0;
     Raw2Mom[i] = 0.0;
     Central2Mom[i] = 0.0;
     for(j = 0; j < p; j++)
     {
         Cov[i][j] = 0.0;
     }
}

  2. Initialize array W of size 2 with zero values.

    This array holds accumulated weights that are important for correct computation of the estimates: 

    W[0] = 0.0; W[1] = 0.0;

  3. Get the first portion of the dataset into array X, and the corresponding weights into array weights: 

    GetNextDataChunk( X, weights );

  4. Follow the common usage model of the Summary Statistics algorithms: 

    /* Create a task */
xstorage = VSL_SS_MATRIX_STORAGE_COLS;
errcode = vsldSSNewTask( &task, &p, &nblock,
                         &xstorage, X, weights, indices );
 
/* Edit the task parameters */
errcode = vsldSSEditTask( task, VSL_SS_ED_ACCUM_WEIGHT, W );
errcode = vsldSSEditTask( task, VSL_SS_ED_VARIATION, Variation );
errcode = vsldSSEditMoments( task, Xmean, Raw2Mom, 0, 0, Central2Mom, 0, 0 );
 
covstorage = VSL_SS_MATRIX_STORAGE_FULL;
errcode = vsldSSEditCovCor( task, Xmean, cov, &covstorage, 0, 0 );
 
/* Compute the estimates for the dataset split into chunks */
estimates = VSL_SS_MEAN | VSL_SS_2C_MOM | VSL_SS_COV | VSL_SS_VARIATION;
for( nchunk = 0;  nchunk++; )
     errcode = vsldSSCompute( task, estimates, VSL_SS_1PASS_METHOD );
     If ( nchunk >= N ) break;
     GetNextDataChunk( X, weights );
}
 
/* Deallocate task resources */
errcode = vslSSDeleteTask( &task );

Summary statistics domain also enables reading the next data block into a different array. The whole computation scheme remains the same. You just need to provide the address of this data block to the library: 

double* NextXChunk[N];
estimates = VSL_SS_MEAN | VSL_SS_2C_MOM | VSL_SS_COV | VSL_SS_VARIATION;
for( nchunk = 0; nchunk++; )
{
     errcode = vsldSSCompute( task, estimates, VSL_SS_1PASS_METHOD );
     If ( nchunk >= N ) break;
     GetNextDataChunk( NextXChunk, [nchunk], weights );
     errcode = vsldSSEditTask( task, VSL_SS_ED_OBSERV, NextXChunk,[nchunk] );
}

For the list of estimators that support processing datasets in blocks, see Table VS Summary Statistics Estimates Obtained with Compute Routine in the Summary Statistics section of [MKLMan].

Product and Performance Information

= = = = = = = = = =

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.

Notice revision #20201201

= = = = = = = = = =

Parent topic: Application Notes for Intel® oneAPI Math Kernel Library Summary Statistics
Level Two Title