A Performance & Scalability Analysis of CNN-Based Deep Learning Inference in the Intel® Distribution of OpenVINO™ Toolkit

Convolutional neural networks (CNNs) are powerful techniques for AI application development, offering the advantage of accuracy in image-recognition problems. In this talk, Intel software engineer Dmitry Matveev analyzes the performance and scalability of several software development tools that provide high-performance CNN-based, deep learning inference on Intel® architectures.

Dmitry focuses on two typical data science problems: image classification¹ and object detection². His experiment plan is as follows:

1. Prepare a set of trained models for several developer tools, including Intel® Distribution of OpenVINO™ toolkit, Intel® Optimization for Caffe*, and OpenCV.
2. Select a large set of images from each dataset to ensure the performance analysis delivers accurate results; experimentally determine the most appropriate parameters (for example, batch size and the number of CPU cores used).
3. Carry out computational experiments on Endeavor, a NASA* shared-memory supercomputer based on 2nd generation Intel® Xeon® Scalable processors (formerly code-named Cascade Lake).

With this experiment, this session covers:

• Performance of the Intel Distribution of OpenVINO toolkit, including comparing it to similar software for CNN-based deep learning inference
• Analysis of the OpenVINO toolkit scaling efficiency using dozens of CPU cores in a throughput mode
• Results of the vector neural network instructions (VNNI) performance acceleration for Intel® Advanced Vector Extensions in Intel Xeon Scalable processors
• Analysis of modern CPU use in CNN-based deep learning inference using the Roofline model included in Intel® Advisor

Check it out.

• Intel Distribution of OpenVINO Toolkit
• Intel Advisor stand-alone download (or get it as part of Intel® Parallel Studio XE or Intel® System Studio)
• Intel® Optimization for Caffe*

Dmitry Matveev

Software engineering manager, Intel Corporation

Dmitry focuses on deep learning application development and optimization. Before joining Intel in 2016, he honed his software knowledge—from functional programming and object-oriented analysis and design to domain-specific languages, digital signal processing, and machine learning—at companies including MERA, SoftDrom, and Itseez. Dmitry holds an MA degree in computer science from Nizhniy Novgorod State Technical University.

1 Image Classification Model: ResNet-50; Dataset: ImageNET
2 Object Detection Model: SSD300; Dataset: PASCAL Visual Object Classes (VOC) 2012