FPGA AI Suite: PCIe-based Design Example User Guide

Download PDF
ID 768977
Date 12/16/2024
Public
Document Table of Contents
Document Table of Contents x
1. FPGA AI Suite PCIe-based Design Example User Guide 2. About the PCIe* -based Design Example 3. Getting Started with the FPGA AI Suite PCIe* -based Design Example 4. Building the FPGA AI Suite Runtime 5. Running the Design Example Demonstration Applications 6. Design Example Components 7. Design Example System Architecture for the Agilex™ 7 FPGA A. FPGA AI Suite PCIe-based Design Example User Guide Archives B. FPGA AI Suite PCIe-based Design Example User Guide Document Revision History
4. Building the FPGA AI Suite Runtime x
4.1. CMake Targets 4.2. Build Options
5. Running the Design Example Demonstration Applications x
5.1. Exporting Trained Graphs from Source Frameworks 5.2. Compiling Exported Graphs Through the FPGA AI Suite 5.3. Compiling the PCIe* -based Example Design 5.4. Programming the FPGA Device ( Agilex™ 7) 5.5. Performing Accelerated Inference with the dla_benchmark Application 5.6. Running the Ported OpenVINO™ Demonstration Applications
5.5. Performing Accelerated Inference with the dla_benchmark Application x
5.5.1. Inference on Image Classification Graphs 5.5.2. Inference on Object Detection Graphs 5.5.3. Additional dla_benchmark Options 5.5.4. The dla_benchmark Performance Metrics
5.5.2. Inference on Object Detection Graphs x
5.5.2.1. The mAP and COCO AP Metrics 5.5.2.2. Specifying Ground Truth 5.5.2.3. Example of Inference on Object Detection Graphs
5.5.4. The dla_benchmark Performance Metrics x
5.5.4.1. Interpreting System Throughput and Latency Metrics
5.6. Running the Ported OpenVINO™ Demonstration Applications x
5.6.1. Example Running the Object Detection Demonstration Application
6. Design Example Components x
6.1. Build Script 6.2. Example Architecture Bitstream Files 6.3. Software Components 6.4. Software Interface to the BSP
6.1. Build Script x
6.1.1. Build Script Options 6.1.2. Script Flow
6.3. Software Components x
6.3.1. OpenVINO™ FPGA Runtime Plugin 6.3.2. FPGA AI Suite Runtime 6.3.3. BSP Driver
7. Design Example System Architecture for the Agilex™ 7 FPGA x
7.1. System Overview 7.2. Hardware
7.2. Hardware x
7.2.1. PLL Adjustment
1. FPGA AI Suite PCIe-based Design Example User Guide
2. About the PCIe* -based Design Example
3. Getting Started with the FPGA AI Suite PCIe* -based Design Example
4. Building the FPGA AI Suite Runtime
5. Running the Design Example Demonstration Applications
6. Design Example Components
7. Design Example System Architecture for the Agilex™ 7 FPGA
A. FPGA AI Suite PCIe-based Design Example User Guide Archives
B. FPGA AI Suite PCIe-based Design Example User Guide Document Revision History

5.5.4.1. Interpreting System Throughput and Latency Metrics

The System throughput and Latency metrics are measured by the host through the OpenVINO™ API. These measurements include any overhead that is incurred by both the API and the FPGA AI Suite runtime. They also account for any time spent waiting to make inference requests and the number of available instances.

In general, the system throughput is defined as follows:

The Batch Size and Images Per Batch values are set by the --batch-size and -niter options, respectively.

For example, consider when -nireq=1 and there is a single IP instance. The System throughput value is approximately the same as the IP-reported throughput value because the runtime can perform only one inference at a time. However, if both the -nireq and the number of IP instances is greater than one, the runtime can perform requests in parallel. As such, the total system throughput is greater than the individual IP throughput.

In general, the -nireq value should be twice the number of IP instances. This setting enables the FPGA AI Suite runtime to pipeline inferences requests, which allows the host to prepare the data for the next request while an IP instance is processing the previous request.

Level Two Title