FPGA AI Suite: SoC Design Example User Guide

ID 768979
Date 7/31/2024
Public
Document Table of Contents

8.1.2.2. External Streaming Mode Buffer Flow

External streaming mode is enabled in the inference application by setting the configuration value DLIA_CONFIG_KEY(EXTERNAL_STREAMING) to CONFIG_VALUE(YES) in the OpenVINO™ FPGA plugin.

In streaming mode, the inference application does not handle any of the input buffers. It still must create inference requests, which allocate the input and output buffers in the EMIF memory as before, but no input blobs are attached to the inference requests.

When the inference request is executed, there are no preprocessing steps required, since they do not have any input blobs.

Inference Execution Steps

  • Instead of writing a source buffer directly to its allocated address, a ScheduleItem command is sent to the Nios® V stream controller which contains details of the input buffer EMIF address.
  • The FPGA AI Suite IP CSR registers are not programmed by the plugin.
  • The plugin waits for the completion count register to increment as before.
  • The results buffer is read directly from the board as before.

Postprocessing Steps

  • The samples in the results buffer (1001 16-bit floating point values) are converted to 32-bit floating point.
  • The inference application receives these buffers, sorts them, and collects the top five results.
  • The same inference request is rescheduled with the inference engine.