FPGA AI Suite: SoC Design Example User Guide

Download PDF
ID 768979
Date 7/31/2024
Public
Document Table of Contents
Document Table of Contents x
1. FPGA AI Suite SoC Design Example User Guide 2. About the SoC Design Example 3. FPGA AI Suite SoC Design Example Quick Start Tutorial 4. FPGA AI Suite SoC Design Example Run Process 5. FPGA AI Suite SoC Design Example Build Process 6. FPGA AI Suite SoC Design Example Quartus® Prime System Architecture 7. FPGA AI Suite Soc Design Example Software Components 8. Streaming-to-Memory (S2M) Streaming Demonstration A. FPGA AI Suite SoC Design Example User Guide Archives B. FPGA AI Suite SoC Design Example User Guide Document Revision History
2. About the SoC Design Example x
2.1. FPGA AI Suite SoC Design Example Prerequisites
2.1. FPGA AI Suite SoC Design Example Prerequisites x
2.1.1. Agilex™ 7 FPGA I-Series Transceiver-SoC Development Kit Hardware Requirements
3. FPGA AI Suite SoC Design Example Quick Start Tutorial x
3.1. Initial Setup 3.2. Initializing a Work Directory 3.3. (Optional) Create an SD Card Image (.wic) 3.4. Writing the SD Card Image (.wic) to an SD Card 3.5. Preparing SoC FPGA Development Kits for the FPGA AI Suite SoC Design Example 3.6. Adding Compiled Graphs (AOT files) to the SD Card 3.7. Verifying FPGA Device Drivers 3.8. Running the Demonstration Applications
3.3. (Optional) Create an SD Card Image (.wic) x
3.3.1. Installing Prerequisite Software for Building an SD Card Image 3.3.2. Building the FPGA Bitstreams 3.3.3. Installing HPS Disk Image Build Prerequisites 3.3.4. (Optional) Downloading the ImageNet Categories 3.3.5. Building the SD Card Image
3.5. Preparing SoC FPGA Development Kits for the FPGA AI Suite SoC Design Example x
3.5.1. Preparing the Agilex™ 7 FPGA I-Series Transceiver-SoC Development Kit 3.5.2. Preparing the Arria® 10 SX SoC FPGA Development Kit 3.5.3. Configuring the SoC FPGA Development Kit UART Connection 3.5.4. Determining the SoC FPGA Development Kit IP Address
3.5.1. Preparing the Agilex™ 7 FPGA I-Series Transceiver-SoC Development Kit x
3.5.1.1. Confirming Agilex™ 7 FPGA I-Series Transceiver-SoC Development Kit Board Set Up 3.5.1.2. Programming the Agilex™ 7FPGA Device with the JTAG Indirect Configuration (.jic) File 3.5.1.3. Connecting the Agilex™ 7 FPGA I-Series Transceiver-SoC Development Kit to the Host Development System
3.5.2. Preparing the Arria® 10 SX SoC FPGA Development Kit x
3.5.2.1. Confirming Arria® 10 SX SoC FPGA Development Kit Board Settings 3.5.2.2. Connecting the Arria® 10 SX SoC FPGA Development Kit to the Host Development System
3.6. Adding Compiled Graphs (AOT files) to the SD Card x
3.6.1. Preparing OpenVINO™ Model Zoo 3.6.2. Preparing a Model 3.6.3. Compiling the Graphs 3.6.4. Copying the Compiled Graphs to the SD card
3.8. Running the Demonstration Applications x
3.8.1. Running the M2M Mode Demonstration Application 3.8.2. Running the S2M Mode Demonstration Application 3.8.3. Troubleshooting the Demonstration Applications
4. FPGA AI Suite SoC Design Example Run Process x
4.1. Exporting Trained Graphs from Source Frameworks 4.2. Compiling Exported Graphs Through the FPGA AI Suite
5. FPGA AI Suite SoC Design Example Build Process x
5.1. Building the Quartus® Prime Project 5.2. Building the Bootable SD Card Image (.wic)
5.1. Building the Quartus® Prime Project x
5.1.1. Quartus® Prime Build Flow 5.1.2. Build Script Options 5.1.3. Build Directory
5.1.1. Quartus® Prime Build Flow x
5.1.1.1. Build Synchronization of FPGA with Software
5.1.3. Build Directory x
5.1.3.1. The create_project.bash Script 5.1.3.2. The generate_sof.bash Script 5.1.3.3. The generate_rbf.bash Script 5.1.3.4. The build_stream_controller.sh Script
6. FPGA AI Suite SoC Design Example Quartus® Prime System Architecture x
6.1. FPGA AI Suite SoC Design Example Inference Sequence Overview 6.2. Memory-to-Memory (M2M) Variant Design 6.3. Streaming-to-Memory (S2M) Variant Design 6.4. Top Level 6.5. The SoC Design Example Platform Designer System 6.6. Fabric EMIF Design Component 6.7. PLL Configuration
6.2. Memory-to-Memory (M2M) Variant Design x
6.2.1. The mSGDMA Intel FPGA IP 6.2.2. RAM considerations
6.3. Streaming-to-Memory (S2M) Variant Design x
6.3.1. Streaming Enablement for FPGA AI Suite 6.3.2. Nios® V Subsystem 6.3.3. Streaming System Operation 6.3.4. Resolving Input Rate Mismatches Between the FPGA AI Suite IP and the Streaming Input 6.3.5. The Layout Transform IP as an Application-Specific Block
6.3.3. Streaming System Operation x
6.3.3.1. Streaming System Buffer Management 6.3.3.2. Streaming System Inference Job Management
6.3.5. The Layout Transform IP as an Application-Specific Block x
6.3.5.1. Layout Transform Considerations 6.3.5.2. Layout Transform IP Register Map 6.3.5.3. Layout Transform Configuration Options
6.4. Top Level x
6.4.1. Clock Domains
6.5. The SoC Design Example Platform Designer System x
6.5.1. The dla_0 Platform Designer Layer (dla.qsys) 6.5.2. The hps_0 Platform Designer Layer (hps.qys)
7. FPGA AI Suite Soc Design Example Software Components x
7.1. Yocto Build and Runtime Linux Environment 7.2. FPGA AI Suite Runtime Plugin 7.3. Runtime Interaction with the MMD Layer 7.4. MMD Layer Hardware Interaction Library
7.1. Yocto Build and Runtime Linux Environment x
7.1.1. Yocto Recipe: recipes-core/images/coredla-image.bb 7.1.2. Yocto Recipe: recipes-bsp/u-boot/u-boot-socfpga_%.bbappend 7.1.3. Yocto Recipe: recipes-drivers/msgdma-userio/msgdma-userio.bb 7.1.4. Yocto Recipe: recipes-drivers/uio-devices/uio-devices.bb 7.1.5. Yocto Recipe: recipes-kernel/linux/linux-socfpga-lts_5.15.bbappend 7.1.6. Yocto Recipe: wic
7.4. MMD Layer Hardware Interaction Library x
7.4.1. MMD Layer Hardware Interaction Library Class mmd_device 7.4.2. MMD Layer Hardware Interaction Library Class uio_device 7.4.3. MMD Layer Hardware Interaction Library Class dma_device
8. Streaming-to-Memory (S2M) Streaming Demonstration x
8.1. Nios® Subsystem 8.2. Building the Stream Controller Module 8.3. Building the Streaming Demonstration Applications 8.4. Running the Streaming Demonstration
8.1. Nios® Subsystem x
8.1.1. Stream Controller Communication Protocol 8.1.2. Buffer Flow in Streaming Mode using Nios® V Software Scheduler
8.1.2. Buffer Flow in Streaming Mode using Nios® V Software Scheduler x
8.1.2.1. Review of M2M mode 8.1.2.2. External Streaming Mode Buffer Flow 8.1.2.3. Nios® V Stream Controller State Machine Buffer Flow New Buffer Received Message Received
8.4. Running the Streaming Demonstration x
8.4.1. The streaming_inference_app Application 8.4.2. The image_streaming_app Application
1. FPGA AI Suite SoC Design Example User Guide
2. About the SoC Design Example
3. FPGA AI Suite SoC Design Example Quick Start Tutorial
4. FPGA AI Suite SoC Design Example Run Process
5. FPGA AI Suite SoC Design Example Build Process
6. FPGA AI Suite SoC Design Example Quartus® Prime System Architecture
7. FPGA AI Suite Soc Design Example Software Components
8. Streaming-to-Memory (S2M) Streaming Demonstration
A. FPGA AI Suite SoC Design Example User Guide Archives
B. FPGA AI Suite SoC Design Example User Guide Document Revision History

8.1.2.3. Nios® V Stream Controller State Machine Buffer Flow

When the network is loaded into the coredla_device, if external streaming has been enabled, a connection to the Nios® V processor is created and an InitializeScheduler message is sent. This message resets the stream controller and sets the size of the raw input buffers and the drop/receive ratio of buffers from the input stream.

The inference application queries the plugin for the number of inference requests to create. When scheduled with the inference engine, these send ScheduleItem commands to the stream controller, and a corresponding CoreDlaJobItem is created. The CoreDlaJobItem keeps details of the input buffer address and size and has flags to indicate if it has a source buffer and to indicate if it has been scheduled for inference on the FPGA AI Suite IP. The CoreDlaJobItem instances are handled as if they are in a circular buffer.

When the Nios® V stream controller has received a ScheduleItem command from all of the inference requests and created a CoreDlaJobItem instance for each of them, it changes to a running state, which arms the mSGDMA stream to receive buffers, and sets a pointer pFillingImageJob that identifies which of the buffers is the next to be filled.

It then enters a loop, waiting for two types of event:

  • A buffer is received through the mSGDMA, which is detected by a callback from an ISR.
  • A message is received from the HPS.

New Buffer Received

The pFillingImageJob pointer is marked as now having a buffer.

If the next job in the circular buffer does not have a buffer, the pFillingImageJob pointer is moved on and the mSGDMA is armed again to receive the next buffer at the address of this next job.

If it does have a buffer, the FPGA AI Suite IP cannot keep up with the input buffer rate, so the pFillingImageJob does not move and the mSGDMA is armed to capture the next buffer at the same address. This means that the previous input buffer is dropped and is not processed by the FPGA AI Suite IP.

Buffers that have not been dropped can now be scheduled for inference on the FPGA AI Suite IP provided that the IP has fewer than two jobs in its pipeline.

Scheduling a job for execution means programming the CSR registers with the configuration address, the configuration size, and the input buffers address in DDR memory. This programming also sets the flag on the job so the controller knows that the job has been scheduled.

Message Received

If the message is a ScheduleItem message type then an inference request has been scheduled by the inference application.

This request happens only if a previous inference request has been completed and rescheduled. The number of jobs in the FPGA AI Suite IP pipeline has decreased by 1, so another job can potentially be scheduled for inference execution, providing it has an input buffer assigned.

If there are no jobs available with valid input buffers, then the FPGA AI Suite IP is processing buffers faster than they are being received by the mSGDMA stream, and consequently all input buffers are processed (that is, none are dropped).

Level Two Title