AN 881: PCI Express* Gen3 x16 Avalon® Memory Mapped (Avalon-MM) DMA with DDR4 SDRAM and HBM2 Memories Reference Design

Download PDF
ID 683291
Date 4/19/2021
Public
Document Table of Contents
Document Table of Contents x
1. Introduction 2. Reference Design Description 3. Understanding PCI Express* throughput 4. Document Revision History for AN 881: PCI Express* Gen3 x16 Avalon® Memory Mapped (Avalon-MM) DMA with External and HBM2 Memories Reference Design
1. Introduction x
1.1. PCI Express Gen3 x16 Avalon-MM DMA with External and HBM2 Memory Hardware and Software Requirements 1.2. Avalon-MM Bridge with DMA Module Description 1.3. DMA Procedure Steps 1.4. Setting Up the Hardware 1.5. Installing the DMA Test Driver and Running the Linux DMA Software
2. Reference Design Description x
2.1. Project Hierarchy 2.2. Parameter Settings for PCI Express* Hard IP Variations 2.3. PCIe Avalon-MM DMA Reference Design with External and HBM2 Memories Platform Designer System 2.4. Intel Stratix 10 MX DMA Memory Throughput
3. Understanding PCI Express* throughput x
3.1. Throughput for Posted Writes 3.2. Throughput for Reads
3.1. Throughput for Posted Writes x
3.1.1. Specifying the Maximum Payload Size
3.2. Throughput for Reads x
3.2.1. Understanding Throughput Measurement
1. Introduction
2. Reference Design Description
3. Understanding PCI Express* throughput
4. Document Revision History for AN 881: PCI Express* Gen3 x16 Avalon® Memory Mapped (Avalon-MM) DMA with External and HBM2 Memories Reference Design

2. Reference Design Description

This application note consists of a reference design using the Avalon® -MM Intel® Stratix® 10 MX Hard IP+ DMA with external DDR4 and HBM2 memories.

Figure 5. Gen3 x16 DMA with DDR4 and HBM2 Memories Reference Design
This Gen3 x16 DMA with DDR4 and HBM2 Platform Designer system (captured in the file g3x16_hbm2_ddr4.qsys) instantiates four Avalon® -MM clock-crossing bridges and an AXI bridge IP core between the PCIe Avalon® -MM Masters, DDR4 and HBMC AXI Slave. The purpose of those IPs is to perform the following functions:
  • PCIe Hard IP and HBMC clock domain crossing
  • PCIe Hard IP and DDR4 clock domain crossing
  • Burst length adaptation
  • Exporting the AXI Master interface
  • Controlling the Read/Write Response FIFO depth

PCIe Hard IP and Memory clock domain crossing

The Gen3 x16 IP user interface is 512-bit @ 250 MHz. The 250 MHz is the frequency of the coreclkout_hip generated by the PCIe Hard IP. The HBM Controller AXI interface in the design is 256-bit @ 300 MHz. The HBM Controller core clock is generated by an IOPLL. Two Avalon® -MM clock-crossing bridges are used to handle the clock crossing.

The DDR4 Controller interface in the design uses 512-bit @ 266.67 MHz. Two Avalon® -MM clock-crossing bridges are used to handle the clock crossing.

Burst length adaptation

The Gen3 x16 IP Write Data Mover (WRDM) and Read Data Mover (RDDM) Avalon® -MM interfaces are bursting masters that issue Read/Write transactions in burst mode (the maximum burst count supported is 8). However, the HBM Controller AXI4 slave only supports single-burst transfers (burst length of 1). To resolve this, the maximum burst size in the Avalon® -MM clock crossing bridges is set to 1.

Exporting the AXI Master interface

The design uses an AXI bridge to export the AXI Master interface from the Platform Designer system. The exported AXI Master interface is connected externally to the HBMC AXI Slave interfaces.

The AXI Bridge Read/Write address drives both HBMC AXI slaves.

The AXI Bridge Read/Write 512-bit data bus is split into two 256-bit data busses.

For a picture of how the AXI Bridge is incorporated into the reference design, refer to Figure 5.

Controlling the Read/Write Response FIFO depth

The AXI Bridge Read/Write Acceptance Capability parameter setting dictates the Interconnect Read/Write Response FIFO depth generated by Platform Designer in the altera_merlin_axi_slave_ni module. The Response FIFO depth affects the Avalon® -MM transaction performance.

If the Read/Write Response FIFO depth is not deep enough and the FIFO becomes full, it creates backpressure, impacting the throughput.

The default Read/Write Acceptance Capability parameter value is set to 16. Intel® Quartus® Prime 19.1 allows up to 32. In this design, the Read/Write Response FIFO depth generated by Platform Designer is manually changed to 64 in the altera_merlin_axi_slave_ni module in order to support Gen3 x16 throughput.

In the 19.3 Intel® Quartus® Prime release, the AXI Bridge will support a higher maximum value.

Level Two Title