Multi Channel DMA Intel® FPGA IP for PCI Express User Guide

Download PDF
ID 683821
Date 11/01/2022
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Before You Begin 2. Introduction 3. Functional Description 4. Interface Overview 5. Parameters (H-Tile) 6. Parameters (P-Tile and F-Tile) 7. Designing with the IP Core 8. Software Programming Model 9. Registers 10. Troubleshooting/Debugging 11. Multi Channel DMA Intel FPGA IP for PCI Express User Guide Archives 12. Revision History for Multi Channel DMA Intel FPGA IP for PCI Express User Guide
1. Before You Begin x
1.1. Terms and Acronyms 1.2. Known Issues
2. Introduction x
2.1. Multi Channel DMA IP for PCI Express Features 2.2. Device Family Support 2.3. Recommended Speed Grades 2.4. Resource Utilization 2.5. Release Information
2.1. Multi Channel DMA IP for PCI Express Features x
2.1.1. Endpoint Mode 2.1.2. Root Port Mode
3. Functional Description x
3.1. Multi Channel DMA 3.2. Bursting Avalon-MM Master (BAM) 3.3. Bursting Avalon-MM Slave (BAS) 3.4. MSI Interrupt 3.5. Config Slave (CS) 3.6. Root Port Address Translation Table Enablement 3.7. Hard IP Reconfiguration Interface 3.8. Config TL Interface 3.9. Configuration Intercept Interface (EP Only) 3.10. Data Mover Only
3.1. Multi Channel DMA x
3.1.1. H2D Data Mover 3.1.2. D2H Data Mover 3.1.3. Descriptors 3.1.4. Avalon-MM PIO Master 3.1.5. Avalon-MM Write (H2D) and Read (D2H) Master 3.1.6. Avalon-ST Source (H2D) and Sink (D2H) 3.1.7. User MSI-X 3.1.8. User Functional Level Reset (FLR) 3.1.9. Control Registers
3.1.2. D2H Data Mover x
3.1.2.1. D2H Descriptor Fetch
3.1.3. Descriptors x
3.1.3.1. Support for Unaligned (or byte-aligned) Data Transfer 3.1.3.2. Metadata Support 3.1.3.3. MSI-X/Writeback
3.1.6. Avalon-ST Source (H2D) and Sink (D2H) x
3.1.6.1. Avalon-ST 1-Port Mode 3.1.6.2. Packet (File) Boundary 3.1.6.3. Metadata
3.4. MSI Interrupt x
3.4.1. Endpoint MSI Support through BAS 3.4.2. MSI Interrupt Controller
3.5. Config Slave (CS) x
3.5.1. 29 bit AVMM address format 3.5.2. 14 bit AVMM address format 3.5.3. Configuration Access Mechanism
3.10. Data Mover Only x
3.10.1. H2D Data Mover 3.10.2. D2H Data Mover 3.10.3. Application Specific Bits
3.10.1. H2D Data Mover x
3.10.1.1. H2D Descriptor Format (h2ddm_desc) 3.10.1.2. H2D Descriptor Completion Packet Format (h2ddm_desc_cmpl) 3.10.1.3. H2D Descriptor Status (h2ddm_desc_status)
3.10.2. D2H Data Mover x
3.10.2.1. D2H Descriptor Format (d2hdm_desc) 3.10.2.2. D2H Descriptor Status (d2hdm_desc_status)
4. Interface Overview x
4.1. Port List 4.2. Clocks 4.3. Resets 4.4. Multi Channel DMA 4.5. Bursting Avalon-MM Master (BAM) Interface 4.6. Bursting Avalon-MM Slave (BAS) Interface 4.7. MSI Interface (EP) 4.8. Config Slave Interface (RP only) 4.9. Hard IP Reconfiguration Interface 4.10. Config TL Interface 4.11. Configuration Intercept Interface (EP Only) 4.12. Data Mover Interface 4.13. Hard IP Status Interface
4.1. Port List x
4.1.1. Port List (H-Tile) 4.1.2. Port List (P-Tile and F-Tile)
4.4. Multi Channel DMA x
4.4.1. Avalon-MM PIO Master 4.4.2. Avalon-MM Write Master (H2D) 4.4.3. Avalon-MM Read Master (D2H) 4.4.4. Avalon-ST Source (H2D) 4.4.5. Avalon-ST Sink (D2H) 4.4.6. User Event MSI-X Interface 4.4.7. User Functional Level Reset (FLR) Interface
4.12. Data Mover Interface x
4.12.1. H2D Data Mover Interface 4.12.2. D2H Data Mover Interface
5. Parameters (H-Tile) x
5.1. IP Settings 5.2. Example Designs
5.1. IP Settings x
5.1.1. System Settings 5.1.2. MCDMA Settings 5.1.3. Device Identification Registers 5.1.4. Multifunction and SR-IOV System Settings Parameters [Endpoint Mode] 5.1.5. Configuration, Debug and Extension Options 5.1.6. PHY Characteristics 5.1.7. PCI Express / PCI Capabilities Parameters
5.1.7. PCI Express / PCI Capabilities Parameters x
5.1.7.1. Device 5.1.7.2. Link 5.1.7.3. MSI-X 5.1.7.4. Power Management 5.1.7.5. Vendor Specific Extended Capability (VSEC)
6. Parameters (P-Tile and F-Tile) x
6.1. Top-Level Settings 6.2. PCIe0 Settings 6.3. PCIe1 Settings 6.4. Example Designs
6.2. PCIe0 Settings x
6.2.1. Base Address Register 6.2.2. PCIe0 Configuration, Debug and Extension Options 6.2.3. PCIe0 Device Identification Registers 6.2.4. PCIe0 PCI Express / PCI Capabilities Parameters 6.2.5. MCDMA Settings
6.2.3. PCIe0 Device Identification Registers x
6.2.3.1. PCIe0 PF0 IDs 6.2.3.2. PCIe0 PF0 VF IDs
6.2.4. PCIe0 PCI Express / PCI Capabilities Parameters x
6.2.4.1. PCIe0 Device 6.2.4.2. PCIe0 Link 6.2.4.3. PCIe0 MSI-X 6.2.4.4. PCIe0 DEV SER 6.2.4.5. PCIe0 PRS 6.2.4.6. PCIe0 ACS Capabilities 6.2.4.7. PCIe0 ATS 6.2.4.8. PCIe0 TPH
6.2.4.6. PCIe0 ACS Capabilities x
6.2.4.6.1. PCIe0 ACS for Physical Functions 6.2.4.6.2. PCIe0 ACS for Virtual Functions
6.2.4.7. PCIe0 ATS x
6.2.4.7.1. PCIe0 ATS for Physical Functions 6.2.4.7.2. PCIe0 ATS for Virtual Functions
6.2.4.8. PCIe0 TPH x
6.2.4.8.1. PCIe0 TPH for Physical Functions 6.2.4.8.2. PCIe0 TPH for Virtual Functions
6.3. PCIe1 Settings x
6.3.1. PCIe1 Configuration, Debug and Extension Options
7. Designing with the IP Core x
7.1. Generating the IP Core 7.2. Simulating the IP Core 7.3. IP Core Generation Output - Intel® Quartus® Prime Pro Edition 7.4. Systems Integration and Implementation
7.4. Systems Integration and Implementation x
7.4.1. Required Supporting IP
8. Software Programming Model x
8.1. Multi Channel DMA IP Custom Driver 8.2. Multi Channel DMA IP DPDK Poll-Mode based Driver 8.3. Multi Channel DMA IP Kernel Mode Character Device Driver 8.4. Multi Channel DMA IP Kernel Mode Network Device Driver
8.1. Multi Channel DMA IP Custom Driver x
8.1.1. Architecture 8.1.2. libmqdma library details 8.1.3. Application 8.1.4. Software Flow Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 8.1.5. API Flow 8.1.6. libmqdma Library API List 8.1.7. Request Structures
8.1.2. libmqdma library details x
8.1.2.1. Channel Initialization 8.1.2.2. Descriptor Memory Management 8.1.2.3. Descriptor Completion Status Update
8.1.5. API Flow x
8.1.5.1. Single Descriptor Load and Submit 8.1.5.2. Multiple Descriptor Load and Submit
8.1.6. libmqdma Library API List x
8.1.6.1. ifc_api_start 8.1.6.2. ifc_mcdma_port_by_name 8.1.6.3. ifc_qdma_device_get 8.1.6.4. ifc_num_channels_get 8.1.6.5. ifc_qdma_channel_get 8.1.6.6. ifc_qdma_acquire_channels 8.1.6.7. ifc_qdma_release_all_channels 8.1.6.8. ifc_qdma_device_put 8.1.6.9. ifc_qdma_channel_put 8.1.6.10. ifc_qdma_completion_poll 8.1.6.11. ifc_qdma_request_start 8.1.6.12. ifc_qdma_request_prepare 8.1.6.13. ifc_qdma_descq_queue_batch_load 8.1.6.14. ifc_qdma_request_submit 8.1.6.15. ifc_qdma_pio_read32 8.1.6.16. ifc_qdma_pio_write32 8.1.6.17. ifc_qdma_pio_read64 8.1.6.18. ifc_qdma_pio_write64 8.1.6.19. ifc_qdma_pio_read128 8.1.6.20. ifc_qdma_pio_write128 8.1.6.21. ifc_qdma_pio_read256 8.1.6.22. ifc_qdma_pio_write256 8.1.6.23. ifc_request_malloc 8.1.6.24. ifc_request_free 8.1.6.25. ifc_app_stop 8.1.6.26. ifc_qdma_poll_init 8.1.6.27. ifc_qdma_poll_add 8.1.6.28. ifc_qdma_poll_wait 8.1.6.29. ifc_mcdma_port_by_name
8.2. Multi Channel DMA IP DPDK Poll-Mode based Driver x
8.2.1. Architecture 8.2.2. MCDMA Poll Mode Driver 8.2.3. DPDK based application 8.2.4. Request Structures 8.2.5. Software Flow 8.2.6. API Flow 8.2.7. API List
8.2.2. MCDMA Poll Mode Driver x
8.2.2.1. Completion Status Update 8.2.2.2. Metadata Support 8.2.2.3. User MSI-X
8.3. Multi Channel DMA IP Kernel Mode Character Device Driver x
8.3.1. Architecture 8.3.2. libmcmem Library Information 8.3.3. Kernel Driver Information 8.3.4. Control Message Structure 8.3.5. Software Flow 8.3.6. API Flow 8.3.7. API List
8.3.3. Kernel Driver Information x
8.3.3.1. Device Management 8.3.3.2. Channel Management 8.3.3.3. Completions Management
8.3.3.3. Completions Management x
8.3.3.3.1. Interrupt Mode 8.3.3.3.2. Poll Mode
8.4. Multi Channel DMA IP Kernel Mode Network Device Driver x
8.4.1. Architecture 8.4.2. Driver Information 8.4.3. Software Flow
8.4.2. Driver Information x
8.4.2.1. Device Management 8.4.2.2. Channel Management 8.4.2.3. Descriptor Memory Management 8.4.2.4. Completions Management 8.4.2.5. ethtool support 8.4.2.6. debugfs support 8.4.2.7. SRIOV Support
8.4.3. Software Flow x
8.4.3.1. Host to Device Flow 8.4.3.2. Device to Host Flow
9. Registers x
9.1. Queue Control (QCSR) 9.2. MSI-X Memory Space 9.3. Control Register (GCSR)
10. Troubleshooting/Debugging x
10.1. Debug Toolkit
10.1. Debug Toolkit x
10.1.1. Overview 10.1.2. Enabling the Debug Toolkit 10.1.3. Launching the Debug Toolkit 10.1.4. Using the Debug Toolkit
10.1.4. Using the Debug Toolkit x
10.1.4.1. Main View 10.1.4.2. Toolkit Parameters 10.1.4.3. Channel Parameters 10.1.4.4. Eye Viewer 10.1.4.5. Link Inspector
1. Before You Begin
2. Introduction
3. Functional Description
4. Interface Overview
5. Parameters (H-Tile)
6. Parameters (P-Tile and F-Tile)
7. Designing with the IP Core
8. Software Programming Model
9. Registers
10. Troubleshooting/Debugging
11. Multi Channel DMA Intel FPGA IP for PCI Express User Guide Archives
12. Revision History for Multi Channel DMA Intel FPGA IP for PCI Express User Guide

8.1.4. Software Flow

Figure 44. Multi Channel DMA IP for PCI Express Software Operation Flow

Step 1

  • Application creates the thread based on the required port of a channel
  • After spawning the thread, the thread tries to acquire the available channel’s port. In case if all channels ports are busy thread waits in poll mode
  • In the context of thread, libqdma allocates descriptors buffer memory in the host
  • libqdma initializes following registers in QCSR associates with queue, which includes Starting address of descriptors, queue size, write back address for Consumed Head, payload size in D2H descriptors and then enables the channels
QCSR registers:
  • Q_RESET (offset 8’h48)
  • Q_TAIL_POINTER (offset 8’h14) Set 0
  • Q_START_ADDR_L (Offset 8’h08)
  • Q_START_ADDR_H (Offset 8’h0C)
  • Q_SIZE (Offset 8’h10)
  • Q_CONSUMED_HEAD_ADDR_L (Offset 8’h20)
  • Q_CONSUMED_HEAD_ADDR_H (Offset 8’h24)
  • Q_BATCH_DELAY (Offset 8’h28)
  • Set q_en, q_wb/intr_en bits Q_CTRL (Offset 8’h00)
  • (Q_PYLD_COUNT) (Offset 8'h44)
GCSR register:
  • WB_INTR_DELAY (Offset 8’h08)

Step 2

  • Threads continuously try to send/receive the data and library keeps checking if channel is busy or descriptor ring is full
  • If channel is not busy and descriptor ring is not full it goes to step 3. If channel is busy or descriptors ring is full thread retries to initiate the transfer again
Descriptor ring full is identified by checking the Consumed Head and Tail pointer registers.

Step 3

Thread requests for new descriptor to submit the request and updates the required field i.e. descriptor index, SOF, EOF, Payload, MSI-X enable and writeback enable.

Step 4

After initializing descriptor ring buffer, the libqdma writes number of descriptor updates into tail register of QCSR region. On every descriptor update the tail pointer is increased by 1.

QCSR tail pointer register: Q_TAIL_POINTER (Offset 8’h14)

Step 5

  • Once the tail pointer write happens, Multi Channel DMA IP for PCI Express fetches descriptors from host memory starting from the programmed Q_START_ADDR_L/H address
  • Multi Channel DMA IP for PCI Express parses the descriptor content to find the sources, destination addresses and length of the data from descriptor and starts DMA operation

Step 6

Once descriptor processing is completed, IP notifies the completion status based on following methods, which can be enabled in each descriptor.
  • Either based on MSI-X Interrupt : Multi Channel DMA IP for PCI Express sends MSI-X interrupt to host if enabled in Q_CTRL.
  • Writeback: Multi Channel DMA for PCI Express IP updates Q_CONSUMED_HEAD_ADDR_L/H, if writeback is enabled in Q_CTRL.
Level Two Title