Multi Channel DMA Intel® FPGA IP for PCI Express User Guide

Download PDF
ID 683821
Date 4/11/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Before You Begin 2. Introduction 3. Functional Description 4. Interface Overview 5. Parameters (H-Tile) 6. Parameters (P-Tile) (F-Tile) (R-Tile) 7. Designing with the IP Core 8. Software Programming Model 9. Registers 10. Troubleshooting/Debugging 11. Multi Channel DMA Intel FPGA IP for PCI Express User Guide Archives 12. Revision History for Multi Channel DMA Intel FPGA IP for PCI Express User Guide
1. Before You Begin x
1.1. Terms and Acronyms 1.2. Known Issues
2. Introduction x
2.1. Multi Channel DMA IP for PCI Express Features 2.2. Device Family Support 2.3. Recommended Speed Grades 2.4. Resource Utilization 2.5. Release Information
2.1. Multi Channel DMA IP for PCI Express Features x
2.1.1. Endpoint Mode 2.1.2. Root Port Mode
3. Functional Description x
3.1. Multi Channel DMA 3.2. Bursting Avalon-MM Master (BAM) 3.3. Bursting Avalon-MM Slave (BAS) 3.4. MSI Interrupt 3.5. Config Slave (CS) 3.6. Root Port Address Translation Table Enablement 3.7. Hard IP Reconfiguration Interface 3.8. Config TL Interface 3.9. Configuration Intercept Interface (EP Only) 3.10. Data Mover Only
3.1. Multi Channel DMA x
3.1.1. H2D Data Mover 3.1.2. D2H Data Mover 3.1.3. Descriptors 3.1.4. Avalon-MM PIO Master 3.1.5. Avalon-MM Write (H2D) and Read (D2H) Master 3.1.6. Avalon-ST Source (H2D) and Sink (D2H) 3.1.7. User MSI-X 3.1.8. User Functional Level Reset (FLR) 3.1.9. Control Registers
3.1.2. D2H Data Mover x
3.1.2.1. D2H Descriptor Fetch
3.1.3. Descriptors x
3.1.3.1. Support for Unaligned (or byte-aligned) Data Transfer 3.1.3.2. Metadata Support 3.1.3.3. MSI-X/Writeback
3.1.6. Avalon-ST Source (H2D) and Sink (D2H) x
3.1.6.1. Avalon-ST 1-Port Mode 3.1.6.2. Packet (File) Boundary 3.1.6.3. Metadata
3.4. MSI Interrupt x
3.4.1. Endpoint MSI Support through BAS 3.4.2. MSI Interrupt Controller
3.5. Config Slave (CS) x
3.5.1. 29 bit AVMM address format 3.5.2. 14 bit AVMM address format 3.5.3. Configuration Access Mechanism
3.10. Data Mover Only x
3.10.1. H2D Data Mover 3.10.2. D2H Data Mover 3.10.3. Application Specific Bits
3.10.1. H2D Data Mover x
3.10.1.1. H2D Descriptor Format (h2ddm_desc) 3.10.1.2. H2D Descriptor Completion Packet Format (h2ddm_desc_cmpl) 3.10.1.3. H2D Descriptor Status (h2ddm_desc_status)
3.10.2. D2H Data Mover x
3.10.2.1. D2H Descriptor Format (d2hdm_desc) 3.10.2.2. D2H Descriptor Status (d2hdm_desc_status)
4. Interface Overview x
4.1. Port List 4.2. Clocks 4.3. Resets 4.4. Multi Channel DMA 4.5. Bursting Avalon-MM Master (BAM) Interface 4.6. Bursting Avalon-MM Slave (BAS) Interface 4.7. MSI Interface 4.8. Config Slave Interface (RP only) 4.9. Hard IP Reconfiguration Interface 4.10. Config TL Interface 4.11. Configuration Intercept Interface (EP Only) 4.12. Data Mover Interface 4.13. Hard IP Status Interface
4.1. Port List x
4.1.1. Port List (H-Tile) 4.1.2. Port List (P-Tile) (F-Tile) (R-Tile)
4.2. Clocks x
4.2.1. F-Tile System PLL Reference Clock Requirements
4.4. Multi Channel DMA x
4.4.1. Avalon-MM PIO Master 4.4.2. Avalon-MM Write Master (H2D) 4.4.3. Avalon-MM Read Master (D2H) 4.4.4. Avalon-ST Source (H2D) 4.4.5. Avalon-ST Sink (D2H) 4.4.6. User Event MSI-X Interface 4.4.7. User Functional Level Reset (FLR) Interface
4.12. Data Mover Interface x
4.12.1. H2D Data Mover Interface 4.12.2. D2H Data Mover Interface
5. Parameters (H-Tile) x
5.1. IP Settings 5.2. Example Designs
5.1. IP Settings x
5.1.1. System Settings 5.1.2. MCDMA Settings 5.1.3. Device Identification Registers 5.1.4. Multifunction and SR-IOV System Settings Parameters [Endpoint Mode] 5.1.5. Configuration, Debug and Extension Options 5.1.6. PHY Characteristics 5.1.7. PCI Express / PCI Capabilities Parameters
5.1.7. PCI Express / PCI Capabilities Parameters x
5.1.7.1. Device 5.1.7.2. Link 5.1.7.3. MSI-X 5.1.7.4. Power Management 5.1.7.5. Vendor Specific Extended Capability (VSEC)
6. Parameters (P-Tile) (F-Tile) (R-Tile) x
6.1. Top-Level Settings 6.2. PCIe0 Settings 6.3. PCIe1 Settings 6.4. Example Designs
6.2. PCIe0 Settings x
6.2.1. Base Address Register 6.2.2. PCIe0 Configuration, Debug and Extension Options 6.2.3. PCIe0 Device Identification Registers 6.2.4. PCIe0 PCI Express / PCI Capabilities Parameters 6.2.5. MCDMA Settings
6.2.3. PCIe0 Device Identification Registers x
6.2.3.1. PCIe0 PF0 IDs 6.2.3.2. PCIe0 PF0 VF IDs
6.2.4. PCIe0 PCI Express / PCI Capabilities Parameters x
6.2.4.1. PCIe0 Device 6.2.4.2. PCIe0 Link 6.2.4.3. PCIe0 MSI-X 6.2.4.4. PCIe0 DEV SER 6.2.4.5. PCIe0 PRS 6.2.4.6. PCIe0 ACS Capabilities 6.2.4.7. PCIe0 ATS 6.2.4.8. PCIe0 TPH
6.2.4.6. PCIe0 ACS Capabilities x
6.2.4.6.1. PCIe0 ACS for Physical Functions 6.2.4.6.2. PCIe0 ACS for Virtual Functions
6.2.4.7. PCIe0 ATS x
6.2.4.7.1. PCIe0 ATS for Physical Functions 6.2.4.7.2. PCIe0 ATS for Virtual Functions
6.2.4.8. PCIe0 TPH x
6.2.4.8.1. PCIe0 TPH for Physical Functions 6.2.4.8.2. PCIe0 TPH for Virtual Functions
6.3. PCIe1 Settings x
6.3.1. PCIe1 Configuration, Debug and Extension Options
7. Designing with the IP Core x
7.1. Generating the IP Core 7.2. Simulating the IP Core 7.3. IP Core Generation Output - Intel® Quartus® Prime Pro Edition 7.4. Systems Integration and Implementation
7.4. Systems Integration and Implementation x
7.4.1. Required Supporting IP
8. Software Programming Model x
8.1. Multi Channel DMA IP Custom Driver 8.2. Multi Channel DMA IP DPDK Poll-Mode based Driver 8.3. Multi Channel DMA IP Kernel Mode Network Device Driver
8.1. Multi Channel DMA IP Custom Driver x
8.1.1. Architecture 8.1.2. libmqdma library details 8.1.3. Application 8.1.4. Software Flow 8.1.5. API Flow 8.1.6. libmqdma Library API List 8.1.7. Request Structures
8.1.2. libmqdma library details x
8.1.2.1. Channel Initialization 8.1.2.2. Descriptor Memory Management 8.1.2.3. Descriptor Completion Status Update
8.1.5. API Flow x
8.1.5.1. Single Descriptor Load and Submit 8.1.5.2. Multiple Descriptor Load and Submit
8.1.6. libmqdma Library API List x
8.1.6.1. ifc_api_start 8.1.6.2. ifc_mcdma_port_by_name 8.1.6.3. ifc_qdma_device_get 8.1.6.4. ifc_num_channels_get 8.1.6.5. ifc_qdma_channel_get 8.1.6.6. ifc_qdma_acquire_channels 8.1.6.7. ifc_qdma_release_all_channels 8.1.6.8. ifc_qdma_device_put 8.1.6.9. ifc_qdma_channel_put 8.1.6.10. ifc_qdma_completion_poll 8.1.6.11. ifc_qdma_request_start 8.1.6.12. ifc_qdma_request_prepare 8.1.6.13. ifc_qdma_descq_queue_batch_load 8.1.6.14. ifc_qdma_request_submit 8.1.6.15. ifc_qdma_pio_read32 8.1.6.16. ifc_qdma_pio_write32 8.1.6.17. ifc_qdma_pio_read64 8.1.6.18. ifc_qdma_pio_write64 8.1.6.19. ifc_qdma_pio_read128 8.1.6.20. ifc_qdma_pio_write128 8.1.6.21. ifc_qdma_pio_read256 8.1.6.22. ifc_qdma_pio_write256 8.1.6.23. ifc_request_malloc 8.1.6.24. ifc_request_free 8.1.6.25. ifc_app_stop 8.1.6.26. ifc_qdma_poll_init 8.1.6.27. ifc_qdma_poll_add 8.1.6.28. ifc_qdma_poll_wait 8.1.6.29. ifc_mcdma_port_by_name
8.2. Multi Channel DMA IP DPDK Poll-Mode based Driver x
8.2.1. Architecture 8.2.2. MCDMA Poll Mode Driver 8.2.3. DPDK based application 8.2.4. Request Structures 8.2.5. Software Flow 8.2.6. API Flow 8.2.7. API List
8.2.2. MCDMA Poll Mode Driver x
8.2.2.1. Completion Status Update 8.2.2.2. Metadata Support 8.2.2.3. User MSI-X
8.3. Multi Channel DMA IP Kernel Mode Network Device Driver x
8.3.1. Architecture 8.3.2. Driver Information 8.3.3. Software Flow
8.3.2. Driver Information x
8.3.2.1. Device Management 8.3.2.2. Channel Management 8.3.2.3. Descriptor Memory Management 8.3.2.4. Completions Management 8.3.2.5. ethtool support 8.3.2.6. debugfs support 8.3.2.7. SRIOV Support
8.3.3. Software Flow x
8.3.3.1. Host to Device Flow 8.3.3.2. Device to Host Flow
9. Registers x
9.1. Queue Control (QCSR) 9.2. MSI-X Memory Space 9.3. Control Register (GCSR)
10. Troubleshooting/Debugging x
10.1. Debug Toolkit
10.1. Debug Toolkit x
10.1.1. Overview 10.1.2. Enabling the Debug Toolkit 10.1.3. Launching the Debug Toolkit 10.1.4. Using the Debug Toolkit
10.1.4. Using the Debug Toolkit x
10.1.4.1. Main View 10.1.4.2. Toolkit Parameters 10.1.4.3. Channel Parameters 10.1.4.4. Eye Viewer 10.1.4.5. Link Inspector
10.1.4.4. Eye Viewer x
10.1.4.4.1. Running Eye Viewer in the P-Tile Debug Toolkit 10.1.4.4.2. Running Eye Viewer in the F-Tile Debug Toolkit
1. Before You Begin
2. Introduction
3. Functional Description
4. Interface Overview
5. Parameters (H-Tile)
6. Parameters (P-Tile) (F-Tile) (R-Tile)
7. Designing with the IP Core
8. Software Programming Model
9. Registers
10. Troubleshooting/Debugging
11. Multi Channel DMA Intel FPGA IP for PCI Express User Guide Archives
12. Revision History for Multi Channel DMA Intel FPGA IP for PCI Express User Guide

12. Revision History for Multi Channel DMA Intel FPGA IP for PCI Express User Guide

Date Intel® Quartus® Prime Version IP Version Changes
2023.04.11 23.1

22.2.0 [H-Tile]

5.1.0 [P-Tile]

5.1.0 [F-Tile]

2.0.0 [R-Tile]
2023.02.11 22.4

22.1.0 [H-Tile]

5.0.0 [P-Tile]

5.0.0 [F-Tile]

1.0.0 [R-Tile]

  • Initial Release and selected feature support for MCDMA R-Tile IP
  • Multi Channel DMA IP Kernel Mode Character Device Driver is no longer supported Intel Quartus Prime 22.4 release onwards. All related information has been removed.
  • Known Issues: Bullet points (3) to (7) added for Intel® Quartus® Prime 22.4 release
  • Endpoint Mode: R-Tile information added
  • Root Port Mode: F/R/P/H-Tile specific information added
  • Recommended Speed Grades: R-Tile information added. Gen5 information added.
  • Resource Utilization: BAM_BAS_MCDMA user mode information added
  • Resource Utilization: Intel Agilex R-Tile PCIe x8 [Avalon-MM Interface] table added
  • Release Information: IP version information updated. R-Tile version added.
  • Functional Description: BAM+BAS+MCDMA user mode information added
  • Endpoint MSI Support through BAS: Note added at the end of the section
  • Configuration Intercept Interface (EP Only): R-Tile information added
  • Interface Overview: R-Tile information added
  • Port List (P-Tile) (F-Tile) (R-Tile): Port List updated
  • Clocks: app_slow_clk and pcie_systempll_clk clock signals added
  • F-Tile System PLL Reference Clock Requirements: New section added
  • Resets: R-Tile reset signals added
  • Resets: i_gpio_perst#_n signal information added
  • Interface clock domain for R-Tile added for following:
    • Avalon-MM PIO Master
    • Avalon-MM Write Master (H2D)
    • Avalon-MM Read Master (D2H)
    • User Event MSI-X Interface
    • User Functional Level Reset (FLR) Interface
  • Bursting Avalon-MM Master (BAM) Interface: BAM Signals table updated
  • Bursting Avalon-MM Slave (BAS) Interface: BAS Signals table updated
  • Hard IP Reconfiguration Interface: Table updated with new signals usr_hip_reconfig_clk_o, usr_hip_reconfig_rst_n_o
  • Hard IP Reconfiguration Interface: Reconfig Address added for R-Tile
  • Configuration Intercept Interface (EP Only): R-Tile information added
  • Hard IP Status Interface: R-Tile information added
  • Hard IP Status Interface: p0_ltssm_st_hipfifo_ovrflw_o new signal added
  • MCDMA Settings: BAM+BAS+MCDMA added for User Mode parameter
  • Top-Level Settings: R-Tile information added. P-Tile and F-Tile specific information added.
  • Base Address Register: BAM+BAS+MCDMA user mode added
  • PCIe0 Configuration, Debug and Extension Options: R-Tile information added
  • PCIeo Link: New row added to the table Enable Modified TS
  • PCIeo MSI-X: New row added VF Table size to table PCIe0 PF MSI-X
  • MCDMA Settings: BAM+BAS+MCDMA added for User Mode parameter. R-Tile specific information added.
  • Example Designs: BAM+BAS+MCDMA User Mode information added. R-Tile specific information added.
  • Registers: Note added about Read/Write access to CSR address space
2022.11.01 22.3

22.0.0 [H-Tile]

4.0.0 [P-Tile]

4.0.0 [F-Tile]
  • Known Issues: Past issues that no longer present in the current release have been removed
  • Endpoint Mode: Feature list updated
  • Root Port Mode: Feature list updated
  • Release Information: IP Version and Intel® Quartus® Prime Version updated
  • Functional Description: IP Block Diagram updated
  • Multi Channel DMA: Note description updated to specify alignment in D2H direction
  • D2H Descriptor Fetch: Information added about Tail pointer updates
  • Support for Unaligned (or byte-aligned) Data Transfer: New section added
  • MSI Interrupt: New section and sub-sections added
  • 29 bit AVMM address format: New sub-section added under Config Slave (CS)
  • 14 bit AVMM address format: New sub-section added under Config Slave (CS)
  • Configuration Access Mechanism: New sub-section added under Config Slave (CS)
  • Root Port Address Translation Table Enablement: New section added
  • Port List (P-Tile) (F-Tile): Figure updated with new MSI Interface added
  • MSI Interface: New section added
  • Top-Level Settings
    • Hard IP mode parameter information updated
    • Number of PCIe parameter information updated
    • PLD Clock Frequency parameter information updated
    • Enable Independent Perst parameter information added
  • PCIe0 Configuration, Debug and Extension Options
    • Default value of parameter Gen 3 Requested equalization far-end TX preset vector for F-Tile updated
    • Default Value of parameter Gen 4 Requested equalization far-end TX preset vector for F-Tile updated
  • MCDMA Settings: New parameters added
    • Enable address byte aligned transfer
    • Enable MSI Capability
    • Enable MSI 64-bit addressing
    • Number of MSI Messages Requested
    • Enable MSI Extended Data Capability
    • Export pld_warm_rst_rdy and link_req_rst_n interface to top level
  • MCDMA Settings: New table added Root Port MCDMA Settings ATT Parameters
  • MCDMA Settings: GUI screenshot updated
  • PCIe1 Settings: New section added
  • Example Designs
    • Information updated for Current development kit parameter
    • GUI screenshot added Example Design Settings [for x16 mode]
    • GUI screenshot added Example Design Settings [for 2x8 mode]
  • Queue Control (QCSR)
    • Table Queue Control Registers: Register Name Q_DEBUG_STATUS_3 information removed
    • Table Queue Control Registers: Register Name Q_DATA_DRP_ERR_CTR information added
    • New register table Q_DATA_DRP_ERR_CTR (Offset 8’h40) added
Troubleshooting/Debugging Chapter updates
  • Unless or otherwise noted, the debug toolkit features described in the Troubleshooting/Debugging Chapter apply to both P-Tile and F-Tile.
  • Enabling the Debug Toolkit: Description of this operation expanded
  • Launching the Debug Toolkit: New note added for about the standalone install of Intel® Quartus® Prime Pro Edition
  • Main View: Channel Mapping for Bifurcated Ports table updated
  • Toolkit Parameters
    • In table Available Parameter Settings, rows Advertised speed, Advertised width, Negotiated speed, Negotiated width have been updated
    • In table Available Parameter Settings, new row PIPE PhyStatus (For F-Tile debug toolkit only) has been added
  • Channel Parameters: Tag (For P-Tile debug toolkit only) has been added wherever required
  • Eye Viewer: F-Tile support information added
  • Link Inspector: New section added
2022.08.19 22.2

21.5.0 [H-Tile]

3.1.0 [P-Tile]

3.0.0 [F-Tile]
  • Release Information: Table updated
  • Functional Description: Block Diagram updated
  • MCDMA Settings: Number of ports parameter info updated
  • Top-Level Settings: Enable Ptile Debug Toolkit (P-Tile) and Enable Debug Toolkit (F-Tile) parameters row added
  • Multifunction and SR-IOV System Settings Paramters [Endpoint Mode: Number of DMA channels allocated to PF0 parameter info updated
  • Example Designs: Currently Selected Example Design parameter info updated for H-Tile
  • Example Designs: Current development kit parameter info updated for P-Tile and F-Tile
  • Eye Viewer: Note added about eye-margining
4 Port AVST Mode has been deprecated from this release. All related 4 Port Mode information has been removed from the following sections:
  • Endpoint Mode
  • Avalon-ST Source (H2D) and Sink (D2H)
  • Packet (File) Boundary
  • Bursting Avalon-MM Slave (BAS)
  • Avalon-ST Source (H2D)
  • Avalon-ST Sink (D2H)
  • Port List (H-Tile)
  • Port List (P-Tile and F-Tile)
2022.04.20 22.1

21.4.0 [H-Tile]

3.0.0 [P-Tile]

2.0.0 [F-Tile]
  • Known Issues [New section added]
  • Release Information [IP Version updated]
  • Recommended Speed Grades [Table updated]
  • Resource Utilization [All tables updated]
  • Descriptors [Software Descriptor Format Table Rows Updated: SRC_ADDR [63:0], DEST_ADDR [127:64], PYLD_CNT [147:128]]
  • Avalon-MM PIO Master [Description updated]
  • Avalon-ST 1-Port Mode [Note added]
  • Bursting Avalon-MM Master [Description updated]
  • H2D Descriptor Format (h2ddm_desc) [Table Rows Updated: RSVD [200:200], RSVD [216:201]]
  • H2D Descriptor Completion Packet Format (h2ddm_desc_cmpl) [Table Rows Updated: En_partial_cmpl_data [82:82], Completion data]
  • D2H Data Mover [Table Row updated: d2hdm_desc]
  • D2H Descriptor Format (d2hdm_desc) [Table Rows updated: SRC_ADDR [63:0], MM_mode [176:176], App_specific_bits [179:177], DESC_IDX1 [195:180], RSVD [196:196], RSVD [212:197]]
  • Application Specific Bits [Table updated]
  • Avalon-MM PIO Master [Note added]
  • Avalon-MM PIO Master [Table updated. Row: rx_pio_address_o[n:0]]
  • MCDMA Settings [H-Tile. New GUI Screesnhot added] [D2H Prefetch channels Row in table updated]
  • Base Address Register [P-Tile and F-Tile] [Note added]
  • MCDMA Settings [P-Tile and F-Tile] [D2H Prefetch channels Row in table updated]
  • Example Designs [Currently Selected Example Design Row in Table updated]
  • Software Flow [QCSR Registers list updated in Step 1]
  • Queue Control (QCSR) [Queue Control Registers Table Rows Updated: Q_PYLD_CNT,Q_RESET]
  • Control Register (GCSR) [Note Added]
2022.01.14 21.4

21.3.0 [H-Tile]

2.2.0 [P-Tile]

1.1.0 [F-Tile]
  • Data Mover Only user mode option added to Endpoint Mode
  • Resource Utilization tables updated
  • IP Version updated in Release Information
  • Data Mover Only user mode option added in Chapter Functional Description
  • Data Mover Interface and Hard IP Status Interface added to Chapter Interface Iverview
  • Port List (P-Tile and F-Tile) figure updated with data mover mode interfaces
  • PCIe0 Configuration, Debug and Extension Options section updated in Parameters (P-Tile and F-Tile) Chapter
  • Enable 10-bit tag support GUI feature added to MCDMA Settings in Parameters (P-Tile and F-Tile) Chapter
  • Data Mover Only user mode option added to Example Designs table in Parameters (P-Tile and F-Tile) Chapter
  • Device Management and Channel Management sections updated for Network Device Driver
  • ethtool support and debugfs support added to Network Device Driver
  • SRIOV Support information added for Network Device Driver
2021.10.29 21.3

21.2.0 [H-Tile]

2.1.0 [P-Tile]

1.0.0 [F-Tile]
  • Recommended Speed Grades table updated with F-Tile support information
  • Resource Utilization tables updated
  • Release Information updated
  • Valid user modes and required functional blocks table updated
  • Address format information added to Config Slave
  • Multi Channel DMA IP for PCI Express Port List (P-Tile and F-Tile) figure updated with F-Tile information
  • Config TL Interface signal table updated
  • F-Tile support information added to Configuration Intercept Interface (EP Only)
  • F-Tile support information added to Parameters (P-Tile and F-Tile) Chapter
  • MCDMA IP Software Driver Differentiation table added
  • Network Device Driver information added in Multi Channel DMA IP Kernel Mode Network Device Driver
  • Debug Toolkit information added
2021.08.16 21.2

21.1.0 [H-Tile]

2.0.0 [P-Tile]

  • Fixed H-Tile IP revision number
  • Added 500 MHz support for P-Tile MCDMA IP
  • Added P-Tile single port Avalon-ST DMA up to 256 channels
  • Added MCDMA IP DPDK Poll-Mode based Driver
  • Added MCDMA IP Kernel Mode (No SRIOV) Driver
2021.05.28 21.1

2.0.0 [H-Tile]

1.0.0 [P-Tile]

  • PCIe Gen4 (P-Tile) Support
  • Support for x8 link width
  • MCDMA 1 port AVST interface
  • BAM, BAS, BAM+BAS, BAM+MCDMA modes
  • SR-IOV support
  • Root Port support (IP only)
  • Config Slave interface for RP
2020.07.20 20.2

20.0.0 [H-Tile]

 Initial Release
Level Two Title