L-Tile and H-Tile Avalon® Memory-mapped Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683667
Date 4/03/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Introduction 2. Quick Start Guide 3. Interface Overview 4. Parameters 5. Designing with the IP Core 6. Block Descriptions 7. Registers 8. Programming Model for the DMA Descriptor Controller 9. Programming Model for the Avalon® -MM Root Port 10. Avalon-MM Testbench and Design Example 11. Troubleshooting and Observing the Link A. PCI Express Core Architecture B. Root Port Enumeration C. Document Revision History
1. Introduction x
1.1. Avalon-MM Interface for PCIe 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Recommended Fabric Speed Grades 1.6. Performance and Resource Utilization 1.7. Transceiver Tiles 1.8. PCI Express IP Core Package Layout 1.9. Channel Availability
2. Quick Start Guide x
2.1. Design Components 2.2. Hardware and Software Requirements 2.3. Directory Structure 2.4. Generating the Design Example 2.5. Simulating the Design Example 2.6. Compiling the Design Example and Programming the Device 2.7. Installing the Linux Kernel Driver 2.8. Running the Design Example Application
3. Interface Overview x
3.1. Avalon-MM DMA Interfaces when Descriptor Controller Is Internally Instantiated 3.2. Avalon-MM DMA Interfaces when Descriptor Controller is Externally Instantiated 3.3. Other Avalon-MM Interfaces 3.4. Clocks and Reset 3.5. System Interfaces
3.3. Other Avalon-MM Interfaces x
3.3.1. Avalon-MM Master Interfaces 3.3.2. Avalon-MM Slave Interfaces 3.3.3. Control Register Access (CRA) Avalon-MM Slave
4. Parameters x
4.1. Avalon-MM Settings 4.2. Base Address Registers 4.3. Device Identification Registers 4.4. PCI Express and PCI Capabilities Parameters 4.5. Configuration, Debug and Extension Options 4.6. PHY Characteristics 4.7. Example Designs
4.4. PCI Express and PCI Capabilities Parameters x
4.4.1. Device Capabilities 4.4.2. Link Capabilities 4.4.3. MSI and MSI-X Capabilities 4.4.4. Slot Capabilities 4.4.5. Power Management 4.4.6. Vendor Specific Extended Capability (VSEC)
5. Designing with the IP Core x
5.1. Generation 5.2. Simulation 5.3. IP Core Generation Output ( Intel® Quartus® Prime Pro Edition) 5.4. Channel Layout and PLL Usage
6. Block Descriptions x
6.1. Interfaces
6.1. Interfaces x
6.1.1. Intel® Stratix® 10 DMA Avalon-MM DMA Interface to the Application Layer 6.1.2. Avalon-MM Interface to the Application Layer 6.1.3. Clocks and Reset 6.1.4. Interrupts 6.1.5. Flush Requests 6.1.6. Serial Data, PIPE, Status, Reconfiguration, and Test Interfaces
6.1.1. Intel® Stratix® 10 DMA Avalon-MM DMA Interface to the Application Layer x
6.1.1.1. Descriptor Controller Interfaces when Instantiated Internally 6.1.1.2. Write DMA Avalon-MM Master Port 6.1.1.3. Descriptor Controller Interfaces when Instantiated Externally
6.1.1.1. Descriptor Controller Interfaces when Instantiated Internally x
6.1.1.1.1. Read Data Mover 6.1.1.1.2. Read Descriptor Controller Avalon-MM Master interface 6.1.1.1.3. Write Descriptor Controller Avalon-MM Master Interface 6.1.1.1.4. Read Descriptor Table Avalon-MM Slave Interface 6.1.1.1.5. Write Descriptor Table Avalon-MM Slave Interface
6.1.1.3. Descriptor Controller Interfaces when Instantiated Externally x
6.1.1.3.1. Avalon® -ST Descriptor Source
6.1.2. Avalon-MM Interface to the Application Layer x
6.1.2.1. Bursting and Non-Bursting Avalon® -MM Module Signals 6.1.2.2. Non-Bursing Slave Module 6.1.2.3. 32-Bit Control Register Access (CRA) Slave Signals 6.1.2.4. Bursting Slave Module
6.1.3. Clocks and Reset x
6.1.3.1. Clocks 6.1.3.2. Resets
6.1.4. Interrupts x
6.1.4.1. MSI Interrupts for Endpoints 6.1.4.2. Legacy Interrupts
6.1.6. Serial Data, PIPE, Status, Reconfiguration, and Test Interfaces x
6.1.6.1. Serial Data Interface 6.1.6.2. PIPE Interface 6.1.6.3. Hard IP Status Interface 6.1.6.4. Hard IP Reconfiguration 6.1.6.5. Test Interface
7. Registers x
7.1. Configuration Space Registers 7.2. Avalon-MM DMA Bridge Registers
7.1. Configuration Space Registers x
7.1.1. Register Access Definitions 7.1.2. PCI Configuration Header Registers 7.1.3. PCI Express Capability Structures 7.1.4. Intel Defined VSEC Capability Header 7.1.5. Uncorrectable Internal Error Status Register 7.1.6. Uncorrectable Internal Error Mask Register 7.1.7. Correctable Internal Error Status Register 7.1.8. Correctable Internal Error Mask Register
7.1.4. Intel Defined VSEC Capability Header x
7.1.4.1. Intel Defined Vendor Specific Header 7.1.4.2. Intel Marker 7.1.4.3. JTAG Silicon ID 7.1.4.4. User Configurable Device and Board ID
7.2. Avalon-MM DMA Bridge Registers x
7.2.1. PCI Express Avalon-MM Bridge Register Address Map 7.2.2. DMA Descriptor Controller Registers
7.2.1. PCI Express Avalon-MM Bridge Register Address Map x
7.2.1.1. Avalon-MM to PCI Express Interrupt Status Registers 7.2.1.2. Avalon-MM to PCI Express Interrupt Enable Registers 7.2.1.3. Address Mapping for High-Performance Avalon-MM 32-Bit Slave Modules 7.2.1.4. PCI Express to Avalon-MM Interrupt Status and Enable Registers for Endpoints 7.2.1.5. PCI Express Configuration Information Registers
7.2.2. DMA Descriptor Controller Registers x
7.2.2.1. Read DMA Internal Descriptor Controller Registers 7.2.2.2. Write DMA Internal Descriptor Controller Registers
8. Programming Model for the DMA Descriptor Controller x
8.1. Read DMA Example 8.2. Write DMA Example 8.3. Software Program for Simultaneous Read and Write DMA 8.4. Read DMA and Write DMA Descriptor Format
9. Programming Model for the Avalon® -MM Root Port x
9.1. Root Port TLP Data Control and Status Registers 9.2. Sending a TLP 9.3. Receiving a Non-Posted Completion TLP 9.4. Example of Reading and Writing BAR0 Using the CRA Interface
10. Avalon-MM Testbench and Design Example x
10.1. Avalon-MM Endpoint Testbench 10.2. Endpoint Design Example 10.3. Avalon® -MM Test Driver Module 10.4. Root Port BFM 10.5. BFM Procedures and Functions
10.2. Endpoint Design Example x
10.2.1. BAR Setup
10.4. Root Port BFM x
10.4.1. Overview 10.4.2. Issuing Read and Write Transactions to the Application Layer 10.4.3. Configuration of Root Port and Endpoint 10.4.4. Configuration Space Bus and Device Numbering 10.4.5. BFM Memory Map
10.5. BFM Procedures and Functions x
10.5.1. ebfm_barwr Procedure 10.5.2. ebfm_barwr_imm Procedure 10.5.3. ebfm_barrd_wait Procedure 10.5.4. ebfm_barrd_nowt Procedure 10.5.5. ebfm_cfgwr_imm_wait Procedure 10.5.6. ebfm_cfgwr_imm_nowt Procedure 10.5.7. ebfm_cfgrd_wait Procedure 10.5.8. ebfm_cfgrd_nowt Procedure 10.5.9. BFM Configuration Procedures 10.5.10. BFM Shared Memory Access Procedures 10.5.11. BFM Log and Message Procedures 10.5.12. Verilog HDL Formatting Functions
10.5.9. BFM Configuration Procedures x
10.5.9.1. ebfm_cfg_rp_ep Procedure 10.5.9.2. ebfm_cfg_decode_bar Procedure
10.5.10. BFM Shared Memory Access Procedures x
10.5.10.1. Shared Memory Constants 10.5.10.2. shmem_write Task 10.5.10.3. shmem_read Function 10.5.10.4. shmem_display Verilog HDL Function 10.5.10.5. shmem_fill Procedure 10.5.10.6. shmem_chk_ok Function
10.5.11. BFM Log and Message Procedures x
10.5.11.1. ebfm_display Verilog HDL Function 10.5.11.2. ebfm_log_stop_sim Verilog HDL Function 10.5.11.3. ebfm_log_set_suppressed_msg_mask Task 10.5.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task 10.5.11.5. ebfm_log_open Verilog HDL Function 10.5.11.6. ebfm_log_close Verilog HDL Function
10.5.12. Verilog HDL Formatting Functions x
10.5.12.1. himage1 10.5.12.2. himage2 10.5.12.3. himage4 10.5.12.4. himage8 10.5.12.5. himage16 10.5.12.6. dimage1 10.5.12.7. dimage2 10.5.12.8. dimage3 10.5.12.9. dimage4 10.5.12.10. dimage5 10.5.12.11. dimage6 10.5.12.12. dimage7
11. Troubleshooting and Observing the Link x
11.1. Troubleshooting 11.2. PCIe Link Inspector Overview
11.1. Troubleshooting x
11.1.1. Simulation Fails To Progress Beyond Polling.Active State 11.1.2. Hardware Bring-Up Issues 11.1.3. Link Training 11.1.4. Use Third-Party PCIe Analyzer
11.2. PCIe Link Inspector Overview x
11.2.1. PCIe* Link Inspector Hardware
11.2.1. PCIe* Link Inspector Hardware x
11.2.1.1. Enabling the PCIe* Link Inspector 11.2.1.2. Launching the PCIe* Link Inspector 11.2.1.3. The PCIe* Link Inspector LTSSM Monitor 11.2.1.4. Accessing the Configuration Space and Transceiver Registers 11.2.1.5. Additional Status Commands
11.2.1.3. The PCIe* Link Inspector LTSSM Monitor x
11.2.1.3.1. LTSSM Monitor Registers 11.2.1.3.2. ltssm_save2file <file_name> 11.2.1.3.3. ltssm_file2console 11.2.1.3.4. ltssm_debug_check 11.2.1.3.5. ltssm_display <num_states> 11.2.1.3.6. ltssm_save_oldstates
11.2.1.4. Accessing the Configuration Space and Transceiver Registers x
11.2.1.4.1. PCIe* Link Inspector Commands 11.2.1.4.2. NPDME PLL and Channel Commands 11.2.1.4.3. Register Address Map
11.2.1.5. Additional Status Commands x
11.2.1.5.1. Displaying PLL Lock and Calibration Status Registers
A. PCI Express Core Architecture x
A.1. Transaction Layer A.2. Data Link Layer A.3. Physical Layer
C. Document Revision History x
C.1. Document Revision History for the L-Tile and H-Tile Avalon® Memory-mapped Intel® FPGA IP for PCI Express* User Guide
1. Introduction
2. Quick Start Guide
3. Interface Overview
4. Parameters
5. Designing with the IP Core
6. Block Descriptions
7. Registers
8. Programming Model for the DMA Descriptor Controller
9. Programming Model for the Avalon® -MM Root Port
10. Avalon-MM Testbench and Design Example
11. Troubleshooting and Observing the Link
A. PCI Express Core Architecture
B. Root Port Enumeration
C. Document Revision History

C.1. Document Revision History for the L-Tile and H-Tile Avalon® Memory-mapped Intel® FPGA IP for PCI Express* User Guide

Document Version Intel® Quartus® Prime Version Changes
2023.04.03 23.1 Changed the IP name from Intel® L-/H-Tile Avalon® memory mapped IP for PCI Express* to L-/H-Tile Avalon® Memory-Mapped Intel® FPGA IP for PCI Express* .

The Intel® Quartus® Prime Platform Designer has been updated to give an explicit warning when the input BAR size exceeds the limit.

2021.11.11 21.1 Removed references to the 64-bit Application interface data width from all chapters since only the 256-bit interface width is supported.
2021.10.19 21.1 Changed the device support level for Intel® Stratix® 10 to Final Support in the Device Family Support section.
2021.05.27 21.1

Added an Appendix chapter on Root Port enumeration.

Added a note to the Features section stating that L- and H-tile Avalon® Memory-mapped IP for PCI Express only supports the Separate Reference Clock With No Spread Spectrum architecture (SRNS), but not the Separate Reference Clock With Independent Spread Spectrum architecture (SRIS).

2021.04.12 20.3 Added a note to the Features and Generating the Design Example sections stating that pin allocations for this IP cannot be changed in the Intel® Quartus® Prime project, but the IP does support lane reversal and polarity inversion on the PCB by default.
2020.10.05 20.3

Updated the IP name to Intel L-/H-tile Avalon Memory-mapped IP for PCI Express.

Removed the Simple DMA design example option from the Generating the Design Example section as that design example is no longer available.

Added a note to the Read Data Mover section stating that Completion TLPs are restricted to a data payload of up to 256 bytes.

2020.06.03 20.1 Added the description for the new input ninit_done to the Resets section under Clocks and Resets. Also added a link to AN 891: Using the Reset Release Intel FPGA IP, which describes the Reset Release IP that is used to drive the ninit_done input.
2020.05.11 20.1 Changed the Altera Debug Master Endpoint (ADME) to Native PHY Debug Master Endpoint (NPDME).
2020.04.23 19.3 Updated the ready latency value in section Avalon® -ST Descriptor Source from 3 cycles to 1 cycle.
2020.04.22 19.3

Updated the document title to Intel® Stratix® 10 H-Tile and L-Tile Avalon® memory mapped Hard IP for PCI Express* User Guide to meet new legal naming guidelines.

Fixed a typo in the byte address of some Reserved bits in Table 51. Correspondence between Configuration Space Capability Structures and the PCIe Base Specification Description.

2020.03.25 19.3

Added notes stating that the Hard IP Reconfiguration interface is not accessible if the PCIe Link Inspector is enabled to the System Interfaces and Hard IP Reconfiguration Interface sections.

2020.01.03 19.3

Updated resource utilization numbers for the Gen1 x1 variant.

Added notes stating that the Gen3 x16 variant is supported by the Intel® Stratix® 10 Avalon® Memory Mapped (Avalon-MM) Hard IP+ for PCI Express* .

2019.09.30 19.3

Added a note to clarify that this User Guide is applicable to H-Tile and L-Tile variants of the Intel® Stratix® 10 devices only.

Added Autonomous Hard IP mode to the Features section.

2019.07.18 19.1 Added a note stating that refclk must be stable and free-running at device power-up for a successful device configuration.
2019.03.30 19.1

Added a chapter on the programming model for Root Ports.

Removed the note stating that Root Port mode is not recommended.

Removed the BIOS Enumeration section from the Troubleshooting chapter.

2019.03.12 18.1.1 Updated the E-Tile PAM-4 frequency to 57.8G and NRZ frequency to 28.9G.
2019.03.04 18.1.1 Updated the commands to run VCS, NCSim and Xcelium simulations in the Simulating the Design Example topic.
2018.12.24 18.1.1

Added the description for the Link Inspector Avalon® -MM Interface.

Added the Avalon® -MM-to-PCIe rxm_irq for MSI feature.

2018.10.26 18.1 Added the statements that the IP core does not support the L1/L2 low-power states, the in-band beacon and sideband WAKE# signal.
2018.09.24 18.1

Added the ltssm_file2console and ltssm_save_oldstates commands for the PCIe* Link Inspector.

Updated the steps to run ModelSim simulations for a design example.

Updated the steps to run a design example.
2018.08.29 18.0 Added the step to invoke vsim to the instructions for running a ModelSim simulation.
Date Version Changes
May 2018 18.0

Made the following changes to the user guide:

  • Edited the chapter on 32-Bit Control Register Access (CRA) to state that in RP mode, application logic must set the Tag field in the TLP Header to 0x10.
  • Added the note that AER is always enabled to the Features chapter.
  • Added a sub-topic in the chapter Interfaces to state that flush requests are not supported.
  • Updated the chapter PCI Express Configuration Information Registers to state that Extended Tag is not supported.
  • Updated the GUI screenshot and the list of steps in Generating the Design Example. Also added a description for the recommended_pinassignments_s10.txt file.
  • Updated the chapter Parameters to add the Application Interface Width parameter and the configurations available when the 64-bit option for that parameter is chosen.
  • Updated the chapters Interface Overview, Avalon-MM Master Interfaces, and Avalon-MM Slave Interfaces to state that DMA operations are available for 256-bit application interface width but not for 64-bit, and add a row for the 64-bit bursting case to the features table.
November 2017 17.1 Removed Enable RX-polarity inversion in soft logic parameter. This parameter is not required for Intel® Stratix® 10 devices.
November 2017 17.1

Made the following changes to the user guide:

  • Revised the Testbench and Design Example for the Avalon-MM Interface chapter. Although the functions and tasks that implement the testbench have not changed, the organization of these functions and task in files is entirely different than in earlier device families.
  • Improved descriptions of the DMA registers.
  • Revised Generating the Avalon® -MM Design to generate the example design from the .ip file. This IP core is now available in the Intel® Quartus® Prime Pro Edition IP Catalog.
  • Added definition for rxm_irq_<n>[15:0]. This signal is available for the Avalon® -MM interface when you enable the CRA port.
  • Added bit encoding for the Expansion ROM which is supported in this release.
  • Corrected address range for Lane Equalization Control Register in the Correspondence between Configuration Space Capability Structures and PCIe Base Specification Correspondence between Configuration Space Capability Structures and PCIe* Base Specification Description table. Up to 16 lanes each have a 4-byte register.
  • Corrected the Legacy Interrupt Assertion and Legacy Interrupt Deassertion figures. Intel® Stratix® 10 devices do not support the app_int_ack signal.
  • Updated maximum throughput for L-Tile transceivers from 17.4 Gbps to 26 Gbps.
  • Removed -3 from the recommended speed grades.
  • Added note that you must treat BAR0 as non-prefetchable when you enable the internal Descriptor Controller.
  • Removed description of testin_zero. This signal is not a top-level signal of the IP.

Made the following changes to the Intel® Stratix® 10 hard IP for PCI Express* IP core:

  • This IP core is now available in the Intel® Quartus® Prime Pro Edition IP Catalog.
May 2017 Quartus®Prime Pro v17.1 Stratix 10 ES Editions Software

Made the following changes to the IP core:

  • Added (*.pof) support for up to Gen3 x8 variants with an .ini file.
  • Added support for the H-Tile transceiver.
  • Added support for a Gen3x16 simulation model that you can use in an Avery testbench.

Made the following changes to the user guide:

  • Added descriptions for DMA Descriptor Controller registers.
  • Replaced the Getting Started with the Avalon-MM DMA static design example with the dynamically generated Quick Start Guide design example.
  • Added Performance and Resource Utilization results.
  • Changed Read DMA Example to use larger data block transfers.
  • Added Write DMA Example.
  • Added Testbench and Design Example for the Avalon-MM Interface chapter.
  • Added reference to AN-811: Using the Avery BFM for PCI Express Gen3x16 Simulation on Intel Stratix 10 Devices.
  • Added figures showing the connections between the Avalon-MM DMA bridge and user application and between the PCIe IP core system interfaces and user application.
  • Revised Generation discussion to match the Quartus® Prime Pro – Stratix 10 Edition 17.1 Interim Release design flow.
  • Added definitions for Advance, Preliminary, and Final timing models.
  • Fixed minor errors and typos.
October 2016 Quartus® Prime Pro – Stratix 10 Edition Beta

Initial release
Level Two Title