L- and H-tile Avalon® Streaming and Single Root I/O Virtualization (SR-IOV) Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683111
Date 3/07/2022
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Introduction 2. Quick Start Guide 3. Interface Overview 4. Parameters 5. Designing with the IP Core 6. Block Descriptions 7. Interrupts 8. Registers 9. Testbench and Design Example 10. Troubleshooting and Observing the Link A. PCI Express Core Architecture B. TX Credit Adjustment Sample Code C. Root Port Enumeration D. Document Revision History
1. Introduction x
1.1. Avalon-ST Interface with Optional SR-IOV for PCIe Introduction 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Recommended Fabric Speed Grades 1.6. Performance and Resource Utilization 1.7. Transceiver Tiles 1.8. PCI Express IP Core Package Layout 1.9. Channel Availability
2. Quick Start Guide x
2.1. Design Components 2.2. Hardware and Software Requirements 2.3. Directory Structure 2.4. Generating the Design Example 2.5. Simulating the Design Example 2.6. Compiling the Design Example and Programming the Device 2.7. Installing the Linux Kernel Driver 2.8. Running the Design Example Application
3. Interface Overview x
3.1. Avalon-ST RX Interface 3.2. Avalon-ST TX Interface 3.3. TX Credit Interface 3.4. TX and RX Serial Data 3.5. Clocks 3.6. Function-Level Reset (FLR) Interface 3.7. Control Shadow Interface for SR-IOV 3.8. Configuration Extension Bus Interface 3.9. Hard IP Reconfiguration Interface 3.10. Interrupt Interfaces 3.11. Power Management Interface 3.12. Reset 3.13. Transaction Layer Configuration Interface 3.14. PLL Reconfiguration Interface 3.15. PIPE Interface (Simulation Only)
4. Parameters x
4.1. Stratix 10 Avalon-ST Settings 4.2. Multifunction and SR-IOV System Settings 4.3. Base Address Registers 4.4. Device Identification Registers 4.5. TPH/ATS Capabilities 4.6. PCI Express and PCI Capabilities Parameters 4.7. Configuration, Debug and Extension Options 4.8. PHY Characteristics 4.9. Example Designs
4.6. PCI Express and PCI Capabilities Parameters x
4.6.1. Device Capabilities 4.6.2. Link Capabilities 4.6.3. MSI and MSI-X Capabilities 4.6.4. Slot Capabilities 4.6.5. Power Management 4.6.6. Vendor Specific Extended Capability (VSEC)
5. Designing with the IP Core x
5.1. Generation 5.2. Simulation 5.3. IP Core Generation Output ( Intel® Quartus® Prime Pro Edition) 5.4. Integration and Implementation 5.5. Required Supporting IP Cores 5.6. Channel Layout and PLL Usage
5.2. Simulation x
5.2.1. Selecting Serial or PIPE Simulation
5.4. Integration and Implementation x
5.4.1. Clock Requirements 5.4.2. Reset Requirements
5.5. Required Supporting IP Cores x
5.5.1. Hard Reset Controller 5.5.2. TX PLL
6. Block Descriptions x
6.1. Interfaces 6.2. Errors reported by the Application Layer 6.3. Power Management 6.4. Transaction Ordering 6.5. RX Buffer
6.1. Interfaces x
6.1.1. TLP Header and Data Alignment for the Avalon-ST RX and TX Interfaces 6.1.2. Avalon-ST 256-Bit RX Interface 6.1.3. Avalon-ST 512-Bit RX Interface 6.1.4. Avalon-ST 256-Bit TX Interface 6.1.5. Avalon-ST 512-Bit TX Interface 6.1.6. TX Credit Interface 6.1.7. Interpreting the TX Credit Interface 6.1.8. Clocks 6.1.9. Update Flow Control Timer and Credit Release 6.1.10. Function-Level Reset (FLR) Interface 6.1.11. Resets 6.1.12. Interrupts 6.1.13. Control Shadow Interface for SR-IOV 6.1.14. Transaction Layer Configuration Space Interface 6.1.15. Configuration Extension Bus Interface 6.1.16. Hard IP Status Interface 6.1.17. Hard IP Reconfiguration 6.1.18. Power Management Interface 6.1.19. Serial Data Interface 6.1.20. PIPE Interface 6.1.21. Test Interface 6.1.22. PLL IP Reconfiguration 6.1.23. Message Handling
6.1.2. Avalon-ST 256-Bit RX Interface x
6.1.2.1. Avalon-ST RX Interface Three- and Four-Dword TLPs 6.1.2.2. Avalon-ST RX Interface rx_st_ready Deasserts for the 256-Bit Interface 6.1.2.3. Avalon-ST RX Interface rx_st_valid Deasserts 6.1.2.4. Avalon-ST RX Back-to-Back Transmission 6.1.2.5. Avalon-ST RX Interface Single-Cycle TLPs
6.1.3. Avalon-ST 512-Bit RX Interface x
6.1.3.1. PCIe TLP Layout Using the 512-Bit Interface
6.1.4. Avalon-ST 256-Bit TX Interface x
6.1.4.1. Avalon-ST TX Three- and Four-Dword TLPs 6.1.4.2. Avalon-ST TX Interface tx_st_ready Deassertion 6.1.4.3. Assertion and Deassertion of Avalon-ST TX Interface tx_st_valid
6.1.12. Interrupts x
6.1.12.1. MSI and Legacy Interrupts
6.1.23. Message Handling x
6.1.23.1. Endpoint Received Messages 6.1.23.2. Endpoint Transmitted Messages
6.2. Errors reported by the Application Layer x
6.2.1. Error Handling
6.3. Power Management x
6.3.1. Endpoint D3 Entry 6.3.2. End Point D3 Exit 6.3.3. Exit from D3 hot 6.3.4. Exit from D3 cold 6.3.5. Active State Power Management
6.4. Transaction Ordering x
6.4.1. TX TLP Ordering 6.4.2. RX TLP Ordering
6.5. RX Buffer x
6.5.1. Retry Buffer 6.5.2. Configuration Retry Status
7. Interrupts x
7.1. Interrupts for Endpoints
7.1. Interrupts for Endpoints x
7.1.1. MSI and Legacy Interrupts 7.1.2. MSI-X 7.1.3. Implementing MSI-X Interrupts 7.1.4. Legacy Interrupts
8. Registers x
8.1. Configuration Space Registers
8.1. Configuration Space Registers x
8.1.1. Register Access Definitions 8.1.2. PCI Configuration Header Registers 8.1.3. PCI Express Capability Structures 8.1.4. Intel Defined VSEC Capability Header 8.1.5. General Purpose Control and Status Register 8.1.6. Uncorrectable Internal Error Status Register 8.1.7. Uncorrectable Internal Error Mask Register 8.1.8. Correctable Internal Error Status Register 8.1.9. Correctable Internal Error Mask Register 8.1.10. SR-IOV Virtualization Extended Capabilities Registers Address Map
8.1.4. Intel Defined VSEC Capability Header x
8.1.4.1. Intel Defined Vendor Specific Header 8.1.4.2. Intel Marker 8.1.4.3. JTAG Silicon ID 8.1.4.4. User Configurable Device and Board ID
8.1.10. SR-IOV Virtualization Extended Capabilities Registers Address Map x
8.1.10.1. ARI Enhanced Capability Header 8.1.10.2. SR-IOV Enhanced Capability Registers 8.1.10.3. Initial VFs and Total VFs Registers 8.1.10.4. VF Device ID Register 8.1.10.5. Page Size Registers 8.1.10.6. VF Base Address Registers (BARs) 0-5 8.1.10.7. Secondary PCI Express Extended Capability Header 8.1.10.8. Lane Status Registers 8.1.10.9. Transaction Processing Hints (TPH) Requester Enhanced Capability Header 8.1.10.10. TPH Requester Capability Register 8.1.10.11. TPH Requester Control Register 8.1.10.12. Address Translation Services ATS Enhanced Capability Header 8.1.10.13. ATS Capability Register and ATS Control Register
9. Testbench and Design Example x
9.1. Endpoint Testbench 9.2. Test Driver Module 9.3. Root Port BFM Overview 9.4. BFM Procedures and Functions
9.1. Endpoint Testbench x
9.1.1. Endpoint Testbench for SR-IOV
9.3. Root Port BFM Overview x
9.3.1. BFM Memory Map 9.3.2. Configuration Space Bus and Device Numbering 9.3.3. Configuration of Root Port and Endpoint 9.3.4. Issuing Read and Write Transactions to the Application Layer
9.4. BFM Procedures and Functions x
9.4.1. ebfm_barwr Procedure 9.4.2. ebfm_barwr_imm Procedure 9.4.3. ebfm_barrd_wait Procedure 9.4.4. ebfm_barrd_nowt Procedure 9.4.5. ebfm_cfgwr_imm_wait Procedure 9.4.6. ebfm_cfgwr_imm_nowt Procedure 9.4.7. ebfm_cfgrd_wait Procedure 9.4.8. ebfm_cfgrd_nowt Procedure 9.4.9. BFM Configuration Procedures 9.4.10. BFM Shared Memory Access Procedures 9.4.11. BFM Log and Message Procedures 9.4.12. Verilog HDL Formatting Functions
9.4.9. BFM Configuration Procedures x
9.4.9.1. ebfm_cfg_rp_ep Procedure 9.4.9.2. ebfm_cfg_decode_bar Procedure
9.4.10. BFM Shared Memory Access Procedures x
9.4.10.1. Shared Memory Constants 9.4.10.2. shmem_write Task 9.4.10.3. shmem_read Function 9.4.10.4. shmem_display Verilog HDL Function 9.4.10.5. shmem_fill Procedure 9.4.10.6. shmem_chk_ok Function
9.4.11. BFM Log and Message Procedures x
9.4.11.1. ebfm_display Verilog HDL Function 9.4.11.2. ebfm_log_stop_sim Verilog HDL Function 9.4.11.3. ebfm_log_set_suppressed_msg_mask Task 9.4.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task 9.4.11.5. ebfm_log_open Verilog HDL Function 9.4.11.6. ebfm_log_close Verilog HDL Function
9.4.12. Verilog HDL Formatting Functions x
9.4.12.1. himage1 9.4.12.2. himage2 9.4.12.3. himage4 9.4.12.4. himage8 9.4.12.5. himage16 9.4.12.6. dimage1 9.4.12.7. dimage2 9.4.12.8. dimage3 9.4.12.9. dimage4 9.4.12.10. dimage5 9.4.12.11. dimage6 9.4.12.12. dimage7
10. Troubleshooting and Observing the Link x
10.1. Troubleshooting 10.2. PCIe Link Inspector Overview
10.1. Troubleshooting x
10.1.1. Simulation Fails To Progress Beyond Polling.Active State 10.1.2. Hardware Bring-Up Issues 10.1.3. Link Training 10.1.4. Link Hangs in L0 State 10.1.5. Use Third-Party PCIe Analyzer
10.2. PCIe Link Inspector Overview x
10.2.1. PCIe* Link Inspector Hardware
10.2.1. PCIe* Link Inspector Hardware x
10.2.1.1. Enabling the PCIe* Link Inspector 10.2.1.2. Launching the PCIe* Link Inspector 10.2.1.3. The PCIe* Link Inspector LTSSM Monitor 10.2.1.4. Accessing the Configuration Space and Transceiver Registers 10.2.1.5. Additional Status Commands
10.2.1.3. The PCIe* Link Inspector LTSSM Monitor x
10.2.1.3.1. LTSSM Monitor Registers 10.2.1.3.2. ltssm_save2file <file_name> 10.2.1.3.3. ltssm_file2console 10.2.1.3.4. ltssm_debug_check 10.2.1.3.5. ltssm_display <num_states> 10.2.1.3.6. ltssm_save_oldstates
10.2.1.4. Accessing the Configuration Space and Transceiver Registers x
10.2.1.4.1. PCIe* Link Inspector Commands 10.2.1.4.2. NPDME PLL and Channel Commands 10.2.1.4.3. Register Address Map
10.2.1.5. Additional Status Commands x
10.2.1.5.1. Displaying PLL Lock and Calibration Status Registers
A. PCI Express Core Architecture x
A.1. Transaction Layer A.2. Data Link Layer A.3. Physical Layer
D. Document Revision History x
D.1. Document Revision History for the Intel® L- and H-tile Avalon® Streaming and Single-Root I/O Virtualization (SR-IOV) IP for PCI Express* User Guide
1. Introduction
2. Quick Start Guide
3. Interface Overview
4. Parameters
5. Designing with the IP Core
6. Block Descriptions
7. Interrupts
8. Registers
9. Testbench and Design Example
10. Troubleshooting and Observing the Link
A. PCI Express Core Architecture
B. TX Credit Adjustment Sample Code
C. Root Port Enumeration
D. Document Revision History

D.1. Document Revision History for the Intel® L- and H-tile Avalon® Streaming and Single-Root I/O Virtualization (SR-IOV) IP for PCI Express* User Guide

Document Version Intel® Quartus® Prime Version Changes
2022.09.26 21.1 Updated the MSI and MSI-X Capabilities section to include a clarification on how the Table Offset and PBA Offset parameters are used in conjunction with the Table BAR Indicator and Pending BAR Indicator parameters to form the corresponding addresses.
2022.03.07 21.1 Added the steps to enable 64-bit addressing support for MSI messages to the MSI and Legacy Interrupts.
2021.10.19 21.1 Changed the device support level for Intel® Stratix® 10 to Final Support in the Device Family Support section.
2021.09.17 21.1

Added a statement to the MSI and Legacy Interrupts section clarifying that the user application logic is responsible for gating the MSI TLP based on the states of the MSI Enable or Bus Master Enable bits in the event of an MSI request.

Updated the description of the apps_ready_entr_l23 signal in the Power Management Interface section.

2021.07.27 21.1 Updated the Function-Level Reset (FLR) Interface section to change the time limits for flr_pf_done and flr_completed_vf signals from 100 microseconds to 100 milliseconds.
2021.05.27 21.1

Added the Appendix chapter on Root Port enumeration.

Added a note to the Features section stating that L- and H-tile Avalon® Streaming IP for PCI Express only supports the Separate Reference Clock With No Spread Spectrum architecture (SRNS), but not the Separate Reference Clock With Independent Spread Spectrum architecture (SRIS).

2021.01.27 20.3 Removed the statement that only BAR4 can access the MSI-X table for multi-function variants from the Implementing MSI-X Interrupts section.
2020.10.23 20.3 Changed the byte parity from odd to even in the Stratix 10 Avalon® -ST Settings and Avalon® -ST 256-Bit TX Interface sections.
2020.10.05 20.3

Updated the IP name to Intel L-/H-tile Avalon Streaming and Single-Root I/O Virtualization (SR-IOV) IP for PCI Express.

2020.06.03 20.1 Added the description for the new input ninit_done to the Resets sections of Chapters 3 and 6. Also added a link to AN 891: Using the Reset Release Intel FPGA IP, which describes the Reset Release IP that is used to drive the ninit_done input.
2020.05.11 20.1

Added a clarification in the PCIe Link Inspector Hardware section on which clocks and resets need to be connected for the PCIe Link Inspector to work.

Remove a statement about using the Transceiver Toolkit from the Launching the PCIe Link Inspector section because the Transceiver Toolkit is not supported in this release of Intel® Quartus® Prime.

Changed the Altera Debug Master Endpoint (ADME) module name to Native PHY Debug Master Endpoint (NPDME).

2020.04.22 19.3

Updated the document title to Intel® Stratix® 10 Avalon® streaming and Single-Root I/O Virtualization (SR-IOV) Interfaces for PCI Express* Solutions User Guide to meet new legal naming guidelines.

Added a note stating that ctl_shdw_* outputs are valid when the signal ctl_shdw_update is asserted.

2020.03.24 19.3 Added notes stating that the Hard IP Reconfiguration interface is not accessible if the PCIe Link Inspector is enabled to the Hard IP Reconfiguration Interface sections in Interface Overview and Block Descriptions.
2020.02.10 19.3

Added a note stating that Root Port mode is not supported if SR-IOV is enabled in the Features section.

Added a note stating that a Root Port design example is not available for the Avalon® Streaming (Avalon-ST) IP for PCIe.

Updated the description and timing diagram of the app_msi_req signal.

2019.11.05 19.3

Added a note stating that Root Port Configuration requests are supported via the Hard IP Reconfiguration interface and not the Avalon-ST TX interface in the Avalon-ST TX Interface section.

Changed the BAR size range to 256 bytes - 8 Ebytes in the Base Address Registers section.

2019.09.30 19.3

Added a note to clarify that this User Guide is applicable to H-Tile and L-Tile variants of the Intel® Stratix® 10 devices only.

Added a note to clarify that the Control Shadow Interface is only used to access VF registers and not PF registers.

Removed the signal xcvr_reconfig_readdatavalid from the Hard IP Reconfiguration Signals table.

Added Autonomous Hard IP mode to the Features section.

Added a note about memory accesses to a disabled Expansion ROM BAR of PF2 or PF3 to the Base Address Registers section.

2019.09.20 19.1 Updated the BAR Size in the Parameters chapter to show the correct supported range of 128 Bytes to 8 EBytes.
2019.07.18 19.1 Added the statement that refclk must be stable and free-running at device power-up for a successful configuration.
2019.03.30 19.1

Removed the BIOS Enumeration section from the Troubleshooting chapter.

Removed the note stating that Root Port mode is not recommended.

2019.03.12 18.1.1 Changed the E-Tile PAM-4 frequency from 56G to 57.8G, and NRZ frequency from 30G to 28.9G.
2018.12.27 18.1.1 Added note stating that users need to implement completion timeout checking functionality in their application logic.
2018.12.24 18.1.1

Added the description for the Link Inspector Avalon® -MM Interface.

2018.10.31 18.1 Changed the infinite header credit net value (tx_cplh_cdts) for H-tile and infinite data credit net value (tx_cpld_cdts) for L-tile from 0 to 0xFF.
2018.10.26 18.1 Added the statements that the IP core does not support the L1/L2 low-power states, the in-band beacon and sideband WAKE# signal.
2018.09.24 18.1

Added the ltssm_file2console and ltssm_save_oldstates commands for the PCIe* Link Inspector.

Updated the steps to run ModelSim simulations for a design example.

Updated the steps to run a design example.

Changed the flr_pf_active_o signal name back to flr_pf_active, and flr_pf_done_i back to flr_pf_done.

2018.09.04 18.0 Changed the flr_pf_active signal name to flr_pf_active_o, and flr_pf_done to flr_pf_done_i.
2018.08.29 18.0 Added the step to invoke vsim to the instructions for running a ModelSim simulation.
Date Version Changes
May 2018 18.0

Made the following changes to the user guide:

  • Updated Section 1.2: Features to state that AER is always enabled for Stratix 10 devices.
  • Updated Section 2.3: Generating the Design Example to describe the recommended_pinassignments_s10.txt file and its function, and to update some steps in the generation flow.
  • Changed the latency of tx_st_ready from 14 to 3 cycles in Section 3.2: Avalon-ST TX Interface. Also added a timing diagram showing the proper assertion and deassertion of tx_st_valid.
  • Added a note to Section 4.2: Multifunction and SR-IOV System Settings to state that different functions cannot use the same tag for their memory read requests.
  • Updated Section 6.1.1: TLP Header and Data Alignment for the Avalon-ST RX and TX Interfaces to show the correct byte ordering for the Header dwords.
  • Changed the rx_st_ready_i to rx_st_valid_o[1:0] latency from 6 to 14 cycles in Section 6.1.3: Avalon-ST 512-Bit RX Interface.
  • Updated Section 6.1.3: Avalon-ST 512-bit RX Interface and Section 6.1.5: Avalon-ST 512-bit TX Interface to note that these interfaces are not strictly Avalon-compliant because they support 2 SOPs and 2 EOPs.
  • Added the requirement for a free-running refclk to meet the PCIe 100 ms wake-up time specification to Section 6.1.8: Clocks.
  • Updated Section 6.1.15: Configuration Extension Bus Interface to state that this interface is not available when SR-IOV is enabled.
December 2017 17.1

Made the following changes to the user guide:

  • Added H-Tile Multiplexed Configuration Register Information Available on tl_cfg_ctl table.
  • Corrected command sequences for the PCIe* Link Inspector in the Launching the Link Inspector and Displaying PLL Lock and Calibration Status Registers topics. All the .tcl scripts are in the TCL directory which must be included in the source command.
November 2017 17.1 Removed Enable RX-polarity inversion in soft logic parameter. This parameter is not required for Intel® Stratix® 10 devices.
November 2017 17.1

Added descriptions for the following new features of the Intel® Stratix® 10 Avalon® -ST and SR-IOV Interfaces for PCI Express* IP core :

  • SR-IOV support, including the following:
    • SR-IOV Virtualization Extended Capabilities Registers
    • Virtual Function Registers
    • Control Shadow Interface for SR-IOV
    • Intel® Stratix® 10 SR-IOV System Settings Parameters
  • Function-Level Reset (FLR) support
  • Configuration Status Interface
  • Interrupt signals to support multiple functions
  • Physical Function TLP Processing Hints (TPH) support
  • Address Translation Services (ATS) support
  • A 512-bit interface to the Application Layer for Gen3 x16 configurations.
  • PCIe* Link Inspector support too monitor the PCIe* link at the Physical, Data Link and Transaction Layers.
  • The following topics to describe the 512-bit interface:
    • Avalon® -ST 512-Bit RX Interface
    • Avalon® -ST 512-Bit TX Interface
    • Transmitting PCIe TLPs Using the 512-Bit Interface
  • Bit encoding for rx_st_bar_range for the Expansion ROM which is supported in this release.
  • Compilation support for dynamically-generated design example.
  • Linux driver to run the dynamically-generated design example.

Made the following changes to the user guide:

  • Revised Generating the Avalon® -ST Design to generate the example design from the .ip file.
  • Removed the tx_st_empty signal from Avalon® -ST TX Interface Cycle Definition for Three-Dword Header TLPs and Avalon® -ST TX Interface Cycle Definition for Four-Dword Header TLPs. The tx_st_empty signal is not available for the 256-bit interface.
  • Removed Chaining DMA Design Examples from the Testbench and Design Example chapter. This design example is not supported for Intel® Stratix® 10 devices.
  • Updated for latest Intel branding conventions.
  • Revised Avalon-ST Stratix 10 Hard IP for PCI Express Top-Level Signals to show signals that are only available in for L-Tile or H-Tile devices.
  • Added fields to the Multiplexed Configuration Register Information Available on tl_cfg_ctl for L-Tile devices.
  • Added separate Multiplexed Configuration Register Information Available on tl_cfg_ctl table for H-Tile devices.
  • Removed description of testin_zero. This signal is not a top-level signal of the IP.
  • Rebranded as Intel
  • Corrected minor errors and typos.
May 2017 Quartus Prime Pro v17.1 Stratix 10 ES Editions Software

Made the following changes to the IP core:

  • Added support for the H-Tile transceiver including example designs available in the installation directory.
  • Added support for a Gen3x16 simulation testbench model that you can use in an Avery testbench.

Made the following changes to the user guide:

  • Added definitions for Advance, Preliminary, and Final timing models.
  • Added Testbench and Design Example for the Avalon-ST Interface chapter.
  • Added figures showing the connections between the Avalon-ST and user application and between Stratix® 10 Hard IP for PCI Express IP Core, system interfaces, and the user application.
  • Revised Generation discussion to match the

    The Quartus Prime Pro v17.1 Stratix 10 ES Editions Software design flow.

  • Changed TX credit interface pseudo code. tx_nph_credit_consume_count_hip_delayed is 3 pld_clk cycle delayed, not 2.
October 2016 Quartus® Prime Pro – Stratix 10 Edition Beta Initial release
Level Two Title