P-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683059
Date 3/28/2022
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. About the P-tile Avalon® Intel® FPGA IPs for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. Testbench 7. Troubleshooting/Debugging 8. Intel® P-tile Avalon® Streaming IP for PCI Express* User Guide Archives 9. Document Revision History for the P-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TLP Bypass Mode E. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations F. Bifurcated Endpoint Support for Independent Warm Resets
1. About the P-tile Avalon® Intel® FPGA IPs for PCI Express x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core and Design Example Support Levels
1.1. Overview x
1.1.1. Standards and Specifications Compliance
2. IP Architecture and Functional Description x
2.1. Architecture 2.2. Functional Description
2.1. Architecture x
2.1.1. Clock Domains 2.1.2. Refclk 2.1.3. Reset
2.2. Functional Description x
2.2.1. PMA/PCS 2.2.2. Data Link Layer Overview 2.2.3. Transaction Layer Overview
3. Advanced Features x
3.1. PCIe* Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TLP Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035 ) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TLP Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TLP Bypass Mode 3.3.3. User Avalon® -MM Interface 3.3.4. Avalon® -ST Interface
3.3.2. Register Settings for the TLP Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x104194) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x104190)
3.3.4. Avalon® -ST Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Overview 4.2. Clocks and Resets 4.3. Serial Data Interface 4.4. Avalon-ST Interface 4.5. Hard IP Status Interface 4.6. Interrupt Interface 4.7. Error Interface 4.8. Hot Plug Interface (RP Only) 4.9. Power Management Interface 4.10. Configuration Output Interface 4.11. Configuration Intercept Interface (EP Only) 4.12. Hard IP Reconfiguration Interface 4.13. PHY Reconfiguration Interface 4.14. Page Request Service (PRS) Interface (EP Only)
4.2. Clocks and Resets x
4.2.1. Interface Clock Signals 4.2.2. Resets
4.2.2. Resets x
4.2.2.1. Interface Reset Signals 4.2.2.2. Function-Level Reset (FLR) Interface (EP Only)
4.4. Avalon-ST Interface x
4.4.1. TLP Header and Data Alignment for the Avalon-ST RX and TX Interfaces 4.4.2. Avalon® -ST RX Interface 4.4.3. Avalon® -ST RX Interface rx_st_ready Behavior 4.4.4. RX Flow Control Interface 4.4.5. Avalon® -ST TX Interface 4.4.6. Avalon® -ST TX Interface tx_st_ready Behavior 4.4.7. TX Flow Control Interface 4.4.8. Tag Allocation
4.4.8. Tag Allocation x
4.4.8.1. Completion Buffer Size
4.6. Interrupt Interface x
4.6.1. Legacy Interrupts 4.6.2. MSI 4.6.3. MSI-X
4.6.3. MSI-X x
4.6.3.1. Implementing MSI-X Interrupts
4.7. Error Interface x
4.7.1. Completion Timeout Interface 4.7.2. VF Error Flags Interface
4.12. Hard IP Reconfiguration Interface x
4.12.1. Address Map for the User Avalon-MM Interface 4.12.2. Configuration Registers Access
4.12.1. Address Map for the User Avalon-MM Interface x
4.12.1.1. User Avalon-MM Control Register (Offset 0x104068)
4.12.2. Configuration Registers Access x
4.12.2.1. Using Direct User Avalon-MM Interface (Byte Access) 4.12.2.2. Using the Debug Register Interface Access (Dword Access)
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters
5.2. Core Parameters x
5.2.1. Avalon® Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters 5.2.4. Device Identification Registers 5.2.5. Configuration, Debug and Extension Options
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) Capabilities 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities
5.2.3.1. Device Capabilities x
5.2.3.1.1. Multifunction and SR-IOV System Parameters
6. Testbench x
6.1. Endpoint Testbench 6.2. Test Driver Module 6.3. Root Port BFM
6.3. Root Port BFM x
6.3.1. BFM Memory Map 6.3.2. Configuration Space Bus and Device Numbering 6.3.3. Configuration of Root Port and Endpoint 6.3.4. Issuing Read and Write Transactions to the Application Layer 6.3.5. BFM Procedures and Functions
6.3.5. BFM Procedures and Functions x
6.3.5.1. ebfm_barwr Procedure 6.3.5.2. ebfm_barwr_imm Procedure 6.3.5.3. ebfm_barrd_wait Procedure 6.3.5.4. ebfm_barrd_nowt Procedure 6.3.5.5. ebfm_cfgwr_imm_wait Procedure 6.3.5.6. ebfm_cfgwr_imm_nowt Procedure 6.3.5.7. ebfm_cfgrd_wait Procedure 6.3.5.8. ebfm_cfgrd_nowt Procedure 6.3.5.9. BFM Configuration Procedures 6.3.5.10. BFM Shared Memory Access Procedures 6.3.5.11. BFM Log and Message Procedures 6.3.5.12. Verilog HDL Formatting Functions
6.3.5.9. BFM Configuration Procedures x
6.3.5.9.1. ebfm_cfg_rp_ep Procedure 6.3.5.9.2. ebfm_cfg_decode_bar Procedure
6.3.5.10. BFM Shared Memory Access Procedures x
6.3.5.10.1. Shared Memory Constants 6.3.5.10.2. shmem_write Task 6.3.5.10.3. shmem_read Function 6.3.5.10.4. shmem_display Verilog HDL Function 6.3.5.10.5. shmem_fill Procedure 6.3.5.10.6. shmem_chk_ok Function
6.3.5.11. BFM Log and Message Procedures x
6.3.5.11.1. ebfm_display Verilog HDL Function 6.3.5.11.2. ebfm_log_stop_sim Verilog HDL Function 6.3.5.11.3. ebfm_log_set_suppressed_msg_mask Task 6.3.5.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task 6.3.5.11.5. ebfm_log_open Verilog HDL Function 6.3.5.11.6. ebfm_log_close Verilog HDL Function
6.3.5.12. Verilog HDL Formatting Functions x
6.3.5.12.1. himage1 6.3.5.12.2. himage2 6.3.5.12.3. himage4 6.3.5.12.4. himage8 6.3.5.12.5. himage16 6.3.5.12.6. dimage1 6.3.5.12.7. dimage2 6.3.5.12.8. dimage3 6.3.5.12.9. dimage4 6.3.5.12.10. dimage5 6.3.5.12.11. dimage6 6.3.5.12.12. dimage7
7. Troubleshooting/Debugging x
7.1. Hardware 7.2. Debug Toolkit
7.1. Hardware x
7.1.1. Debugging Link Training Issues 7.1.2. Debugging Data Transfer and Performance Issues
7.1.1. Debugging Link Training Issues x
7.1.1.1. Generic Tools and Utilities 7.1.1.2. Signal Tap II Logic Analyzer 7.1.1.3. Additional Debug Tools
7.1.2. Debugging Data Transfer and Performance Issues x
7.1.2.1. Advanced Error Reporting (AER) 7.1.2.2. Second-Level Debug Tools
7.2. Debug Toolkit x
7.2.1. Overview 7.2.2. Enabling the P-Tile Debug Toolkit 7.2.3. Launching the P-Tile Debug Toolkit 7.2.4. Using the P-Tile Debug Toolkit 7.2.5. Using the P-Tile Link Inspector
A. Configuration Space Registers x
A.1. Configuration Space Registers A.2. Configuration Space Registers for Virtualization A.3. Intel-Defined VSEC Capability Registers
A.1. Configuration Space Registers x
A.1.1. Register Access Definitions A.1.2. PCIe Configuration Header Registers A.1.3. PCI Express Capability Structures A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure A.1.5. MSI-X Registers
A.2. Configuration Space Registers for Virtualization x
A.2.1. SR-IOV Virtualization Extended Capabilities Registers Address Map A.2.2. PCIe Configuration Registers for Each Virtual Function
A.2.2. PCIe Configuration Registers for Each Virtual Function x
A.2.2.1. Alternative Routing ID (ARI) Capability Structure A.2.2.2. TLP Processing Hint (TPH) Capability Structure A.2.2.3. Address Translation Services (ATS) Capability Structure A.2.2.4. Access Control Services (ACS) Capability Structure
A.2.2.1. Alternative Routing ID (ARI) Capability Structure x
A.2.2.1.1. ARI Enhanced Capability Header Register (Offset 0x0) A.2.2.1.2. ARI Capability and Control Register (Offset 0x4)
A.2.2.2. TLP Processing Hint (TPH) Capability Structure x
A.2.2.2.1. TPH Requester Enhanced Capability Header (Offset 0x0) A.2.2.2.2. TPH Requester Capability Register (Offset 0x4) A.2.2.2.3. TPH Requester Control Register (Offset 0x8)
A.2.2.3. Address Translation Services (ATS) Capability Structure x
A.2.2.3.1. ATS Enhanced Capability Header (Offset 0x0) A.2.2.3.2. ATS Capability Register and ATS Control Register (Offset 0x4)
A.2.2.4. Access Control Services (ACS) Capability Structure x
A.2.2.4.1. ACS Extended Capability Header (Offset 0x0) A.2.2.4.2. ACS Capability Register (Offset 0x4) A.2.2.4.3. ACS Control Register (Offset 0x6) A.2.2.4.4. Egress Control Vector (Offset 0x8)
A.3. Intel-Defined VSEC Capability Registers x
A.3.1. Intel-Defined VSEC Capability Header (Offset 00h) A.3.2. Intel-Defined Vendor Specific Header (Offset 04h) A.3.3. Intel Marker (Offset 08h) A.3.4. JTAG Silicon ID (Offset 0x0C - 0x18) A.3.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D) A.3.6. General Purpose Control and Status Register (Offset 0x30) A.3.7. Uncorrectable Internal Error Status Register (Offset 0x34) A.3.8. Uncorrectable Internal Error Mask Register (Offset 0x38) A.3.9. Correctable Internal Error Status Register (Offset 0x3C) A.3.10. Correctable Internal Error Mask Register (Offset 0x40)
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TLP Bypass Mode x
D.1. EP TLP Bypass Mode (Upstream) D.2. RC TLP Bypass Mode (Downstream)
E. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations x
E.1. Overview E.2. Generate the PCIe PIO Example Design E.3. Integrate Avery BFMs (Root Complex) E.4. Configure the Avery BFM and Update the Simulation Script E.5. Compile and Simulate E.6. View the Results
E.2. Generate the PCIe PIO Example Design x
E.2.1. Avalon® -ST PCIe PIO Example Design
F. Bifurcated Endpoint Support for Independent Warm Resets x
F.1. PCI Express Resets F.2. P-tile PCIe Design Constraints
F.2. P-tile PCIe Design Constraints x
F.2.1. P-tile Dual-Endpoint System Configurations
1. About the P-tile Avalon® Intel® FPGA IPs for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. Testbench
7. Troubleshooting/Debugging
8. Intel® P-tile Avalon® Streaming IP for PCI Express* User Guide Archives
9. Document Revision History for the P-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TLP Bypass Mode
E. Using the Avery BFM for Intel P-Tile PCI Express Gen4 x16 Simulations
F. Bifurcated Endpoint Support for Independent Warm Resets

4.1. Overview

You can determine which core each interface in this section belongs to by looking at the prefixes in the signal names:
  • p0 : x16 core
  • p1 : x8 core
  • p2 : x4_0 core
  • p3 : x4_1 core

P-Tile Avalon® -ST IP for PCI Express Top-Level Signals shows the top-level signals of this IP. Note that the signal names in the figure will get the appropriate prefix pn (where n = 0, 1, 2 or 3) depending on which of the three supported configurations (1x16, 2x8, or 4x4) the P-Tile Avalon® -ST IP for PCI Express* is in.

As an example, the rx_st_data_o bus can take on the following names:
  • In the 1x16 configuration, only the x16 core is active. In this case, this bus appears as p0_rx_st_data_o[511:0].
  • In the 2x8 configuration, both the x16 core and x8 core are active. In this case, this bus is split into p0_rx_st_data_o[255:0] and p1_rx_st_data_o[255:0].
  • In the 4x4 configuration, all four cores are active. In this case, this bus is split into p0_rx_st_data_o[127:0], p1_rx_st_data_o[127:0], p2_rx_st_data_o[127:0] and p3_rx_st_data_o[127:0].

The only cases where the interface signal names do not get the pn prefixes are the interfaces that are common for all the cores, like the PHY reconfiguration interface, clocks and resets. For example, there is only one xcvr_reconfig_clk that is shared by all the cores.

You can enable the PHY reconfiguration interface from the Top Level Settings in the GUI.

Each of the cores has its own Avalon® -ST interface to the user logic. The number of IP-to-User Logic interfaces exposed to the FPGA fabric are different based on the configuration modes:

Table 44.  IP to FPGA Fabric Interfaces Summary
Mode Avalon-ST Interface Count Data Width (each Interface) Header Width (each Interface) TLP Prefix Width (each Interface) Application Clock Frequency
Gen4 x16 EP/RP mode 1 512-bit 256-bit 64-bit

175 MHz / 200 MHz / 225 MHz / 350 MHz / 400 MHz / 450 MHz ( Intel® Stratix® 10 DX)

175 MHz / 200 MHz / 225 MHz / 250 MHz / 350 MHz / 400 MHz / 450 MHz / 500 MHz ( Intel® Agilex™ )

3
Gen3 x16 EP/RP mode 1 512-bit 256-bit 64-bit 250 MHz
Gen3 x16 EP/RP mode 1 256-bit 128-bit 32-bit 250 MHz 4
Gen4 x8 x8 EP mode 2 256-bit 128-bit 32-bit

175 MHz / 200 MHz / 225 MHz / 350 MHz / 400 MHz / 450 MHz ( Intel® Stratix® 10 DX)

175 MHz / 200 MHz / 225 MHz / 250 MHz / 350 MHz / 400 MHz / 450 MHz / 500 MHz ( Intel® Agilex™ )

3
Gen4 x8 x8 EP mode 2 512-bit 256-bit 64-bit

175 MHz / 200 MHz / 225 MHz ( Intel® Stratix® 10 DX)

175 MHz / 200 MHz / 225 MHz / 250 MHz ( Intel® Agilex™ )

3
Gen3 x8 x8 EP mode 2 256-bit 128-bit 32-bit 250 MHz
Gen4 x4 x4 x4 x4 RP mode 4 128-bit 128-bit 32-bit

350 MHz / 400 MHz / 450 MHz ( Intel® Stratix® 10 DX)

350 MHz / 400 MHz / 450 MHz / 500 MHz ( Intel® Agilex™ )

3
Gen3 x4 x4 x4 x4 RP mode 4 128-bit 128-bit 32-bit 250 MHz
Figure 14. P-Tile Avalon® -ST IP for PCI Express* Top-Level Signals
Note: The table below shows the variables that are used to define the bus indices for top-level signal busses shown in the top-level block diagram above. The values of these variables change depending on which configuration is active (1x16, 2x8 or 4x4). For example, for the 4x4 configuration, using w=1 and n=1 will give Avalon® -ST RX bus widths of p0_rx_st_data_o[127:0], p0_rx_st_hdr_o[127:0] and p0_rx_st_tlp_prfx_o[31:0].
Table 45.  Variables Used in the Bus Indices
Variable 1x16 Configuration 2x8 Configuration 4x4 Configuration
w 4 2 1
n 2 1 1
p 6 3 2
b 16 8 4
3 Select the highest clock frequency to achieve maximum PCIe Gen4 bandwidth.
4 In this configuration, interface efficiency is traded off for lower interface width.
Level Two Title