R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 7/08/2024
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. Troubleshooting/Debugging 7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 8. Document Revision History for the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TL Bypass Mode E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express x
1.1. Glossary 1.2. Overview 1.3. Features 1.4. Release Information 1.5. Device Family Support 1.6. Performance and Resource Utilization 1.7. IP Core Support Levels
1.2. Overview x
1.2.1. Standards and Specifications Compliance
1.3. Features x
1.3.1. Multifunction and Virtualization Features
2. IP Architecture and Functional Description x
2.1. Overview 2.2. PCIe Hard IP Mode 2.3. PIPE Direct Mode
2.2. PCIe Hard IP Mode x
2.2.1. Clocking 2.2.2. Reset 2.2.3. PCI Express Protocol Stack
2.2.2. Reset x
2.2.2.1. Single PERST 2.2.2.2. Independent PERST Pins 2.2.2.3. Independent GPIO PERST
2.2.2.2. Independent PERST Pins x
2.2.2.2.1. REFCLK and PERST Guidelines When Using Independent PERST Pins
2.2.2.3. Independent GPIO PERST x
2.2.2.3.1. REFCLK and PERST Guidelines when Using Independent GPIO PERST
2.2.3. PCI Express Protocol Stack x
2.2.3.1. PMA 2.2.3.2. Data Link Layer Overview 2.2.3.3. Transaction Layer Overview
2.2.3.3. Transaction Layer Overview x
2.2.3.3.1. Deferrable Memory Write (DMWr)
2.3. PIPE Direct Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PIPE Layer
2.3.3. PIPE Layer x
2.3.3.1. Preset Mappings
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TL Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TL Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TL Bypass Mode 3.3.3. Hard IP Reconfiguration Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TL Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Clocks and Resets 4.2. Serial Data Interface 4.3. PCI Express Mode 4.4. PIPE Direct Mode
4.1. Clocks and Resets x
4.1.1. Clocks 4.1.2. Resets
4.3. PCI Express Mode x
4.3.1. Avalon® Streaming Interface 4.3.2. Precision Time Measurement (PTM) Interface (Endpoint Only) 4.3.3. Hot Plug Interface 4.3.4. Interrupt Interface 4.3.5. Hard IP Reconfiguration Interface 4.3.6. Error Interface 4.3.7. Completion Timeout Interface 4.3.8. Configuration Intercept Interface 4.3.9. Power Management Interface 4.3.10. Hard IP Status Interface 4.3.11. Page Request Services (PRS) Interface (Endpoint Only) 4.3.12. Function-Level Reset (FLR) Interface (Endpoint Only) 4.3.13. SR-IOV VF Error Flag Interface (Endpoint Only) 4.3.14. General Purpose VSEC Interface
4.3.1. Avalon® Streaming Interface x
4.3.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.3.1.2. Credit Control 4.3.1.3. Avalon® Streaming RX Interface 4.3.1.4. Avalon® Streaming TX Interface 4.3.1.5. Tag Allocation
4.3.1.2. Credit Control x
4.3.1.2.1. Credit Interface - Data and Header 4.3.1.2.2. Credit Initialization 4.3.1.2.3. Credit Update/Release 4.3.1.2.4. RX Flow Control Interface 4.3.1.2.5. TX Flow Control Interface
4.3.1.3. Avalon® Streaming RX Interface x
4.3.1.3.1. Application Logic Guidelines for the Avalon Streaming RX Interface
4.3.1.4. Avalon® Streaming TX Interface x
4.3.1.4.1. Application Logic Guidelines for the Avalon Streaming TX Interface 4.3.1.4.2. Avalon® Streaming TX Interface pX_tx_st_ready_o Behavior
4.3.1.5. Tag Allocation x
4.3.1.5.1. Completion Buffer Size
4.3.4. Interrupt Interface x
4.3.4.1. Legacy Interrupts 4.3.4.2. MSI 4.3.4.3. MSI-X
4.3.4.3. MSI-X x
4.3.4.3.1. Implementing MSI-X Interrupts
4.3.9. Power Management Interface x
4.3.9.1. D3Hot Entry 4.3.9.2. D3Hot Exit 4.3.9.3. D3Cold Entry 4.3.9.4. D3Cold Exit
4.3.9.2. D3Hot Exit x
4.3.9.2.1. D3Hot Exit Initiated by Host 4.3.9.2.2. D3Hot Exit Initiated by EP
4.4. PIPE Direct Mode x
4.4.1. Data Signals 4.4.2. Command and Status Signals 4.4.3. Message Bus Signals 4.4.4. Deskew Signals 4.4.5. PIPE Direct Reset Sequence 4.4.6. PIPE Direct Speed Change 4.4.7. PIPE Lane Reset Staggering 4.4.8. Unused Lanes in PIPE Direct Mode
4.4.1. Data Signals x
4.4.1.1. Transmit Signals 4.4.1.2. Receive Signals
4.4.4. Deskew Signals x
4.4.4.1. MAC to PHY (M2P) Signals 4.4.4.2. PHY to MAC (P2M) Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters 5.3. Design Examples
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Measurement (PTM)
5.3. Design Examples x
5.3.1. Design Example Files 5.3.2. Generated HDL Format 5.3.3. Target Development Kit 5.3.4. Currently Selected Design Example
6. Troubleshooting/Debugging x
6.1. Hardware Debug 6.2. Operating System Tools and Utilities 6.3. Signal Tap Logic Analyzer 6.4. Using the Hard IP Reconfiguration Interface for Debugging 6.5. Other Interfaces That Can Be Used for Debug Activities 6.6. Debug Toolkit
6.1. Hardware Debug x
6.1.1. Debugging Link Training Issues 6.1.2. Debugging Device Enumeration Issues 6.1.3. Debugging Data Transfer and Performance Issues
6.1.3. Debugging Data Transfer and Performance Issues x
6.1.3.1. Advanced Error Reporting (AER)
6.2. Operating System Tools and Utilities x
6.2.1. Using lspci Utility to Read Negotiated Link Speed
6.4. Using the Hard IP Reconfiguration Interface for Debugging x
6.4.1. Using the Hard IP Reconfiguration Interface to Enable and Read ECRC and LCRC Error Counts
6.6. Debug Toolkit x
6.6.1. Overview 6.6.2. Enabling the R-Tile Debug Toolkit 6.6.3. Launching the R-Tile Debug Toolkit 6.6.4. Using the R-Tile Debug Toolkit
6.6.4. Using the R-Tile Debug Toolkit x
6.6.4.1. R-Tile Information 6.6.4.2. Event Counters 6.6.4.3. Link Inspector 6.6.4.4. Configuration Space 6.6.4.5. Channel Parameters
6.6.4.5. Channel Parameters x
6.6.4.5.1. General PHY 6.6.4.5.2. Tx Path 6.6.4.5.3. Rx Path 6.6.4.5.4. Lane Margining
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TL Bypass Mode x
D.1. Upstream TL Bypass Mode D.2. Downstream TL Bypass Mode
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express x
E.1. Margin Masks Overview E.2. 5x5 Testing Efficiency E.3. Debug Toolkit Margin Methodology
E.2. 5x5 Testing Efficiency x
E.2.1. First Time Testing E.2.2. Spot Check Testing
E.3. Debug Toolkit Margin Methodology x
E.3.1. Debug Toolkit Testing Process
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. Troubleshooting/Debugging
7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
8. Document Revision History for the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TL Bypass Mode
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express

2.2.1. Clocking

In the PCIe Hard IP mode, the R-Tile Avalon Streaming Intel FPGA IP for PCI Express* has four primary clock domains:
  • PHY clock domain (i.e. core_clk domain): this clock is synchronous to the SerDes parallel clock.
  • EMIB/FPGA fabric interface clock domain (i.e. pld_clk domain): this clock is derived depending on the OPN being used and the parameter Enable Independent perst pins. If this parameter is set to disabled, this clock is derived from refclk0. On the other hand, if Enable Independent perst pins is set to enabled, this clock is derived from refclk2. Refer to the Agilex™ 7 Device Family Pin Connection Guidelines for further details on the implementation of the refclk pins available for your specific OPN.
  • Application clock domain (coreclkout_hip) for in-band signals: this clock is an output from the R-Tile IP, and it has the same frequency as pld_clk.
  • Application clock domain (slow_clk) for sideband signals: this clock is another output from the R-Tile IP. It is a divide-by-2/4 version of coreclkout_hip.
Figure 2. Clock Domains in PCIe Modes
Table 11.  PHY Clock and Application Clock Frequencies
Mode PHY Clock Frequency Application Clock Frequency
PCIe Gen1 1000 MHz Gen1 is supported only via link down-training and not natively. Hence, the application clock frequency depends on the configuration you choose in the IP Parameter Editor. For example, if you choose a Gen3 configuration, the application clock frequency is 250 MHz - 300 MHz.
PCIe Gen2 1000 MHz Gen2 is supported only via link down-training and not natively. Hence, the application clock frequency depends on the configuration you choose in the IP Parameter Editor. For example, if you choose a Gen3 configuration, the application clock frequency is 250 MHz - 300 MHz.
PCIe Gen3 1000 MHz 250 MHz - 500 MHz (*)
PCIe Gen4 1000 MHz 250 MHz - 500 MHz (*)
PCIe Gen5 1000 MHz 400 MHz - 500 MHz
Note:
(*) The highest frequencies at Gen3 and Gen4 for the Application Clock Frequency are only available in Production devices or Engineering Samples with the following OPNs:
  • AGIx027R29AxxxxR2
  • AGIx027R29AxxxxR3
  • AGIx027R29BxxxxR3
  • AGIx023R18AxxxxR0
  • AGIx041R29DxxxxR0
  • AGIx041R29DxxxxR1
  • AGMx039R47AxxR0
For additional details, refer to the Available Options section of the Agilex™ 7 FPGAs and SoCs Device Overview.
Note: For a link down-training scenario when R-Tile is configured at Gen3, Gen4 or Gen5 and the link gets down-trained to a lower speed, the application clock frequency will continue to run at the configured frequency set in the PLD Clock Frequency parameter. For example, when the PCIe Hard IP Mode parameter is set as Gen5 1x16 and the PLD Clock Frequency parameter as 500 MHz, the PLD clock frequency will continue to run at 500 MHz even if the link is down-trained to Gen4 or less.

R-Tile may have up to three reference clock inputs at the package level, refclk0, refclk1, and refclk2, depending on the OPN being used. Refer to the Agilex™ 7 Device Family Pin Connection Guidelines for further details on the implementation of the refclk pins available for your specific OPN.

The connection requirements for these input clocks depend on the Configuration mode being used and the Enable Independent perst pins parameter. You must consider the following guidelines:
  • Connect a 100 MHz reference clock source to refclk0 and refclk1.
  • If using Configuration Mode 0 (1x16) or Configuration Mode 2 (4x4), drive refclk0 and refclk1 using a single clock source.
Figure 3. Using a Single 100 MHz Clock Source in 1x16 and 4x4 Modes

In Configuration Mode 1 (2x8), you can drive the refclk0 and refclk1 inputs with either a single 100 MHz clock source as shown above, or two independent 100 MHz sources (see Using Independent 100 MHz Clock Sources in 2x8 Mode) depending on your system architecture. For example, if your system has each x8 port connected to a separate CPU/Root Complex, it may be required to drive these refclk inputs using independent clock sources. In that case, if the parameter Enable Independent perst pins is set to disabled, the refclk0 input for Port 0 must always be running because it feeds the reference clock for the R-Tile core PLL that controls the data transfers between the R-Tile and FPGA fabric via the EMIB. If this clock goes down, Port 0 link will go down and Port 1 will not be able to communicate with the FPGA fabric.

On the other hand, if the parameter Enable Independent perst pins is set to enabled, the refclk2 input must always be running because it feeds the R-Tile core PLL that controls the data transfers between the R-Tile and FPGA fabric via the EMIB.

Following are the guidelines for implementing two independent refclks in Configuration Mode 1 (2x8) with the parameter Enable Independent perst pins set to disabled:
  • If the link can handle two separate reference clocks, drive the refclk0 of R-Tile with the on-board free-running oscillator.
  • If the link needs to use a common reference clock, then PERST# needs to indicate the stability of this reference clock. If this reference clock goes down, the entire R-Tile must be reset.
Following are the guidelines for implementing two independent refclks in Configuration Mode 1 (2x8) with the parameter Enable Independent perst pins set to enabled:
  • Drive the additional refclk2 of R-Tile with an on-board free-running oscillator.
  • The pin pin_perst_n indicates the stability of this reference clock. If this reference clock goes down, the entire R-Tile must be reset.
  • You must have a weak pull-down on the pin_perst0_n/pin_perst1_n pins if the corresponding port is not supplied with a reference clock (refclk0/1) during the FPGA configuration. Failing to meet this requirement may cause both ports to fail to link up when pin_perst_n is toggled once configuration is complete and the reference clock (refclk0/1) is not available. Toggling pin_perst_n or the corresponding pin_perst0_n/pin_perst1_n after the FPGA configuration completion will not recover the links and the FPGA will need to be reconfigured.
Figure 4. Using Independent 100 MHz Clock Sources in 2x8 Mode
Level Two Title