R-tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 3/28/2022
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. About the R-tile Avalon® Streaming Intel® FPGA IP for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 7. Document Revision History for the R-tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TLP Bypass Mode
1. About the R-tile Avalon® Streaming Intel® FPGA IP for PCI Express x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core Support Levels
1.1. Overview x
1.1.1. Standards and Specifications Compliance
2. IP Architecture and Functional Description x
2.1. Overview 2.2. Bias Temperature Instability (BTI) Protection Mode 2.3. PCIe Hard IP Mode 2.4. PIPE Direct Mode
2.3. PCIe Hard IP Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PCI Express Protocol Stack
2.3.3. PCI Express Protocol Stack x
2.3.3.1. PMA/PCS 2.3.3.2. Data Link Layer Overview 2.3.3.3. Transaction Layer Overview
2.4. PIPE Direct Mode x
2.4.1. Clocking 2.4.2. Reset 2.4.3. PIPE Layer
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TLP Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TLP Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TLP Bypass Mode 3.3.3. Hard IP Reconfiguration Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TLP Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Overview 4.2. Clocks and Resets 4.3. Serial Data Interface 4.4. PCI Express Mode 4.5. PIPE Direct Mode
4.2. Clocks and Resets x
4.2.1. Clocks 4.2.2. Resets
4.4. PCI Express Mode x
4.4.1. Avalon® Streaming Interface 4.4.2. Precision Time Measurement (PTM) Interface (Endpoint Only) 4.4.3. Interrupt Interface 4.4.4. Hard IP Reconfiguration Interface 4.4.5. Error Interface 4.4.6. Completion Timeout Interface 4.4.7. Configuration Intercept Interface 4.4.8. Power Management Interface 4.4.9. Hard IP Status Interface 4.4.10. Page Request Services (PRS) Interface (Endpoint Only) 4.4.11. Function-Level Reset (FLR) Interface (Endpoint Only) 4.4.12. SR-IOV VF Error Flag Interface (Endpoint Only) 4.4.13. General Purpose VSEC Interface
4.4.1. Avalon® Streaming Interface x
4.4.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.4.1.2. Credit Control 4.4.1.3. Avalon® Streaming RX Interface 4.4.1.4. Avalon® Streaming TX Interface 4.4.1.5. Tag Allocation
4.4.1.2. Credit Control x
4.4.1.2.1. Credit Interface - Data and Header 4.4.1.2.2. Credit Initialization 4.4.1.2.3. Credit Update/Release 4.4.1.2.4. RX Flow Control Interface 4.4.1.2.5. TX Flow Control Interface
4.4.1.4. Avalon® Streaming TX Interface x
4.4.1.4.1. Avalon® Streaming TX Interface tx_st_ready_o Behavior
4.4.1.5. Tag Allocation x
4.4.1.5.1. Completion Buffer Size
4.4.3. Interrupt Interface x
4.4.3.1. Legacy Interrupts 4.4.3.2. MSI 4.4.3.3. MSI-X
4.4.3.3. MSI-X x
4.4.3.3.1. Implementing MSI-X Interrupts
4.4.8. Power Management Interface x
4.4.8.1. D3Hot Entry 4.4.8.2. D3Hot Exit 4.4.8.3. D3Cold Entry 4.4.8.4. D3Cold Exit
4.4.8.2. D3Hot Exit x
4.4.8.2.1. D3Hot Exit Initiated by Host 4.4.8.2.2. D3Hot Exit Initiated by EP
4.5. PIPE Direct Mode x
4.5.1. Data Signals 4.5.2. Message Channel 4.5.3. Deskew Channel 4.5.4. PIPE Direct Reset Sequence 4.5.5. PIPE Direct Speed Change 4.5.6. PIPE Lane Reset Staggering 4.5.7. Unused Lanes in PIPE Direct Mode
4.5.1. Data Signals x
4.5.1.1. Transmit Signals 4.5.1.2. Receive Signals
4.5.3. Deskew Channel x
4.5.3.1. MAC to PHY (M2P) Signals 4.5.3.2. PHY to MAC (P2M) Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Management (PTM)
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TLP Bypass Mode x
D.1. EP TLP Bypass Mode (Upstream) D.2. RC TLP Bypass Mode (Downstream)
1. About the R-tile Avalon® Streaming Intel® FPGA IP for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
7. Document Revision History for the R-tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TLP Bypass Mode

2.3.1. Clocking

In the PCIe Hard IP mode, the R-tile Avalon Streaming IP for PCI Express* has four primary clock domains:
  • PHY clock domain (i.e. core_clk domain): this clock is synchronous to the SerDes parallel clock.
  • EMIB/FPGA fabric interface clock domain (i.e. pld_clk domain): this clock is derived from the same reference clock (refclk0) as the one used by the SerDes. However, this clock is generated from a stand-alone core PLL.
  • Application clock domain (coreclkout_hip) for in-band signals: this clock is an output from the R-tile IP, and it has the same frequency as pld_clk.
  • Application clock domain (slow_clk) for sideband signals: this clock is another output from the R-tile IP. It is a divide-by-2/4 version of coreclkout_hip.
Figure 2. Clock Domains in PCIe Modes
Table 10.  PHY Clock and Application Clock Frequencies
Mode PHY Clock Frequency Application Clock Frequency
PCIe Gen1 1000 MHz Gen1 is supported only via link down-training and not natively. Hence, the application clock frequency depends on the configuration you choose in the IP Parameter Editor. For example, if you choose a Gen3 configuration, the application clock frequency is 250 MHz - 300 MHz.
PCIe Gen2 1000 MHz Gen2 is supported only via link down-training and not natively. Hence, the application clock frequency depends on the configuration you choose in the IP Parameter Editor. For example, if you choose a Gen3 configuration, the application clock frequency is 250 MHz - 300 MHz.
PCIe Gen3 1000 MHz 250 MHz - 500 MHz (*)
PCIe Gen4 1000 MHz 250 MHz - 500 MHz (*)
PCIe Gen5 1000 MHz 400 MHz - 500 MHz
Note: (*) The highest frequencies at Gen3 and Gen4 for the Application Clock Frequency are only available in devices with the suffix R2 or R3 in their OPN numbers. For additional details, refer to Intel® Agilex™ FPGAs and SoCs Device Overview.
Note: For a link down-training scenario when R-tile is configured at Gen3, Gen4 or Gen5 and the link gets down-trained to a lower speed, the application clock frequency will continue to run at the configured frequency set in the PLD Clock Frequency parameter. For example, when the PCIe Hard IP Mode parameter is set as Gen5 1x16 and the PLD Clock Frequency parameter as 500 MHz, the PLD clock frequency will continue to run at 500 MHz even if the link is down-trained to Gen4 or less.

R-tile has two reference clock inputs at the package level, refclk0 and refclk1. You must connect a 100 MHz reference clock source to these two inputs. Depending on the port mode, you can drive the two refclk inputs using either a single clock source or two independent clock sources.

In 1x16 and 4x4 modes, drive the refclk inputs with a single clock source (through a fanout buffer) as shown in the figure below.

Figure 3. Using a Single 100 MHz Clock Source in 1x16 and 4x4 Modes
In 2x8 mode, you can drive the refclk inputs with either a single 100 MHz clock source as shown above, or two independent 100 MHz sources (see Using Independent 100 MHz Clock Sources in 2x8 Mode) depending on your system architecture. For example, if your system has each x8 port connected to a separate CPU/Root Complex, it may be required to drive these refclk inputs using independent clock sources. In that case, the refclk0 input for Port 0 must always be running because it feeds the reference clock for the R-tile core PLL that controls the data transfers between the R-tile and FPGA fabric via the EMIB. If this clock goes down, Port 0 link will go down and Port 1 will not be able to communicate with the FPGA fabric. Following are the guidelines for implementing two independent refclks in 2x8 mode:
  • If the link can handle two separate reference clocks, drive the refclk0 of R-tile with the on-board free-running oscillator.
  • If the link needs to use a common reference clock, then PERST# needs to indicate the stability of this reference clock. If this reference clock goes down, the entire R-tile must be reset.
Figure 4. Using Independent 100 MHz Clock Sources in 2x8 Mode
Level Two Title