R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 10/07/2022
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. Troubleshooting/Debugging 7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 8. Document Revision History for the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TLP Bypass Mode
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core Support Levels
1.1. Overview x
1.1.1. Standards and Specifications Compliance
1.2. Features x
1.2.1. Multifunction and Virtualization Features
2. IP Architecture and Functional Description x
2.1. Overview 2.2. Bias Temperature Instability (BTI) Protection Mode 2.3. PCIe Hard IP Mode 2.4. PIPE Direct Mode
2.3. PCIe Hard IP Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PCI Express Protocol Stack
2.3.2. Reset x
2.3.2.1. Single PERST 2.3.2.2. Independent PERST
2.3.2.2. Independent PERST x
2.3.2.2.1. REFCLK and PERST Guidelines when Using Independent PERST
2.3.3. PCI Express Protocol Stack x
2.3.3.1. PMA/PCS 2.3.3.2. Data Link Layer Overview 2.3.3.3. Transaction Layer Overview
2.3.3.3. Transaction Layer Overview x
2.3.3.3.1. Deferrable Memory Write (DMWr)
2.4. PIPE Direct Mode x
2.4.1. Clocking 2.4.2. Reset 2.4.3. PIPE Layer
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TLP Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TLP Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TLP Bypass Mode 3.3.3. Hard IP Reconfiguration Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TLP Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Clocks and Resets 4.2. Serial Data Interface 4.3. PCI Express Mode 4.4. PIPE Direct Mode
4.1. Clocks and Resets x
4.1.1. Clocks 4.1.2. Resets
4.3. PCI Express Mode x
4.3.1. Avalon® Streaming Interface 4.3.2. Precision Time Measurement (PTM) Interface (Endpoint Only) 4.3.3. Interrupt Interface 4.3.4. Hard IP Reconfiguration Interface 4.3.5. Error Interface 4.3.6. Completion Timeout Interface 4.3.7. Configuration Intercept Interface 4.3.8. Power Management Interface 4.3.9. Hard IP Status Interface 4.3.10. Page Request Services (PRS) Interface (Endpoint Only) 4.3.11. Function-Level Reset (FLR) Interface (Endpoint Only) 4.3.12. SR-IOV VF Error Flag Interface (Endpoint Only) 4.3.13. General Purpose VSEC Interface
4.3.1. Avalon® Streaming Interface x
4.3.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.3.1.2. Credit Control 4.3.1.3. Avalon® Streaming RX Interface 4.3.1.4. Avalon® Streaming TX Interface 4.3.1.5. Tag Allocation
4.3.1.2. Credit Control x
4.3.1.2.1. Credit Interface - Data and Header 4.3.1.2.2. Credit Initialization 4.3.1.2.3. Credit Update/Release 4.3.1.2.4. RX Flow Control Interface 4.3.1.2.5. TX Flow Control Interface
4.3.1.4. Avalon® Streaming TX Interface x
4.3.1.4.1. Avalon® Streaming TX Interface pX_tx_st_ready_o Behavior
4.3.1.5. Tag Allocation x
4.3.1.5.1. Completion Buffer Size
4.3.3. Interrupt Interface x
4.3.3.1. Legacy Interrupts 4.3.3.2. MSI 4.3.3.3. MSI-X
4.3.3.3. MSI-X x
4.3.3.3.1. Implementing MSI-X Interrupts
4.3.8. Power Management Interface x
4.3.8.1. D3Hot Entry 4.3.8.2. D3Hot Exit 4.3.8.3. D3Cold Entry 4.3.8.4. D3Cold Exit
4.3.8.2. D3Hot Exit x
4.3.8.2.1. D3Hot Exit Initiated by Host 4.3.8.2.2. D3Hot Exit Initiated by EP
4.4. PIPE Direct Mode x
4.4.1. Data Signals 4.4.2. Command and Status Signals 4.4.3. Message Bus Signals 4.4.4. Deskew Signals 4.4.5. PIPE Direct Reset Sequence 4.4.6. PIPE Direct Speed Change 4.4.7. PIPE Lane Reset Staggering 4.4.8. Unused Lanes in PIPE Direct Mode
4.4.1. Data Signals x
4.4.1.1. Transmit Signals 4.4.1.2. Receive Signals
4.4.4. Deskew Signals x
4.4.4.1. MAC to PHY (M2P) Signals 4.4.4.2. PHY to MAC (P2M) Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Management (PTM)
6. Troubleshooting/Debugging x
6.1. Hardware Debug 6.2. Operating System Tools and Utilities 6.3. Signal Tap Logic Analyzer 6.4. Using the Hard IP Reconfiguration Interface for Debugging 6.5. Other Interfaces That Can Be Used for Debug Activities 6.6. Debug Toolkit
6.1. Hardware Debug x
6.1.1. Debugging Link Training Issues 6.1.2. Debugging Device Enumeration Issues 6.1.3. Debugging Data Transfer and Performance Issues
6.1.3. Debugging Data Transfer and Performance Issues x
6.1.3.1. Advanced Error Reporting (AER)
6.2. Operating System Tools and Utilities x
6.2.1. Using lspci Utility to Read Negotiated Link Speed
6.4. Using the Hard IP Reconfiguration Interface for Debugging x
6.4.1. Using the Hard IP Reconfiguration Interface to Enable and Read ECRC and LCRC Error Counts
6.6. Debug Toolkit x
6.6.1. Overview 6.6.2. Enabling the R-Tile Debug Toolkit 6.6.3. Launching the R-tile Debug Toolkit 6.6.4. Using the R-Tile Debug Toolkit
6.6.4. Using the R-Tile Debug Toolkit x
6.6.4.1. R-Tile Information 6.6.4.2. Event Counters 6.6.4.3. Configuration Space
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TLP Bypass Mode x
D.1. EP TLP Bypass Mode (Upstream) D.2. RC TLP Bypass Mode (Downstream)
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. Troubleshooting/Debugging
7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
8. Document Revision History for the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TLP Bypass Mode

4.3.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces

The TLP prefix, header and data are sent and received on the Avalon® streaming TX and RX interfaces. There are four Avalon® streaming TX/RX interfaces corresponding to the four PCIe cores. The signal names on these interfaces are prefixed with "p0" for the x16 core, "p1" for the x8 core, "p2" for the x4_0 core and "p3" for the x4_1 core.

Each of these interfaces can contain up to four segments. The signal names for these segments are notated with *_st0_*, *_st1_*, *_st2_* and *_st3_*.

A Start of Packet (SOP) symbol demarcates the start of a packet. A TLP with or without data is terminated by an End of Packet (EOP) symbol. An EOP happening on the st0, st1, st2 or st3 segment implies that the last data with data valid = 1 is located on the corresponding data bus.

A TLP without data is qualified by a header valid indicator. A TLP with data is qualified by a data valid indicator. There is also a prefix valid indicator to qualify a TLP Prefix.

If a global valid signal is required for backward compatibility to P-tile, the header valid ORed with data valid will provide the global valid indicator.

This is defined to enable pipelined Header/Prefix and Data transfers to meet the bandwidth target. For example, while transferring the Data for one TLP, the Header and Prefix for the next TLP can also be transferred.

Figure 24. Generic TLP Format

The Avalon® streaming Header and TLP Prefix bus packet format follows the TLP packet format as defined by the PCIe specification for Memory, Configuration and Message TLPs. Refer to TLP Prefix, Header and Data when PCIe Header Format Checkbox is Disabled and TLP Prefix, Header and Data when PCIe Header Format Checkbox is Enabled below for more details.

In the R-tile Parameter Editor, there is a checkbox labeled PCIe Header format (in the PCIe Avalon® Settings tab) that allows you to specify whether the Header will follow the Big Endian format or Little Endian format. If this checkbox is not enabled, the Header follows the Little Endian format as shown in TLP Prefix, Header and Data when PCIe Header Format Checkbox is Disabled (note that the diagram shows the Prefix, Header and Data for one segment of the Avalon® streaming interface, not the whole interface):

Figure 25. TLP Prefix, Header and Data when PCIe Header Format Checkbox is Disabled

If the PCIe Header format checkbox is enabled, the Header follows the Big Endian format as shown in TLP Prefix, Header and Data when PCIe Header Format Checkbox is Enabled:

Figure 26. TLP Prefix, Header and Data when PCIe Header Format Checkbox is Enabled
Table 50.  IP to FPGA Fabric Interface Summary
Configuration Link Rate Width Mode Header Data Application Clock Frequency (MHz)
1x16 Gen5 Double-Width

st3_hdr[127:0]

st2_hdr[127:0]

st1_hdr[127:0]

st0_hdr[127:0]

st3_data[255:0]

st2_data[255:0]

st1_data[255:0]

st0_data[255:0]

400 MHz

425 MHz

450 MHz

475 MHz

500 MHz
Gen4 Double-Width

st3_hdr[127:0]

st2_hdr[127:0]

st1_hdr[127:0]

st0_hdr[127:0]

st3_data[255:0]

st2_data[255:0]

st1_data[255:0]

st0_data[255:0]

250 MHz

275 MHz

300 MHz
Single-Width 7

st1_hdr[127:0]

st0_hdr[127:0]

st1_data[255:0]

st0_data[255:0]

400 MHz

425 MHz

450 MHz

475 MHz

500 MHz
Gen3 Double-Width

st3_hdr[127:0]

st2_hdr[127:0]

st1_hdr[127:0]

st0_hdr[127:0]

st3_data[255:0]

st2_data[255:0]

st1_data[255:0]

st0_data[255:0]

250 MHz

275 MHz

300 MHz
Single-Width 7

st1_hdr[127:0]

st0_hdr[127:0]

st1_data[255:0]

st0_data[255:0]

400 MHz

425 MHz

450 MHz

475 MHz

500 MHz
2x8 Gen5 Double-Width

st1_hdr[127:0]

st0_hdr[127:0]

st1_data[255:0]

st0_data[255:0]

400 MHz

425 MHz

450 MHz

475 MHz

500 MHz
Gen4 Double-Width

st1_hdr[127:0]

st0_hdr[127:0]

st1_data[255:0]

st0_data[255:0]

250 MHz

275 MHz

300 MHz
Single-Width 7 st0_hdr[127:0] st0_data[255:0]

400 MHz

425 MHz

450 MHz

475 MHz

500 MHz
Gen3 Double-Width

st1_hdr[127:0]

st0_hdr[127:0]

st1_data[255:0]

st0_data[255:0]

250 MHz

275 MHz

300 MHz
Single-Width 7 st0_hdr[127:0] st0_data[255:0]

250 MHz

275 MHz

300 MHz
4x4 Gen5 Double-width

st1_hdr[127:0]

st0_hdr[127:0]

st1_data[127:0]

st0_data[127:0]

400 MHz

425 MHz

450 MHz

475 MHz

500 MHz
Gen4 Double-width

st1_hdr[127:0]

st0_hdr[127:0]

st1_data[127:0]

st0_data[127:0]

400 MHz

425 MHz

450 MHz

475 MHz

500 MHz
Single-Width 7 st0_hdr[127:0] st0_data[127:0]

400 MHz

425 MHz

450 MHz

475 MHz

500 MHz
Gen3 Double-width

st1_hdr[127:0]

st0_hdr[127:0]

st1_data[127:0]

st0_data[127:0]

250 MHz

275 MHz

300 MHz
Single-Width 7 st0_hdr[127:0] st0_data[127:0]

250 MHz

275 MHz

300 MHz
Note:
  • For Configuration Mode 0 (1x16), the SOP for a TLP can happen only on segment 0 (st0) or segment 2 (st2) (i.e., a given TLP cannot start on segment 1 or segment 3).
  • For Configuration Mode 0 (1x16) in double-width mode, the header for segment 2 (st2_hdr) is only allowed if segment 0 and segment 1 are also used (i.e., st0_hdr, st1_hdr and st0_data, st1_data are also used).
  • For Configuration Mode 0 (1x16) in single-width mode, only one segment can be used per clock cycle (i.e., st0_hdr/st0_data or st1_hdr/st1_data). In addition, if segment 1 is used, st0_data must be used by the previous TLP.
7 Single-width mode is only available in devices with the suffix R2 or R3 in their OPN number. For additional details on OPN decoding, refer to Intel® Agilex™ FPGAs and SoCs Device Overview.
Level Two Title