R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 6/26/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. Troubleshooting/Debugging 7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 8. Document Revision History for the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TLP Bypass Mode E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core Support Levels
1.1. Overview x
1.1.1. Standards and Specifications Compliance
1.2. Features x
1.2.1. Multifunction and Virtualization Features
2. IP Architecture and Functional Description x
2.1. Overview 2.2. PCIe Hard IP Mode 2.3. PIPE Direct Mode
2.2. PCIe Hard IP Mode x
2.2.1. Clocking 2.2.2. Reset 2.2.3. PCI Express Protocol Stack
2.2.2. Reset x
2.2.2.1. Single PERST 2.2.2.2. Independent PERST Pins 2.2.2.3. Independent GPIO PERST
2.2.2.2. Independent PERST Pins x
2.2.2.2.1. REFCLK and PERST Guidelines When Using Independent PERST Pins
2.2.2.3. Independent GPIO PERST x
2.2.2.3.1. REFCLK and PERST Guidelines when Using Independent GPIO PERST
2.2.3. PCI Express Protocol Stack x
2.2.3.1. PMA 2.2.3.2. Data Link Layer Overview 2.2.3.3. Transaction Layer Overview
2.2.3.3. Transaction Layer Overview x
2.2.3.3.1. Deferrable Memory Write (DMWr)
2.3. PIPE Direct Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PIPE Layer
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TLP Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TLP Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TLP Bypass Mode 3.3.3. Hard IP Reconfiguration Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TLP Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Clocks and Resets 4.2. Serial Data Interface 4.3. PCI Express Mode 4.4. PIPE Direct Mode
4.1. Clocks and Resets x
4.1.1. Clocks 4.1.2. Resets
4.3. PCI Express Mode x
4.3.1. Avalon® Streaming Interface 4.3.2. Precision Time Measurement (PTM) Interface (Endpoint Only) 4.3.3. Interrupt Interface 4.3.4. Hard IP Reconfiguration Interface 4.3.5. Error Interface 4.3.6. Completion Timeout Interface 4.3.7. Configuration Intercept Interface 4.3.8. Power Management Interface 4.3.9. Hard IP Status Interface 4.3.10. Page Request Services (PRS) Interface (Endpoint Only) 4.3.11. Function-Level Reset (FLR) Interface (Endpoint Only) 4.3.12. SR-IOV VF Error Flag Interface (Endpoint Only) 4.3.13. General Purpose VSEC Interface
4.3.1. Avalon® Streaming Interface x
4.3.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.3.1.2. Credit Control 4.3.1.3. Avalon® Streaming RX Interface 4.3.1.4. Avalon® Streaming TX Interface 4.3.1.5. Tag Allocation
4.3.1.2. Credit Control x
4.3.1.2.1. Credit Interface - Data and Header 4.3.1.2.2. Credit Initialization 4.3.1.2.3. Credit Update/Release 4.3.1.2.4. RX Flow Control Interface 4.3.1.2.5. TX Flow Control Interface
4.3.1.3. Avalon® Streaming RX Interface x
4.3.1.3.1. Application Logic Guidelines for the Avalon Streaming RX Interface
4.3.1.4. Avalon® Streaming TX Interface x
4.3.1.4.1. Application Logic Guidelines for the Avalon Streaming TX Interface 4.3.1.4.2. Avalon® Streaming TX Interface pX_tx_st_ready_o Behavior
4.3.1.5. Tag Allocation x
4.3.1.5.1. Completion Buffer Size
4.3.3. Interrupt Interface x
4.3.3.1. Legacy Interrupts 4.3.3.2. MSI 4.3.3.3. MSI-X
4.3.3.3. MSI-X x
4.3.3.3.1. Implementing MSI-X Interrupts
4.3.8. Power Management Interface x
4.3.8.1. D3Hot Entry 4.3.8.2. D3Hot Exit 4.3.8.3. D3Cold Entry 4.3.8.4. D3Cold Exit
4.3.8.2. D3Hot Exit x
4.3.8.2.1. D3Hot Exit Initiated by Host 4.3.8.2.2. D3Hot Exit Initiated by EP
4.4. PIPE Direct Mode x
4.4.1. Data Signals 4.4.2. Command and Status Signals 4.4.3. Message Bus Signals 4.4.4. Deskew Signals 4.4.5. PIPE Direct Reset Sequence 4.4.6. PIPE Direct Speed Change 4.4.7. PIPE Lane Reset Staggering 4.4.8. Unused Lanes in PIPE Direct Mode
4.4.1. Data Signals x
4.4.1.1. Transmit Signals 4.4.1.2. Receive Signals
4.4.4. Deskew Signals x
4.4.4.1. MAC to PHY (M2P) Signals 4.4.4.2. PHY to MAC (P2M) Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Measurement (PTM)
6. Troubleshooting/Debugging x
6.1. Hardware Debug 6.2. Operating System Tools and Utilities 6.3. Signal Tap Logic Analyzer 6.4. Using the Hard IP Reconfiguration Interface for Debugging 6.5. Other Interfaces That Can Be Used for Debug Activities 6.6. Debug Toolkit
6.1. Hardware Debug x
6.1.1. Debugging Link Training Issues 6.1.2. Debugging Device Enumeration Issues 6.1.3. Debugging Data Transfer and Performance Issues
6.1.3. Debugging Data Transfer and Performance Issues x
6.1.3.1. Advanced Error Reporting (AER)
6.2. Operating System Tools and Utilities x
6.2.1. Using lspci Utility to Read Negotiated Link Speed
6.4. Using the Hard IP Reconfiguration Interface for Debugging x
6.4.1. Using the Hard IP Reconfiguration Interface to Enable and Read ECRC and LCRC Error Counts
6.6. Debug Toolkit x
6.6.1. Overview 6.6.2. Enabling the R-Tile Debug Toolkit 6.6.3. Launching the R-Tile Debug Toolkit 6.6.4. Using the R-Tile Debug Toolkit
6.6.4. Using the R-Tile Debug Toolkit x
6.6.4.1. R-Tile Information 6.6.4.2. Event Counters 6.6.4.3. Link Inspector 6.6.4.4. Configuration Space 6.6.4.5. Channel Parameters
6.6.4.5. Channel Parameters x
6.6.4.5.1. General PHY 6.6.4.5.2. Tx Path 6.6.4.5.3. Rx Path 6.6.4.5.4. Lane Margining
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TLP Bypass Mode x
D.1. EP TLP Bypass Mode (Upstream) D.2. RC TLP Bypass Mode (Downstream)
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express x
E.1. Margin Masks Overview E.2. 5x5 Testing Efficiency E.3. Debug Toolkit Margin Methodology
E.2. 5x5 Testing Efficiency x
E.2.1. First Time Testing E.2.2. Spot Check Testing
E.3. Debug Toolkit Margin Methodology x
E.3.1. Debug Toolkit Testing Process
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. Troubleshooting/Debugging
7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
8. Document Revision History for the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TLP Bypass Mode
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express

4.3.1.3. Avalon® Streaming RX Interface

The Application Layer receives data from the Transaction Layer of the R-Tile PCI Express IP core over the Avalon® Streaming RX interface. For R-Tile, the rx_st_ready_i has to be always high. The buffer control in the application logic needs to be handled by the RX Flow Control interface. Refer to RX Flow Control Interface for more details.

This interface supports four rx_st_sop_o signals and four rx_st_eop_o signals per cycle when the R-Tile IP is operating in a 1x16 configuration.

In Configuration Mode 0 (1x16) with a double-width configuration, the core provides four segments with each one having 256 bits of data (pX_rx_stN_data_o[255:0]), 128 bits of header (pX_rx_stN_hdr_o[127:0]), and 32 bits of TLP prefix (pX_rx_stN_prefix_o[31:0]). If this core is configured in the 1x16 mode, the data bus becomes a 1024-bit bus.

In Configuration Mode 1 (2x8) with a double-width configuration, there are still four Avalon® Streaming segments (two for each x8 port).

Parity generation is done via a 32:1 XOR (i.e. there is one parity bit for every 32 data, header or prefix bits).

Table 55.  Avalon Streaming RX Interface Signals
Signal Name Direction Description EP/RP/BP Clock Domain
pX_rx_stN_data_o[W:0] where

X = 0,1,2,3 (IP core number) and W varies based on the core.

N = 0,1,2,3 (segment number)

Output This is the Receive data bus. The Application Layer receives data from the Transaction Layer of the IP core on this bus. EP/RP/BP coreclkout_hip
pX_rx_stN_hdr_o[127:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output This is the received header, which follows the TLP header format of the PCIe specifications. EP/RP/BP coreclkout_hip
pX_rx_stN_prefix_o[31:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output

This is the first TLP prefix received, which follows the TLP prefix format of the PCIe specifications. PASID is supported.

These signals are valid when the corresponding rx_st_sop_o is asserted.

The TLP prefix uses a Big Endian implementation (i.e, the Fmt field is in bits [31:29] and the Type field is in bits [28:24]).

If no prefix is present for a given TLP, that dword (including the Fmt field) is all zeros.

EP/RP/BP coreclkout_hip
pX_rx_stN_sop_o where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output

Signals the first cycle of the TLP when asserted in conjunction with the corresponding bit of rx_stN_valid_o.

rx_stN_sop_o: When asserted, signals the start of a TLP on rx_stN_data_o[255:0].

For example, when asserted, rx_st2_sop_o signals the start of a TLP on rx_st2_data_o[255:0].

EP/RP/BP coreclkout_hip
pX_rx_stN_eop_o where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output

Signals the last cycle of the TLP when asserted in conjunction with the corresponding bit of rx_stN_valid_o.

rx_stN_eop_o: When asserted, signals the end of a TLP on rx_stN_data_o[255:0].

For example, when asserted, rx_st2_eop_o signals the end of a TLP on rx_st2_data_o[255:0].

EP/RP/BP coreclkout_hip
pX_rx_stN_dvalid_o where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output These signals qualify the rx_stN_data_o signals going into the Application Layer. EP/RP/BP coreclkout_hip
pX_rx_stN_hvalid_o where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output These signals qualify the rx_stN_hdr_o signals going into the Application Layer. EP/RP/BP coreclkout_hip
pX_rx_stN_pvalid_o where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output These signals qualify the rx_stN_prefix_o signals going into the Application Layer. EP/RP/BP coreclkout_hip
pX_rx_stN_data_par_o[Z:0] where

X = 0,1,2,3 (IP core number) and Z varies based on the core.

N = 0,1,2,3 (segment number)

Output Parity signals for rx_stN_data_o. EP/RP/BP coreclkout_hip
pX_rx_stN_hdr_par_o[3:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output Parity signals for rx_stN_hdr_o. EP/RP/BP coreclkout_hip
pX_rx_stN_prefix_par_o where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output Parity signals for rx_stN_prefix_o. EP/RP/BP coreclkout_hip
pX_rx_st_ready_i Input Indicates the Application Layer is ready to accept data. This signal should always be set to 1. The Flow Control on the RX side is handled through the Credit Control Interface. EP/RP/BP coreclkout_hip
pX_rx_stN_empty_o[2:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output

Specifies the number of dwords that are empty during cycles when the rx_stN_eop_o signals are asserted. These signals are not valid when the rx_stN_eop_o signals are not asserted.

EP/RP/BP coreclkout_hip
pX_rx_stN_bar_o[2:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output

Specify the BAR for the TLP being output.

These outputs are valid when both rx_stN_sop_o and rx_stN_valid_o are asserted.

EP/RP coreclkout_hip
pX_rx_stN_vfactive_o where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output

When asserted, these signals indicate that the received TLP is targeting a virtual function. When these signals are deasserted, the received TLP is targeting a physical function and the rx_stN_pfnum_o signals indicate the function number.

These signals are valid when the corresponding rx_stN_sop_o is asserted.

EP/RP coreclkout_hip
pX_rx_stN_vfnum_o[10:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output

Specify the target VF number for the received TLP. The application uses this information for both request and completion TLPs. For a completion TLP, these bits specify the VF number of the requester for this completion TLP.

These signals are valid when rx_stN_vf_active_o and the corresponding rx_stN_sop_o are asserted.

EP/RP coreclkout_hip
pX_rx_stN_pfnum_o[2:0] where

X = 0,1,2,3 (IP core number)

N = 0,1,2,3 (segment number)

 Output

Specify the target physical function number for the received TLP.

These signals are valid when the corresponding rx_stN_sop_o is asserted.

EP/RP coreclkout_hip
As an example, Avalon® Streaming RX Interface Timings below shows the behavior of the Avalon Streaming RX interface with multiple TLPs, and each of those TLPs spanning across multiple segments. The following text describes the waveforms per clock cycle:
  1. Clock cycle 1: Application logic asserts p0_rx_st_ready_i signal. This signal must be set high all the time. The RX flow control must be handled by the Application logic using the RX Flow Control interface (refer to RX Flow Control Interface for details).
  2. Clock cycle 2:
    1. The start of the first TLP (T0) arrives in segment 1, when p0_rx_st1_sop_o is asserted.
    2. The signal p0_rx_st1_hvalid_o is asserted to validate the header of this first TLP (T0H0) in the p0_rx_st1_hdr_o bus.
    3. The signal p0_rx_st1_dvalid_o is asserted to validate the data of this first TLP (T0D0) in the p0_rx_st1_data_o bus.
    4. The end of this first TLP (T0) is in segment 2, denoted by the assertion of p0_rx_st2_eop_o.
    5. The signal p0_rx_st2_dvalid_o is asserted to validate the data of this first TLP (T0D1) in the p0_rx_st2_data_o bus.
    6. The bus p0_rx_st2_empty_o indicates the number of dwords that are not valid in the p0_rx_st2_data_o bus (T0D1).
  3. Clock cycle 3:
    1. The next TLP (T1), arrives in segment 1, as denoted by the assertion of p0_rx_st1_sop_o.
    2. The signal p0_rx_st1_hvalid_o is asserted to validate the header of this TLP (T1H0) in the p0_rx_st1_hdr_o bus.
    3. The signal p0_rx_st1_dvalid_o is asserted to validate the data of this TLP (T1D0) in the p0_rx_st1_data_o bus.
    4. The signal p0_rx_st2_dvalid_o is asserted to validate the data of this TLP (T1D1) in the p0_rx_st2_data_o bus.
    5. The signal p0_rx_st3_dvalid_o is asserted to validate the data of this TLP (T1D2) in the p0_rx_st3_data_o bus.
  4. Clock cycle 4:
    1. The end of the T1 TLP is in segment 0, denoted by the assertion of p0_rx_st0_eop_o.
    2. The signal p0_rx_st0_dvalid_o is asserted to validate the data of this TLP (T1D3) in the p0_rx_st0_data_o bus.
    3. The bus p0_rx_st0_empty_o indicates the number of dwords that are not valid in the p0_rx_st0_data_o bus (T1D3).

The next TLP arrives in the next clock cycle in segment 1 and finishes in segment 0.

Figure 31.  Avalon® Streaming RX Interface Timings

Section Content
Application Logic Guidelines for the Avalon Streaming RX Interface

Level Two Title