R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 4/10/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. Troubleshooting/Debugging 7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 8. Document Revision History for the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TLP Bypass Mode E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core Support Levels
1.1. Overview x
1.1.1. Standards and Specifications Compliance
1.2. Features x
1.2.1. Multifunction and Virtualization Features
2. IP Architecture and Functional Description x
2.1. Overview 2.2. PCIe Hard IP Mode 2.3. PIPE Direct Mode
2.2. PCIe Hard IP Mode x
2.2.1. Clocking 2.2.2. Reset 2.2.3. PCI Express Protocol Stack
2.2.2. Reset x
2.2.2.1. Single PERST 2.2.2.2. Independent PERST
2.2.2.2. Independent PERST x
2.2.2.2.1. REFCLK and PERST Guidelines when Using Independent PERST
2.2.3. PCI Express Protocol Stack x
2.2.3.1. PMA 2.2.3.2. Data Link Layer Overview 2.2.3.3. Transaction Layer Overview
2.2.3.3. Transaction Layer Overview x
2.2.3.3.1. Deferrable Memory Write (DMWr)
2.3. PIPE Direct Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PIPE Layer
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TLP Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TLP Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TLP Bypass Mode 3.3.3. Hard IP Reconfiguration Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TLP Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Clocks and Resets 4.2. Serial Data Interface 4.3. PCI Express Mode 4.4. PIPE Direct Mode
4.1. Clocks and Resets x
4.1.1. Clocks 4.1.2. Resets
4.3. PCI Express Mode x
4.3.1. Avalon® Streaming Interface 4.3.2. Precision Time Measurement (PTM) Interface (Endpoint Only) 4.3.3. Interrupt Interface 4.3.4. Hard IP Reconfiguration Interface 4.3.5. Error Interface 4.3.6. Completion Timeout Interface 4.3.7. Configuration Intercept Interface 4.3.8. Power Management Interface 4.3.9. Hard IP Status Interface 4.3.10. Page Request Services (PRS) Interface (Endpoint Only) 4.3.11. Function-Level Reset (FLR) Interface (Endpoint Only) 4.3.12. SR-IOV VF Error Flag Interface (Endpoint Only) 4.3.13. General Purpose VSEC Interface
4.3.1. Avalon® Streaming Interface x
4.3.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.3.1.2. Credit Control 4.3.1.3. Avalon® Streaming RX Interface 4.3.1.4. Avalon® Streaming TX Interface 4.3.1.5. Tag Allocation
4.3.1.2. Credit Control x
4.3.1.2.1. Credit Interface - Data and Header 4.3.1.2.2. Credit Initialization 4.3.1.2.3. Credit Update/Release 4.3.1.2.4. RX Flow Control Interface 4.3.1.2.5. TX Flow Control Interface
4.3.1.3. Avalon® Streaming RX Interface x
4.3.1.3.1. Application Logic Guidelines for the Avalon Streaming RX Interface
4.3.1.4. Avalon® Streaming TX Interface x
4.3.1.4.1. Application Logic Guidelines for the Avalon Streaming TX Interface 4.3.1.4.2. Avalon® Streaming TX Interface pX_tx_st_ready_o Behavior
4.3.1.5. Tag Allocation x
4.3.1.5.1. Completion Buffer Size
4.3.3. Interrupt Interface x
4.3.3.1. Legacy Interrupts 4.3.3.2. MSI 4.3.3.3. MSI-X
4.3.3.3. MSI-X x
4.3.3.3.1. Implementing MSI-X Interrupts
4.3.8. Power Management Interface x
4.3.8.1. D3Hot Entry 4.3.8.2. D3Hot Exit 4.3.8.3. D3Cold Entry 4.3.8.4. D3Cold Exit
4.3.8.2. D3Hot Exit x
4.3.8.2.1. D3Hot Exit Initiated by Host 4.3.8.2.2. D3Hot Exit Initiated by EP
4.4. PIPE Direct Mode x
4.4.1. Data Signals 4.4.2. Command and Status Signals 4.4.3. Message Bus Signals 4.4.4. Deskew Signals 4.4.5. PIPE Direct Reset Sequence 4.4.6. PIPE Direct Speed Change 4.4.7. PIPE Lane Reset Staggering 4.4.8. Unused Lanes in PIPE Direct Mode
4.4.1. Data Signals x
4.4.1.1. Transmit Signals 4.4.1.2. Receive Signals
4.4.4. Deskew Signals x
4.4.4.1. MAC to PHY (M2P) Signals 4.4.4.2. PHY to MAC (P2M) Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Management (PTM)
6. Troubleshooting/Debugging x
6.1. Hardware Debug 6.2. Operating System Tools and Utilities 6.3. Signal Tap Logic Analyzer 6.4. Using the Hard IP Reconfiguration Interface for Debugging 6.5. Other Interfaces That Can Be Used for Debug Activities 6.6. Debug Toolkit
6.1. Hardware Debug x
6.1.1. Debugging Link Training Issues 6.1.2. Debugging Device Enumeration Issues 6.1.3. Debugging Data Transfer and Performance Issues
6.1.3. Debugging Data Transfer and Performance Issues x
6.1.3.1. Advanced Error Reporting (AER)
6.2. Operating System Tools and Utilities x
6.2.1. Using lspci Utility to Read Negotiated Link Speed
6.4. Using the Hard IP Reconfiguration Interface for Debugging x
6.4.1. Using the Hard IP Reconfiguration Interface to Enable and Read ECRC and LCRC Error Counts
6.6. Debug Toolkit x
6.6.1. Overview 6.6.2. Enabling the R-Tile Debug Toolkit 6.6.3. Launching the R-tile Debug Toolkit 6.6.4. Using the R-Tile Debug Toolkit
6.6.4. Using the R-Tile Debug Toolkit x
6.6.4.1. R-Tile Information 6.6.4.2. Event Counters 6.6.4.3. Link Inspector 6.6.4.4. Configuration Space 6.6.4.5. Channel Parameters
6.6.4.5. Channel Parameters x
6.6.4.5.1. General PHY 6.6.4.5.2. Tx Path 6.6.4.5.3. Rx Path 6.6.4.5.4. Lane Margining
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TLP Bypass Mode x
D.1. EP TLP Bypass Mode (Upstream) D.2. RC TLP Bypass Mode (Downstream)
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express x
E.1. Margin Masks Overview E.2. 5x5 Testing Efficiency E.3. Debug Toolkit Margin Methodology
E.2. 5x5 Testing Efficiency x
E.2.1. First Time Testing E.2.2. Spot Check Testing
E.3. Debug Toolkit Margin Methodology x
E.3.1. Debug Toolkit Testing Process
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. Troubleshooting/Debugging
7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
8. Document Revision History for the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TLP Bypass Mode
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express

4.3.1.2.4. RX Flow Control Interface

The RX flow control interface provides information on the application's available RX buffer space for Posted (P), Non-Posted (NP) and Completion (CPL) transactions to the PCIe Hard IP. It reports the space available in number of credits as specified by the PCIe Specification.

The Application logic must ensure that enough credits are provided to the R-Tile Avalon-ST IP for PCIe during the Credit Initialization phase. This is to prevent a performance impact caused by the lack of credits available between the IP and the Application logic. Depending on the number of P, NP and CPL transactions happening at the PCIe link level, a lack of credit scenario may happen if the number of credits advertised by the Application logic is less than the credits advertised by the R-Tile Avalon-ST IP for PCIe to the link partner.

Flow control credits are available for the following TLP categories:
  • Posted (P) transactions: TLPs that do not required a response.
  • Non-poseted (NP) transactions: TLPs that require a completion.
  • Completions (CPL): TLPs that respond to non-posted transactions.
Table 52.  Credits Advertised by the R-Tile Avalon-ST IP for PCIe to the Link PartnerNote: The credit unit in this table follows the PCIe Specification where 1 credit = 4 DWs = 16 Bytes.
Port Hard IP Mode Posted Headers Posted Data Non-Posted Headers Non-Posted Data Completion Headers Completion Data
Port 0 Endpoint 784 1456 784 392 0 (infinite) 0 (infinite)
Root Port 784 1456 784 392 0 (infinite) 0 (infinite)
Upstream Port 784 1456 784 392 1024 2816
Downstream Port 784 1456 784 392 1024 2816
Port 1 Endpoint 392 760 392 196 0 (infinite) 0 (infinite)
Root Port 392 760 392 196 0 (infinite) 0 (infinite)
Upstream Port 392 760 392 196 512 1408
Downstream Port 392 760 392 196 512 1408
Port 2 Endpoint 224 444 224 112 0 (infinite) 0 (infinite)
Root Port 224 444 224 112 0 (infinite) 0 (infinite)
Upstream Port 224 444 224 112 256 704
Downstream Port 224 444 224 112 256 704
Port 3 Endpoint 224 444 224 112 0 (infinite) 0 (infinite)
Root Port 224 444 224 112 0 (infinite) 0 (infinite)
Upstream Port 224 444 224 112 256 704
Downstream Port 224 444 224 112 256 704

For more information on how credit control in general is implemented in this IP, refer to Credit Control.

Note: In some systems, there are broadcast Message TLPs being sent by the link partner during the PCIe enumeration process. Failing to properly initialize and return the corresponding credits from the Application logic to the R-Tile Avalon® Streaming IP may cause enumeration issues due to the priority order between Message TLPs and the Completions required for Configuration TLPs. Application logic must ensure proper credits are returned to the R-Tile Avalon® Streaming IP for any TLP that it receives.
Table 53.  Categorization of Transaction Types
TLP Type Category
Memory Write Posted
Memory Read Non-Posted
Memory Read Lock
I/O Read
I/O Write
Configuration Read
Configuration Write
Fetch and Add AtomicOp
Message Posted
Completion Completion
Completion with Data
Completion Lock
Completion Lock with Data
Table 54.  RX Flow Control Interface Signals
Signal Name Direction Description EP/RP/BP Clock Domain
pX_rx_st_hcrdt_update_i[2:0] where

X = 0, 1, 2, 3 (IP core number)

Input

Indicates credit is made available for the different types of Header.

Each Header (including the TLP Prefix, if any) consumes one credit.

[0] : Posted Header (PH)

[1] : Non-Posted Header (NPH)

[2] : Completion Header (CPLH)

To advertise infinite credits for a specific TLP type between the IP and the Application logic, the corresponding bit within this signal bus must be asserted for one clock cycle, with a value of 0 on the corresponding bits for the targeted TLP type in the pX_rx_st_hcrdt_update_cnt_i signal bus during the credit initialization phase.

EP/RP/BP coreclkout_hip
pX_rx_st_hcrdt_update_cnt_i[5:0] where

X = 0, 1, 2, 3 (IP core number)

Input

Indicates number of credits released.

[1:0] : number of PH credits released

[3:2] : number of NPH credits released

[5:4] : number of CPLH credits released

Valid when the corresponding pX_rx_st_hcrdt_update_i bit = 1.

The maximum number of credits released is three.

To advertise infinite credits for a specific TLP type between the IP and the Application logic, the corresponding bits within this signal bus must be set to 0 during the credit initialization phase (when the corresponding pX_rx_st_hcrdt_update_i bit is asserted for one clock cycle).

EP/RP/BP coreclkout_hip
pX_rx_st_hcrdt_init_i[2:0] where

X = 0, 1, 2, 3 (IP core number)

Input

Credit Initialization indicator. These signals remain high for entire initialization phase. A High to Low transition indicates the completion of the credit initialization phase.

[0] : PH

[1] : NPH

[2] : CPLH

 EP/RP/BP coreclkout_hip
pX_rx_st_hcrdt_init_ack_o[2:0] where

X = 0, 1, 2, 3 (IP core number)

Output

Indicates the Host is ready for the credit initialization phase

[0] : PH

[1] : NPH

[2] : CPLH

 EP/RP/BP coreclkout_hip
pX_rx_st_dcrdt_update_i[2:0] where

X = 0, 1, 2, 3 (IP core number)

Input

Indicates credit is made available for the different types of Data.

[0] : Posted Data (PD)

[1] : Non-Posted Data (NPD)

[2] : Completion Data (CPLD)

To advertise infinite credits for a specific TLP type between the IP and the Application logic, the corresponding bit within this signal bus must be asserted for one clock cycle, with a value of 0 on the corresponding bits for the targeted TLP type in the pX_rx_st_dcrdt_update_cnt_i signal bus during the credit initialization phase.

EP/RP/BP coreclkout_hip
pX_rx_st_dcrdt_update_cnt_i[11:0] where

X = 0, 1, 2, 3 (IP core number)

Input

Indicates number of credits released.

[3:0] : number of PD credits released

[7:4] : number of NPD credits released

[11:8] : number of CPLD credits released

Valid when the corresponding pX_rx_st_dcrdt_update_i bit = 1.

The maximum number of credits released is 15.

To advertise infinite credits for a specific TLP type between the IP and the Application logic, the corresponding bits within this signal bus must be set to 0 during the credit initialization phase (when the corresponding pX_rx_st_dcrdt_update_i bit is asserted for one clock cycle).

Note: The initial RX NPD credits must be equal or larger than the Maximum Payload Size.
EP/RP/BP coreclkout_hip
pX_rx_st_dcrdt_init_i[2:0] where

X = 0, 1, 2, 3 (IP core number)

Input

Credit Initialization indicator. These signals remain high for entire initialization phase. A High to Low transition indicates the completion of the credit initialization phase.

[0] : PD

[1] : NPD

[2] : CPLD

 EP/RP/BP coreclkout_hip
pX_rx_st_dcrdt_init_ack_o[2:0] where

X = 0, 1, 2, 3 (IP core number)

Input

Indicates the Host is ready for the credit initialization phase

[0] : PD

[1] : NPD

[2] : CPLD

 EP/RP/BP coreclkout_hip
Level Two Title