R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 10/07/2024
Public
Document Table of Contents
Document Table of Contents x
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. Troubleshooting/Debugging 7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 8. Document Revision History for the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TL Bypass Mode E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express x
1.1. Glossary 1.2. Overview 1.3. Features 1.4. Release Information 1.5. Device Family Support 1.6. Performance and Resource Utilization 1.7. IP Core Support Levels
1.2. Overview x
1.2.1. Standards and Specifications Compliance
1.3. Features x
1.3.1. Multifunction and Virtualization Features
2. IP Architecture and Functional Description x
2.1. Overview 2.2. PCIe Hard IP Mode 2.3. PIPE Direct Mode
2.2. PCIe Hard IP Mode x
2.2.1. Clocking 2.2.2. Reset 2.2.3. PCI Express Protocol Stack
2.2.2. Reset x
2.2.2.1. Single PERST 2.2.2.2. Independent PERST Pins 2.2.2.3. Independent GPIO PERST
2.2.2.2. Independent PERST Pins x
2.2.2.2.1. REFCLK and PERST Guidelines When Using Independent PERST Pins
2.2.2.3. Independent GPIO PERST x
2.2.2.3.1. REFCLK and PERST Guidelines when Using Independent GPIO PERST
2.2.3. PCI Express Protocol Stack x
2.2.3.1. PMA 2.2.3.2. Data Link Layer Overview 2.2.3.3. Transaction Layer Overview
2.2.3.3. Transaction Layer Overview x
2.2.3.3.1. Deferrable Memory Write (DMWr)
2.3. PIPE Direct Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PIPE Layer
2.3.3. PIPE Layer x
2.3.3.1. Preset Mappings
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TL Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TL Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TL Bypass Mode 3.3.3. Hard IP Reconfiguration Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TL Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Clocks and Resets 4.2. Serial Data Interface 4.3. PCI Express Mode 4.4. PIPE Direct Mode
4.1. Clocks and Resets x
4.1.1. Clocks 4.1.2. Resets
4.3. PCI Express Mode x
4.3.1. Avalon® Streaming Interface 4.3.2. Precision Time Measurement (PTM) Interface (Endpoint Only) 4.3.3. Hot Plug Interface 4.3.4. Interrupt Interface 4.3.5. Hard IP Reconfiguration Interface 4.3.6. Error Interface 4.3.7. Completion Timeout Interface 4.3.8. Configuration Intercept Interface 4.3.9. Power Management Interface 4.3.10. Hard IP Status Interface 4.3.11. Page Request Services (PRS) Interface (Endpoint Only) 4.3.12. Function-Level Reset (FLR) Interface (Endpoint Only) 4.3.13. SR-IOV VF Error Flag Interface (Endpoint Only) 4.3.14. General Purpose VSEC Interface
4.3.1. Avalon® Streaming Interface x
4.3.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.3.1.2. Credit Control 4.3.1.3. Avalon® Streaming RX Interface 4.3.1.4. Avalon® Streaming TX Interface 4.3.1.5. Tag Allocation
4.3.1.2. Credit Control x
4.3.1.2.1. Credit Interface - Data and Header 4.3.1.2.2. Credit Initialization 4.3.1.2.3. Credit Update/Release 4.3.1.2.4. RX Flow Control Interface 4.3.1.2.5. TX Flow Control Interface
4.3.1.3. Avalon® Streaming RX Interface x
4.3.1.3.1. Application Logic Guidelines for the Avalon Streaming RX Interface
4.3.1.4. Avalon® Streaming TX Interface x
4.3.1.4.1. Application Logic Guidelines for the Avalon Streaming TX Interface 4.3.1.4.2. Avalon® Streaming TX Interface pX_tx_st_ready_o Behavior
4.3.1.5. Tag Allocation x
4.3.1.5.1. Completion Buffer Size
4.3.4. Interrupt Interface x
4.3.4.1. Legacy Interrupts 4.3.4.2. MSI 4.3.4.3. MSI-X
4.3.4.3. MSI-X x
4.3.4.3.1. Implementing MSI-X Interrupts
4.3.9. Power Management Interface x
4.3.9.1. D3Hot Entry 4.3.9.2. D3Hot Exit 4.3.9.3. D3Cold Entry 4.3.9.4. D3Cold Exit
4.3.9.2. D3Hot Exit x
4.3.9.2.1. D3Hot Exit Initiated by Host 4.3.9.2.2. D3Hot Exit Initiated by EP
4.4. PIPE Direct Mode x
4.4.1. Data Signals 4.4.2. Command and Status Signals 4.4.3. Message Bus Signals 4.4.4. Deskew Signals 4.4.5. PIPE Direct Reset Sequence 4.4.6. PIPE Direct Speed Change 4.4.7. PIPE Lane Reset Staggering 4.4.8. Unused Lanes in PIPE Direct Mode
4.4.1. Data Signals x
4.4.1.1. Transmit Signals 4.4.1.2. Receive Signals
4.4.4. Deskew Signals x
4.4.4.1. MAC to PHY (M2P) Signals 4.4.4.2. PHY to MAC (P2M) Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters 5.3. Design Examples
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Measurement (PTM)
5.3. Design Examples x
5.3.1. Design Example Files 5.3.2. Generated HDL Format 5.3.3. Target Development Kit 5.3.4. Currently Selected Design Example
6. Troubleshooting/Debugging x
6.1. Hardware Debug 6.2. Operating System Tools and Utilities 6.3. Signal Tap Logic Analyzer 6.4. Using the Hard IP Reconfiguration Interface for Debugging 6.5. Other Interfaces That Can Be Used for Debug Activities 6.6. Debug Toolkit
6.1. Hardware Debug x
6.1.1. Debugging Link Training Issues 6.1.2. Debugging Device Enumeration Issues 6.1.3. Debugging Data Transfer and Performance Issues
6.1.3. Debugging Data Transfer and Performance Issues x
6.1.3.1. Advanced Error Reporting (AER)
6.2. Operating System Tools and Utilities x
6.2.1. Using lspci Utility to Read Negotiated Link Speed
6.4. Using the Hard IP Reconfiguration Interface for Debugging x
6.4.1. Using the Hard IP Reconfiguration Interface to Enable and Read ECRC and LCRC Error Counts
6.6. Debug Toolkit x
6.6.1. Overview 6.6.2. Enabling the R-Tile Debug Toolkit 6.6.3. Launching the R-Tile Debug Toolkit 6.6.4. Using the R-Tile Debug Toolkit
6.6.4. Using the R-Tile Debug Toolkit x
6.6.4.1. R-Tile Information 6.6.4.2. Event Counters 6.6.4.3. Link Inspector 6.6.4.4. Configuration Space 6.6.4.5. Channel Parameters
6.6.4.5. Channel Parameters x
6.6.4.5.1. General PHY 6.6.4.5.2. Tx Path 6.6.4.5.3. Rx Path 6.6.4.5.4. Lane Margining
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TL Bypass Mode x
D.1. Upstream TL Bypass Mode D.2. Downstream TL Bypass Mode
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express x
E.1. Margin Masks Overview E.2. 5x5 Testing Efficiency E.3. Debug Toolkit Margin Methodology
E.2. 5x5 Testing Efficiency x
E.2.1. First Time Testing E.2.2. Spot Check Testing
E.3. Debug Toolkit Margin Methodology x
E.3.1. Debug Toolkit Testing Process
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. Troubleshooting/Debugging
7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
8. Document Revision History for the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TL Bypass Mode
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express

4.4.5. PIPE Direct Reset Sequence

In PIPE Direct mode, your application logic is responsible for managing most of the PHY reset sequence in the FPGA fabric. The following figure describes the required sequence.

Note:
In case you are using any of the PIPE Direct bundle modes (such as 2x8 or 4x4, refer to Topologies Supported by R-Tile) and there is no link partner connected in the lower 8 lanes, or your Soft IP Controller is only using the lanes 8-15, the following procedure is required for lane 0:
  1. Deassert ln0_pipe_direct_pld_pcs_rst_n_i as described in Step d of the reset sequence shown in PIPE Direct Reset Sequence.
  2. Set ln0_pipe_direct_powerdown_i = 2'b00, until any of the 8-15 lanes has reached Step f of the reset sequence shown in PIPE Direct Reset Sequence.
  3. After this, set ln0_pipe_direct_powerdown_i = 2'b11 as required in Unused Lanes in PIPE Direct Mode.
Figure 45. PIPE Direct Reset Release Sequence
Note: As per the PHY Interface for PCI Express Specification, Version 5.1.1, the TX data valid signal must be asserted whenever TX electrical idle toggles. The PHY will mask off the TX data as long as TX electrical idle is asserted.

Below are the steps required for the reset sequence and the TX/RX data transfer for lane 0 in the R-Tile Avalon Streaming IP when configured in PIPE-D mode. This behavior applies in the same way for other lanes.

1. ninit_done is driven low by the Reset Release IP indicating the FGPA fabric is configured. The Soft IP controller should be in reset until this signal is low.

2. pin_perst_n_o is driven high by the R-Tile Avalon Streaming IP. This signal reflects the PERST# signal at the board level.

3. lnX_pipe_direct_tx_transfer_en_o is driven high by the R-Tile Avalon Streaming IP indicating the EMIB bridge between the R-Tile Avalon Streaming IP and the FPGA fabric is ready.

4. The Soft IP controller drives high the per-lane

lnX_pipe_direct_pld_pcs_rst_n_i signal to signal reset release after the per-lane

lnX_pipe_direct_tx_transfer_en_o signal is asserted.

----

Step (a) : pipe_direct_pld_tx_clk_out_o becomes active as the TX clock output to be used by the Soft IP controller for the TX path.

Step (b) : lnX_pipe_direct_phystatus_o is driven low by the R-Tile Avalon Streaming IP indicating a reset exit.

Step (c) : The Soft IP controller drives lnX_pipe_direct_txdetectrx_i low.

Step (d) : lnX_pipe_direct_phystatus_o is pulsed to indicate receiver detection.

Step (e) : lnX_pipe_direct_rx_status_o is pulsed as well. Both pulses confirm to the Soft IP controller the receiver detection.

Step (f) : The soft IP controller changes lnX_pipe_direct_powerdown_i to P0.

Step (g) : lnX_pipe_direct_phystatus_o is pulsed to acknowledge power down state transition.

Step (h, i, j, k) : The soft IP controller asserts lnX_pipe_direct_txdatavalid0_i and lnX_pipe_direct_txdatavalid1_i. While the lnX_pipe_direct_txdatavalid0_i and lnX_pipe_direct_txdatavalid1_i are asserted, the soft IP controller deasserts lnX_pipe_direct_txelecidle_i. The soft IP controller de-asserts lnX_pipe_direct_txelecidle_i and starts sending valid data on lnX_pipe_direct_txdata while lnX_pipe_direct_txdatavalid0_i and lnX_pipe_direct_txdatavalid1_i are asserted

Step (l) : After TX data is transmitted from the Soft IP controller and once enough RX data has been received from the link partner to recover the clock, the lnX_pipe_direct_cdrlockstatus_o signal is driven high.

Step (m) : The lnX_pipe_direct_cdrlock2data_o signal is driven high indicating the CDR has locked to the data being received. The lnX_pipe_direct_rx_clk_out_o signal becomes active as the RX clock output to be used by the Soft IP controller for the RX data path.

Step (n) : The ln_pipe_direct_reset_status_n_o signal is driven high by the R-Tile Avalon Streaming IP indicating the RX data path is out of reset.

Step (o, p, q) : The Soft IP controller starts sampling data on the lnX_pipe_direct_pipe_rxdata_o while qualifying the data with its corresponding lnX_pipe_direct_rxdatavalid0_i and lnX_pipe_direct_rxdatavalid1_i signals.

Application logic needs to wait for the assertion of the corresponding lane's ln_pipe_direct_reset_status_n_o[15:0] to sample the Rx data. Refer to PIPE Direct RX Datapath for additional details.

Figure 46. PIPE Direct Reset Entry Sequence
Note: As per the PHY Interface for PCI Express Specification, Version 5.1.1, the TX data valid signal must be asserted whenever TX electrical idle toggles. The PHY will mask off the TX data as long as TX electrical idle is asserted.
Step (a) : The Soft IP controller initiates reset sequence by transitioning LTSSM State (ex. Into Detect state from hot reset, etc).
  • The soft IP controller drives lnX_pipe_direct_txelecidle_i high while lnX_pipe_direct_txdatavalid0_i and lnX_pipe_direct_txdatavalid1_i are high.
  • The soft IP controller drives lnX_pipe_direct_txdatavalid0_i, lnX_pipe_direct_txdatavalid1_i signals low after asserting lnX_pipe_direct_txelecidle_i.
  • The R-tile Avalon Streaming IP drives lnX_pipe_direct_rxelecidle_o signal high indicating the RX path is in electrical idle.
  • The R-tile Avalon Streaming IP drives lnX_pipe_direct_rxdatavalid0_o, lnX_pipe_direct_rxdatavalid1_o and lnX_pipe_direct_pipe_rxdata low after lnX_pipe_direct_rxelecidle_o signal high indicating no valid data is sent on the RX datapath.

Step (b) : The Soft IP controller drives the rate change signal to speed change to Gen1 after entering Detect state.

Step (c) : lnX_pipe_direct_phystatus_o is pulsed to acknowledge rate change to Gen1.

Step (d) : The soft IP controller changes lnX_pipe_direct_powerdown_i to P1 state from P0 state.

Step (e) : lnX_pipe_direct_phystatus_o is pulsed to acknowledge power state transition.

Step (f,g) : The soft IP controller drives lnX_pipe_direct_pld_pcs_rst_n_i low to transition the TX and RX paths into reset (minimum of 100ns).

Step (h) : lnX_pipe_direct_reset_status_n_o is driven low to indicate the R-tile Avalon Streaming IP RX path is in reset.

Level Two Title