R-tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 12/13/2021
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Introduction 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. R-tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 7. Document Revision History for the R-tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TLP Bypass Mode
1. Introduction x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core Support Levels
1.1. Overview x
1.1.1. Standards and Specifications Compliance
2. IP Architecture and Functional Description x
2.1. Overview 2.2. Bias Temperature Instability (BTI) Protection Mode 2.3. PCIe Hard IP Mode 2.4. PIPE Direct Mode
2.3. PCIe Hard IP Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PCI Express Protocol Stack
2.3.3. PCI Express Protocol Stack x
2.3.3.1. PMA/PCS 2.3.3.2. Data Link Layer Overview 2.3.3.3. Transaction Layer Overview
2.4. PIPE Direct Mode x
2.4.1. Clocking 2.4.2. Reset 2.4.3. PIPE Layer
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TLP Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TLP Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TLP Bypass Mode 3.3.3. User Avalon Memory-mapped Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TLP Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Overview 4.2. Clocks and Resets 4.3. Serial Data Interface 4.4. PCI Express Mode 4.5. PIPE Direct Mode
4.2. Clocks and Resets x
4.2.1. Clocks 4.2.2. Resets
4.4. PCI Express Mode x
4.4.1. Avalon® Streaming Interface 4.4.2. Precision Time Measurement (PTM) Interface 4.4.3. Interrupt Interface 4.4.4. Hard IP Reconfiguration Interface 4.4.5. Error Interface 4.4.6. Completion Timeout Interface 4.4.7. Configuration Intercept Interface 4.4.8. Power Management Interface 4.4.9. Hard IP Status Interface 4.4.10. Page Request Services (PRS) Interface 4.4.11. Function-Level Reset (FLR) Interface 4.4.12. SR-IOV VF Error Flag Interface 4.4.13. General Purpose VSEC Interface
4.4.1. Avalon® Streaming Interface x
4.4.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.4.1.2. Credit Control 4.4.1.3. Avalon® Streaming RX Interface 4.4.1.4. Avalon® Streaming TX Interface 4.4.1.5. Tag Allocation
4.4.1.2. Credit Control x
4.4.1.2.1. Credit Interface - Data and Header 4.4.1.2.2. Credit Initialization 4.4.1.2.3. Credit Update/Release 4.4.1.2.4. RX Flow Control Interface 4.4.1.2.5. TX Flow Control Interface
4.4.1.4. Avalon® Streaming TX Interface x
4.4.1.4.1. Avalon® Streaming TX Interface tx_st_ready_o Behavior
4.4.1.5. Tag Allocation x
4.4.1.5.1. Completion Buffer Size
4.4.3. Interrupt Interface x
4.4.3.1. Legacy Interrupts 4.4.3.2. MSI 4.4.3.3. MSI-X
4.4.3.3. MSI-X x
4.4.3.3.1. Implementing MSI-X Interrupts
4.4.8. Power Management Interface x
4.4.8.1. Endpoint D3Hot Entry 4.4.8.2. Endpoint D3Hot Exit
4.5. PIPE Direct Mode x
4.5.1. Data Signals 4.5.2. Message Channel 4.5.3. Deskew Channel 4.5.4. PIPE Direct Reset Sequence 4.5.5. PIPE Direct Speed Change 4.5.6. PIPE Lane Reset Staggering 4.5.7. Unused Lanes in PIPE Direct Mode 4.5.8. PIPE Direct Mode TX/RX Datapaths
4.5.1. Data Signals x
4.5.1.1. Transmit Signals 4.5.1.2. Receive Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Management (PTM)
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TLP Bypass Mode x
D.1. EP TLP Bypass Mode (Upstream) D.2. RC TLP Bypass Mode (Downstream)
1. Introduction
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. R-tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
7. Document Revision History for the R-tile Avalon® Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TLP Bypass Mode

4.5.3. Deskew Channel

In PIPE Direct mode, the PHY wrapper removes any lane-to-lane skew introduced while crossing the EMIB. A dedicated deskew marker is used for detecting and compensating for any multi-lane skew introduced by EMIB. The deskew logic can account for a maximum of up to 3 cycles of parallel skew. After a cold/warm/hot reset or CvP update, the deskew process will start.

The user application logic needs to send a deskew marker every 16 clock cycles for the purpose of deskewing the data on the EMIB channels. The PHY deskew logic will run the deskew process every time it receives the deskew marker.

Table 79.  EMIB TX Deskew Grouping for R-tile Topologies
PIPE Direct Tx Deskew Bundle Octet 1 Octet 0
Lane 15 Lane 14 Lane 13 Lane 12 Lane 11 Lane 10 Lane 9 Lane 8 Lane 7 Lane 6 Lane 5 Lane 4 Lane 3 Lane 2 Lane 1 Lane 0
1X16 Octet1_Dsk_0 Octet0_Dsk_0
2X8 Octet1_Dsk_0 Octet0_Dsk_0
4X4 Octet1_Dsk_2 Octet1_Dsk_0 Octet0_Dsk_2 Octet0_Dsk_0
8X2 Octet1_Dsk_3 Octet1_Dsk_2 Octet1_Dsk_1 Octet1_Dsk_0 Octet0_Dsk_3 Octet0_Dsk_2 Octet0_Dsk_1 Octet0_Dsk_0
16X1 No Tx Deskew
2X4; 1X8 Octet1_Dsk_2 Octet1_Dsk_0 Octet0_Dsk_0
4X2; 1X8 Octet1_Dsk_3 Octet1_Dsk_2 Octet1_Dsk_1 Octet1_Dsk_0 Octet0_Dsk_0
8X1; 1X8 No Tx Deskew Octet0_Dsk_0
1X8; 2X4 Octet1_Dsk_0 Octet0_Dsk_2 Octet0_Dsk_0
4X2; 2X4 Octet1_Dsk_3 Octet1_Dsk_2 Octet1_Dsk_1 Octet1_Dsk_0 Octet0_Dsk_2 Octet0_Dsk_0
8X1; 2X4 No Tx Deskew Octet0_Dsk_0 Octet0_Dsk_0
1X8; 4X2 Octet1_Dsk_0 Octet0_Dsk_3 Octet0_Dsk_2 Octet0_Dsk_1 Octet0_Dsk_0
2X4; 4X2 Octet1_Dsk_2 Octet1_Dsk_0 Octet0_Dsk_3 Octet0_Dsk_2 Octet0_Dsk_1 Octet0_Dsk_0
8X1; 4X2 No Tx Deskew Octet0_Dsk_3 Octet0_Dsk_2 Octet0_Dsk_1 Octet0_Dsk_0
1X8; 8X1 Octet1_Dsk_0 No Tx Deskew
2X4; 8X1 Octet1_Dsk_2 Octet1_Dsk_0 No Tx Deskew
4X2; 8X1 Octet1_Dsk_3 Octet1_Dsk_2 Octet1_Dsk_1 Octet1_Dsk_0 No Tx Deskew

MAC to PHY (M2P) Signals

Table 80.  PIPE Direct EMIB Control Deskew Channel M2P Signals
Signal Name Direction Descriptions/Notes Clock Domain
lnX_pipe_direct_deskew_clear_0/1/2/3_i Input When asserted, the current tx_dsk_eval_done and tx_dsk_status are cleared. A new deskew evaluation is expected after the current status is cleared. This signal is asserted for two clock cycles. pipe_direct_pld_tx_clk_out_o

PHY to MAC (P2M) Signals

Table 81.  PIPE Direct EMIB Control Deskew Channel P2M Signals
Signal Name Direction Descriptions/Notes Clock Domain
lnX_pipe_direct_txdeskewmarker_i Input Tx Deskew marker used to deskew EMIB routings per bundle mode. This is a simple repeating pulse that provides a protocol-agnostic mechanism to detect EMIB channel skew and perform alignment. The marker fans out and appears on all bundle channels simultaneously once every 16 clock cycles. The deskew module looks for the deskew marker from each EMIB channel and adds delays on the early channels to compensate for the delays of the late channels. pipe_direct_pld_tx_clk_out_o
octet#_pipe_direct_phy_dsk_active_chans_o Output Indicates which channels received a deskew marker. pipe_direct_pld_tx_clk_out_o
octet#_pipe_direct_phy_dsk_monitor_err_o Output

Value is latched upon an error, and held until the state machine is restarted via i_dsk_clear or async reset.

Monitor this signal only after 16 pclk (pipe_direct_pld_tx_clk_out_o) cycles following the assertion of octet#_pipe_direct_phy_dsk_eval_done_[3:0]_o.

pipe_direct_pld_tx_clk_out_o
octet#_pipe_direct_phy_dsk_monitor_err_status_[3:0]_o Output

Indicates a deskew monitor error.

Monitor this signal only after 16 pclk (pipe_direct_pld_tx_clk_out_o) cycles following the assertion of octet#_pipe_direct_phy_dsk_eval_done_[3:0]_o.

pipe_direct_pld_tx_clk_out_o
octet#_pipe_direct_phy_dsk_status_[3:0]_o Output

Indicates the deskew evaluation result.

Monitor this signal only after 16 pclk (pipe_direct_pld_tx_clk_out_o) cycles following the assertion of octet#_pipe_direct_phy_dsk_eval_done_[3:0]_o.

Note: octet#_pipe_direct_phy_dsk_status should be monitored only after octet#_pipe_direct_phy_dsk_valid is asserted.
pipe_direct_pld_tx_clk_out_o
octet#_pipe_direct_phy_dsk_valid_[3:0]_o Output

Indicates the deskew operation status.

When using x16, the octet#_pipe_direct_phy_dsk_valid_o from each of the octets must be ANDed together by the user logic.

pipe_direct_pld_tx_clk_out_o
octet#_pipe_direct_phy_dsk_eval_done_[3:0]_o Output Indicates the deskew process is complete. This signal is for debugging purpose. When using x16, the octet#_pipe_direct_phy_dsk_eval_done_o from each of the octets must be ANDed together by the user logic. pipe_direct_pld_tx_clk_out_o
To use the deskew interface, follow these steps:
  1. The controller in the application logic sends the deskew marker for each lane of the bundle every 16 pclk (pipe_direct_pld_tx_clk_out_o) cycles using the signal ln*_pipe_direct_txdeskewmarker_i.
  2. After the data from the EMIB is deskewed, octet*_pipe_direct_phy_dsk_valid_o is asserted, indicating deskew done status.
    Note: (*) When using x16, the octet*_pipe_direct_phy_dsk_valid_o from each of the octets must be ANDed together.
  3. In addition to the octet*_pipe_direct_phy_dsk_valid_o signals, the PIPE interface provides octet*_pipe_direct_phy_dsk_eval_done_o and octet*_pipe_direct_phy_dsk_status_*_o signals to show the details of the deskew status.
    Note: (#) These signals are for debugging purposes only. The user application logic should rely only on the octet*_pipe_direct_phy_dsk_valid_o signals.
  4. The octet*_pipe_direct_deskew_clear_i signals on both octets can be used to clear the current deskew state to allow additional deskew evaluations. When using x16, the octet*_pipe_direct_deskew_clear_i for each of the octets must be used.
  5. After the pulse on octet*_pipe_direct_deskew_clear_i, the deskew state on octet*_pipe_direct_phy_dsk_monitor_err_o bus is cleared.
Level Two Title