R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683501
Date 12/06/2024
Public
Document Table of Contents
Document Table of Contents x
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express 2. IP Architecture and Functional Description 3. Advanced Features 4. Interfaces 5. Parameters 6. Troubleshooting/Debugging 7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives 8. Document Revision History for the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide A. Configuration Space Registers B. Root Port Enumeration C. Implementation of Address Translation Services (ATS) in Endpoint Mode D. Packets Forwarded to the User Application in TL Bypass Mode E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express x
1.1. Glossary 1.2. Overview 1.3. Features 1.4. Release Information 1.5. Device Family Support 1.6. Performance and Resource Utilization 1.7. IP Core Support Levels
1.2. Overview x
1.2.1. Standards and Specifications Compliance
1.3. Features x
1.3.1. Multifunction and Virtualization Features
2. IP Architecture and Functional Description x
2.1. Overview 2.2. PCIe Hard IP Mode 2.3. PIPE Direct Mode
2.2. PCIe Hard IP Mode x
2.2.1. Clocking 2.2.2. Reset 2.2.3. PCI Express Protocol Stack
2.2.2. Reset x
2.2.2.1. Single PERST 2.2.2.2. Independent PERST Pins 2.2.2.3. Independent GPIO PERST
2.2.2.2. Independent PERST Pins x
2.2.2.2.1. REFCLK and PERST Guidelines When Using Independent PERST Pins
2.2.2.3. Independent GPIO PERST x
2.2.2.3.1. REFCLK and PERST Guidelines when Using Independent GPIO PERST
2.2.3. PCI Express Protocol Stack x
2.2.3.1. PMA 2.2.3.2. Data Link Layer Overview 2.2.3.3. Transaction Layer Overview
2.2.3.3. Transaction Layer Overview x
2.2.3.3.1. Deferrable Memory Write (DMWr)
2.3. PIPE Direct Mode x
2.3.1. Clocking 2.3.2. Reset 2.3.3. PIPE Layer
2.3.3. PIPE Layer x
2.3.3.1. Preset Mappings
3. Advanced Features x
3.1. PCIe Port Bifurcation and PHY Channel Mapping 3.2. Virtualization Support 3.3. TL Bypass Mode
3.2. Virtualization Support x
3.2.1. SR-IOV Support 3.2.2. VirtIO Support
3.2.1. SR-IOV Support x
3.2.1.1. SR-IOV Supported Features List 3.2.1.2. Implementation 3.2.1.3. VF to PF Mapping 3.2.1.4. Function Level Reset (FLR)
3.2.1.2. Implementation x
3.2.1.2.1. VF Error Flag Interface (for x16/x8 Cores Only)
3.2.2. VirtIO Support x
3.2.2.1. VirtIO Supported Features List 3.2.2.2. Overview 3.2.2.3. Parameters 3.2.2.4. VirtIO PCI Configuration Access Interface 3.2.2.5. Registers
3.2.2.5. Registers x
3.2.2.5.1. VirtIO Common Configuration Capability Register (Address: 0x012) 3.2.2.5.2. VirtIO Common Configuration BAR Indicator Register (Address: 0x013) 3.2.2.5.3. VirtIO Common Configuration BAR Offset Register (Address: 0x014) 3.2.2.5.4. VirtIO Common Configuration Structure Length Register (Address 0x015) 3.2.2.5.5. VirtIO Notifications Capability Register (Address: 0x016) 3.2.2.5.6. VirtIO Notifications BAR Indicator Register (Address: 0x017) 3.2.2.5.7. VirtIO Notifications BAR Offset Register (Address: 0x018) 3.2.2.5.8. VirtIO Notifications Structure Length Register (Address: 0x019) 3.2.2.5.9. VirtIO Notifications Notify Off Multiplier Register (Address: 0x01A) 3.2.2.5.10. VirtIO ISR Status Capability Register (Address: 0x02F) 3.2.2.5.11. VirtIO ISR Status BAR Indicator Register (Address: 0x030) 3.2.2.5.12. VirtIO ISR Status BAR Offset Register (Address: 0x031) 3.2.2.5.13. VirtIO ISR Status Structure Length Register (Address: 0x032) 3.2.2.5.14. VirtIO Device Specific Capability Register (Address: 0x033) 3.2.2.5.15. VirtIO Device Specific BAR Indicator Register (Address: 0x034) 3.2.2.5.16. VirtIO Device Specific BAR Offset Register (Address 0x035) 3.2.2.5.17. VirtIO Device Specific Structure Length Register (Address: 0x036) 3.2.2.5.18. VirtIO PCI Configuration Access Capability Register (Address: 0x037) 3.2.2.5.19. VirtIO PCI Configuration Access BAR Indicator Register (Address: 0x038) 3.2.2.5.20. VirtIO PCI Configuration Access BAR Offset Register (Address: 0x039) 3.2.2.5.21. VirtIO PCI Configuration Access Structure Length Register (Address: 0x03A) 3.2.2.5.22. VirtIO PCI Configuration Access Data Register (Address: 0x03B)
3.3. TL Bypass Mode x
3.3.1. Overview 3.3.2. Register Settings for the TL Bypass Mode 3.3.3. Hard IP Reconfiguration Interface 3.3.4. Avalon® Streaming Interface
3.3.2. Register Settings for the TL Bypass Mode x
3.3.2.1. TLPBYPASS_ERR_EN (Address 0x1314) 3.3.2.2. TLPBYPASS_ERR_STATUS (Address 0x1310)
3.3.4. Avalon® Streaming Interface x
3.3.4.1. Configuration TLP 3.3.4.2. Transmit Interface 3.3.4.3. Receive Interface 3.3.4.4. Malformed TLP 3.3.4.5. ECRC
4. Interfaces x
4.1. Clocks and Resets 4.2. Serial Data Interface 4.3. PCI Express Mode 4.4. PIPE Direct Mode
4.1. Clocks and Resets x
4.1.1. Clocks 4.1.2. Resets
4.3. PCI Express Mode x
4.3.1. Avalon® Streaming Interface 4.3.2. Precision Time Measurement (PTM) Interface (Endpoint Only) 4.3.3. Hot Plug Interface 4.3.4. Interrupt Interface 4.3.5. Hard IP Reconfiguration Interface 4.3.6. Error Interface 4.3.7. Completion Timeout Interface 4.3.8. Configuration Intercept Interface 4.3.9. Power Management Interface 4.3.10. Hard IP Status Interface 4.3.11. Page Request Services (PRS) Interface (Endpoint Only) 4.3.12. Function-Level Reset (FLR) Interface (Endpoint Only) 4.3.13. SR-IOV VF Error Flag Interface (Endpoint Only) 4.3.14. General Purpose VSEC Interface
4.3.1. Avalon® Streaming Interface x
4.3.1.1. TLP Header and Data Alignment for the Avalon® streaming RX and TX Interfaces 4.3.1.2. Credit Control 4.3.1.3. Avalon® Streaming RX Interface 4.3.1.4. Avalon® Streaming TX Interface 4.3.1.5. Tag Allocation
4.3.1.2. Credit Control x
4.3.1.2.1. Credit Interface - Data and Header 4.3.1.2.2. Credit Initialization 4.3.1.2.3. Credit Update/Release 4.3.1.2.4. RX Flow Control Interface 4.3.1.2.5. TX Flow Control Interface
4.3.1.3. Avalon® Streaming RX Interface x
4.3.1.3.1. Application Logic Guidelines for the Avalon Streaming RX Interface
4.3.1.4. Avalon® Streaming TX Interface x
4.3.1.4.1. Application Logic Guidelines for the Avalon Streaming TX Interface 4.3.1.4.2. Avalon® Streaming TX Interface pX_tx_st_ready_o Behavior
4.3.1.5. Tag Allocation x
4.3.1.5.1. Completion Buffer Size
4.3.4. Interrupt Interface x
4.3.4.1. Legacy Interrupts 4.3.4.2. MSI 4.3.4.3. MSI-X
4.3.4.3. MSI-X x
4.3.4.3.1. Implementing MSI-X Interrupts
4.3.9. Power Management Interface x
4.3.9.1. D3Hot Entry 4.3.9.2. D3Hot Exit 4.3.9.3. D3Cold Entry 4.3.9.4. D3Cold Exit
4.3.9.2. D3Hot Exit x
4.3.9.2.1. D3Hot Exit Initiated by Host 4.3.9.2.2. D3Hot Exit Initiated by EP
4.4. PIPE Direct Mode x
4.4.1. Data Signals 4.4.2. Command and Status Signals 4.4.3. Message Bus Signals 4.4.4. Deskew Signals 4.4.5. PIPE Direct Reset Sequence 4.4.6. PIPE Direct Speed Change 4.4.7. PIPE Lane Reset Staggering 4.4.8. Unused Lanes in PIPE Direct Mode
4.4.1. Data Signals x
4.4.1.1. Transmit Signals 4.4.1.2. Receive Signals
4.4.4. Deskew Signals x
4.4.4.1. MAC to PHY (M2P) Signals 4.4.4.2. PHY to MAC (P2M) Signals
5. Parameters x
5.1. Top-Level Settings 5.2. Core Parameters 5.3. Design Examples
5.2. Core Parameters x
5.2.1. Avalon Parameters 5.2.2. Base Address Registers 5.2.3. PCI Express and PCI Capabilities Parameters
5.2.3. PCI Express and PCI Capabilities Parameters x
5.2.3.1. Device Capabilities 5.2.3.2. VirtIO Parameters 5.2.3.3. Link Capabilities 5.2.3.4. Legacy Interrupt Pin Register 5.2.3.5. MSI Capabilities 5.2.3.6. MSI-X Capabilities 5.2.3.7. Slot Capabilities 5.2.3.8. Latency Tolerance Reporting (LTR) 5.2.3.9. Process Address Space ID (PASID) 5.2.3.10. Device Serial Number Capability 5.2.3.11. Page Request Service (PRS) 5.2.3.12. Access Control Service (ACS) 5.2.3.13. Power Management 5.2.3.14. Vendor Specific Extended Capability (VSEC) Registers 5.2.3.15. TLP Processing Hints (TPH) 5.2.3.16. Address Translation Services (ATS) Capabilities 5.2.3.17. Precision Time Measurement (PTM)
5.3. Design Examples x
5.3.1. Design Example Files 5.3.2. Generated HDL Format 5.3.3. Target Development Kit 5.3.4. Currently Selected Design Example
6. Troubleshooting/Debugging x
6.1. Hardware Debug 6.2. Operating System Tools and Utilities 6.3. Signal Tap Logic Analyzer 6.4. Using the Hard IP Reconfiguration Interface for Debugging 6.5. Other Interfaces That Can Be Used for Debug Activities 6.6. Debug Toolkit
6.1. Hardware Debug x
6.1.1. Debugging Link Training Issues 6.1.2. Debugging Device Enumeration Issues 6.1.3. Debugging Data Transfer and Performance Issues
6.1.3. Debugging Data Transfer and Performance Issues x
6.1.3.1. Advanced Error Reporting (AER)
6.2. Operating System Tools and Utilities x
6.2.1. Using lspci Utility to Read Negotiated Link Speed
6.4. Using the Hard IP Reconfiguration Interface for Debugging x
6.4.1. Using the Hard IP Reconfiguration Interface to Enable and Read ECRC and LCRC Error Counts
6.6. Debug Toolkit x
6.6.1. Overview 6.6.2. Enabling the R-Tile Debug Toolkit 6.6.3. Launching the R-Tile Debug Toolkit 6.6.4. Using the R-Tile Debug Toolkit
6.6.4. Using the R-Tile Debug Toolkit x
6.6.4.1. R-Tile Information 6.6.4.2. Event Counters 6.6.4.3. Link Inspector 6.6.4.4. Configuration Space 6.6.4.5. Channel Parameters
6.6.4.5. Channel Parameters x
6.6.4.5.1. General PHY 6.6.4.5.2. Tx Path 6.6.4.5.3. Rx Path 6.6.4.5.4. Lane Margining
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
D. Packets Forwarded to the User Application in TL Bypass Mode x
D.1. Upstream TL Bypass Mode D.2. Downstream TL Bypass Mode
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express x
E.1. Margin Masks Overview E.2. 5x5 Testing Efficiency E.3. Debug Toolkit Margin Methodology
E.2. 5x5 Testing Efficiency x
E.2.1. First Time Testing E.2.2. Spot Check Testing
E.3. Debug Toolkit Margin Methodology x
E.3.1. Debug Toolkit Testing Process
1. About the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express
2. IP Architecture and Functional Description
3. Advanced Features
4. Interfaces
5. Parameters
6. Troubleshooting/Debugging
7. R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide Archives
8. Document Revision History for the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide
A. Configuration Space Registers
B. Root Port Enumeration
C. Implementation of Address Translation Services (ATS) in Endpoint Mode
D. Packets Forwarded to the User Application in TL Bypass Mode
E. Margin Masks for the R-Tile Avalon Streaming Intel FPGA IP for PCI Express

8. Document Revision History for the R-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Document Version Quartus® Prime Version IP Version Changes
2024.12.06 24.3 11.4.0 Fixed the configuration descriptions in Lane Margining.
2024.10.07 24.3 11.4.0
  • Updated the table of supported features in SR-IOV Supported Features List.
  • Updated the signal names in VF Error Flag Interface (for x16/x8 Cores Only).
  • Updated the signal names and timing diagram in PIPE Direct Reset Sequence.
  • Updated the timing diagram in PIPE Direct Speed Change.
2024.07.08 24.2 11.3.0
  • Added an example of BDF translation to the Avalon Streaming TX Interface section.
  • Fixed the directions of some signals in the PIPE Direct Reset Entry Sequence diagram under the PIPE Direct Reset Sequence section.
2024.04.10 24.1 11.2.0
  • Updated values in the Preset Mappings section.
  • Updated the timing diagram and text in the PIPE Direct Reset Sequence section.
  • Updated the timing diagram and text in the PIPE Direct Speed Change section.
  • Updated TLP Prefix, Header and Data when PCIe Header Format Checkbox is Disabled in the TLP Header and Data Alignment for the Avalon streaming RX and TX Interfaces section.
2023.12.04 23.4 11.1.0
  • Updated the description in the REFCLK and PERST Guidelines When Using Independent PERST Pins section.
  • Updated the refclk0, refclk1, refclk2 description in the Clocks section.
  • Added a description of the behavior of the pX_rx_stN_dvalid_o signal to the Application Logic Guidelines for the Avalon Streaming RX Interface section.
  • Updated the signal descriptions in the Power Management Interface section.
  • Updated the description in the Margin Masks Overview section.
2023.10.02 23.3 11.0.0
  • Updated the list of devices or OPNs supported for various features throughout the document.
  • Updated the screenshots in the Debug Toolkit section.
  • Updated some section headings in the Appendix Packets Forwarded to the User Application in TL Bypass Mode.
2023.06.26 23.2 10.0.0
  • Updated the Clocking section to include connection requirements for the input clocks refclk0, refclk1, and refclk2 for different Configuration Modes and different values of the Enable Independent Perst Pins parameter.
  • Added the Independent Perst Pins sub-section to the Reset section.
  • Added the requirement of using the Hard IP Reconfiguration Interface to access TL Bypass registers to the Register Settings for the TL Bypass Mode section.
  • Added some implementation considerations to the Precision Time Measurement (PTM) Interface (Endpoint Only) section.
  • Added the Enable Debug Toolkit, Enable Warm Perst, Enable Independent GPIO Perst, and Enable Independent Perst Pins parameters to the Top-Level Settings section. Updated corresponding screenshots of the IP Parameter Editor.
  • Updated IP Parameter Editor screenshots in the Core Parameters section.
2023.04.03 23.1 9.0.0
  • Updated product family name to "Agilex 7".
  • Updated all references to OPN numbers to differentiate between AGI OPNs and AGM OPNs.
  • Removed the section Bias Temperature Instability (BTI) Protection Mode.
  • Changed the section PMA/PCS to PMA since the PCS responsibilities are implemented by the MAC layer. Also changed all references to PCS to MAC, and some instances of PHY to PMA.
  • Updated the description of the pX_tx_st_ready_o signal in the Avalon Streaming TX Interface section.
  • Updated the guidelines in the Application Logic Guidelines for the Avalon Streaming TX Interface section.
  • Added example cases to the Avalon Streaming TX Interface pX_tx_st_ready_o Behavior section.
  • Added a Note on temperature monitoring to the Hard IP Reconfiguration Interface section.
  • Added a Note on temperature monitoring to the Overview section under the Debug Toolkit section.
  • Added a Note about the PHY Reconfiguration interface being automatically enabled to the Enabling the R-Tile Debug Toolkit section.
  • Added a Note about the availability of the Channel Parameters tab to the Channel Parameters section.
2022.12.19 22.4 8.0.0
  • Added guidelines for the Application logic to the Avalon Streaming RX Interface and Avalon Streaming TX Interface sections.
  • Moved the Enable the PHY Reconfiguration Interface from the Top-Level Settings section to the Avalon Parameters section to match the 22.4 IP Parameter Editor.
  • Updated screenshots in the Core Parameters section to match the 22.4 IP Parameter Editor.
  • Added the Link Inspector and the Channel Parameters sections.
2022.10.07 22.3 7.0.0 Updated the links to the configuration space register maps in the Configuration Space Registers section.
2022.09.26 22.3 7.0.0
  • Updated the information in the table Agilex Recommended FPGA Fabric Speed Grades for All Avalon Streaming Widths and Frequencies.
  • Added guidelines that the Application logic must follow to the Avalon Streaming RX Interface section.
  • Added guidelines that the Application logic must follow to the Avalon Streaming TX Interface section.
  • Updated the Troubleshooting/Debugging chapter to:
    • Add the information regarding hardware debugging, specifically how to debug link training issues, device enumeration issues, and performance and data transfer issues.
    • Add the list of top-level signals that can be monitored via the Signal Tap Logic Analyzer.
    • Add the list of interfaces that can be used for debug activities.
2022.06.20 22.2 6.0.0
  • Updated the Debug Toolkit section to document the additional capabilities for users to verify in hardware the status of the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express.
  • Documented the new ports pX_cold_perst_n_i, pX_warm_perst_n_i, and pX_ip_rst_n_o.
  • Updated the Independent PERST section with new timing diagrams showing the behaviors of the pX_cold_perst_n_i and pX_warm_perst_n_i signals.
  • Updated the Hard IP Reconfiguration Interface section with a new timing diagram showing the behavior of the signals on this interface while doing a read of the configuration space registers.

2022.03.28 22.1 5.0.0
  • Removed Questa* FPGA Edition from the list of supported simulators in the Features section.
  • Updated the FPGA fabric speed grades table in the Performance and Resource Utilization section.
  • Updated the application clock frequencies table in the Clocking section.
  • Updated the list of PCI Express Capability Structure registers that need to be implemented in the Application logic in the Hard IP Reconfiguration Interface.
  • Updated the tables in the Completion Buffer Size section.
  • Added the subsections D3Hot Exit Initiated by Host, D3Hot Exit Initiated by EP, D3Cold Entry and D3Cold Exit to the Power Management Interface section.
  • Modified the list of steps given in the PIPE Direct Reset Sequence section.
  • Modified the description given in the PIPE Direct Speed Change section.
2021.12.13 21.4 4.0.0
  • Added the new section BTI Protection Mode.
  • Updated the Completion Buffer Size section to show the correct buffer sizes. Also added examples showing the amount of Completion buffer entries consumed for Memory Read requests. Finally, added a suggested flow for the Application logic to track the completion buffer entries and based on this, schedule Non-Posted (NP) requests to the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express.
  • Updated the text descriptions and timing waveforms for the Avalon Streaming RX Interface and the Avalon Streaming TX Interface to show how the user application logic can properly use these interfaces.
  • Updated the signal descriptions and timing waveforms for the Deskew Channel section to show how the user application logic can properly use this interface.
2021.10.06 21.3 3.0.0 Removed the section ECRC due to missing information on the register offsets for the ECRC or LCRC counters.
2021.10.04 21.3 3.0.0
  • Updated the block diagrams in the PCI Express Mode and PIPE Direct Mode sections to match the interface signals on the 21.3 block symbol for the R-Tile Avalon® Streaming Intel FPGA IP for PCI Express.
  • Added a Note to the Avalon Streaming TX Interface section stating that for this interface, the SOP can only be sent on segments 0 and 2.
  • Added the Root Port Enumeration Appendix chapter.
2021.07.12 21.2 2.0.0 Initial release.
Level Two Title