GTS AXI Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 813754
Date 11/04/2024
Public
Document Table of Contents
Document Table of Contents x
1. Introduction 2. Features 3. Getting Started with GTS AXI Streaming IP 4. IP Architecture and Functional Description 5. IP Parameters 6. Interfaces and Signals 7. Registers 8. Document Revision History for the GTS AXI Streaming Intel® FPGA IP for PCI Express* User Guide A. Troubleshooting/Debugging B. PIPE Mode Simulation C. Implementation of Address Translation Services (ATS) in Endpoint Mode
1. Introduction x
1.1. About PCI Express* 1.2. Goal of the GTS AXI Streaming Intel® FPGA IP for PCI Express* User Guide 1.3. Example Use Models
2. Features x
2.1. Supported Features 2.2. IP Support Status 2.3. Release Information 2.4. Recommended FPGA Fabric Speed Grades 2.5. Resource Utilization 2.6. IP Core and Design Example Support Levels
3. Getting Started with GTS AXI Streaming IP x
3.1. Downloading and Installing Quartus® Prime Software 3.2. Configuring and Generating the GTS AXI Streaming IP 3.3. Configuring and Generating GTS System PLL Clocks Intel® FPGA IP 3.4. Configuring and Generating GTS Reset Sequencer Intel® FPGA IP 3.5. Instantiating and Connecting GTS AXI Streaming IP Interfaces (and Other IPs) 3.6. Simulating the GTS AXI Streaming IP Variant 3.7. Compiling the GTS AXI Streaming IP Variant
4. IP Architecture and Functional Description x
4.1. Clocking 4.2. Resets 4.3. PCIe* Hard IP 4.4. Hard IP Interface (IF) Adaptor 4.5. Interrupts 4.6. Transaction Ordering 4.7. TX Non-Posted Metering Requirement on Application 4.8. AXI4-Stream Interface 4.9. Tag Allocation 4.10. Power Management 4.11. Config Retry Status Enable 4.12. Hot-Plug 4.13. Configuration Space Extension 4.14. Page Request Service (EP only) 4.15. Precision Time Measurement (PTM) 4.16. Single Root I/O Virtualization (SR-IOV) 4.17. Transaction Layer Packet (TLP) Bypass Mode 4.18. Scalable IOV
4.1. Clocking x
4.1.1. Reference Clock
4.5. Interrupts x
4.5.1. Legacy Interrupt 4.5.2. Message Signaled Interrupts (MSI) 4.5.3. Message Signal Interrupt Extended (MSI-X)
4.8. AXI4-Stream Interface x
4.8.1. Header Format 4.8.2. Prefix Format 4.8.3. Function Number Format 4.8.4. BAR Number Format 4.8.5. Data and Header Packing Scheme
4.16. Single Root I/O Virtualization (SR-IOV) x
4.16.1. SR-IOV Implementation
4.17. Transaction Layer Packet (TLP) Bypass Mode x
4.17.1. AXI-Lite Usage for TLP Bypass Mode 4.17.2. Configuration TLP 4.17.3. Transmit Interface 4.17.4. Receive Interface 4.17.5. Malformed TLP 4.17.6. ECRC
5. IP Parameters x
5.1. System Settings 5.2. PCIe0/PCIe1 5.3. Diagnostics
5.2. PCIe0/PCIe1 x
5.2.1. Interfaces Port 0/Interfaces Port 1 5.2.2. PCIe0/PCIe1 Settings
5.2.1. Interfaces Port 0/Interfaces Port 1 x
5.2.1.1. AXI Interfaces 0/AXI Interface 1 Settings
5.2.2. PCIe0/PCIe1 Settings x
5.2.2.1. PCIe0/PCIe1 Base Address Registers 5.2.2.2. PCIe0/PCIe1 Device Identification Registers 5.2.2.3. PCIe0/PCIe1 PCI Express / PCI Capabilities
5.2.2.3. PCIe0/PCIe1 PCI Express / PCI Capabilities x
5.2.2.3.1. PCIe0/PCIe1 Device 5.2.2.3.2. PCIe0/PCIe1 Link 5.2.2.3.3. PCIe0/PCIe1 Slot 5.2.2.3.4. PCIe0/PCIe1 Legacy Interrupt Pin Register 5.2.2.3.5. PCIe0/PCIe1 PTM 5.2.2.3.6. PCIe0/PCIe1 LTR 5.2.2.3.7. PCIe0/PCIe1 MSI 5.2.2.3.8. PCIe0/PCIe1 MSI-X 5.2.2.3.9. PCIe0/PCIe1 PASID 5.2.2.3.10. PCIe0/PCIe1 DEV SER 5.2.2.3.11. PCIe0/PCIe1 PRS 5.2.2.3.12. PCIe0/PCIe1 Power Management 5.2.2.3.13. PCIe0/PCIe1 VSEC 5.2.2.3.14. PCIe0/PCIe1 ATS 5.2.2.3.15. PCIe0/PCIe1 TPH 5.2.2.3.16. PCIe0/PCIe1 ACS 5.2.2.3.17. PCIe0/PCIe1 Hot-Plug 5.2.2.3.18. PCIe0/PCIe1 VIRTIO
5.2.2.3.18. PCIe0/PCIe1 VIRTIO x
5.2.2.3.18.1. PCIe0/PCIe1 PF<n> VIRTIO STRUCTURES
6. Interfaces and Signals x
6.1. Overview 6.2. Clocks and Resets 6.3. AXI4-Stream Interfaces 6.4. Configuration Intercept Interface 6.5. Configuration Extension Bus (CEB) Interface 6.6. Control Shadow Interface 6.7. Transmit Flow Control Credit Interface 6.8. Completion Timeout Interface 6.9. Control and Status Register Responder Interface 6.10. Function Level Reset Interface 6.11. TLP Bypass Error Reporting Interface 6.12. Error Interface 6.13. VF Error Flag Interface 6.14. VIRTIO PCI* Configuration Access Interface 6.15. Precision Time Measurement (PTM) Interface 6.16. Serial Data Signals 6.17. Miscellaneous Signals
6.2. Clocks and Resets x
6.2.1. Interface Clock Signals 6.2.2. Interface Reset Signals
6.3. AXI4-Stream Interfaces x
6.3.1. AXI4-Stream Receive (RX) Interface 6.3.2. AXI4-Stream Transmit (TX) Interface
6.4. Configuration Intercept Interface x
6.4.1. Configuration Intercept Request Interface 6.4.2. Configuration Intercept Response Interface
6.5. Configuration Extension Bus (CEB) Interface x
6.5.1. Configuration Extension Bus Request Interface 6.5.2. Configuration Extension Bus Response Interface
6.10. Function Level Reset Interface x
6.10.1. Function Level Reset Received Interface 6.10.2. Function Level Reset Completion Interface
6.14. VIRTIO PCI* Configuration Access Interface x
6.14.1. VIRTIO PCI Configuration Access Request Interface 6.14.2. VIRTIO PCI Configuration Access Completion Interface
7. Registers x
7.1. Register Address Map 7.2. PCI Express* Configuration Space 7.3. HIP Port and Status Registers 7.4. Soft Register Address Map 7.5. Indirect Register Access 7.6. Virtual Function (VF) PCI Express* Configuration Space 7.7. Intel-Defined VSEC Capability Register
7.2. PCI Express* Configuration Space x
7.2.1. PCI Express* Configuration Header Registers 7.2.2. PCI Express* Capability Structures 7.2.3. Physical Layer 16.0 GT/s Extended Capability Structures
7.3. HIP Port and Status Registers x
7.3.1. Root Port Interrupt Registers 7.3.2. PTM Registers
7.4. Soft Register Address Map x
7.4.1. Control Registers 7.4.2. Debug Registers 7.4.3. Performance Monitor Registers
7.6. Virtual Function (VF) PCI Express* Configuration Space x
7.6.1. VF PCI-Compatible Configuration Space Header Type0 7.6.2. VF PCI Express* Capability Structure 7.6.3. VF Message Signal Interrupt Extended (MSI-X) Capability Structure 7.6.4. VF Alternative Routing ID (ARI) Capability Structure 7.6.5. VF TLP Processing Hints (TPH) Capability Structure 7.6.6. VF Address Translation Services (ATS) Capability Structure 7.6.7. VF Access Control Services (ACS) Capability Structure
7.6.1. VF PCI-Compatible Configuration Space Header Type0 x
7.6.1.1. Vendor ID and Device ID Registers 7.6.1.2. Command Registers 7.6.1.3. Status Registers 7.6.1.4. Revision ID and Class Code Register 7.6.1.5. BIST, Header Type, Latency Timer, and Cache Line Size Registers 7.6.1.6. Subsystem Vendor ID and Subsystem ID Register 7.6.1.7. Capabilities Pointer
7.6.2. VF PCI Express* Capability Structure x
7.6.2.1. PCI Express* Capability List Register 7.6.2.2. PCI Express* Device Capabilities Register 7.6.2.3. PCI Express* Device Control and Status Register 7.6.2.4. Link Capabilities Register 7.6.2.5. Link Control and Status Register 7.6.2.6. PCI Express* Device Capabilities 2 Register 7.6.2.7. PCI Express* Device Control and Status 2 Register 7.6.2.8. Link Capabilities 2 Register 7.6.2.9. Link Control and Status 2 Register
7.6.3. VF Message Signal Interrupt Extended (MSI-X) Capability Structure x
7.6.3.1. MSI-X Control Register 7.6.3.2. MSI-X Table Offset Register 7.6.3.3. MSI-X Pending Bit Array Register
7.6.4. VF Alternative Routing ID (ARI) Capability Structure x
7.6.4.1. ARI Enhanced Capability Header Register 7.6.4.2. ARI Capability and Control Registers
7.6.5. VF TLP Processing Hints (TPH) Capability Structure x
7.6.5.1. TPH Requester Enhanced Capability Header 7.6.5.2. TPH Requester Capability Register 7.6.5.3. TPH Requester Control Register
7.6.6. VF Address Translation Services (ATS) Capability Structure x
7.6.6.1. ATS Enhanced Capability Header 7.6.6.2. ATS Capability Register and Control Register
7.6.7. VF Access Control Services (ACS) Capability Structure x
7.6.7.1. ACS Extended Capability Header 7.6.7.2. ACS Capability and Control Register 7.6.7.3. Egress Control Vector
A. Troubleshooting/Debugging x
A.1. Hardware A.2. Debug Toolkit
A.1. Hardware x
A.1.1. Debugging Link Training Issues A.1.2. Debugging Data Transfer and Performance Issues
A.1.1. Debugging Link Training Issues x
A.1.1.1. Generic Tools and Utilities A.1.1.2. Signal Tap Logic Analyzer
A.2. Debug Toolkit x
A.2.1. Overview A.2.2. Enabling the Agilex™ 5 Debug Toolkit A.2.3. Launching the Agilex™ 5 Debug Toolkit A.2.4. Using the Agilex™ 5 Debug Toolkit A.2.5. Using the Agilex™ 5 Link Inspector
A.2.4. Using the Agilex™ 5 Debug Toolkit x
A.2.4.1. Main View A.2.4.2. Toolkit Parameters A.2.4.3. Channel Parameters A.2.4.4. Eye Viewer
A.2.4.2. Toolkit Parameters x
A.2.4.2.1. Agilex™ 5 Information A.2.4.2.2. Event Counter A.2.4.2.3. P0 Configuration Space
A.2.4.3. Channel Parameters x
A.2.4.3.1. General PHY A.2.4.3.2. TX Path A.2.4.3.3. RX Path
C. Implementation of Address Translation Services (ATS) in Endpoint Mode x
C.1. Sending Translated/Untranslated Requests C.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) C.3. Invalidating Requests/Completions
1. Introduction
2. Features
3. Getting Started with GTS AXI Streaming IP
4. IP Architecture and Functional Description
5. IP Parameters
6. Interfaces and Signals
7. Registers
8. Document Revision History for the GTS AXI Streaming Intel® FPGA IP for PCI Express* User Guide
A. Troubleshooting/Debugging
B. PIPE Mode Simulation
C. Implementation of Address Translation Services (ATS) in Endpoint Mode

A.1.2. Debugging Data Transfer and Performance Issues

There are many possible reasons causing the PCIe* link to stop transmitting data. The PCI Express* base specification defines three types of errors, outlined in the table below:

Table 116.  Error Types Defined by the PCI Express* Base Specification
Type Responsible Agent Description

Correctable

Hardware While correctable errors may affect system performance, data integrity is maintained.

Uncorrectable, nonfatal

 Device software

Uncorrectable, non-fatal errors are defined as errors in which data is lost, but system integrity is maintained. For example, the fabric may lose a particular TLP, but it still works without problems.

Uncorrectable, fatal

System software Errors generated by a loss of data and system failure are considered uncorrectable and fatal. Software must determine how to handle such errors: whether to reset the link or implement other means to minimize the problem.

Each PCI Express* compliant device must implement a basic level of error management and can optionally implement advanced error management. The PCI Express* Advanced Error Reporting Capability is an optional Extended Capability that may be implemented by PCI Express* device functions supporting advanced error control and reporting.

The GTS AXI Streaming IP implements both basic and advanced error reporting. Error handling for a Root Port is more complex than that of an Endpoint. In this IP, the Physical Functions (PFs) are always capable of AER (enabled by default). There is no AER implementation for Virtual Functions (VFs). Use the AER capability of the IP to identify the type of error and the protocol stack layer in which the error may have occurred.

Refer to the PCI Express* Capability Structures section of the Configuration Space Registers appendix for the AER Extended Capability Structure and the associated registers.

Table 117.  Correctable Error Status Register (AER)
Observation Issue Resolution

Receiver error bit set

Physical layer error which may be due to a PCS error when a lane is in L0, or a Control symbol being received in the wrong lane, or signal Integrity issues where the link may transition from L0 to the Recovery state.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Bad DLLP bit set

Data link layer error which may occur when a CRC verification fails.

 Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Bad TLP bit set

Data link layer error which may occur when an LCRC verification fails or when a sequence number error occurs.

 Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Replay_num_rollover bit set

Data link layer error which may be due to TLPs sent without success (no ACK) four times in a row.

 Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

replay timer timeout status bit set

Data link layer error which may occur when no ACK or NAK was received within the timeout period for the TLPs transmitted.

 Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Corrected internal error bits set

Transaction layer error which may be due to an ECC error in the internal Hard IP RAM.

 Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.
Table 118.  Uncorrectable Error Status Register (AER)
Observation Issue Resolution

Data link protocol error

Data link layer error which may be due to transmitter receiving an ACK/NAK whose Seq ID does not correspond to an unacknowledged TLP or ACK sequence number.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Surprise down error

Data link layer error which may be due to link_up_o getting deasserted during L0, indicating the physical layer link is going down unexpectedly.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Flow control protocol error

Transaction layer error which can be due to the receiver reporting more than the allowed credit limit. This error occurs when a component does not receive updated flow control credits with the 200 μs limit.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Poisoned TLP received

Transaction layer error which can be due to a received TLP with the EP bit set.

 Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Completion timeout

Transaction layer error which can be due to a completion not received within the required amount of time after a non-posted request was sent.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Completer abort

Transaction layer error which can be due to a completer being unable to fulfill a request due to a problem with the requester or a failure of the completer.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Unexpected completion

Transaction layer error which can be due to a requester receiving a completion that does not match any request awaiting a completion. The TLP is deleted by the Hard IP and not presented to the Application Layer.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Receiver overflow

Transaction layer error which can be due to a receiver receiving more TLPs than the available receive buffer space. The TLP is deleted by the Hard IP and not presented to the Application Layer.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Malformed TLP

Transaction layer error which can be due to errors in the received TLP header. The TLP is deleted by the Hard IP and not presented to the Application Layer.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

ECRC error

Transaction layer error which can be due to an ECRC check failure at the receiver despite the fact that the TLP is not malformed and the LCRC check is valid. The Hard IP block handles this TLP automatically. If the TLP is a non-posted request, the Hard IP block generates a completion with a completer abort status. The TLP is deleted by the Hard IP and not presented to the Application Layer.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Unsupported request

Transaction layer error which can be due to the completer being unable to fulfill the request. The TLP is deleted in the Hard IP block and not presented to the Application Layer. If the TLP is a non-posted request, the Hard IP block generates a completion with Unsupported Request status.

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

ACS violation

Transaction layer error which can be due to access control error in the received posted or non-posted request.

 Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Uncorrectable internal error

Transaction layer error which can be due to an internal error that cannot be corrected by the hardware.

 Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

TLP prefix blocked

EP or RP only

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.

Poisoned TLP egress blocked

EP or RP only

Use the AXI4-Lite Control and Status Register Responder interface to obtain more information about the error.
Related Information
Level Two Title