P-Tile Avalon® Memory-mapped Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683268
Date 7/14/2021
Public
Document Table of Contents
Document Table of Contents x
1. About the P-tile Avalon® Intel® FPGA IPs for PCI Express 2. IP Architecture and Functional Description 3. Parameters 4. Interfaces 5. Advanced Features 6. Troubleshooting/Debugging 7. Document Revision History for the P-tile Avalon® Memory-mapped Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers
1. About the P-tile Avalon® Intel® FPGA IPs for PCI Express x
1.1. Overview 1.2. Features 1.3. Release Information 1.4. Device Family Support 1.5. Performance and Resource Utilization 1.6. IP Core and Design Example Support Levels
2. IP Architecture and Functional Description x
2.1. Top-Level Architecture 2.2. Functional Description
2.1. Top-Level Architecture x
2.1.1. Avalon® -MM Bridge Architecture 2.1.2. Clock Domains 2.1.3. Refclk 2.1.4. Reset
2.2. Functional Description x
2.2.1. PMA/PCS 2.2.2. Data Link Layer Overview 2.2.3. Transaction Layer Overview 2.2.4. Avalon® -MM Bridge
2.2.4. Avalon® -MM Bridge x
2.2.4.1. Endpoint Mode with Data Movers 2.2.4.2. Endpoint Mode 2.2.4.3. Root Port Mode
3. Parameters x
3.1. Top-Level Settings 3.2. Core Parameters 3.3. Avalon-MM Settings
3.2. Core Parameters x
3.2.1. Base Address Registers 3.2.2. PCI Express and PCI Capabilities Parameters 3.2.3. Device Identification Registers 3.2.4. Configuration, Debug and Extension Options
3.2.2. PCI Express and PCI Capabilities Parameters x
3.2.2.1. Device Capabilities 3.2.2.2. Link Capabilities 3.2.2.3. Legacy Interrupt Pin Register 3.2.2.4. MSI Capabilities 3.2.2.5. MSI-X Capabilities 3.2.2.6. Device Serial Number Capability 3.2.2.7. TLP Processing Hints (TPH)
4. Interfaces x
4.1. Overview 4.2. Clocks and Resets 4.3. Avalon® -MM Interface 4.4. Serial Data Interface 4.5. Hard IP Status Interface 4.6. Interrupt Interface 4.7. Hot Plug Interface (RP Only) 4.8. Power Management Interface 4.9. Configuration Output Interface 4.10. Hard IP Reconfiguration Interface 4.11. PHY Reconfiguration Interface
4.2. Clocks and Resets x
4.2.1. Interface Clock Signals 4.2.2. Interface Reset Signals
4.3. Avalon® -MM Interface x
4.3.1. Endpoint Mode Interface (512-bit Avalon® -MM Interface) 4.3.2. Root Port Mode Interface (256-bit Avalon® -MM Interface)
4.3.1. Endpoint Mode Interface (512-bit Avalon® -MM Interface) x
4.3.1.1. Bursting Avalon® -MM Master and Conduit 4.3.1.2. Bursting Avalon® -MM Slave and Conduit 4.3.1.3. Read Data Mover 4.3.1.4. Write Data Mover 4.3.1.5. Descriptor Format for Data Movers 4.3.1.6. Avalon® -MM DMA Operations
4.3.1.1. Bursting Avalon® -MM Master and Conduit x
4.3.1.1.1. Bursting Avalon® -MM Master and Conduit in Non-Bursting Mode
4.3.1.2. Bursting Avalon® -MM Slave and Conduit x
4.3.1.2.1. Bursting Avalon® -MM Slave and Conduit in Non-Bursting Mode
4.3.1.3. Read Data Mover x
4.3.1.3.1. Read Data Mover Avalon® -MM Write Master and Conduit 4.3.1.3.2. Read Data Mover Avalon® -ST Descriptor Sinks 4.3.1.3.3. Read Data Mover Status Avalon® -ST Source
4.3.1.4. Write Data Mover x
4.3.1.4.1. Write Data Mover Avalon® -MM Read Master and Conduit 4.3.1.4.2. Write Data Mover Avalon® -ST Descriptor Sinks 4.3.1.4.3. Write Data Mover Status Avalon® -ST Source
4.3.2. Root Port Mode Interface (256-bit Avalon® -MM Interface) x
4.3.2.1. High Performance Avalon® -MM Slave (HPTXS) Interface 4.3.2.2. High Performance Avalon® -MM Master (HPRXM) Interface 4.3.2.3. 32-Bit Control Register Access (CRA) Slave (Root Port only)
4.6. Interrupt Interface x
4.6.1. Legacy Interrupts 4.6.2. MSI 4.6.3. MSI-X
4.6.3. MSI-X x
4.6.3.1. Implementing MSI-X Interrupts
4.10. Hard IP Reconfiguration Interface x
4.10.1. Address Map for the User Avalon-MM Interface 4.10.2. Configuration Registers Access
4.10.2. Configuration Registers Access x
4.10.2.1. Using Direct User Avalon-MM Interface (Byte Access) 4.10.2.2. Using the Debug Register Interface Access
5. Advanced Features x
5.1. PCIe* Port Bifurcation and PHY Channel Mapping
6. Troubleshooting/Debugging x
6.1. Hardware 6.2. Debug Toolkit
6.1. Hardware x
6.1.1. Debugging Link Training Issues 6.1.2. Debugging Data Transfer and Performance Issues
6.1.1. Debugging Link Training Issues x
6.1.1.1. Generic Tools and Utilities 6.1.1.2. SignalTapII Logic Analyzer 6.1.1.3. Additional Debug Tools
6.1.2. Debugging Data Transfer and Performance Issues x
6.1.2.1. Advanced Error Reporting (AER) 6.1.2.2. Second-Level Debug Tools
6.2. Debug Toolkit x
6.2.1. Overview 6.2.2. Enabling the P-Tile Debug Toolkit 6.2.3. Launching the P-Tile Debug Toolkit 6.2.4. Using the P-Tile Debug Toolkit 6.2.5. Enabling the P-Tile Link Inspector 6.2.6. Using the P-Tile Link Inspector
A. Configuration Space Registers x
A.1. Configuration Space Registers A.2. Intel-Defined VSEC Capability Registers
A.1. Configuration Space Registers x
A.1.1. Register Access Definitions A.1.2. PCIe Configuration Header Registers A.1.3. PCI Express Capability Structures A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure A.1.5. MSI-X Registers
A.2. Intel-Defined VSEC Capability Registers x
A.2.1. Intel-Defined VSEC Capability Header (Offset 00h) A.2.2. Intel-Defined Vendor Specific Header (Offset 04h) A.2.3. Intel Marker (Offset 08h) A.2.4. JTAG Silicon ID (Offset 0x0C - 0x18) A.2.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D) A.2.6. General Purpose Control and Status Register (Offset 0x30) A.2.7. Uncorrectable Internal Error Status Register (Offset 0x34) A.2.8. Uncorrectable Internal Error Mask Register (Offset 0x38) A.2.9. Correctable Internal Error Status Register (Offset 0x3C) A.2.10. Correctable Internal Error Mask Register (Offset 0x40)
1. About the P-tile Avalon® Intel® FPGA IPs for PCI Express
2. IP Architecture and Functional Description
3. Parameters
4. Interfaces
5. Advanced Features
6. Troubleshooting/Debugging
7. Document Revision History for the P-tile Avalon® Memory-mapped Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers

6.1.2. Debugging Data Transfer and Performance Issues

There are many possible reasons causing the PCIe link to stop transmitting data. The PCI Express base specification defines three types of errors, outlined in the table below:

Table 64.  Error Types Defined by the PCI Express Base Specification
Type Responsible Agent Description
Correctable Hardware While correctable errors may affect system performance, data integrity is maintained.
Uncorrectable, non-fatal Device software Uncorrectable, non-fatal errors are defined as errors in which data is lost, but system integrity is maintained. For example, the fabric may lose a particular TLP, but it still works without problems.
Uncorrectable, fatal System software Errors generated by a loss of data and system failure are considered uncorrectable and fatal. Software must determine how to handle such errors: whether to reset the link or implement other means to minimize the problem.
Table 65.  Correctable Error Status Register (AER)
Observation Issue Resolution
Receiver error bit set

Physical layer error which may be due to a PCS error when a lane is in L0, or a Control symbol being received in the wrong lane, or signal Integrity issues where the link may transition from L0 to the Recovery state.

Use the Hard IP reconfiguration interface and the flow chart in Figure 35 to obtain more information about the error.
Bad DLLP bit set

Data link layer error which may occur when a CRC verification fails.

Use the Hard IP reconfiguration interface to obtain more information about the error.
Bad TLP bit set Data link layer error which may occur when an LCRC verification fails or when a sequence number error occurs. Use the Hard IP reconfiguration interface to obtain more information about the error.
Replay_num_rollover bit set Data link layer error which may be due to TLPs sent without success (no ACK) four times in a row. Use the Hard IP reconfiguration interface to obtain more information about the error.
replay timer timeout status bit set

Data link layer error which may occur when no ACK or NAK was received within the timeout period for the TLPs transmitted.

Use the Hard IP reconfiguration interface to obtain more information about the error.
Advisory non-fatal Transaction layer error which may be due to higher priority uncorrectable error detected.  
Corrected internal error bits set Transaction layer error which may be due to an ECC error in the internal Hard IP RAM. Use the Hard IP reconfiguration interface and DBI registers to obtain more information about the error.
Table 66.  Uncorrectable Error Status Register (AER)
Observation Issue Resolution
Data link protocol error Data link layer error which may be due to transmitter receiving an ACK/NAK whose Seq ID does not correspond to an unacknowledged TLP or ACK sequence number. Use the Hard IP reconfiguration interface to obtain more information about the error.
Surprise down error Data link layer error which may be due to link_up_o getting deasserted during L0, indicating the physical layer link is going down unexpectedly. Use the Hard IP reconfiguration interface and DBI registers to obtain more information about the error.
Flow control protocol error

Transaction layer error which can be due to the receiver reporting more than the allowed credit limit.

This error occurs when a component does not receive updated flow control credits with the 200 μs limit.

Use the TX/RX flow control interface, Hard IP reconfiguration interface to obtain more information about the error.
Poisoned TLP received Transaction layer error which can be due to a received TLP with the EP bit set. Use the Hard IP reconfiguration interface to obtain more information on the error and determine the appropriate action.
Completion timeout Transaction layer error which can be due to a completion not received within the required amount of time after a non-posted request was sent. Use the Hard IP reconfiguration interface to obtain more information on the error.
Completer abort Transaction layer error which can be due to a completer being unable to fulfill a request due to a problem with the requester or a failure of the completer. Use the Hard IP reconfiguration interface to obtain more information on the error.
Unexpected completion

Transaction layer error which can be due to a requester receiving a completion that doesn’t match any request awaiting a completion.

The TLP is deleted by the Hard IP and not presented to the Application Layer.

Use the Hard IP reconfiguration interface to obtain more information on the error.
Receiver overflow

Transaction layer error which can be due to a receiver receiving more TLPs than the available receive buffer space.

The TLP is deleted by the Hard IP and not presented to the Application Layer.

Use the TX/RX flow control interface and Hard IP reconfiguration interface to obtain more information on the error.
Malformed TLP

Transaction layer error which can be due to errors in the received TLP header.

The TLP is deleted by the Hard IP and not presented to the Application Layer.

Use the Hard IP reconfiguration interface to obtain more information on the error.
ECRC error

Transaction layer error which can be due to an ECRC check failure at the receiver despite the fact that the TLP is not malformed and the LCRC check is valid.

The Hard IP block handles this TLP automatically. If the TLP is a non-posted request, the Hard IP block generates a completion with a completer abort status. The TLP is deleted by the Hard IP and not presented to the Application Layer.

Use the Hard IP reconfiguration interface to obtain more information on the error.
Unsupported request

Transaction layer error which can be due to the completer being unable to fulfill the request.

The TLP is deleted in the Hard IP block and not presented to the Application Layer. If the TLP is a non-posted request, the Hard IP block generates a completion with Unsupported Request status.

Use the Hard IP reconfiguration interface to obtain more information on the error.
ACS violation Transaction layer error which can be due to access control error in the received posted or non-posted request. Use the Hard IP reconfiguration interface to obtain more information on the error.
Uncorrectable internal error Transaction layer error which can be due to an internal error that cannot be corrected by the hardware. Use the Hard IP reconfiguration interface and DBI registers to obtain more information on the error.
Atomic egress blocked Use the Hard IP reconfiguration interface to obtain more information on the error.
TLP prefix blocked EP or RP only Use the Hard IP reconfiguration interface to obtain more information on the error.
Poisoned TLP egress blocked EP or RP only Use the Hard IP reconfiguration interface to obtain more information on the error.

Use the debug tools mentioned in the next two sections for debugging link training issues observed on the PCI Express link when using the P-Tile Avalon® -MM IP for PCI Express.


Section Content
Advanced Error Reporting (AER)
Second-Level Debug Tools

Level Two Title