F-Tile Avalon® Streaming Intel® FPGA IP for PCI Express* User Guide

Download PDF
ID 683140
Date 10/22/2021
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Acronyms 2. Introduction 3. IP Architecture and Functional Description 4. Advanced Features 5. Interfaces 6. Parameters 7. Testbench 8. Troubleshooting/Debugging 9. F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide Archives 10. Revision History of the F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide A. Configuration Space Registers B. Implementation of Address Translation Services (ATS) in Endpoint Mode C. Packets Forwarded to the User Application in TLP Bypass Mode D. Root Port Enumeration
2. Introduction x
2.1. Overview 2.2. Features 2.3. Release Information 2.4. Device Family Support 2.5. Resource Utilization 2.6. IP Core and Design Example Support Levels
3. IP Architecture and Functional Description x
3.1. Architecture 3.2. Functional Description 3.3. Avalon-ST TX/RX 3.4. Interrupts 3.5. Completion Timeout 3.6. Hot Plug 3.7. Power Management 3.8. Configuration Output Interface (COI) 3.9. Configuration Intercept Interface (EP Only) 3.10. Hard IP Reconfiguration Interface 3.11. PHY Reconfiguration Interface 3.12. Page Request Service (PRS) (EP Only)
3.1. Architecture x
3.1.1. Clocks 3.1.2. Refclk 3.1.3. Reset
3.2. Functional Description x
3.2.1. PMA/PCS 3.2.2. Data Link Layer 3.2.3. Transaction Layer
3.3. Avalon-ST TX/RX x
3.3.1. TLP Header and Data Alignment 3.3.2. Avalon-ST RX 3.3.3. Avalon-ST TX 3.3.4. Tag Allocation 3.3.5. Completion Buffer Size
3.3.2. Avalon-ST RX x
3.3.2.1. RX Flow Control
3.3.3. Avalon-ST TX x
3.3.3.1. TX Flow Control
3.4. Interrupts x
3.4.1. Legacy Interrupts 3.4.2. MSI 3.4.3. MSI-X
3.10. Hard IP Reconfiguration Interface x
3.10.1. Configuration Registers Access
3.10.1. Configuration Registers Access x
3.10.1.1. Direct User Avalon-MM Interface (Byte Access) 3.10.1.2. Debug Register Interface Access (Dword Access)
4. Advanced Features x
4.1. Virtualization Support 4.2. TLP Bypass Mode 4.3. Precision Time Measurement (PTM) 4.4. Scalable IOV
4.1. Virtualization Support x
4.1.1. Single Root I/O Virtualization (SR-IOV) 4.1.2. VirtIO
4.1.1. Single Root I/O Virtualization (SR-IOV) x
4.1.1.1. SR-IOV Implementation
4.1.1.1. SR-IOV Implementation x
4.1.1.1.1. Functional Level Reset (FLR)
4.1.2. VirtIO x
4.1.2.1. VirtIO Implementation 4.1.2.2. VirtIO Registers
5. Interfaces x
5.1. Overview 5.2. Clocks and Resets 5.3. Serial Data Interface 5.4. Avalon-ST Interface 5.5. Interrupt Interface 5.6. Hard IP Status Interface 5.7. Error Interface 5.8. 10-bit Tag Support Interface 5.9. Completion Timeout Interface 5.10. Power Management Interface 5.11. Hot Plug Interface (RP Only) 5.12. Configuration Output Interface 5.13. Configuration Intercept Interface (EP Only) 5.14. Hard IP Reconfiguration Interface 5.15. PHY Reconfiguration Interface 5.16. Page Request Service (PRS) Interface (EP Only) 5.17. FLR Interface Signals 5.18. PTM Interface Signals 5.19. VF Error Flag Interface Signals 5.20. VirtIO PCI Configuration Access Interface Signals
5.4. Avalon-ST Interface x
5.4.1. Avalon-ST RX Interface Signals 5.4.2. Avalon-ST TX Interface Signals
5.5. Interrupt Interface x
5.5.1. Legacy Interrupt Interface Signals 5.5.2. MSI Pending Bits Interface Signals
6. Parameters x
6.1. Top-Level Settings 6.2. Core Parameters
6.2. Core Parameters x
6.2.1. Avalon Parameters 6.2.2. Base Address Registers 6.2.3. PCI Express and PCI Capabilities Parameters 6.2.4. Device Identification Registers 6.2.5. Configuration, Debug and Extension Options
6.2.3. PCI Express and PCI Capabilities Parameters x
6.2.3.1. Device Capabilities 6.2.3.2. Link Capabilities 6.2.3.3. Legacy Interrupt Pin Register 6.2.3.4. MSI Capabilities 6.2.3.5. MSI-X Capabilities 6.2.3.6. Slot Capabilities 6.2.3.7. Latency Tolerance Reporting (LTR) 6.2.3.8. Process Address Space ID (PASID) 6.2.3.9. Device Serial Number Capability 6.2.3.10. Page Request Service (PRS) 6.2.3.11. Access Control Service (ACS) Capabilities 6.2.3.12. Power Management 6.2.3.13. Vendor Specific Extended Capability (VSEC) Registers 6.2.3.14. Precision Time Measurement (PTM) 6.2.3.15. Address Translation Services (ATS) 6.2.3.16. TLP Processing Hints (TPH) 6.2.3.17. VirtIO Parameters
7. Testbench x
7.1. Endpoint Testbench 7.2. Test Driver Module 7.3. Root Port BFM 7.4. BFM Procedures and Functions
7.3. Root Port BFM x
7.3.1. BFM Memory Map 7.3.2. Configuration Space Bus and Device Numbering 7.3.3. Configuration of Root Port and Endpoint 7.3.4. Issuing Read and Write Transactions to the Application Layer
7.4. BFM Procedures and Functions x
7.4.1. ebfm_barwr Procedure 7.4.2. ebfm_barwr_imm Procedure 7.4.3. ebfm_barrd_wait Procedure 7.4.4. ebfm_barrd_nowt Procedure 7.4.5. ebfm_cfgwr_imm_wait Procedure 7.4.6. ebfm_cfgwr_imm_nowt Procedure 7.4.7. ebfm_cfgrd_wait Procedure 7.4.8. ebfm_cfgrd_nowt Procedure 7.4.9. BFM Configuration Procedures 7.4.10. BFM Shared Memory Access Procedures 7.4.11. BFM Log and Message Procedures 7.4.12. Verilog HDL Formatting Functions
7.4.9. BFM Configuration Procedures x
7.4.9.1. ebfm_cfg_rp_ep Procedure 7.4.9.2. ebfm_cfg_decode_bar Procedure
7.4.10. BFM Shared Memory Access Procedures x
7.4.10.1. Shared Memory Constants 7.4.10.2. shmem_write Task 7.4.10.3. shmem_read Function 7.4.10.4. shmem_display Verilog HDL Function 7.4.10.5. shmem_fill Procedure 7.4.10.6. shmem_chk_ok Function
7.4.11. BFM Log and Message Procedures x
7.4.11.1. ebfm_display Verilog HDL Function 7.4.11.2. ebfm_log_stop_sim Verilog HDL Function 7.4.11.3. ebfm_log_set_suppressed_msg_mask Task 7.4.11.4. ebfm_log_set_stop_on_msg_mask Verilog HDL Task 7.4.11.5. ebfm_log_open Verilog HDL Function 7.4.11.6. ebfm_log_close Verilog HDL Function
7.4.12. Verilog HDL Formatting Functions x
7.4.12.1. himage1 7.4.12.2. himage2 7.4.12.3. himage4 7.4.12.4. himage8 7.4.12.5. himage16 7.4.12.6. dimage1 7.4.12.7. dimage2 7.4.12.8. dimage3 7.4.12.9. dimage4 7.4.12.10. dimage5 7.4.12.11. dimage6 7.4.12.12. dimage7
8. Troubleshooting/Debugging x
8.1. Hardware
8.1. Hardware x
8.1.1. Debugging Link Training Issues 8.1.2. Debugging Data Transfer and Performance Issues
8.1.1. Debugging Link Training Issues x
8.1.1.1. Generic Tools and Utilities 8.1.1.2. SignalTapII Logic Analyzer 8.1.1.3. Additional Debug Tools
8.1.1.3. Additional Debug Tools x
8.1.1.3.1. Using the Hard IP Reconfiguration Interface 8.1.1.3.2. Enable and Read LCRC and ECRC Error Count
8.1.2. Debugging Data Transfer and Performance Issues x
8.1.2.1. Advanced Error Reporting (AER) 8.1.2.2. Second-Level Debug Tools
A. Configuration Space Registers x
A.1. Configuration Space Registers A.2. Configuration Space Registers for Virtualization A.3. Intel-Defined VSEC Capability Registers
A.1. Configuration Space Registers x
A.1.1. Register Access Definitions A.1.2. PCIe Configuration Header Registers A.1.3. PCI Express Capability Structures A.1.4. Physical Layer 16.0 GT/s Extended Capability Structure A.1.5. MSI-X Registers
A.2. Configuration Space Registers for Virtualization x
A.2.1. SR-IOV Virtualization Extended Capabilities Registers Address Map A.2.2. PCIe Configuration Registers for Each Virtual Function
A.2.2. PCIe Configuration Registers for Each Virtual Function x
A.2.2.1. Alternative Routing ID (ARI) Capability Structure A.2.2.2. TLP Processing Hint (TPH) Capability Structure A.2.2.3. Address Translation Services (ATS) Capability Structure A.2.2.4. Access Control Services (ACS) Capability Structure
A.2.2.1. Alternative Routing ID (ARI) Capability Structure x
A.2.2.1.1. ARI Enhanced Capability Header Register (Offset 0x0) A.2.2.1.2. ARI Capability and Control Register (Offset 0x4)
A.2.2.2. TLP Processing Hint (TPH) Capability Structure x
A.2.2.2.1. TPH Requester Enhanced Capability Header (Offset 0x0) A.2.2.2.2. TPH Requester Capability Register (Offset 0x4) A.2.2.2.3. TPH Requester Control Register (Offset 0x8)
A.2.2.3. Address Translation Services (ATS) Capability Structure x
A.2.2.3.1. ATS Enhanced Capability Header (Offset 0x0) A.2.2.3.2. ATS Capability Register and ATS Control Register (Offset 0x4)
A.2.2.4. Access Control Services (ACS) Capability Structure x
A.2.2.4.1. ACS Extended Capability Header (Offset 0x0) A.2.2.4.2. ACS Capability Register (Offset 0x4) A.2.2.4.3. ACS Control Register (Offset 0x6) A.2.2.4.4. Egress Control Vector (Offset 0x8)
A.3. Intel-Defined VSEC Capability Registers x
A.3.1. Intel-Defined VSEC Capability Header (Offset 00h) A.3.2. Intel-Defined Vendor Specific Header (Offset 04h) A.3.3. Intel Marker (Offset 08h) A.3.4. JTAG Silicon ID (Offset 0x0C - 0x18) A.3.5. User Configurable Device and Board ID (Offset 0x1C - 0x1D) A.3.6. General Purpose Control and Status Register (Offset 0x30) A.3.7. Uncorrectable Internal Error Status Register (Offset 0x34) A.3.8. Uncorrectable Internal Error Mask Register (Offset 0x38) A.3.9. Correctable Internal Error Status Register (Offset 0x3C) A.3.10. Correctable Internal Error Mask Register (Offset 0x40)
B. Implementation of Address Translation Services (ATS) in Endpoint Mode x
B.1. Sending Translated/Untranslated Requests B.2. Sending a Page Request Message from the Endpoint (EP) to the Root Complex (RC) B.3. Invalidating Requests/Completions
C. Packets Forwarded to the User Application in TLP Bypass Mode x
C.1. EP TLP Bypass Mode (Upstream) C.2. RC TLP Bypass Mode (Downstream)
1. Acronyms
2. Introduction
3. IP Architecture and Functional Description
4. Advanced Features
5. Interfaces
6. Parameters
7. Testbench
8. Troubleshooting/Debugging
9. F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide Archives
10. Revision History of the F-Tile Avalon Streaming Intel FPGA IP for PCI Express User Guide
A. Configuration Space Registers
B. Implementation of Address Translation Services (ATS) in Endpoint Mode
C. Packets Forwarded to the User Application in TLP Bypass Mode
D. Root Port Enumeration

5.7. Error Interface

This is an optional interface in the Intel F-Tile Avalon-ST IP for PCI Express that allows the Application Layer to report errors to the IP core and vice versa. Specifically, the Application Layer can report the different types of errors defined by the app_error_info_i signal to the IP. For Advanced Error Reporting (AER), the Application Layer can provide the information to log the TLP header and the error log request via the app_err_* interface.

The Intel F-Tile Avalon-ST IP for PCI Express enables the AER capability for Physical Functions (PFs) by default. There is no AER implementation for Virtual Functions (VFs). Use the VF Error Flag Interface instead of AER when using VFs.

Note: The Error Interface is not available for Topology H.
Table 64.  Error Interface Signals
Signal Name Direction EP/RP/BP Clock Domain Description
p#_serr_out_o Output EP/RP/BP coreclkout_hip

Indicates system error is detected.

RP mode: A one-clock-cycle pulse on this signal indicates if any device in the hierarchy reports any of the following errors and the associated enable bit is set in the Root Control register: ERR_COR, ERR_FATAL, ERR_NONFATAL.

Also asserted when an internal error is detected. The source of the error will be logged in the Root Port Error Status registers in the Port Configuration and Status registers.

EP mode: Asserted when the F-Tile PCIe Hard IP sends a message of correctable/non-fatal/fatal error.

BP mode: The transaction layer or data link layer errors detected by the Hard IP core trigger this signal. Detailed information are logged in the Bypass Mode Error Status registers in the Port Configuration and Status registers.

p#_hip_enter_err_mode_o Output EP/RP/BP coreclkout_hip

Asserted when the Hard IP enters the error mode. This usually happens when the Hard IP detects an uncorrectable RAM ECC error. Upon seeing the assertion of this signal, user should discard all the TLPs received.

p#_app_err_valid_i Input EP/RP coreclkout_hip

A one-cycle pulse on this signal indicates that the data on app_err_info_i, app_err_hdr_i, and app_err_func_num_i are valid in that cycle and app_err_hdr_i will be valid during the following four cycles.

p#_app_err_hdr_i[31:0] Input EP/RP coreclkout_hip

This bus contains the header and TLP prefix information for the error TLP. The 128-bit header and 32-bit TLP prefix are sent to the Hard IP over five cycles (32 bits of information are sent in each clock cycle).

Cycle 1 : header[31:0]

Cycle 2 : header[63:32]

Cycle 3 : header[95:64]

Cycle 4 : header[127:96]

Cycle 5 : TLP prefix
p#_app_err_info_i[12:0] Input EP/RP coreclkout_hip
This error bus carries the following information:
  • [0]: Malformed TLP
  • [1]: Receiver overflow
  • [2]: Unexpected completion
  • [3]: Completer abort
  • [4]: Completion timeout
  • [5]: Unsupported request
  • [6]: Poisoned TLP received
  • [7]: AtomicOp egress blocked
  • [8]: Uncorrectable internal error
  • [9]: Correctable internal error
  • [10]: Advisory error
  • [11]: TLP prefix blocked
  • [12]: ACS violation
p#_app_err_func_num_i[2:0] Input EP/RP coreclkout_hip
Note: Not available for p2 and p3.

This bus contains the function number for the function that asserts the error valid signal.

Figure 56. Error Interface Timing Diagram
For Topology H [1x4], error reporting is via the Hard IP Reconfiguration Interface. Your application needs to write to the APP_ERR registers follow the steps listed below.
  1. Write the error TLP header [127:96] to APP_ERR_HDR3 register
  2. Write the error TLP header [95:64] to APP_ERR_HDR2 register
  3. Write the error TLP header [63:32] to APP_ERR_HDR1 register
  4. Write the error TLP header [31:0] to APP_ERR_HDR0 register
  5. Write the error TLP prefix to APP_ERR PRFX register
  6. Write error valid, error function number, and error information to APP_ERR_BUS register
Table 65.  APP_ERR regsiters for Topology H
Register Offset Address Bit Description Access Type Default Value
APP_ERR_HDR0 0x14300 31:0 TLP header [31:0] RW 0x0000_0000
APP_ERR_HDR1 0x14304 31:0 TLP header [63:32] RW 0x0000_0000
APP_ERR_HDR2 0x14308 31:0 TLP header [95:64] RW 0x0000_0000
APP_ERR_HDR3 0x1430C 31:0 TLP header [127:96] RW 0x0000_0000
APP_ERR_PRFX 0x14310 31:0 TLP prefix RW 0x0000_0000
APP_ERR_BUS 0x14314 31:17 RSVD RO 0x0000
16:4

Error Information

Note: Refer to the error information defined for p#_app_err_info_i in Error Interface Signals Table
RW 0x0000
3:1 Function Number RW 0x0
0 Valid RW 0x0
Figure 57. Topology H Error Interface Hard IP Write
Level Two Title