Intel® Acceleration Stack User Guide: Intel® FPGA Programmable Acceleration Card N3000-N/2

Download PDF
ID 683362
Date 11/01/2021
Public
Document Table of Contents
Document Table of Contents x
1. About this Document 2. System Requirements 3. Hardware Installation 4. Installing the OPAE Software 5. Identify the Intel® MAX® 10 BMC Version 6. Intel XL710 Driver Installation and Firmware Update 7. Updating the Retimer Firmware 8. OPAE Tools 9. Sample Test: Native Loopback 10. Configuring Ethernet Interfaces 11. Testing Network Loopback Using Data Plane Development Kit (DPDK) 12. Graceful Shutdown 13. Single Event Upset (SEU) 14. Document Revision History for Intel Acceleration Stack User Guide: Intel® FPGA PAC N3000-N/2 A. Troubleshooting B. fpgabist Sample Output
1. About this Document x
1.1. Acronym List
3. Hardware Installation x
3.1. Installing the Intel® FPGA PAC N3000-N/2
4. Installing the OPAE Software x
4.1. Install the Release Package
4.1. Install the Release Package x
4.1.1. Remove Previous OPAE Packages 4.1.2. Install the Acceleration Stack for Runtime 4.1.3. Install the Acceleration Stack for Development 4.1.4. Verify the OPAE Installation
4.1.2. Install the Acceleration Stack for Runtime x
4.1.2.1. For RHEL 4.1.2.2. For CentOS
4.1.2.2. For CentOS x
4.1.2.2.1. Download kernel 4.19 4.1.2.2.2. Build and Install kernel 4.19 4.1.2.2.3. Install OPAE in kernel 4.19
5. Identify the Intel® MAX® 10 BMC Version x
5.1. FPGA Factory Image Overview
6. Intel XL710 Driver Installation and Firmware Update x
6.1. Installing the Intel XL710 Driver 6.2. Updating the Intel XL710 Firmware
8. OPAE Tools x
8.1. Using fpgasupdate 8.2. Using fpgainfo 8.3. Test PCIe and External Memories with fpgabist 8.4. Test Network Loopback using fpgadiag
10. Configuring Ethernet Interfaces x
10.1. Modifying the Interface Maximum Transmission Unit (MTU) Size 10.2. Setting Forward Error Correction (FEC) Mode 10.3. Ethernet Pause Flow Control 10.4. Get Link Statistics 10.5. Ethernet Link Status
11. Testing Network Loopback Using Data Plane Development Kit (DPDK) x
11.1. Test Using an External Traffic Generator 11.2. Revert Back from DPDK to OPAE
12. Graceful Shutdown x
12.1. Using OPAE
13. Single Event Upset (SEU) x
13.1. OPAE Handling of SEU
A. Troubleshooting x
A.1. If OPAE installation verification fails, how to install OPAE manually?
B. fpgabist Sample Output x
B.1. For 4x25G Configuration B.2. For 2x2x25G Configuration
1. About this Document
2. System Requirements
3. Hardware Installation
4. Installing the OPAE Software
5. Identify the Intel® MAX® 10 BMC Version
6. Intel XL710 Driver Installation and Firmware Update
7. Updating the Retimer Firmware
8. OPAE Tools
9. Sample Test: Native Loopback
10. Configuring Ethernet Interfaces
11. Testing Network Loopback Using Data Plane Development Kit (DPDK)
12. Graceful Shutdown
13. Single Event Upset (SEU)
14. Document Revision History for Intel Acceleration Stack User Guide: Intel® FPGA PAC N3000-N/2
A. Troubleshooting
B. fpgabist Sample Output

B.1. For 4x25G Configuration

==========================================================

Beginning FPGA Built-In Self-Test

==========================================================
Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** FME ******//
Object Id                     : 0xF100000
PCIe s:b:d.f                  : 0000:3e:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
Boot Page                     : user


Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** PORT ******//
Object Id                     : 0xF000000
PCIe s:b:d.f                  : 0000:3e:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
Accelerator Id                : 9aeffe5f-8457-0612-c000-c9660d824272


Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** TEMP ******//
Object Id                     : 0xF100000
PCIe s:b:d.f                  : 0000:3e:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
(12) FPGA Core Temperature        : 46.00 Celsius
(13) Board Temperature            : 30.00 Celsius
(15) QSFP A Temperature           : N/A
(38) QSFP B Temperature           : N/A
(44) Retimer A Core Temperature   : 54.50 Celsius
(45) Retimer A Serdes Temperature : 55.50 Celsius
(46) Retimer B Core Temperature   : 0.00 Celsius
(47) Retimer B Serdes Temperature : 0.00 Celsius


Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** POWER ******//
Object Id                     : 0xF100000
PCIe s:b:d.f                  : 0000:3e:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
( 1) Board Power              : 67.57 Watts
( 2) 12V Backplane Current    : 3.10 Amps
( 3) 12V Backplane Voltage    : 12.10 Volts
( 4) 1.2V Voltage             : 1.19 Volts
( 6) 1.8V Voltage             : 1.80 Volts
( 8) 3.3V Voltage             : 3.26 Volts
(10) FPGA Core Voltage        : 0.90 Volts
(11) FPGA Core Current        : 14.31 Amps
(14) QSFP A Voltage           : N/A
(24) 12V AUX Current          : 2.49 Amps
(25) 12V AUX Voltage          : 12.10 Volts
(37) QSFP B Voltage           : N/A


Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** FME ERRORS ******//
Object Id                     : 0xF100000
PCIe s:b:d.f                  : 0000:3e:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
PCIe0 Errors                  : 0x0
PCIe1 Errors                  : 0x0
Catfatal Errors               : 0x0
Seu Emr                       : 0x0
Inject Error                  : 0x0
Nonfatal Errors               : 0x0
Next Error                    : 0x0
First Error                   : 0x0
Errors                        : 0x0
Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** PORT ERRORS ******//
Object Id                     : 0xF000000
PCIe s:b:d.f                  : 0000:3e:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
Accelerator Id                : 9aeffe5f-8457-0612-c000-c9660d824272
First Malformed Req           : 0x0
First Error                   : 0x0
Errors                        : 0x0


Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** PHY ******//
Object Id                     : 0xF100000
PCIe s:b:d.f                  : 0000:3e:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
//****** PHY GROUP 0 ******//
Direction                     : Line side
Speed                         : 25 Gbps
Number of PHYs                : 4
//****** PHY GROUP 1 ******//
Direction                     : Host side
Speed                         : 40 Gbps
Number of PHYs                : 4
//****** Intel C827 Retimer ******//
Port0 25G                     : Up
Port1 25G                     : Up
Port2 25G                     : Up
Port3 25G                     : Up
Retimer A Version             : 101c.1064
Retimer B Version             : 0000.0000


Board Management Controller, MAX10 NIOS FW version D.2.1.24
Board Management Controller, MAX10 Build version D.2.0.7
//****** MAC ******//
Object Id                     : 0xF100000
PCIe s:b:d.f                  : 0000:3e:00.0
Device Id                     : 0x0b30
Numa Node                     : 0
Ports Num                     : 01
Bitstream Id                  : 0x23000110010310
Bitstream Version             : 0.2.3
Pr Interface Id               : f3c99413-5081-4aad-bced-07eb84a6d0bb
Number of MACs                : 8
MAC address 0                 : 64:4C:36:12:9D:D0
MAC address 1                 : 64:4C:36:12:9D:D1
MAC address 2                 : 64:4C:36:12:9D:D2
MAC address 3                 : 64:4C:36:12:9D:D3
MAC address 4                 : 64:4C:36:12:9D:D4
MAC address 5                 : 64:4C:36:12:9D:D5
MAC address 6                 : 64:4C:36:12:9D:D6
MAC address 7                 : 64:4C:36:12:9D:D7


Running mode: nlb
Running fpgadiag lpbk1 vh0-vh0 test...
found the NLB offset=0x28000

Cachelines Read_Count Write_Count Cache_Rd_Hit Cache_Wr_Hit Cache_Rd_Miss Cache_Wr_Miss   Eviction 'Clocks(@200 MHz)'   Rd_Bandwidth   Wr_Bandwidth
      1024   97681728    97680872            0            0             0             0          0        200032043     6.251 GB/s     6.251 GB/s

VH0_Rd_Count VH0_Wr_Count VH1_Rd_Count VH1_Wr_Count VL0_Rd_Count VL0_Wr_Count 
    97681740     97680873            0            0            0            0 

Running fpgadiag lpbk1 vh0-vh1 test...
found the NLB offset=0x28000

Cachelines Read_Count Write_Count Cache_Rd_Hit Cache_Wr_Hit Cache_Rd_Miss Cache_Wr_Miss   Eviction 'Clocks(@200 MHz)'   Rd_Bandwidth   Wr_Bandwidth
      1024   97694604    97693772            0            0             0             0          0        200030128     6.252 GB/s     6.251 GB/s

VH0_Rd_Count VH0_Wr_Count VH1_Rd_Count VH1_Wr_Count VL0_Rd_Count VL0_Wr_Count 
    97694616     97693773            0            0            0            0 

Running fpgadiag lpbk1 vh1-vh0 test...
found the NLB offset=0x28000

Cachelines Read_Count Write_Count Cache_Rd_Hit Cache_Wr_Hit Cache_Rd_Miss Cache_Wr_Miss   Eviction 'Clocks(@200 MHz)'   Rd_Bandwidth   Wr_Bandwidth
      1024   97676772    97675900            0            0             0             0          0        200031350     6.250 GB/s     6.250 GB/s

VH0_Rd_Count VH0_Wr_Count VH1_Rd_Count VH1_Wr_Count VL0_Rd_Count VL0_Wr_Count 
    97676772     97675901            0            0            0            0 

Running fpgadiag lpbk1 vh1-vh1 test...
found the NLB offset=0x28000

Cachelines Read_Count Write_Count Cache_Rd_Hit Cache_Wr_Hit Cache_Rd_Miss Cache_Wr_Miss   Eviction 'Clocks(@200 MHz)'   Rd_Bandwidth   Wr_Bandwidth
      1024   97687552    97686640            0            0             0             0          0        200031698     6.251 GB/s     6.251 GB/s

VH0_Rd_Count VH0_Wr_Count VH1_Rd_Count VH1_Wr_Count VL0_Rd_Count VL0_Wr_Count 
    97687552     97686641            0            0            0            0 

Finished Executing NLB (FPGA DIAG) Tests

Running mode: dma_afu
Running fpga_dma_test test on DDR4_A...

Running test in HW mode
Buffer Verification Success!
Buffer Verification Success!
Running DDR sweep test
Buffer pointer = 0x7fb65c5fc000, size = 0x100000000 (0x7fb65c5fc000 through 0x7fb75c5fc000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6028.495598 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6238.024430 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
DDR sweep with unaligned pointer and size
Buffer pointer = 0x7fb65cffd03d, size = 0xffffffbe (0x7fb65cffd03d through 0x7fb75cffcffb)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6023.680411 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6227.207827 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fb65cffd003, size = 0xfffffffd (0x7fb65cffd003 through 0x7fb75cffd000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6040.148285 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6278.841591 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fb65cffd007, size = 0xfffffff6 (0x7fb65cffd007 through 0x7fb75cffcffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6075.907357 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6193.636128 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fb65cffd000, size = 0xfffffffd (0x7fb65cffd000 through 0x7fb75cffcffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6075.847398 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6286.483893 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fb65cffd000, size = 0xffffffc3 (0x7fb65cffd000 through 0x7fb75cffcfc3)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6077.481368 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6239.577290 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fb65cffd000, size = 0xfffffff9 (0x7fb65cffd000 through 0x7fb75cffcff9)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6075.056363 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6282.301315 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Running fpga_dma_test test on DDR4_B...

Running test in HW mode
Buffer Verification Success!
Buffer Verification Success!
Running DDR sweep test
Buffer pointer = 0x7fe2a87fc000, size = 0x100000000 (0x7fe2a87fc000 through 0x7fe3a87fc000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6075.915975 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6354.786148 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
DDR sweep with unaligned pointer and size
Buffer pointer = 0x7fe2a91fd03d, size = 0xffffffbe (0x7fe2a91fd03d through 0x7fe3a91fcffb)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6008.463097 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6317.263668 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fe2a91fd003, size = 0xfffffffd (0x7fe2a91fd003 through 0x7fe3a91fd000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6077.102232 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6295.423917 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fe2a91fd007, size = 0xfffffff6 (0x7fe2a91fd007 through 0x7fe3a91fcffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6076.641461 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6308.306961 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fe2a91fd000, size = 0xfffffffd (0x7fe2a91fd000 through 0x7fe3a91fcffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6078.123251 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6403.779597 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fe2a91fd000, size = 0xffffffc3 (0x7fe2a91fd000 through 0x7fe3a91fcfc3)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6077.587999 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6442.061446 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7fe2a91fd000, size = 0xfffffff9 (0x7fe2a91fd000 through 0x7fe3a91fcff9)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6076.830038 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6434.268346 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Running fpga_dma_test test on DDR4_C...

Running test in HW mode
Buffer Verification Success!
Buffer Verification Success!
Running DDR sweep test
Buffer pointer = 0x7f9f08bfc000, size = 0x40000000 (0x7f9f08bfc000 through 0x7f9f48bfc000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2359.378051 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2441.195532 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
DDR sweep with unaligned pointer and size
Buffer pointer = 0x7f9f095fd03d, size = 0x3fffffbe (0x7f9f095fd03d through 0x7f9f495fcffb)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2358.980944 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2439.859968 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f9f095fd003, size = 0x3ffffffd (0x7f9f095fd003 through 0x7f9f495fd000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2359.307560 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2440.822373 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f9f095fd007, size = 0x3ffffff6 (0x7f9f095fd007 through 0x7f9f495fcffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2359.178753 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2440.677334 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f9f095fd000, size = 0x3ffffffd (0x7f9f095fd000 through 0x7f9f495fcffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2359.269882 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2440.602513 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f9f095fd000, size = 0x3fffffc3 (0x7f9f095fd000 through 0x7f9f495fcfc3)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2359.528387 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2441.371818 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f9f095fd000, size = 0x3ffffff9 (0x7f9f095fd000 through 0x7f9f495fcff9)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2359.232654 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2399.931406 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Running fpga_dma_test test on QDR...

Running test in HW mode
Buffer Verification Success!
Buffer Verification Success!
Running DDR sweep test
Buffer pointer = 0x7f2de37fc000, size = 0x1000000 (0x7f2de37fc000 through 0x7f2de47fc000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 933.236849 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 913.672934 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Finished Executing DMA Tests


Built-in Self-Test Completed.
Level Two Title