Intel® Acceleration Stack User Guide: Intel FPGA Programmable Acceleration Card N3000

Download PDF
ID 683040
Date 6/14/2021
Public
Document Table of Contents
Document Table of Contents x
1. About this Document 2. System Requirements 3. Hardware Installation 4. Installing the OPAE Software 5. OPAE Tools 6. Sample Test: Native Loopback 7. Installing the Intel XL710 Driver 8. Configuring Ethernet Interfaces 9. Testing Network Loopback Using Data Plane Development Kit (DPDK) 10. Graceful Shutdown 11. Single Event Upset (SEU) 12. Document Revision History for Intel Acceleration Stack User Guide: Intel® FPGA PAC N3000 A. Troubleshooting B. Upgrade your Intel® FPGA PAC N3000 with Production Version of BMC and Intel® Arria® 10 Image C. Configure the 4.19 Kernel D. fpgabist Sample Output
1. About this Document x
1.1. Acronym List
2. System Requirements x
2.1. Cooling Requirements
3. Hardware Installation x
3.1. Installing the Intel® FPGA PAC N3000
4. Installing the OPAE Software x
4.1. Install Additional Packages 4.2. Install the Release Package 4.3. Identify the Intel® MAX® 10 Version on your Intel® FPGA PAC N3000
4.2. Install the Release Package x
4.2.1. Remove Previous OPAE Packages 4.2.2. Install the Acceleration Stack for Runtime 4.2.3. Install the Acceleration Stack for Development 4.2.4. Verify the OPAE Installation 4.2.5. Install the Configuration Files
4.3. Identify the Intel® MAX® 10 Version on your Intel® FPGA PAC N3000 x
4.3.1. FPGA Factory Image Overview
5. OPAE Tools x
5.1. Using fpgasupdate 5.2. Using fpgainfo 5.3. Test PCIe and External Memories with fpgabist 5.4. Test Network Loopback using fpgadiag
7. Installing the Intel XL710 Driver x
7.1. Updating the Intel XL710 Firmware
8. Configuring Ethernet Interfaces x
8.1. Modifying the Interface Maximum Transmission Unit (MTU) Size 8.2. Setting Forward Error Correction (FEC) Mode 8.3. Ethernet Pause Flow Control 8.4. Get Link Status and Statistics
9. Testing Network Loopback Using Data Plane Development Kit (DPDK) x
9.1. Test Using an External Traffic Generator 9.2. Test Using a Packet Generator 9.3. Troubleshooting in DPDK 9.4. Revert Back from DPDK to OPAE
10. Graceful Shutdown x
10.1. Background 10.2. Using OPAE 10.3. Using DPDK
11. Single Event Upset (SEU) x
11.1. OPAE Handling of SEU 11.2. DPDK Handling of SEU
A. Troubleshooting x
A.1. If OPAE installation verification fails, how to install OPAE manually?
B. Upgrade your Intel® FPGA PAC N3000 with Production Version of BMC and Intel® Arria® 10 Image x
B.1. Upgrading from 1.1 Beta to Production Version B.2. Upgrading from 1.1 Alpha-2 or Older to Production Version
B.1. Upgrading from 1.1 Beta to Production Version x
B.1.1. Root Entry Hash Programmed B.1.2. Root Entry Hash Not Programmed
D. fpgabist Sample Output x
D.1. For 2x2x25G Configuration D.2. For 8x10G Configuration
1. About this Document
2. System Requirements
3. Hardware Installation
4. Installing the OPAE Software
5. OPAE Tools
6. Sample Test: Native Loopback
7. Installing the Intel XL710 Driver
8. Configuring Ethernet Interfaces
9. Testing Network Loopback Using Data Plane Development Kit (DPDK)
10. Graceful Shutdown
11. Single Event Upset (SEU)
12. Document Revision History for Intel Acceleration Stack User Guide: Intel® FPGA PAC N3000
A. Troubleshooting
B. Upgrade your Intel® FPGA PAC N3000 with Production Version of BMC and Intel® Arria® 10 Image
C. Configure the 4.19 Kernel
D. fpgabist Sample Output

D.2. For 8x10G Configuration

==========================================================
Beginning FPGA Built-In Self-Test
==========================================================
Board Management Controller, 
               Intel® 
               MAX® 10 NIOS FW version D.2.0.19
Board Management Controller, 
               Intel® 
               MAX® 10 Build version D.2.0.6
//****** FME ******//
Object Id                     : 0xF000000
PCIe s:b:d.f                  : 0000:8a:00.0
Device Id                     : 0x0b30
Numa Node                     : 1
Ports Num                     : 01
Bitstream Id                  : 0x23000010010309
Bitstream Version             : 0.2.3
Pr Interface Id               : 901dd697-ca79-4b05-b843-8138cefa2846
Boot Page                     : user

Board Management Controller, 
               Intel® 
               MAX® 10 NIOS FW version D.2.0.19
Board Management Controller, 
               Intel® 
               MAX® 10 Build version D.2.0.6
//****** PORT ******//
Object Id                     : 0xEF00000
PCIe s:b:d.f                  : 0000:8a:00.0
Device Id                     : 0x0b30
Numa Node                     : 1
Ports Num                     : 01
Bitstream Id                  : 0x23000010010309
Bitstream Version             : 0.2.3
Pr Interface Id               : 901dd697-ca79-4b05-b843-8138cefa2846
Accelerator Id                : 9aeffe5f-8457-0612-c000-c9660d824272

Board Management Controller, 
               Intel® 
               MAX® 10 NIOS FW version D.2.0.19
Board Management Controller, 
               Intel® 
               MAX® 10 Build version D.2.0.6
//****** TEMP ******//
Object Id                     : 0xF000000
PCIe s:b:d.f                  : 0000:8a:00.0
Device Id                     : 0x0b30
Numa Node                     : 1
Ports Num                     : 01
Bitstream Id                  : 0x23000010010309
Bitstream Version             : 0.2.3
Pr Interface Id               : 901dd697-ca79-4b05-b843-8138cefa2846
(12) FPGA Die Temperature     : 51.00 Celsius
(13) Board Temperature        : 32.00 Celsius
(15) QSFP0 Temperature        : N/A
(38) QSFP1 Temperature        : N/A
(44) PKVL0 Core Temperature   : 49.00 Celsius
(45) PKVL0 SerDes Temperature : 49.50 Celsius
(46) PKVL1 Core Temperature   : 49.50 Celsius
(47) PKVL1 SerDes Temperature : 50.50 Celsius

Board Management Controller, 
               Intel® 
               MAX® 10 NIOS FW version D.2.0.19
Board Management Controller, 
               Intel® 
               MAX® 10 Build version D.2.0.6
//****** POWER ******//
Object Id                     : 0xF000000
PCIe s:b:d.f                  : 0000:8a:00.0
Device Id                     : 0x0b30
Numa Node                     : 1
Ports Num                     : 01
Bitstream Id                  : 0x23000010010309
Bitstream Version             : 0.2.3
Pr Interface Id               : 901dd697-ca79-4b05-b843-8138cefa2846
( 1) Board Power              : 60.10 Watts
( 2) 12V Backplane Current    : 2.77 Amps
( 3) 12V Backplane Voltage    : 12.14 Volts
( 4) 1.2V Voltage             : 1.19 Volts
( 6) 1.8V Voltage             : 1.80 Volts
( 8) 3.3V Voltage             : 3.27 Volts
(10) FPGA Core Voltage        : 0.90 Volts
(11) FPGA Core Current        : 12.18 Amps
(14) QSFP0 Supply Voltage     : N/A
(24) 12V AUX Current          : 2.17 Amps
(25) 12V AUX Voltage          : 12.18 Volts
(37) QSFP1 Supply Voltage     : N/A

Board Management Controller, 
               Intel® 
               MAX® 10 NIOS FW version D.2.0.19
Board Management Controller, 
               Intel® 
               MAX® 10 Build version D.2.0.6
//****** PORT ERRORS ******//
Object Id                     : 0xEF00000
PCIe s:b:d.f                  : 0000:8a:00.0
Device Id                     : 0x0b30
Numa Node                     : 1
Ports Num                     : 01
Bitstream Id                  : 0x23000010010309
Bitstream Version             : 0.2.3
Pr Interface Id               : 901dd697-ca79-4b05-b843-8138cefa2846
Accelerator Id                : 9aeffe5f-8457-0612-c000-c9660d824272
First Error                   : 0x0
First Malformed Req           : 0x0
Errors                        : 0x0
Board Management Controller, 
               Intel® 
               MAX® 10 NIOS FW version D.2.0.19
Board Management Controller, 
               Intel® 
               MAX® 10 Build version D.2.0.6
//****** FME ERRORS ******//
Object Id                     : 0xF000000
PCIe s:b:d.f                  : 0000:8a:00.0
Device Id                     : 0x0b30
Numa Node                     : 1
Ports Num                     : 01
Bitstream Id                  : 0x23000010010309
Bitstream Version             : 0.2.3
Pr Interface Id               : 901dd697-ca79-4b05-b843-8138cefa2846
Seu Emr                       : 0x0
First Error                   : 0x0
Next Error                    : 0x0
Errors                        : 0x0
PCIe1 Errors                  : 0x0
Nonfatal Errors               : 0x0
Inject Error                  : 0x0
Catfatal Errors               : 0x0
PCIe0 Errors                  : 0x0

Board Management Controller, 
               Intel® 
               MAX® 10 NIOS FW version D.2.0.19
Board Management Controller, 
               Intel® 
               MAX® 10 Build version D.2.0.6
//****** PHY ******//
Object Id                     : 0xF000000
PCIe s:b:d.f                  : 0000:8a:00.0
Device Id                     : 0x0b30
Numa Node                     : 1
Ports Num                     : 01
Bitstream Id                  : 0x23000010010309
Bitstream Version             : 0.2.3
Pr Interface Id               : 901dd697-ca79-4b05-b843-8138cefa2846
//****** PHY GROUP 0 ******//
Direction                     : Line side
Speed                         : 10 Gbps
Number of PHYs                : 8
//****** PHY GROUP 1 ******//
Direction                     : Host side
Speed                         : 10 Gbps
Number of PHYs                : 8
//****** Intel C827 Retimer ******//
Port0 10G                     : Down
Port1 10G                     : Down
Port2 10G                     : Down
Port3 10G                     : Down
Port4 10G                     : Down
Port5 10G                     : Down
Port6 10G                     : Down
Port7 10G                     : Down
Retimer A Version             : 101c.1064
Retimer B Version             : 101c.1064

Board Management Controller, 
               Intel® 
               MAX® 10 NIOS FW version D.2.0.19
Board Management Controller, 
               Intel® 
               MAX® 10 Build version D.2.0.6
//****** MAC ******//
Object Id                     : 0xF000000
PCIe s:b:d.f                  : 0000:8a:00.0
Device Id                     : 0x0b30
Numa Node                     : 1
Ports Num                     : 01
Bitstream Id                  : 0x23000010010309
Bitstream Version             : 0.2.3
Pr Interface Id               : 901dd697-ca79-4b05-b843-8138cefa2846
Number of MACs                : 8
MAC address 0                 : 64:4c:36:00:16:e0
MAC address 1                 : 64:4c:36:00:16:e1
MAC address 2                 : 64:4c:36:00:16:e2
MAC address 3                 : 64:4c:36:00:16:e3
MAC address 4                 : 64:4c:36:00:16:e4
MAC address 5                 : 64:4c:36:00:16:e5
MAC address 6                 : 64:4c:36:00:16:e6
MAC address 7                 : 64:4c:36:00:16:e7

Running mode: nlb
Running fpgadiag lpbk1 vh0-vh0 test...
found the NLB offset=0x28000

Cachelines Read_Count Write_Count Cache_Rd_Hit Cache_Wr_Hit Cache_Rd_Miss Cache_Wr_Miss   Eviction 'Clocks(@200 MHz)'   Rd_Bandwidth   Wr_Bandwidth
      1024   97724300    97723396            0            0             0             0          0        200174911     6.249 GB/s     6.249 GB/s

VH0_Rd_Count VH0_Wr_Count VH1_Rd_Count VH1_Wr_Count VL0_Rd_Count VL0_Wr_Count 
    97724300     97723397            0            0            0            0 

Running fpgadiag lpbk1 vh0-vh1 test...
found the NLB offset=0x28000

Cachelines Read_Count Write_Count Cache_Rd_Hit Cache_Wr_Hit Cache_Rd_Miss Cache_Wr_Miss   Eviction 'Clocks(@200 MHz)'   Rd_Bandwidth   Wr_Bandwidth
      1024   97717928    97717048            0            0             0             0          0        200161726     6.249 GB/s     6.249 GB/s

VH0_Rd_Count VH0_Wr_Count VH1_Rd_Count VH1_Wr_Count VL0_Rd_Count VL0_Wr_Count 
    97717932     97717049            0            0            0            0 

Running fpgadiag lpbk1 vh1-vh0 test...
found the NLB offset=0x28000

Cachelines Read_Count Write_Count Cache_Rd_Hit Cache_Wr_Hit Cache_Rd_Miss Cache_Wr_Miss   Eviction 'Clocks(@200 MHz)'   Rd_Bandwidth   Wr_Bandwidth
      1024   97909184    97908288            0            0             0             0          0        200496312     6.251 GB/s     6.251 GB/s

VH0_Rd_Count VH0_Wr_Count VH1_Rd_Count VH1_Wr_Count VL0_Rd_Count VL0_Wr_Count 
    97909184     97908289            0            0            0            0 

Running fpgadiag lpbk1 vh1-vh1 test...
found the NLB offset=0x28000

Cachelines Read_Count Write_Count Cache_Rd_Hit Cache_Wr_Hit Cache_Rd_Miss Cache_Wr_Miss   Eviction 'Clocks(@200 MHz)'   Rd_Bandwidth   Wr_Bandwidth
      1024   97911048    97910212            0            0             0             0          0        200494947     6.251 GB/s     6.251 GB/s

VH0_Rd_Count VH0_Wr_Count VH1_Rd_Count VH1_Wr_Count VL0_Rd_Count VL0_Wr_Count 
    97911048     97910213            0            0            0            0 

Finished Executing NLB (FPGA DIAG) Tests

Running mode: dma_afu
Running fpga_dma_test test on DDR4_A...

Running test in HW mode
Buffer Verification Success!
Buffer Verification Success!
Running DDR sweep test
Buffer pointer = 0x7f3c4aede000, size = 0x100000000 (0x7f3c4aede000 through 0x7f3d4aede000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6047.228482 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6818.460343 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
DDR sweep with unaligned pointer and size
Buffer pointer = 0x7f3c4b8df03d, size = 0xffffffbe (0x7f3c4b8df03d through 0x7f3d4b8deffb)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6043.044307 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6684.135801 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f3c4b8df003, size = 0xfffffffd (0x7f3c4b8df003 through 0x7f3d4b8df000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6047.563358 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6663.338529 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f3c4b8df007, size = 0xfffffff6 (0x7f3c4b8df007 through 0x7f3d4b8deffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6045.153009 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6673.172429 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f3c4b8df000, size = 0xfffffffd (0x7f3c4b8df000 through 0x7f3d4b8deffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6043.433517 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6751.913703 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f3c4b8df000, size = 0xffffffc3 (0x7f3c4b8df000 through 0x7f3d4b8defc3)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6047.493452 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6787.509760 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f3c4b8df000, size = 0xfffffff9 (0x7f3c4b8df000 through 0x7f3d4b8deff9)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6048.778852 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6746.716503 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Running fpga_dma_test test on DDR4_B...

Running test in HW mode
Buffer Verification Success!
Buffer Verification Success!
Running DDR sweep test
Buffer pointer = 0x7f734649e000, size = 0x100000000 (0x7f734649e000 through 0x7f744649e000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 5998.758802 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6475.479900 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
DDR sweep with unaligned pointer and size
Buffer pointer = 0x7f7346e9f03d, size = 0xffffffbe (0x7f7346e9f03d through 0x7f7446e9effb)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 5993.467581 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6445.968784 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f7346e9f003, size = 0xfffffffd (0x7f7346e9f003 through 0x7f7446e9f000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 5995.302867 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6411.985356 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f7346e9f007, size = 0xfffffff6 (0x7f7346e9f007 through 0x7f7446e9effd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 5997.001248 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6427.978131 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f7346e9f000, size = 0xfffffffd (0x7f7346e9f000 through 0x7f7446e9effd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 6001.461969 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6451.031043 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f7346e9f000, size = 0xffffffc3 (0x7f7346e9f000 through 0x7f7446e9efc3)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 5979.483548 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6482.630084 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f7346e9f000, size = 0xfffffff9 (0x7f7346e9f000 through 0x7f7446e9eff9)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 5993.585033 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 6394.897393 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Running fpga_dma_test test on DDR4_C...

Running test in HW mode
Buffer Verification Success!
Buffer Verification Success!
Running DDR sweep test
Buffer pointer = 0x7f836ebd7000, size = 0x40000000 (0x7f836ebd7000 through 0x7f83aebd7000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2501.897592 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2582.127891 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
DDR sweep with unaligned pointer and size
Buffer pointer = 0x7f836f5d803d, size = 0x3fffffbe (0x7f836f5d803d through 0x7f83af5d7ffb)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2501.362552 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2582.615815 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f836f5d8003, size = 0x3ffffffd (0x7f836f5d8003 through 0x7f83af5d8000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2500.993490 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2582.841768 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f836f5d8007, size = 0x3ffffff6 (0x7f836f5d8007 through 0x7f83af5d7ffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2502.907991 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2581.905891 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f836f5d8000, size = 0x3ffffffd (0x7f836f5d8000 through 0x7f83af5d7ffd)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2500.809100 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2582.694133 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f836f5d8000, size = 0x3fffffc3 (0x7f836f5d8000 through 0x7f83af5d7fc3)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2500.802500 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2583.019243 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Buffer pointer = 0x7f836f5d8000, size = 0x3ffffff9 (0x7f836f5d8000 through 0x7f83af5d7ff9)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 2501.453270 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 2583.127684 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Running fpga_dma_test test on QDR...

Running test in HW mode
Buffer Verification Success!
Buffer Verification Success!
Running DDR sweep test
Buffer pointer = 0x7ff375fb3000, size = 0x1000000 (0x7ff375fb3000 through 0x7ff376fb3000)
Allocated test buffer
Fill test buffer
DDR Sweep Host to FPGA
Measured bandwidth = 936.309401 Megabytes/sec 
Clear buffer
DDR Sweep FPGA to Host
Measured bandwidth = 915.727217 Megabytes/sec 
Verifying buffer..
Buffer Verification Success!
Finished Executing DMA Tests

Built-in Self-Test Completed.
Level Two Title