Intel® Acceleration Stack User Guide: Intel FPGA Programmable Acceleration Card N3000

Download PDF
ID 683040
Date 6/14/2021
Public
Document Table of Contents
Document Table of Contents x
1. About this Document 2. System Requirements 3. Hardware Installation 4. Installing the OPAE Software 5. OPAE Tools 6. Sample Test: Native Loopback 7. Installing the Intel XL710 Driver 8. Configuring Ethernet Interfaces 9. Testing Network Loopback Using Data Plane Development Kit (DPDK) 10. Graceful Shutdown 11. Single Event Upset (SEU) 12. Document Revision History for Intel Acceleration Stack User Guide: Intel® FPGA PAC N3000 A. Troubleshooting B. Upgrade your Intel® FPGA PAC N3000 with Production Version of BMC and Intel® Arria® 10 Image C. Configure the 4.19 Kernel D. fpgabist Sample Output
1. About this Document x
1.1. Acronym List
2. System Requirements x
2.1. Cooling Requirements
3. Hardware Installation x
3.1. Installing the Intel® FPGA PAC N3000
4. Installing the OPAE Software x
4.1. Install Additional Packages 4.2. Install the Release Package 4.3. Identify the Intel® MAX® 10 Version on your Intel® FPGA PAC N3000
4.2. Install the Release Package x
4.2.1. Remove Previous OPAE Packages 4.2.2. Install the Acceleration Stack for Runtime 4.2.3. Install the Acceleration Stack for Development 4.2.4. Verify the OPAE Installation 4.2.5. Install the Configuration Files
4.3. Identify the Intel® MAX® 10 Version on your Intel® FPGA PAC N3000 x
4.3.1. FPGA Factory Image Overview
5. OPAE Tools x
5.1. Using fpgasupdate 5.2. Using fpgainfo 5.3. Test PCIe and External Memories with fpgabist 5.4. Test Network Loopback using fpgadiag
7. Installing the Intel XL710 Driver x
7.1. Updating the Intel XL710 Firmware
8. Configuring Ethernet Interfaces x
8.1. Modifying the Interface Maximum Transmission Unit (MTU) Size 8.2. Setting Forward Error Correction (FEC) Mode 8.3. Ethernet Pause Flow Control 8.4. Get Link Status and Statistics
9. Testing Network Loopback Using Data Plane Development Kit (DPDK) x
9.1. Test Using an External Traffic Generator 9.2. Test Using a Packet Generator 9.3. Troubleshooting in DPDK 9.4. Revert Back from DPDK to OPAE
10. Graceful Shutdown x
10.1. Background 10.2. Using OPAE 10.3. Using DPDK
11. Single Event Upset (SEU) x
11.1. OPAE Handling of SEU 11.2. DPDK Handling of SEU
A. Troubleshooting x
A.1. If OPAE installation verification fails, how to install OPAE manually?
B. Upgrade your Intel® FPGA PAC N3000 with Production Version of BMC and Intel® Arria® 10 Image x
B.1. Upgrading from 1.1 Beta to Production Version B.2. Upgrading from 1.1 Alpha-2 or Older to Production Version
B.1. Upgrading from 1.1 Beta to Production Version x
B.1.1. Root Entry Hash Programmed B.1.2. Root Entry Hash Not Programmed
D. fpgabist Sample Output x
D.1. For 2x2x25G Configuration D.2. For 8x10G Configuration
1. About this Document
2. System Requirements
3. Hardware Installation
4. Installing the OPAE Software
5. OPAE Tools
6. Sample Test: Native Loopback
7. Installing the Intel XL710 Driver
8. Configuring Ethernet Interfaces
9. Testing Network Loopback Using Data Plane Development Kit (DPDK)
10. Graceful Shutdown
11. Single Event Upset (SEU)
12. Document Revision History for Intel Acceleration Stack User Guide: Intel® FPGA PAC N3000
A. Troubleshooting
B. Upgrade your Intel® FPGA PAC N3000 with Production Version of BMC and Intel® Arria® 10 Image
C. Configure the 4.19 Kernel
D. fpgabist Sample Output

9. Testing Network Loopback Using Data Plane Development Kit (DPDK)

Before starting Data Plane Development Kit (DPDK), you must perform configuration steps described in Configuring Ethernet Interfaces as it relies on the FPGA being bound to OPAE driver (pac_n3000_net). While using DPDK, FPGA is unbound from this driver and bound to the vfio-pci driver.

Follow these steps to install DPDK for testing network loopback:
  1. You must enable the Intel IOMMU driver on the host. Complete the following steps to enable the Intel IOMMU driver:
    1. Add iommu=pt intel_iommu=on to the GRUB_CMDLINE_LINUX entry by editing /etc/default/grub. For example:
      GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/\
swap rhgb quiet pci=realloc intel_iommu=pt"

      For RHEL: Additionally, add pci=realloc to GRUB_CMDLINE_LINUX entry.

    2. GRUB reads its configuration from either the /boot/grub2/grub.cfg file on traditional BIOS-based machines or from the /boot/efi/EFI/redhat/ grub.cfg file on UEFI machines. Depending on your system, execute one of the instructions below as root:
      • BIOS based machine: 
        grub2-mkconfig -o /boot/grub2/grub.cfg
      • UEFI based machine: 
        grub2-mkconfig -o /boot/efi/EFI/redhat/ grub.cfg
        # grub2-mkconfig -o /boot/efi/EFI/redhat/
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-594cabaaf9a84c6ea0a5167c89ad916d
Found initrd image: /boot/initramfs-0-rescue-594cabaaf9a84c6ea0a5167c89ad916d.img
/usr/sbin/grub2-mkconfig: line 290: /boot/efi/EFI/redhat/: Is a directory
    3. Reboot the server to apply the new GRUB configuration file.
    4. To verify the GRUB update, run the following command:
      $ cat /proc/cmdline
      The sample output below shows intel_iommu=on on the kernel command line.
      BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/rhel-root ro default_hugepagesz=1G hugepagesz=1G hugepages=64 hugepagesz=2M hugepages=2048 nosoftlockup mce=ignore_ce audit=0 isolcpus=1-11,24-35,13-23,36-47 nohz_full=1-11,24-35,13-23,36-47 rcu_nocbs=1-11,24-35,13-23,36-47 pci=realloc intel_iommu=on iommu=pt enforcing=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet skew_tick=1
  2. Install the required packages:
    $	sudo yum install readline-devel libpcap libpcap-devel numactl-devel
    You need to download these two extra Fedora packages: To install: 
    $ sudo rpm -i <RPM file>
    To check installation: 
    $ rpm -qa | grep libfdt
  3. Compile and bind drivers:
    1. Download DPDK code from DPDK community and checkout release 19.08.
      $ git clone https://github.com/DPDK/dpdk.git
      $ cd dpdk
      $ git pull
      Already up-to-date.
      $ git reset --hard 31b798a6f08e9b333b94b8bb26910209aa810b73
      HEAD is now at 31b798a build: avoid overlinking
      $ wget https://patches.dpdk.org/series/6821/mbox/
      Length: 179795 (176K) [text/plain]
Saving to: \u2018index.html\u2019

100%[==============>] 179,795      554KB/s   in 0.3s   

(554 KB/s) - \u2018index.html\u2019 saved [179795/179795]
      $ git am index.html
      Applying: net/i40e: i40e support ipn3ke FPGA port bonding
Applying: raw/ifpga/base: add irq support
Applying: raw/ifpga/base: clear pending bit
Applying: raw/ifpga/base: add SEU error support
Applying: raw/ifpga/base: add device tree support
Applying: raw/ifpga/base: align the send buffer for SPI
Applying: raw/ifpga/base: add sensor support
Applying: raw/ifpga/base: introducing sensor APIs
Applying: raw/ifpga/base: update SEU register definition
Applying: raw/ifpga: add SEU error handler
Applying: raw/ifpga: add PCIe BDF devices tree scan
Applying: net/ipn3ke: remove configuration for i40e port bonding
Applying: raw/ifpga/base: add secure support
Applying: raw/ifpga/base: configure FEC mode
Applying: raw/ifpga/base: clean fme errors
Applying: raw/ifpga/base: add new API get board info
Applying: raw/ifpga: add lightweight fpga image support
Applying: raw/ifpga/base: add multiple cards support

      $ vim config/common_linux
// modify CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=n to CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y
      $ vim config/common_base
// modify CONFIG_RTE_LIBRTE_IPN3KE_PMD=n to CONFIG_RTE_LIBRTE_IPN3KE_PMD=y
    2. Build DPDK and export RTE_SDK path to point to dpdk directory:
      Note: Ignore the message “Build complete [x86_64-native-linuxapp-gcc] Installation cannot run with T defined and DESTDIR undefined" 
      $	export RTE_SDK=$PWD
      $	export RTE_TARGET=x86_64-native-linuxapp-gcc
      $	make config T=x86_64-native-linuxapp-gcc
      $	make install -j8 T=x86_64-native-linuxapp-gcc
      Note: $RTE_SDK points to the extracted dpdk source location.
    3. Bind FPGA and NIC to vfio-pci driver as shown below. The sample output below shows result for 2x2x25G configuration. The 8x10G configuration produces similar output with following exception:
      Table 11.  Output Differences
      Configuration Ports per Intel Ethernet Controller XL710-BM2 NIC Device ID for Intel Ethernet Controller XL710-BM2 NIC Ports
      2x2x25G 2 0d58
      8x10G 4 0cf8 
      $	cd $RTE_SDK
      Check the binding between driver and device with the following command: 
      $	./usertools/dpdk-devbind.py --status-dev net
      Sample output:
Network devices using DPDK-compatible driver
============================================
<none>
Network devices using kernel driver
===================================
0000:18:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em1 drv=tg3 unused= 
0000:18:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em2 drv=tg3 unused= *Active*
0000:19:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em3 drv=tg3 unused= 
0000:19:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em4 drv=tg3 unused= 
0000:14:00.0 'Device 0d58' if=p1p1 drv=i40e unused= 
0000:14:00.1 'Device 0d58' if=p1p2 drv=i40e unused= 
0000:16:00.0 'Device 0d58' if=p1p3 drv=i40e unused= 
0000:16:00.1 'Device 0d58' if=p1p4 drv=i40e unused= 
Other Network devices
=====================
0000:15:00.0 'Device 0b30' unused=intel_fpga_pci
      Install the vfio-pci kernel driver: 
      $	sudo modprobe vfio-pci
      Bind the FPGA and FVL PFs to DPDK driver. Replace Bus:Device.Function with your output from above step. In case of 8x10G image, bind all 8 FVL B:D.F and the FPGA to vfio_pci: 
      $	sudo ./usertools/dpdk-devbind.py -b vfio-pci 14:00.0 \
14:00.1 15:00.0 16:00.0 16:00.1
      Rerun ./usertools/dpdk-devbind.py –-status-dev net to check that FPGA and FVL PF’s are bound to vfio-pci driver.
      Figure 17. Sample Output
    4. Reserve hugepages:
      $ sudo mkdir -p /mnt/huge
      $ sudo mount -t hugetlbfs nodev /mnt/huge
      $ sudo sh -c "echo 2048 > /sys/kernel/mm/hugepages/\
hugepages-2048kB/nr_hugepages"

For more information on how the FPGA support is enabled in DPDK, refer to Data Plane Development Kit Reference Manual: Intel FPGA Programmable Acceleration Card N3000 .


Section Content
Test Using an External Traffic Generator
Test Using a Packet Generator
Troubleshooting in DPDK
Revert Back from DPDK to OPAE

Level Two Title