1. Add iommu=pt intel_iommu=on to the GRUB_CMDLINE_LINUX entry by editing /etc/default/grub. For example:

   GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/\
   swap rhgb quiet pci=realloc intel_iommu=on iommu=pt"
   

   For RHEL: Additionally, add pci=realloc to GRUB_CMDLINE_LINUX entry.

2. GRUB reads its configuration from either the /boot/grub2/grub.cfg file on traditional BIOS-based machines or from the /boot/efi/EFI/redhat/ grub.cfg file on UEFI machines. Depending on your system, execute one of the instructions below as root:
   • BIOS based machine:

     grub2-mkconfig -o /boot/grub2/grub.cfg

   • UEFI based machine:

     grub2-mkconfig -o /boot/efi/EFI/redhat/ grub.cfg

     # grub2-mkconfig -o /boot/efi/EFI/redhat/
     Generating grub configuration file ...
     Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64
     Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img
     Found linux image: /boot/vmlinuz-0-rescue-594cabaaf9a84c6ea0a5167c89ad916d
     Found initrd image: /boot/initramfs-0-rescue-594cabaaf9a84c6ea0a5167c89ad916d.img
     /usr/sbin/grub2-mkconfig: line 290: /boot/efi/EFI/redhat/: Is a directory
     

3. Reboot the server to apply the new GRUB configuration file.
4. To verify the GRUB update, run the following command:

   $ cat /proc/cmdline

   The sample output below shows intel_iommu=on on the kernel command line.

   BOOT_IMAGE=/vmlinuz-3.10.0-957.el7.x86_64 root=/dev/mapper/rhel-root ro default_hugepagesz=1G hugepagesz=1G hugepages=64 hugepagesz=2M hugepages=2048 nosoftlockup mce=ignore_ce audit=0 isolcpus=1-11,24-35,13-23,36-47 nohz_full=1-11,24-35,13-23,36-47 rcu_nocbs=1-11,24-35,13-23,36-47 pci=realloc intel_iommu=on iommu=pt enforcing=0 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet skew_tick=1

$	sudo yum install readline-devel libpcap libpcap-devel numactl-devel

$ sudo rpm -i <RPM file>

$ rpm -qa | grep libfdt

1. Download DPDK code from DPDK community and checkout release 19.11.

   $ git clone https://github.com/DPDK/dpdk.git

   $ cd dpdk

   $ git pull

   Already up-to-date.

   $ git reset --hard 31b798a6f08e9b333b94b8bb26910209aa810b73

   HEAD is now at 31b798a build: avoid overlinking

   $ wget https://patches.dpdk.org/series/6821/mbox/

   Length: 179795 (176K) [text/plain]
   Saving to: \u2018index.html\u2019
   
   100%[==============>] 179,795      554KB/s   in 0.3s   
   
   (554 KB/s) - \u2018index.html\u2019 saved [179795/179795]

   $ git am index.html

   Applying: net/i40e: i40e support ipn3ke FPGA port bonding
   Applying: raw/ifpga/base: add irq support
   Applying: raw/ifpga/base: clear pending bit
   Applying: raw/ifpga/base: add SEU error support
   Applying: raw/ifpga/base: add device tree support
   Applying: raw/ifpga/base: align the send buffer for SPI
   Applying: raw/ifpga/base: add sensor support
   Applying: raw/ifpga/base: introducing sensor APIs
   Applying: raw/ifpga/base: update SEU register definition
   Applying: raw/ifpga: add SEU error handler
   Applying: raw/ifpga: add PCIe BDF devices tree scan
   Applying: net/ipn3ke: remove configuration for i40e port bonding
   Applying: raw/ifpga/base: add secure support
   Applying: raw/ifpga/base: configure FEC mode
   Applying: raw/ifpga/base: clean fme errors
   Applying: raw/ifpga/base: add new API get board info
   Applying: raw/ifpga: add lightweight fpga image support
   Applying: raw/ifpga/base: add multiple cards support
   

   $ vim config/common_linux
   // modify CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=n to CONFIG_RTE_LIBRTE_PMD_IFPGA_RAWDEV=y

   $ vim config/common_base
   // modify CONFIG_RTE_LIBRTE_IPN3KE_PMD=n to CONFIG_RTE_LIBRTE_IPN3KE_PMD=y

2. Build DPDK and export RTE_SDK path to point to dpdk directory:
   Note: Ignore the message “Build complete [x86_64-native-linuxapp-gcc] Installation cannot run with T defined and DESTDIR undefined"

   $	export RTE_SDK=$PWD

   $	export RTE_TARGET=x86_64-native-linuxapp-gcc

   $	make config T=x86_64-native-linuxapp-gcc

   $	make install -j8 T=x86_64-native-linuxapp-gcc

   Note: $RTE_SDK points to the extracted dpdk source location.
3. Bind FPGA and NIC to vfio-pci driver as shown below. The sample output below shows result for 2x2x25G configuration.

   Table 8.  Output Differences

   Configuration	Ports per Intel Ethernet Controller XL710-BM2 NIC	Device ID for Intel Ethernet Controller XL710-BM2 NIC Ports
   2x2x25G	2	0d58
   4x25G	2	0d58

   $	cd $RTE_SDK

   Check the binding between driver and device with the following command:

   $	./usertools/dpdk-devbind.py --status-dev net

   Sample output:
   Network devices using DPDK-compatible driver
   ============================================
   <none>
   Network devices using kernel driver
   ===================================
   0000:18:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em1 drv=tg3 unused= 
   0000:18:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em2 drv=tg3 unused= *Active*
   0000:19:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em3 drv=tg3 unused= 
   0000:19:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe 165f' if=em4 drv=tg3 unused= 
   0000:14:00.0 'Device 0d58' if=p1p1 drv=i40e unused= 
   0000:14:00.1 'Device 0d58' if=p1p2 drv=i40e unused= 
   0000:16:00.0 'Device 0d58' if=p1p3 drv=i40e unused= 
   0000:16:00.1 'Device 0d58' if=p1p4 drv=i40e unused= 
   Other Network devices
   =====================
   0000:15:00.0 'Device 0b30' unused=intel_fpga_pci
   

   Install the vfio-pci kernel driver:

   $	sudo modprobe vfio-pci
   

   Bind the FPGA and FVL PFs to DPDK driver. Replace Bus:Device.Function with your output from above step.

   $	sudo ./usertools/dpdk-devbind.py -b vfio-pci 14:00.0 \
   14:00.1 15:00.0 16:00.0 16:00.1
   

   Rerun ./usertools/dpdk-devbind.py –-status-dev net to check that FPGA and FVL PF’s are bound to vfio-pci driver.
   Figure 11. Sample Output

   
4. Reserve hugepages:

   $ sudo mkdir -p /mnt/huge

   $ sudo mount -t hugetlbfs nodev /mnt/huge

   $ sudo sh -c "echo 2048 > /sys/kernel/mm/hugepages/\
   hugepages-2048kB/nr_hugepages"