Multi Channel DMA Intel® FPGA IP for PCI Express Design Example User Guide

ID 683517
Date 2/14/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents

3.5.2.3.1.3. Set the Boot Parameters

Follow the steps below to modify the default hugepages setting in the grub files:
  1. Edit the /etc/default/grub file

    Append the highlighted parameters to the GRUB_CMDLINE_LINUX line in the /etc/default/grub file

    GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on iommu=pt

    This is what the file looks like after the edit:

    GRUB_TIMEOUT=5

    GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

    GRUB_DEFAULT=saved

    GRUB_DISABLE_SUBMENU=true

    GRUB_TERMINAL_OUTPUT="console"

    GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on iommu=pt

    GRUB_DISABLE_RECOVERY="true"

    In the case of memory allocation failure at the time of Virtual Function creation, add the following boot parameters: "pci=hpbussize=10,hpmemsize=2M,nocrs,realloc=on"

    To bind the device to vfio-pci and use IOMMU, enable the following parameter: intel_iommu=on

    To use UIO and not enable the IOMMU lookup, add the following parameter: iommu=pt

    To use the AMD platform and the UIO driver, add the following parameter at boot time: iommu=soft

  2. Generate GRUB configuration files.

    To check whether the boot system is legacy or EFI-based, check the existence of the following file:

    $ls -al /sys/firmware/efi

    If this file is present, the boot system is EFI-based. Otherwise, it is a legacy system.
    1. In case of a legacy system, execute the following command:

      $ grub2-mkconfig -o /boot/grub2/grub.cfg

    2. In case of an EFI-based system, execute the following command:

      $ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

    3. In case of Ubuntu, execute the following command:
      grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
      or
      grub2-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg 
      
      or
      sudo grub update
  3. Reboot the system.
  4. Verify the changes above:

    $ cat /proc/cmdline

  5. Set the huge pages:

    $ echo 40 > /proc/sys/vm/nr_hugepages

  6. If host supports multiple NUMAs, follow the following steps:
    1. Check how many NUMAs enabled on host.
      $lscpu | grep NUMA
      NUMA node(s): 2
      NUMA node0 CPU(s): 0-15, 32-47
      NUMA node1 CPU(s): 16-31, 48-63

      In this example, we have 2 NUMAs. If only one NUMA is present, ignore this step:

    2. Check which device is provisioned:
      $cat /sys/class/pci_bus/<Domain:Bus>\
      /device/numa_node
    3. Enable the Huge pages, which ever NUMA, device is located:
      $echo 40> /sys/devices/system/node/node<nodenum>\
      /hugepages/hugepages-1048576kB/nr_hugepages
    4. Configure thread sequence based on which NUMA device is located, for example:
      #define THREAD_SEQ "0-15" in software/user/cli/\
      perfq_app/perfq_app.h