Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Guide

ID 683517
Date 7/30/2024
Public
Document Table of Contents

3.5.2.3.1.3. Set the Boot Parameters

Follow the steps below to modify the default hugepages setting in the grub files:
  1. Edit the /etc/default/grub file

    Append the highlighted parameters to the GRUB_CMDLINE_LINUX line in the /etc/default/grub file

    CentOS: GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on iommu=pt

    Ubuntu: GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 intel_iommu=on iommu=pt panic=1 quiet splash vt.handoff=7"

    File contents after the edit for CentOS is shown below:

    GRUB_TIMEOUT=5
    GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
    GRUB_DEFAULT=saved
    GRUB_DISABLE_SUBMENU=true
    GRUB_TERMINAL_OUTPUT="console"
    GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on iommu=pt
    GRUB_DISABLE_RECOVERY="true"

    In the case of memory allocation failure at the time of Virtual Function creation, add the following boot parameters: "pci=hpbussize=10,hpmemsize=2M,nocrs,realloc=on"

    To bind the device to vfio-pci and use IOMMU, enable the following parameter: intel_iommu=on

    To use UIO and not enable the IOMMU lookup, add the following parameter: iommu=pt

    To use the AMD platform and the UIO driver, add the following parameter at boot time: iommu=soft

    An example /etc/default/grub file on ubuntu after the edits can be seen below:
    root@bapvecise042:~# cat /etc/default/grub
    # If you change this file, run 'update-grub' afterwards to update 
    
    # /boot/grub/grub.cfg.
    # For full documentation of the options in this file, see: 
    
    #   info -f grub -n 'Simple configuration' 
    
    GRUB_DEFAULT="1>2" 
    
    GRUB_TIMEOUT_STYLE=hidden 
    
    GRUB_TIMEOUT=0 
    
    GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` 
    
    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" 
    
    GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 intel_iommu=on iommu=pt panic=1" 
    
    # Uncomment to enable BadRAM filtering, modify to suit your needs 
    
    # This works with Linux (no patch required) and with any kernel that obtains 
    
    # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) 
    
    #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" 
    
    # Uncomment to disable graphical terminal (grub-pc only) 
    
    #GRUB_TERMINAL=console 
      
    
    # The resolution used on graphical terminal 
    
    # note that you can use only modes which your graphic card supports via VBE 
    
    # you can see them in real GRUB with the command `vbeinfo' 
    
    #GRUB_GFXMODE=640x480 
    
    # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux 
    
    #GRUB_DISABLE_LINUX_UUID=true 
      
    
    # Uncomment to disable generation of recovery mode menu entries 
    
    #GRUB_DISABLE_RECOVERY="true" 
      
    
    # Uncomment to get a beep at grub start 
    
    #GRUB_INIT_TUNE="480 440 1" 
  2. Generate GRUB configuration files.

    To check whether the boot system is legacy or EFI-based, check the existence of the following file:

    $ls -al /sys/firmware/efi

    If this file is present, the boot system is EFI-based. Otherwise, it is a legacy system.
    1. In case of a legacy system, execute the following command:

      $ grub2-mkconfig -o /boot/grub2/grub.cfg

    2. In case of an EFI-based system, execute the following command:

      $ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

    3. In case of Ubuntu, execute the following command:
      grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
      or
      grub2-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg 
      
      or
      sudo grub update
  3. Reboot the system.
  4. Verify the changes above:

    $ cat /proc/cmdline

  5. Set the huge pages:

    $ echo 40 > /proc/sys/vm/nr_hugepages

  6. If host supports multiple NUMAs, follow the following steps:
    1. Check how many NUMAs enabled on host.
      $lscpu | grep NUMA
      NUMA node(s): 2
      NUMA node0 CPU(s): 0-15, 32-47
      NUMA node1 CPU(s): 16-31, 48-63

      In this example, we have 2 NUMAs. If only one NUMA is present, ignore this step:

    2. Check which device is provisioned:
      $cat /sys/class/pci_bus/<Domain:Bus>\
      /device/numa_node
    3. Enable the Huge pages, which ever NUMA, device is located:
      $echo 40> /sys/devices/system/node/node<nodenum>\
      /hugepages/hugepages-1048576kB/nr_hugepages
    4. Configure thread sequence based on which NUMA device is located, for example:
      #define THREAD_SEQ "0-15" in software/user/cli/\
      perfq_app/perfq_app.h