Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Guide

Download PDF
ID 683517
Date 10/06/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Introduction 2. Design Example Detailed Description 3. Design Example Quick Start Guide 4. Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide Archives 5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide
1. Introduction x
1.1. Terms and Acronyms 1.2. MCDMA IP Modes
2. Design Example Detailed Description x
2.1. Design Example Overview 2.2. Hardware and Software Requirements 2.3. PIO using MCDMA Bypass Mode 2.4. Avalon-ST Packet Generate/Check 2.5. Avalon-ST Device-side Packet Loopback 2.6. Avalon-MM DMA 2.7. BAM_BAS Traffic Generator and Checker 2.8. External Descriptor Controller
2.1. Design Example Overview x
2.1.1. H-Tile MCDMA IP - Design Examples for Endpoint 2.1.2. P-Tile MCDMA IP - Design Examples for Endpoint 2.1.3. F-Tile MCDMA IP - Design Examples for Endpoint 2.1.4. R-Tile MCDMA IP - Design Examples for Endpoint
2.3. PIO using MCDMA Bypass Mode x
2.3.1. Avalon-ST PIO using MCDMA Bypass Mode 2.3.2. Avalon-MM PIO using MCDMA Bypass Mode
2.3.1. Avalon-ST PIO using MCDMA Bypass Mode x
2.3.1.1. Single-Port Avalon-ST PIO using MCDMA Bypass Mode
2.3.2. Avalon-MM PIO using MCDMA Bypass Mode x
2.3.2.1. Simulation Results 2.3.2.2. Hardware Test Results
2.4. Avalon-ST Packet Generate/Check x
2.4.1. Single-Port Avalon-ST Packet Generate/Check
2.4.1. Single-Port Avalon-ST Packet Generate/Check x
2.4.1.1. Simulation Waveforms 2.4.1.2. Simulation Log 2.4.1.3. Hardware Test Result
2.5. Avalon-ST Device-side Packet Loopback x
2.5.1. Simulation Results 2.5.2. Hardware Test Results
2.6. Avalon-MM DMA x
2.6.1. Simulation Results 2.6.2. Hardware Test Results
2.7. BAM_BAS Traffic Generator and Checker x
2.7.1. BAM_BAS Traffic Generator and Checker Example Design Register Map
2.8. External Descriptor Controller x
2.8.1. Registers 2.8.2. Hardware Test Results
2.8.1. Registers x
2.8.1.1. GCSR Registers (Base address 64’h00000) 2.8.1.2. QCSR Registers(Base address 64’h10000)
3. Design Example Quick Start Guide x
3.1. Design Example Directory Structure 3.2. Generating the Example Design using Intel® Quartus® Prime 3.3. Simulating the Design Example 3.4. Compiling the Example Design in Intel® Quartus® Prime 3.5. Running the Design Example Application on a Hardware Setup
3.2. Generating the Example Design using Intel® Quartus® Prime x
3.2.1. Procedure
3.3. Simulating the Design Example x
3.3.1. Testbench Overview 3.3.2. Supported Simulators 3.3.3. Example Testbench Flow for DMA Test with Avalon-ST Packet Generate/Check Design Example 3.3.4. Run the Simulation Script 3.3.5. Steps to Run the Simulation 3.3.6. View the Results
3.3.5. Steps to Run the Simulation x
3.3.5.1. Steps to Run the Simulation : QuestaSim* / Questa* Intel® FPGA Edition 3.3.5.2. Steps to Run the Simulation : VCS* / VCS* MX 3.3.5.3. Steps to Run the Simulation : Xcelium*
3.5. Running the Design Example Application on a Hardware Setup x
3.5.1. Program the FPGA 3.5.2. Quick Start Guide
3.5.2. Quick Start Guide x
3.5.2.1. Software Test Setup 3.5.2.2. Driver Support 3.5.2.3. MCDMA Custom Driver 3.5.2.4. MCDMA DPDK Poll Mode Driver 3.5.2.5. MCDMA Kernel Mode Network Device Driver
3.5.2.3. MCDMA Custom Driver x
3.5.2.3.1. Prerequisites 3.5.2.3.2. Software Setup 3.5.2.3.3. Run the Reference Example Application 3.5.2.3.4. Testing Bitstream Configuration beyond 256 Channels (for MCDMA Custom Driver) 3.5.2.3.5. External Descriptor Mode Verification
3.5.2.3.1. Prerequisites x
3.5.2.3.1.1. Configuration Changes from BIOS 3.5.2.3.1.2. External Packages 3.5.2.3.1.3. Set the Boot Parameters
3.5.2.3.2. Software Setup x
3.5.2.3.2.1. Installing the Linux Kernel Driver 3.5.2.3.2.2. Build and Install User Space Library 3.5.2.3.2.3. Enabling VFs and Create Guest VM by Using QEMU 3.5.2.3.2.4. Establish Communication Between Host and QEMU
3.5.2.3.3. Run the Reference Example Application x
3.5.2.3.3.1. Meta Data Test 3.5.2.3.3.2. Custom PIO Read Write Test 3.5.2.3.3.3. BAM Test 3.5.2.3.3.4. BAS Test 3.5.2.3.3.5. Packet Gen Test
3.5.2.4. MCDMA DPDK Poll Mode Driver x
3.5.2.4.1. Prerequisites 3.5.2.4.2. Install PMD and Test Application (for CentOS) 3.5.2.4.3. Install PMD and Test Application (for Ubuntu) 3.5.2.4.4. Create VM Environment 3.5.2.4.5. Run the Reference Example Application
3.5.2.4.1. Prerequisites x
3.5.2.4.1.1. Set the Boot Parameters 3.5.2.4.1.2. Configuration Changes from BIOS 3.5.2.4.1.3. Install and Build Testpmd
3.5.2.4.4. Create VM Environment x
3.5.2.4.4.1. Enable VFs 3.5.2.4.4.2. Create Guest VM by using QEMU 3.5.2.4.4.3. Establish Communication Between Host and VM
3.5.2.4.5. Run the Reference Example Application x
3.5.2.4.5.1. Channel ID VF PF Verification 3.5.2.4.5.2. Example of Verifying on an AVMM Design 3.5.2.4.5.3. Testing Bitstream Configuration beyond 256 Channels (for DPDK Poll Mode Driver) 3.5.2.4.5.4. BAM Test 3.5.2.4.5.5. BAS Test
3.5.2.5. MCDMA Kernel Mode Network Device Driver x
3.5.2.5.1. Build and Install Netdev Driver 3.5.2.5.2. Enable VFs if SRIOV is Supported 3.5.2.5.3. Configure the Number of Channels Supported on the Device 3.5.2.5.4. Configure the MTU Value 3.5.2.5.5. Configure the Device Communication 3.5.2.5.6. Configure Transmit Queue Selection Mechanism 3.5.2.5.7. Test Procedure by Using Name Space Environment 3.5.2.5.8. PIO Test
3.5.2.5.6. Configure Transmit Queue Selection Mechanism x
3.5.2.5.6.1. MCDMA Queue Selection 3.5.2.5.6.2. Transmit Packet Steering (XPS) 3.5.2.5.6.3. Default
1. Introduction
2. Design Example Detailed Description
3. Design Example Quick Start Guide
4. Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide Archives
5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide

3.5.2.3.1.3. Set the Boot Parameters

Follow the steps below to modify the default hugepages setting in the grub files:
  1. Edit the /etc/default/grub file

    Append the highlighted parameters to the GRUB_CMDLINE_LINUX line in the /etc/default/grub file

    CentOS: GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on iommu=pt

    Ubuntu: GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 intel_iommu=on iommu=pt panic=1 quiet splash vt.handoff=7"

    File contents after the edit for CentOS is shown below: 

    GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX=" rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb default_hugepagesz=1G hugepagesz=1G hugepages=40 panic=1 intel_iommu=on iommu=pt
GRUB_DISABLE_RECOVERY="true"

    In the case of memory allocation failure at the time of Virtual Function creation, add the following boot parameters: "pci=hpbussize=10,hpmemsize=2M,nocrs,realloc=on"

    To bind the device to vfio-pci and use IOMMU, enable the following parameter: intel_iommu=on

    To use UIO and not enable the IOMMU lookup, add the following parameter: iommu=pt

    To use the AMD platform and the UIO driver, add the following parameter at boot time: iommu=soft

    An example /etc/default/grub file on ubuntu after the edits can be seen below: 
    root@bapvecise042:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update 

# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see: 

#   info -f grub -n 'Simple configuration' 

GRUB_DEFAULT="1>2" 

GRUB_TIMEOUT_STYLE=hidden 

GRUB_TIMEOUT=0 

GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` 

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" 

GRUB_CMDLINE_LINUX="default_hugepagesz=1G hugepagesz=1G hugepages=20 intel_iommu=on iommu=pt panic=1" 

# Uncomment to enable BadRAM filtering, modify to suit your needs 

# This works with Linux (no patch required) and with any kernel that obtains 

# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) 

#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" 

# Uncomment to disable graphical terminal (grub-pc only) 

#GRUB_TERMINAL=console 
  

# The resolution used on graphical terminal 

# note that you can use only modes which your graphic card supports via VBE 

# you can see them in real GRUB with the command `vbeinfo' 

#GRUB_GFXMODE=640x480 

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux 

#GRUB_DISABLE_LINUX_UUID=true 
  

# Uncomment to disable generation of recovery mode menu entries 

#GRUB_DISABLE_RECOVERY="true" 
  

# Uncomment to get a beep at grub start 

#GRUB_INIT_TUNE="480 440 1"
  2. Generate GRUB configuration files.

    To check whether the boot system is legacy or EFI-based, check the existence of the following file:

    $ls -al /sys/firmware/efi

    If this file is present, the boot system is EFI-based. Otherwise, it is a legacy system.
    1. In case of a legacy system, execute the following command:

      $ grub2-mkconfig -o /boot/grub2/grub.cfg

    2. In case of an EFI-based system, execute the following command:

      $ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

    3. In case of Ubuntu, execute the following command: 
      grub-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
      or 
      grub2-mkconfig -o /boot/efi/EFI/ubuntu/grub.cfg
      or 
      sudo grub update
  3. Reboot the system.
  4. Verify the changes above:

    $ cat /proc/cmdline

  5. Set the huge pages:

    $ echo 40 > /proc/sys/vm/nr_hugepages

  6. If host supports multiple NUMAs, follow the following steps:
    1. Check how many NUMAs enabled on host.
      $lscpu | grep NUMA
NUMA node(s): 2
NUMA node0 CPU(s): 0-15, 32-47
NUMA node1 CPU(s): 16-31, 48-63

      In this example, we have 2 NUMAs. If only one NUMA is present, ignore this step:

    2. Check which device is provisioned:
      $cat /sys/class/pci_bus/<Domain:Bus>\
/device/numa_node
    3. Enable the Huge pages, which ever NUMA, device is located:
      $echo 40> /sys/devices/system/node/node<nodenum>\
/hugepages/hugepages-1048576kB/nr_hugepages
    4. Configure thread sequence based on which NUMA device is located, for example:
      #define THREAD_SEQ "0-15" in software/user/cli/\
perfq_app/perfq_app.h
Level Two Title