Multi Channel DMA Intel® FPGA IP for PCI Express* Design Example User Guide

Download PDF
ID 683517
Date 10/06/2023
Public

A newer version of this document is available. Customers should click here to go to the newest version.

Document Table of Contents
Document Table of Contents x
1. Introduction 2. Design Example Detailed Description 3. Design Example Quick Start Guide 4. Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide Archives 5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide
1. Introduction x
1.1. Terms and Acronyms 1.2. MCDMA IP Modes
2. Design Example Detailed Description x
2.1. Design Example Overview 2.2. Hardware and Software Requirements 2.3. PIO using MCDMA Bypass Mode 2.4. Avalon-ST Packet Generate/Check 2.5. Avalon-ST Device-side Packet Loopback 2.6. Avalon-MM DMA 2.7. BAM_BAS Traffic Generator and Checker 2.8. External Descriptor Controller
2.1. Design Example Overview x
2.1.1. H-Tile MCDMA IP - Design Examples for Endpoint 2.1.2. P-Tile MCDMA IP - Design Examples for Endpoint 2.1.3. F-Tile MCDMA IP - Design Examples for Endpoint 2.1.4. R-Tile MCDMA IP - Design Examples for Endpoint
2.3. PIO using MCDMA Bypass Mode x
2.3.1. Avalon-ST PIO using MCDMA Bypass Mode 2.3.2. Avalon-MM PIO using MCDMA Bypass Mode
2.3.1. Avalon-ST PIO using MCDMA Bypass Mode x
2.3.1.1. Single-Port Avalon-ST PIO using MCDMA Bypass Mode
2.3.2. Avalon-MM PIO using MCDMA Bypass Mode x
2.3.2.1. Simulation Results 2.3.2.2. Hardware Test Results
2.4. Avalon-ST Packet Generate/Check x
2.4.1. Single-Port Avalon-ST Packet Generate/Check
2.4.1. Single-Port Avalon-ST Packet Generate/Check x
2.4.1.1. Simulation Waveforms 2.4.1.2. Simulation Log 2.4.1.3. Hardware Test Result
2.5. Avalon-ST Device-side Packet Loopback x
2.5.1. Simulation Results 2.5.2. Hardware Test Results
2.6. Avalon-MM DMA x
2.6.1. Simulation Results 2.6.2. Hardware Test Results
2.7. BAM_BAS Traffic Generator and Checker x
2.7.1. BAM_BAS Traffic Generator and Checker Example Design Register Map
2.8. External Descriptor Controller x
2.8.1. Registers 2.8.2. Hardware Test Results
2.8.1. Registers x
2.8.1.1. GCSR Registers (Base address 64’h00000) 2.8.1.2. QCSR Registers(Base address 64’h10000)
3. Design Example Quick Start Guide x
3.1. Design Example Directory Structure 3.2. Generating the Example Design using Intel® Quartus® Prime 3.3. Simulating the Design Example 3.4. Compiling the Example Design in Intel® Quartus® Prime 3.5. Running the Design Example Application on a Hardware Setup
3.2. Generating the Example Design using Intel® Quartus® Prime x
3.2.1. Procedure
3.3. Simulating the Design Example x
3.3.1. Testbench Overview 3.3.2. Supported Simulators 3.3.3. Example Testbench Flow for DMA Test with Avalon-ST Packet Generate/Check Design Example 3.3.4. Run the Simulation Script 3.3.5. Steps to Run the Simulation 3.3.6. View the Results
3.3.5. Steps to Run the Simulation x
3.3.5.1. Steps to Run the Simulation : QuestaSim* / Questa* Intel® FPGA Edition 3.3.5.2. Steps to Run the Simulation : VCS* / VCS* MX 3.3.5.3. Steps to Run the Simulation : Xcelium*
3.5. Running the Design Example Application on a Hardware Setup x
3.5.1. Program the FPGA 3.5.2. Quick Start Guide
3.5.2. Quick Start Guide x
3.5.2.1. Software Test Setup 3.5.2.2. Driver Support 3.5.2.3. MCDMA Custom Driver 3.5.2.4. MCDMA DPDK Poll Mode Driver 3.5.2.5. MCDMA Kernel Mode Network Device Driver
3.5.2.3. MCDMA Custom Driver x
3.5.2.3.1. Prerequisites 3.5.2.3.2. Software Setup 3.5.2.3.3. Run the Reference Example Application 3.5.2.3.4. Testing Bitstream Configuration beyond 256 Channels (for MCDMA Custom Driver) 3.5.2.3.5. External Descriptor Mode Verification
3.5.2.3.1. Prerequisites x
3.5.2.3.1.1. Configuration Changes from BIOS 3.5.2.3.1.2. External Packages 3.5.2.3.1.3. Set the Boot Parameters
3.5.2.3.2. Software Setup x
3.5.2.3.2.1. Installing the Linux Kernel Driver 3.5.2.3.2.2. Build and Install User Space Library 3.5.2.3.2.3. Enabling VFs and Create Guest VM by Using QEMU 3.5.2.3.2.4. Establish Communication Between Host and QEMU
3.5.2.3.3. Run the Reference Example Application x
3.5.2.3.3.1. Meta Data Test 3.5.2.3.3.2. Custom PIO Read Write Test 3.5.2.3.3.3. BAM Test 3.5.2.3.3.4. BAS Test 3.5.2.3.3.5. Packet Gen Test
3.5.2.4. MCDMA DPDK Poll Mode Driver x
3.5.2.4.1. Prerequisites 3.5.2.4.2. Install PMD and Test Application (for CentOS) 3.5.2.4.3. Install PMD and Test Application (for Ubuntu) 3.5.2.4.4. Create VM Environment 3.5.2.4.5. Run the Reference Example Application
3.5.2.4.1. Prerequisites x
3.5.2.4.1.1. Set the Boot Parameters 3.5.2.4.1.2. Configuration Changes from BIOS 3.5.2.4.1.3. Install and Build Testpmd
3.5.2.4.4. Create VM Environment x
3.5.2.4.4.1. Enable VFs 3.5.2.4.4.2. Create Guest VM by using QEMU 3.5.2.4.4.3. Establish Communication Between Host and VM
3.5.2.4.5. Run the Reference Example Application x
3.5.2.4.5.1. Channel ID VF PF Verification 3.5.2.4.5.2. Example of Verifying on an AVMM Design 3.5.2.4.5.3. Testing Bitstream Configuration beyond 256 Channels (for DPDK Poll Mode Driver) 3.5.2.4.5.4. BAM Test 3.5.2.4.5.5. BAS Test
3.5.2.5. MCDMA Kernel Mode Network Device Driver x
3.5.2.5.1. Build and Install Netdev Driver 3.5.2.5.2. Enable VFs if SRIOV is Supported 3.5.2.5.3. Configure the Number of Channels Supported on the Device 3.5.2.5.4. Configure the MTU Value 3.5.2.5.5. Configure the Device Communication 3.5.2.5.6. Configure Transmit Queue Selection Mechanism 3.5.2.5.7. Test Procedure by Using Name Space Environment 3.5.2.5.8. PIO Test
3.5.2.5.6. Configure Transmit Queue Selection Mechanism x
3.5.2.5.6.1. MCDMA Queue Selection 3.5.2.5.6.2. Transmit Packet Steering (XPS) 3.5.2.5.6.3. Default
1. Introduction
2. Design Example Detailed Description
3. Design Example Quick Start Guide
4. Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide Archives
5. Revision History for the Multi Channel DMA Intel FPGA IP for PCI Express Design Example User Guide

3.5.2.4.4.2. Create Guest VM by using QEMU

Follow the steps needed to create a guest environment and assign VF device to VM by using QEMU.

  1. Unbind the device from UIO driver: 
    $./dpdk-devbind.py -u <bdf>
  2. Install vfio-pci module. 
    $modprobe vfio-pci
  3. Bind the device to vfio-pci.
    1. In case the device is binded to ifc_uio, unbind with the following command: 
      $ echo "<bdf>" > /sys/bus/pci/devices/\
<bdf>/driver/unbind
      For example: 
      echo "0000:01:00.0" > /sys/bus/pci/devices/\
0000\:01\:00.0/driver/unbind
    2. Bind the device to vfio-pci. 
      $ echo <PCI Vendor ID> <PCI Device ID> > /sys/bus/pci/drivers/vfio-pci/new_id
      For example: 
      echo 1172 0000 > /sys/bus/pci/drivers/vfio-pci/new_id
  4. For CentOS VMs: Use QEMU version 3.0.0 rc0 on Intel machines. Creating a VM with 8 GB RAM is advisable.

    ./qemu-3.0.0-rc0/x86_64-softmmu/qemu-system-x86_64 -smp 8 -m 10240M -boot c -machine q35,kernel-irqchip=split -cpu host -enable-kvm -nographic -L /home/dev/QEMU/qemu-3.0.0-rc0/pc-bios -name offload1 -hda /home/dev/QEMU/vm1.qcow2 -device vfio-pci,host=01:00.4 -netdev type=tap,id=net6551,script=no,downscript=no,vhost=on,ifname=net6551 -device virtio-net-pci,netdev=net6551 -device intel-iommu,intremap=on,caching-mode=on -serial telnet::5551,server,nowait -object memory-backend-file,id=mem,size=10240M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -monitor telnet::3331,server,nowait&

    On another terminal (vm) 
    telnet localhost 5551
ifconfig eth0 up
user:root
pass:root
Bring up interface.
$ifconfig eth0 up
Assign the IP address to eth0.
$ifconfig eth0 <1.1.1.11>
    On Host 
    $ifconfig net6551 up
$ifconfig net6551 <1.1.1.12>

    Copy the code by using scp.

    Example: $scp -r <directory> root@<GuestIP>:/ <path_to_the_directory>/

  5. For Ubuntu VMs:
    Get the qemu 7.0.0 version and compile using below steps 
    wget  https://download.qemu.org/qemu-7.0.0.tar.xz 
tar xvJf qemu-7.0.0.tar.xz 
cd qemu-7.0.0 
 ./configure 
 make -j$nproc 
cd ..
    Get Ubuntu VM image 
    wget https://cloud-images.ubuntu.com/releases/jammy/release/ubuntu-22.04-server-cloudimg-amd64.img 

mv ubuntu-22.04-server-cloudimg-amd64.img ubuntu-22.04.qcow2 

sudo qemu-img create -f qcow2 -F qcow2 -o  backing_file=./ubuntu-22.04.qcow2 mcdma_VM-1.qcow2 50G 

qemu-img info mcdma_VM-1.qcow2 

sudo genisoimage -output VM-1-cidata.iso -volid cidata -joliet -rock user-data meta-data

    Command to launch the VM

    sudo qemu-7.0.0/build/x86_64-softmmu/qemu-system-x86_64 -smp 8 -machine accel=kvm,type=q35,kernel-irqchip=split -cpu host -enable-kvm -nographic -boot c -L qemu-7.0.0/build/pc-bios -cpu host -m 20G -hda mcdma_VM-1.qcow2 -cdrom VM-1-cidata.iso -device vfio-pci,host=$BDF -serial telnet::5551,server,nowait -monitor telnet::3331,server,nowait&

    Note:
    • On AMD machines, the parameter -device intel-iommu,intremap=on,caching-mode=on is not required.
    • If multiple devices are in the same group, check for ACS enablement in root port and bridge.
    • Below are the Host and Guest VM configurations used for verification:
      Table 43.  Host System Configuration
      Host System Configuration Details
      Operating System

      CentOS Linux release 7.8

      Ubuntu 22.04 LTS
      Linux Kernel

      CentOS: 3.10.0-1127.10.1.el7.28.x86_64

      Ubuntu: 5.15.0-52-generic x86_64
      CPU Cores 96
      RAM 128 GB (64 GB on single NUMA)
      Hypervisor KVM
      Qemu Version QEMU version 3.0.0-rc0
      Table 44.  Guest System Configuration
      Guest System Configuration Details
      Operating System CentOS Linux release 7.8
      Linux Kernel 3.10.0-1127.10.1.el7.28.x86_64
      CPU Cores 2
      RAM 8 GB
Level Two Title