40Gbps Ethernet Accelerator Functional Unit (AFU) Design Example User Guide: Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA

Download PDF
ID 683436
Date 4/30/2019
Public
Document Table of Contents
Document Table of Contents x
1. About this Document 2. Overview 3. Running the Design Example Tests 4. Using the Design Example as a Platform for Further Evaluation 5. 40Gbps Ethernet AFU Design Example User Guide Archives 6. Document Revision History for 40Gbps Ethernet AFU Design Example User Guide
1. About this Document x
1.1. Intended Audience 1.2. Conventions 1.3. Acronym List 1.4. Acceleration Glossary
2. Overview x
2.1. 40GbE Design Example AFU Hardware 2.2. 40GbE Design Example AFU Software
3. Running the Design Example Tests x
3.1. Setup Prerequisites 3.2. Running 40GbE Internal Loopback Test in Single Intel® PAC System 3.3. Running 40GbE External Loopback Test in Single Intel® PAC System 3.4. Running 40GbE Intel® PAC-to-PAC Test between two connected Intel® PACs
4. Using the Design Example as a Platform for Further Evaluation x
4.1. Prerequisite while Evaluating with the Intel FPGA MAC/PHY IP 4.2. Evaluation with an Alternate MAC/PHY IP
1. About this Document
2. Overview
3. Running the Design Example Tests
4. Using the Design Example as a Platform for Further Evaluation
5. 40Gbps Ethernet AFU Design Example User Guide Archives
6. Document Revision History for 40Gbps Ethernet AFU Design Example User Guide

3.4. Running 40GbE Intel® PAC-to-PAC Test between two connected Intel® PACs

In this procedure, you can install the Intel® PACs in the same system or two separate systems with the Acceleration Stack. Unless shown otherwise, you can expect the commands to return similar outputs as the internal loopback test.

Figure 2. System Setup with Intel® PACs Installed on Same Host
Figure 3. System Setup with Intel® PACs Installed on Separate Hosts

  1. Install a QSFP+ optical module in each Intel® PAC and connect the QSFP+ ports with an optical cable.
  2. Assuming the two Intel® PACs are installed in the same system, find their PCI Bus:Device:Function mappings. 
    $ lspci | grep 09c4
    Sample output:
04:00.0 Processing accelerators:Intel Corporation Device 09c4
06:00.0 Processing accelerators:Intel Corporation Device 09c4
  3. Load the AF for the 40GbE AFU example on both Intel® PACs. 
    $ cd $OPAE_PLATFORM_ROOT
$ sudo fpgaconf hw/samples/eth_e2e_e40/bin/eth_e2e_e40.gbs -b 0x04
$ sudo fpgaconf hw/samples/eth_e2e_e40/bin/eth_e2e_e40.gbs -b 0x06
  4. cd $OPAE_PLATFORM_ROOT/hw/samples/eth_e2e_e40/sw
  5. Run the following steps on your both Intel® PAC:
    1. Complie the library and application using the command: 
      $ make
    2. To configure the transceiver channel into 40G mode, write 40 to the following sysfs entry: 
      $ sudo sh -c "echo 40 > /sys/class/fpga/intel-fpga-dev.<instance_id>\
/intel-fpga-fme.<instance_id>/intel-pac-hssi.<instance_id>.\
auto/hssi_mgmt/config"

      <instance_id> represents the consecutive numbering of device, fme, and hssi instances. 

      For example:
sudo sh -c "echo 40 > /sys/class/fpga/intel-fpga-dev.0\
/intel-fpga-fme.0/intel-pac-hssi.2.auto/hssi_mgmt/config"
    3. To allow non-root users to access the 40GbE AFU instance, you can provide read and write privileges to the port (/dev/intel-fpga-port.\*) where \* denotes the respective socket. For example, to provide read and write privileges on Port 0: 
      $ sudo chmod 666 /dev/intel-fpga-port.0
    4. To resolve library dependency: 
      export LD_LIBRARY_PATH=`pwd`:$LD_LIBRARY_PATH
    5. To disable the internal loopback on B:D:F - 00:0a:0b, 
      $ ./pac_hssi_e40 -b 00 -d 0a -f 0b --action=loopback_disable
    6. To trigger the DFE: 
      sudo sh -c "echo 1 > /sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/intel-pac-hssi.2.auto/hssi_mgmt/dfe_kickstart"
      Verify that some of the DFE tap values are non-zero. This ensures that the script run is successful. 
      cat /sys/class/fpga/intel-fpga-dev.0/intel-fpga-fme.0/intel-pac-hssi.2.auto/hssi_mgmt/dfe_kickstart
    7. To clear PHY, transmit, and receive statistics: 
      $ ./pac_hssi_e40 -b 00 -d 0a -f 0b --action=stat_clear
      Sample output: 
      Cleared TX stats on channel 0
Cleared RX stats on channel 0
    8. To transmit 0x1000 packets: 
      $ ./pac_hssi_e40 -b 00 -d 0a -f 0b --action=pkt_send
      Sample output: 
      Sent 0x10000 packets on channel 0
      Note: After programming the eth_e2e_e40 AFU, the initial send of packets may drop the first packet. Subsequent packet sends do not drop any packets.
      Note: In pac_hssi_e40 [-h] [-b <bus>] [-d <device>] [-f <function>] [-m Dest. MAC] -a action, the Dest (destination) MAC address is user configurable. By default, the broadcast address is used as the destination MAC address.
    9. To get PHY, transmit and receive statistics: 
      $ ./pac_hssi_e40 -b 00 -d 0a -f 0b --action=stat
    Note: After every hot plug/unplug of the cables, you must trigger the DFE as discussed above after disabling internal loopback.
    For more details, refer to the README file located in the sw subdirectory to: 
    $OPAE_PLATFORM_ROOT/hw/samples/eth_e2e_e40/sw/README.md

    To run this example on a virtual machine:

    1. Program the eth_e2e_e40 AFU and configure the transceiver channel to 40G mode from the Host machine by referencing the previous substeps.
    2. Disable internal loopback and trigger DFE from the Host by refereccing the previous substeps.
    3. Follow the steps in the Running the OPAE in a Virtualized Environment section of the Intel® Acceleration Stack Quick Start Guide for Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA to create a virtual function and attach the virtual function to a virtual machine.
    4. Run the external loopback test on the virtual machine.
Level Two Title