How to Set Up Intel® Ethernet Flow Director

Clayne B Robison

Introduction

Intel® Ethernet Flow Director (Intel® Ethernet FD) directs Ethernet packets to the core where the packet consuming process, application, container, or microservice is running. It is a step beyond receive side scaling (RSS) in which packets are sent to different cores for interrupt processing, and then subsequently forwarded to cores on which the consuming process is running.

Intel Ethernet FD supports advanced filters that direct received packets to different queues, and enables tight control on flow in the platform. It matches flows and CPU cores where the processing application is running for flow affinity, and supports multiple parameters for flexible flow classification and load balancing. When operating in Application Targeting Routing (ATR) mode, Intel Ethernet FD is essentially the hardware offloaded version of Receive Flow Steering available on Linux* systems, and when running in this mode, Receive Packet Steering and Receive Flow Steering are disabled.

It provides the most benefit on Linux bare-metal usages (that is, not using virtual machines (VMs)) where packets are small and traffic is heavy. And because the packet processing is offloaded to the network interface card (NIC), Intel Ethernet FD could be used to avert denial-of-service attacks.

Supported Devices

Intel Ethernet FD is supported on devices that use the ixgbe driver, including the following:

Intel® Ethernet Converged Network Adapter X520
Intel® Ethernet Converged Network Adapter X540
Intel® Ethernet Controller 10 Gigabit 82599 family

It is also supported on devices that use the i40e driver:

Intel® Ethernet Controller X710 family
Intel® Ethernet Controller XL710 family

The Data Plane Development Kit (DPDK) includes support for Intel Ethernet FD on the devices listed above. See the DPDK documentation for how to use DPDK and testpmd with Intel Ethernet FD.

In order to determine whether your device supports Intel Ethernet FD, use the ethtool command with the --show-features or -k parameter on the network interface you want to use:

# ethtool --show-features <interface name> | grep ntuple

Screenshot of using ethool command to detect Intel Flow Director support.

If the ntuple-filters feature is followed by off or on, Intel Ethernet FD is supported on your Ethernet adapter. However, if the ntuple-filters feature is followed by off [fixed], Intel Ethernet FD is not supported on your network interface.

Enabling Intel® Ethernet Flow Director

Driver Parameters for Devices Supported by the ixgbe Driver

On devices that are supported by the ixgbe driver, there are two parameters that can be passed-in when the driver is loaded into the kernel that will affect Intel Ethernet FD:

FdirPballoc
AtrSampleRate

FdirPballoc

This driver parameter specifies the packet buffer size allocated to Intel Ethernet FD. The valid range is 1–3, where 1 specifies that 64k should be allocated for the packet buffer, 2 specifies a 128k packet buffer, and 3 specifies a 256k packet buffer. If this parameter is not explicitly passed to the driver when it is loaded into the kernel, the default value is 1 for a 64k packet buffer.

AtrSampleRate

The AtrSampleRate parameter indicates how many Tx packets will be skipped before a sample is taken. The valid range is from 0 to 255. If the parameter is not passed to the driver when it is loaded into the kernel, the default value is 20, meaning that every 20^th packet will be sampled to determine if a new flow should be created. Passing a value of 0 will disable ATR mode, and no samples will be taken from the Tx queues.

The above driver parameters are not supported on devices that use the i40e driver.

To enable these parameters, first unload the ixgbe module from the kernel. Note, if you are connecting to the system over ssh, this may disconnect your session:

# rmmod ixgbe

Then re-load the ixgbe driver into the kernel with the desired parameters listed above:

# modprobe ixgbe FdirPballoc=3,2,2,3 AtrSampleRate=31,63,127,255

Note that, in this example, for each parameter there are four values. This is because on my test system, I have two network adapters that are using the ixgbe driver--an Intel Ethernet Controller 10 Gigabit 82599, and an Intel® Ethernet Controller 10 Gigabit X540--each of which has two ports. The order in which the parameters are applied is in PCI Bus/Device/Function order. To determine the PCI BDF order on your system, use the following command:

# lshw -c network -businfo

Screenshot of lshw command showing PCI Bus, Device Function information for NICs

Based on this system configuration, using the modprobe command above, the Intel Ethernet Controller 10 Gigabit X540-AT2 port at PCI address 00:03.0 is allocated the FdirPballoc and AtrSampleRate parameters of 3 and 31, respectively, and the Intel Ethernet Controller 10 Gigabit 82599 port at PCI address 81:00.1 is allocated the FdirPballoc and AtrSampleRate parameters of 3 and 255, respectively.

Once you have determined that your Intel branded server network adapter supports Intel Ethernet FD and you have loaded the desired parameters into the driver (on supported models), execute the following command to enable Intel Ethernet FD:

# ethtool --features enp4s0f0 ntuple on

Screenshot of using ethtool command to turn Intel Flow Director on

Because the commands below only indicate which Rx queue a matched packet should be sent to, ideally an additional step should be taken to pin both Rx queues and the process, application, or container that is consuming the network traffic to the same CPU. Pinning an application/process/container to a CPU is beyond the scope of this document, but it can be done using the taskset command. Pinning IRQs to a CPU can be done using the set_irq_affinity script that is included with the freely available sources of the i40e and ixgbe drivers. See Intel Support: Drivers and Software for the latest versions of these drivers. See also the IRQ Affinity section in this tuning guide for how to set IRQ affinity.

Using Intel Ethernet Flow Director

Intel Ethernet FD can run in one of two modes: externally programmed (EP) mode, and ATR mode. Once Intel Ethernet FD is enabled as shown above, ATR mode is the default mode, provided that the driver is in multiple Tx queue mode. When running in EP mode, the user or management/orchestration software can manually set how flows are handled. In either mode, fields are intelligently selected from the packets in the Rx queues to index into the Perfect-Match filter table. For more information on how Intel Ethernet FD works, see this whitepaper.

Application Targeting Routing

In ATR mode, Intel Ethernet FD uses fields from the outgoing packets in the Tx queues to populate the 8K-entry Perfect-Match filter table. The fields that are selected depend on the packet type; for example, fields to filter TCP traffic will be different than those used to filter user diagram protocol (UDP) traffic. Intel Ethernet FD then uses the Perfect-Match filter table to intelligently route incoming traffic to the Rx queues.

To disable ATR mode and switch to EP mode, simply use the ethtool command shown under Adding Filters to manually add a filter, and the driver will automatically enter EP mode. To automatically re-enable ATR mode, use the ethtool command under Removing Filters until the Perfect-Match filter table is empty.

Externally Programmed Mode

When Intel Ethernet FD runs in EP mode, flows are manually entered by an administrator or by management/orchestration software (for example, OpenFlow*). As mentioned above, once enabled, Intel Ethernet FD automatically enters EP mode when a flow is manually entered using the ethtool command listed under Adding Filters.

Adding Filters

The following commands illustrate how to add flows/filters to Intel Ethernet FD using the -U,
-N, or --config-ntuple switch to ethtool.

To specify that all traffic from 10.23.4.6 to 10.23.4.18 be placed in queue 4, issue this command:

# ethtool --config-ntuple flow-type tcp4 src-ip 10.23.4.6 dst-ip 10.23.4.18 action 4

Note: Without the ‘loc’ parameter, the rule is placed at position 1 of the Perfect-Match filter table. If a rule is already in that position, it is overwritten.

Forwards to queue 2 all IPv4 TCP traffic from 192.168.10.1:2000 that is going to 192.168.10.2:2001, placing the filter at position 33 of the Perfect-Match filter table (and overwriting any rule currently in that position):

# ethtool --config-ntuple <interface name> flow-type tcp4 src-ip 192.168.10.1 dst-ip 192.168.10.2 src-port 2000 dst-port 2001 action 2 loc 33

Drops all UDP packets from 10.4.83.2:

# ethtool --config-ntuple flow-type udp4 src-ip 10.4.82.2 action -1

Note: The VLAN field is not a supported filter with the i40e driver (Intel Ethernet Controller XL710 and Intel Ethernet Controller X710 NICs).

For more information and options, see the ethtool man page documentation on the -U, -N, or --config-ntuple option.

Note: The Intel Ethernet Controller XL710 and the Intel Ethernet Controller X710, of the Intel® Ethernet Adapter family, provide extended cloud filter flow support for more complex cloud networks. For more information on this feature, please see the Cloud Filter Support section in this ReadMe document, or in the ReadMe document in the root folder of the i40e driver sources.

Removing Filters

In EP mode, to remove a filter from the Perfect-Match filter table, execute the following command against the appropriate interface. ‘N’ in the rule below is the numeric location in the table that contains the rule you want to delete:

# ethtool --config-ntuple <interface name> delete N

Listing Filters

To list the filters that have been manually entered in EP mode, execute the following command against the desired interface:

# ethtool --show-ntuple <interface name>

Disabling Intel Ethernet Flow Director

Disabling Intel Ethernet FD is done with this command:

# ethtool --features  enp4s0f0 ntuple off

This flushes all entries from the Perfect-Filter flow table.

Conclusion

Intel Ethernet FD directs Ethernet packets to the core where the packet consuming process, application, container, or microservice is running. This functionality is a step beyond RSS, in which packets are simply sent to different cores for interrupt processing, and then subsequently forwarded to cores on which the consuming process is running. It can be explicitly programmed by administrators and control plane management software, or it can intelligently sample outgoing traffic and automatically create Perfect-Match filters for incoming packets. When operating in automatic ATR mode, Intel Ethernet FD is essentially the hardware offloaded version of Receive Flow Steering available on Linux systems.

Intel Ethernet FD can provide additional performance benefit, particularly in workloads where packets are small and traffic is heavy (for example, in Telco environments). And because it can be used to filter and drop packets at the network interface card (NIC), it could be used to avert denial-of-service attacks.