Using DPDK for Tests
Download
Focus Mode
Font Size
Scenarios
This document describes how to use DPDK to test CVM instances for high-throughput network performance.
Directions
Step 1: Preparing and Installing the Environment
This chapter guides you through setting up the DPDK test environment, which includes downloading the source code, extracting it, installing dependency libraries, and compiling it.
Preparing the Environment
A CVM requires public network access. You can assign a public IP address when creating the CVM, or configure an EIP for it after creation.
If a CVM is configured with multiple NICs, it is recommended to use eth0 for testing and the remaining NICs for remote login.
The relevant operations require system root privileges. Please apply for the required permissions in advance.
The following operational guidelines apply to TencentOS 3.0 and later operating systems.
Prepare two test servers, one as the receiver and the other as the sender.
For information on instance creation, see Creating an Instance via the Purchase Page.
For information on how to log in to an instance, see Connecting to a Linux Instance Using OrcaTerm (Recommended).
Downloading and Extracting the DPDK Source Package
Use the
wget command to download the DPDK 23.11 source code tarball from the official repository and extract it.Note:
Perform subsequent operations in the directory where DPDK is extracted.
wget https://fast.dpdk.org/rel/dpdk-23.11.tar.gztar -xf dpdk-23.11.tar.gzcd dpdk-23.11
Installing Dependencies and Compiling
Install the necessary dependency libraries and compilation tools.
Install the
numactl-devel dependency library (root privileges are required).yum install numactl-devel -y
Install the Python build tools.
pip3 install meson ninja pyelftools
Use
meson to build the project, setting the maximum number of logical cores to 1024.meson build -Dmax_lcores=1024
Use
ninja to compile the project.ninja -C build
Attention:
If your test server is an instance type later than Standard SA4 and has the highest CPU specifications, see Special Scenario Handling and add the relevant configuration steps.
Step 2: Binding Kernel Driver Modules
Bind a dedicated kernel driver module to DPDK to take over the network interface card.
Downloading the UIO Module
Clone the kernel module code from the DPDK official repository.
git clone http://dpdk.org/git/dpdk-kmods
Or use the git protocol (alternative).
git clone git://dpdk.org/dpdk-kmods
Compiling the UIO Module
Go to the
igb_uio directory and compile the code.cd ./dpdk-kmods/linux/igb_uiomake
Loading the UIO Module
Load the
uio and igb_uio kernel modules (root privileges are required).modprobe uioinsmod ./dpdk-kmods/linux/igb_uio/igb_uio.ko
Querying and Recording the NIC PCI Address
Before binding a network interface card, you need to obtain and record the PCI address of the target NIC. Run the following command to view and record all network interface card device information:
ethtool -i eth0 | grep bus-info
Binding the NIC to the UIO Driver
Use the tool provided by DPDK to bind the network interface card to the igb_uio driver.
1. Disable the network interface card.
Attention:
Before disabling a network interface card, check whether you are currently logged in to the CVM remotely via the IP address of another ENI. If you are logged in via eth0, the connection will be interrupted.
ifconfig eth0 down
2. Bind the network interface card to the
igb_uio driver (replace with the actual PCI address)../usertools/dpdk-devbind.py --bind=igb_uio 0000:02:01.0 # Replace 0000:02:01.0 with the actual PCI address of your network interface card.
Step 3: Huge Page Memory Configuration
DPDK uses huge pages (Huge Pages) to improve memory access efficiency. Configuring huge pages is a prerequisite for running DPDK.
Run the following command to allocate huge pages (this example uses 2MB huge pages, allocating 8192 pages for a total of approximately 16GB):
echo 8192 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
Step 4: Running Performance Tests
DPDK performance testing uses the RX-TX dual-server mode: one server acts as the receiver (RX), and the other server acts as the transmitter (TX).
Note:
RX is the receiver, and TX is the transmitter. The two servers must run their respective commands.
The RX receiver runs the following command:
C=4; Q=32; ./build/app/dpdk-testpmd -l 0-47 --proc-type primary --file-prefix bench -a 0000:1b:00.0 -- --burst=128 --nb-cores=$C --txd=4096 --rxd=4096 --txq=$Q --rxq=$Q --forward-mode=rxonly --stats-period=1
The TX transmitter runs the following command:
C=4; Q=32; S=10.31.0.80; D=10.31.0.240; ./build/app/dpdk-testpmd -l 0-47 --proc-type primary --file-prefix bench -a 0000:1b:00.0 -- --burst=128 --nb-cores=$C --tx-ip $S,$D --txonly-multi-flow --txpkts=64 --txd=4096 --rxd=4096 --txq=$Q --rxq=$Q --forward-mode=txonly --stats-period=1
Parameter Details
Parameter | Description |
-a 0000:1b:00.0 | Specify the PCI address of the network card to be used, which must be replaced according to the actual environment. |
--tx-ip S,D | The destination IP address for packets sent from the TX end, which must be replaced according to the CVM's network card address. |
--txonly-multi-flow | Enable multi-flow transmission mode (if not set, only a single flow is sent). |
--txpkts=64 | Packet length (in bytes, including the MAC header). A value of 64 is recommended for PPS testing scenarios. |
-l 0-47 | Specify the CPU core number range to be used, adjust it as needed, and in cross-NUMA scenarios, you can obtain the information by viewing the queue cross-NUMA distribution. |
--txq / --rxq = $Q | The number of transmit/receive queues, which can be viewed by running ethtool -l eth0 | grep Combined. |
--nb-cores=$C | The number of CPU cores actually participating in the test. A value of C between 2 and Q (inclusive) is recommended. |
--stats-period=1 | Outputs statistical information once per second. |
The test result example is shown below, with the RX receiver on the left and the TX transmitter on the right:
Step 5: Test Environment Restoration
After testing is complete, restore the environment to its initial state, release resources, and avoid impacting other services.
Restoring the NIC Driver
Use the
dpdk-devbind.py tool to restore the network interface card to its default driver:./usertools/dpdk-devbind.py -b virtio-pci 0000:00:08.0 # Note: Replace 0000:00:08.0 with the actual PCI address.
If the preceding command does not work, you can manually run the following command to unbind vfio-pci.
echo 0000:1b:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind #Replace 0000:1b:00.0 as needed.echo virtio-pci > /sys/bus/pci/devices/0000\\:1b\\:00.0/driver_override #Set driver_override to virtio-pci.echo 0000:1b:00.0 > /sys/bus/pci/drivers/virtio-pci/bind #Rebind to virtio-pci.
Restoring Huge Page Memory
Set the huge page memory quantity to zero.
echo 0 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
Re-enable the network interface card.
ifconfig eth1 up
Special Scenario Handling
If your test server is an instance type later than Standard SA4 and has the highest CPU specifications, you need to add the following configuration steps.
Viewing Queue Distribution Across NUMA
Note:
For servers with multi-socket CPUs, ensure that the CPU cores used by DPDK and the network interface card are located on the same NUMA node. Otherwise, performance will degrade significantly.
1. Download and view the script.
wget https://lag-dpdk-1326316016.cos.ap-guangzhou.myqcloud.com/dpdk_core_advisor.sh
2. Run the script to view cross-queue NUMA distribution.
sh dpdk_core_advisor.sh eth0
If the returned result shows that queues are distributed across NUMA nodes, install
lag_dpdk.patch.Installing lag_dpdk.patch
1. Download the
lag_dpdk.patch patch script.wget https://lag-dpdk-1326316016.cos.ap-guangzhou.myqcloud.com/lag_dpdk.patch
2. Apply the patch.
git apply lag_dpdk.patch# Equivalent commands (choose one)patch -p1 < lag_dpdk.patch
3. Recompile.
meson build -Dmax_lcores=1024ninja -C build
Help and Support
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback