Overview
This document describes the key factors that affect TCP transfer rate and provides troubleshooting approaches and solutions for common issues, helping you optimize TCP transfer performance on CVM. TCP transfer rate is affected by factors at multiple layers, mainly including:
Network link: Bandwidth, RTT (round-trip time), packet loss, and out-of-order packets
Window size: Receive window, send window, and congestion window
Buffer configuration: Send/receive buffer watermarks for TCP sockets
Congestion control algorithms: Performance differences of different algorithms in different network environments
Kernel protocol stack parameters: Features such as timestamps, initial window, and GSO
Host resources: CPU scheduling latency and softIRQ processing latency
Among these, window size and buffer configuration are the most common areas where tuning can deliver quick results.
Bandwidth-Delay Product (BDP) and Window Size
BDP Formula
TCP uses a window mechanism to prevent a fast sender from overwhelming a slow receiver. As network bandwidth increases and links become longer, it takes more time to acknowledge received data. This relationship between bandwidth and latency is called the bandwidth-delay product (BDP), which represents the optimal amount of data to send to fill the network pipe:
BDP (bits) = bandwidth (bits/second) × RTT (seconds)
Calculation Examples
Assume the bandwidth between two devices is 10 Gbps, the RTT is 50 ms, and the default TCP window size is 65535 bytes. Can the bandwidth be fully utilized?
Bandwidth = (65535 bytes × 8 bits/byte) / 0.050 seconds ≈ 10.49 Mbit/s
Clearly, the default window size is far from sufficient to achieve the theoretical bandwidth of 10 Gbps. To fully utilize the link, the required window size is:
Window size = (10000 Mbps × 0.050 seconds) / 8 bits/byte ≈ 62.5 MB
Note:
62.5 MB is a theoretical value. Actual transfer performance is also affected by factors such as CPU scheduling, application-layer overhead, packet size, and kernel window watermarks.
Window Scaling Factor
To address the problem of insufficient window size in high-latency networks, RFC 1323 defines the Window Scale option, which adds a scaling factor to the TCP option to increase the original 16-bit window size by several times.
Kernel control parameters:
net.ipv4.tcp_window_scaling = 1
Enabled by default. It is recommended to keep it enabled to support window scaling in high-latency networks.
TCP Buffer Parameter Tuning
Parameter Description
|
net.ipv4.tcp_rmem
| 4096 131072 6291456
| TCP socket receive buffer watermarks (min/default/max, in bytes) |
net.ipv4.tcp_wmem
| 4096 16384 4194304
| TCP socket send buffer watermarks (min/default/max, in bytes) |
The meanings of the three values are as follows:
min: The minimum buffer memory reserved for TCP sockets, which is retained even when memory is under pressure.
default: The default buffer size for TCP sockets, which is typically smaller than net.core.rmem_max / net.core.wmem_max
max: The maximum TCP socket buffer size, which must be less than or equal to rmem_max / wmem_max
Tuning Suggestion
For high-latency, high-bandwidth networks (such as cross-region transfer): Increase the max value to more than twice the BDP.
View the current configuration: sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem
Example of dynamic adjustment (increase the send/receive buffer upper limit to 20 MB):
sysctl -w net.ipv4.tcp_rmem='4096 131072 20971520'
sysctl -w net.ipv4.tcp_wmem='4096 16384 20971520'
To make the changes permanent, write them to /etc/sysctl.conf and then run sysctl -p.
Diagnostic Methods
Identifying Window Bottlenecks with the ss Command
The ss command can display detailed TCP connection status and is the preferred tool for identifying window bottlenecks. When traffic cannot increase, pay attention to the following fields:
|
rwnd_limited
| Percentage of time paused from sending due to peer receive window limit |
sndbuf_limited
| Percentage of time paused from sending due to local send buffer insufficiency |
cwnd
| Congestion window size |
snd_wnd
| Current send window |
bytes_sent / bytes_acked
| Bytes sent/acked |
retrans
| Number of retransmissions |
busy
| Connection busy duration |
pacing_rate / delivery_rate
| Send rate / delivery rate |
View command:
Assessment logic:
If rwnd_limited accounts for a high proportion, it indicates that the peer receive window is insufficient, and you need to increase the receiver's tcp_rmem.
If sndbuf_limited accounts for a high proportion, it indicates that the local send buffer is insufficient, and you need to increase the sender's tcp_wmem.
If both are low but the busy duration is high and the bandwidth cannot increase, this is usually caused by link latency or packet loss. Check the network quality or qdisc rate limiting.
Sample Output (Simplified)
ss -ie 'dst 10.0.0.16'
tcp ESTAB 0 3932288 10.0.0.15:49550 10.0.0.16:5201
ts sack cubic wscale:7,7 rto:251 rtt:50.53/0.051 mss:1412
cwnd:4474 send 1Gbps
rwnd_limited:4981ms(63.5%) sndbuf_limited:101ms(1.3%)
In the example above, rwnd_limited accounts for 63.5%, indicating that the receive window is the primary bottleneck.
Verifying Bandwidth with iperf3
iperf3 -c <server IP> -i 1 -t 60
You can quickly identify the bottleneck by testing the baseline bandwidth with iperf3 and then using ss to observe window limitations.
Troubleshooting Common Issues
Low TCP Performance on High-Latency Networks (Insufficient Window)
Symptom
Over cross-region or high-latency links, the bandwidth measured by iperf3 is far lower than the physical bandwidth. For example, a 10 Gbps link achieves only about 480 Mbps.
Possible Causes
The default TCP window size is insufficient to fill a high-latency pipe (the BDP calculation is inadequate), and the window scaling factor is not in effect.
Troubleshooting Methods
1. Use ss -ie to check the proportions of rwnd_limited and sndbuf_limited for the connection.
2. If the proportion of rwnd_limited is high (for example, 63.5%), confirm that the receive window is the bottleneck.
3. Confirm that net.ipv4.tcp_window_scaling = 1 is enabled.
Solution
Increase the TCP buffer size limits on both the sender and receiver so that they cover the window size required by the BDP:
sysctl -w net.ipv4.tcp_rmem='4096 131072 20971520'
sysctl -w net.ipv4.tcp_wmem='4096 16384 20971520'
After tuning, retest the connection. The bandwidth should improve significantly.
TCP Transfer Speed Limited by Buffer Size
Symptom
The ss output shows that sndbuf_limited accounts for a high proportion, and the traffic cannot increase.
Possible Causes
The send buffer upper limit (the max value of tcp_wmem) is set too small, causing application-layer data to accumulate in the send queue.
Troubleshooting Methods
ss -ie 'dst <peer IP>' | grep sndbuf_limited
Check the proportion of sndbuf_limited and the backlog of Send-Q.
Solution
Increase the max value of net.ipv4.tcp_wmem, and ensure that net.core.wmem_max is not less than this value:
sysctl -w net.core.wmem_max=20971520
sysctl -w net.ipv4.tcp_wmem='4096 16384 20971520'
Congestion Control Algorithm Selection
Symptom
Transmission performance varies significantly under different network conditions, and throughput changes considerably after algorithms are switched on the same link.
Possible Causes
Default congestion control algorithms (such as cubic) perform poorly in specific scenarios, such as long fat pipes and high packet loss.
Troubleshooting Methods
View the current congestion control algorithm:
sysctl net.ipv4.tcp_congestion_control
View available algorithms:
sysctl net.ipv4.tcp_available_congestion_control
Solution
Select an appropriate algorithm based on the scenario:
High-latency, high-bandwidth long-fat pipes: Consider using BBR.
General scenarios: cubic is sufficient.
sysctl -w net.ipv4.tcp_congestion_control=bbr
Traffic Shaping with tc qdisc
Symptom
Bandwidth is stuck at a fixed value. The ss output shows that the window is not limited, and the proportions of rwnd_limited / sndbuf_limited are low, but the busy duration remains high.
Possible Causes
The network interface is configured with tc qdisc rate limiting rules, or the default limit of the netem rule is too small, causing packet loss.
Troubleshooting Methods
View the qdisc configuration and packet loss statistics:
tc -s qdisc show dev eth0
Pay close attention to the dropped count and the backlog queue. If the default limit of the netem rule is 1000, packets are dropped when the backlog exceeds this value.
Solution
If it is business rate limiting, confirm whether it meets expectations.
For netem delay simulation scenarios, increase the limit parameter:
tc qdisc del dev eth0 root netem 2>/dev/null
tc qdisc add dev eth0 root netem delay 25ms limit 10000
Impact of Kernel GSO Feature
Symptom
Under the same hardware and configuration, TCP transmission performance varies significantly across different kernel versions. After tc netem delay rules are configured, packet loss occurs on lower kernel versions, and the bandwidth cannot increase.
Possible Causes
The kernel GSO (Generic Segmentation Offload) feature is not enabled, so the protocol stack cannot aggregate multiple TCP packets and hand them over to the driver for segmentation, resulting in tc queue backlog and packet loss.
Troubleshooting Methods
1. Compare the tc packet loss statistics across different kernel versions:
tc -s qdisc show dev eth0
2. On a normal kernel, dropped is 0 and backlog is small. On an abnormal kernel, dropped increases and backlog accumulates severely.
3. Check whether GSO is enabled:
ethtool -k eth0 | grep gso
Solution
Upgrade to a newer version of the TencentOS Server kernel. GSO is enabled by default and enforced on higher kernel versions.
Or manually enable the NIC GSO feature:
Summary: Quick Reference of Key Tuning Parameters
|
net.ipv4.tcp_rmem
| TCP receive buffer watermarks (min/default/max) | Set max to more than twice the BDP in high-latency networks. |
net.ipv4.tcp_wmem
| TCP send buffer watermarks (min/default/max) | Same as above. Ensure that the send buffer is sufficiently large. |
net.core.rmem_max
| Maximum receive buffer limit | Not less than the max value of tcp_rmem |
net.core.wmem_max
| Maximum send buffer limit | Not less than the max value of tcp_wmem |
net.ipv4.tcp_window_scaling
| Window scaling factor | Keep it enabled (=1). |
net.ipv4.tcp_congestion_control
| Congestion control algorithm | For high-latency links, try bbr. |
net.ipv4.tcp_timestamps
| TCP timestamps | Enable it. This helps with RTT measurement and out-of-order detection. |
net.ipv4.tcp_slow_start_after_idle
| Slow start after idle | Disable it in long-connection scenarios (=0). |
NIC GSO | Generic segmentation offload | Keep it enabled. |