tencent cloud

TencentOS Server

DokumentasiTencentOS ServerUser GuideKernel FeaturesI/O Performance and Storage Optimization

I/O Performance and Storage Optimization

Unduh
Mode fokus
Ukuran font
Terakhir diperbarui: 2026-10-19 17:22:48
Diterjemahkan oleh AI

Feature Overview

IO performance tuning covers IO schedulers, cgroup IO control, file system optimization, io_uring, and block device performance testing, helping users achieve better IO throughput and more stable latency in high-concurrency storage, database, and container colocation scenarios. Common issues in typical scenarios include:
IO interference between services: A single container saturates the disk bandwidth, causing IO jitter for other services on the shared storage.
Additional write overhead: The default ext4 mount options update the access time (atime), causing unnecessary write amplification.
High system call overhead under high concurrency: The traditional aio interface incurs excessive system call overhead in high-concurrency scenarios, making it difficult to meet low-latency requirements.
TencentOS Server 4 is built on the Linux 6.6 kernel. By default, it uses the mq-deadline scheduler for NVMe, fully supports cgroup v2 IO bandwidth/IOPS limits, and provides the io_uring asynchronous IO interface. By properly adjusting the IO scheduler, file system mount options, and cgroup IO limits, you can ensure stable QoS for critical workloads on shared storage.

Must-Knows

IO tuning parameters must be tested and validated against actual business workloads. It is recommended that you use fio to run benchmark tests before adjusting production parameters.
For NVMe devices, the none scheduler is recommended to achieve optimal latency. For HDDs, the bfq scheduler is recommended to achieve fair scheduling.
The ext4 data=writeback mode may lose recently written data in the event of a system crash. For databases and other strongly consistent workloads, use the default ordered mode.
The cgroup io.max limit takes effect only when the device is busy. Setting it to max means no limit is applied to the corresponding dimension.
Switching IO schedulers and applying cgroup IO limits inside a container require privileged mode or operations on the host. Regular containers may fail because /sys is read-only.

API and Parameter Description

I/O Scheduler

Scheduler
Scenario
Description
mq-deadline
NVMe / SSD
Deadline-based multi-queue scheduling, the default option
none
NVMe (high performance)
Pass-through mode, no scheduling, suitable for low-latency devices.
bfq
HDD / hybrid
Fair queue scheduling, suitable for interactive scenarios.
kyber
NVMe
Adaptive scheduling, suitable for high concurrency.

cgroup v2 I/O Control Interface

Interface File
Value Range
Default Value
Read/Write Description
io.max
MAJ:MIN rbps=<N> wbps=<N> riops=<N> wiops=<N>
Unlimited (max)
Read/write: set the upper limit for device read/write bandwidth and IOPS.
io.weight
default 1-10000
default 100
Read/write: set the IO weight, which takes effect only when the device is busy.
io.stat
Read-Only
-
Read: view current IO statistics (bytes/IOPS).

File System Mount Options (ext4)

Parameter
Value
Default Value
Description
noatime
Mount Option
Not enabled
Do not update access time, reducing writes.
data=writeback
ordered/writeback/journal
ordered
Writeback mode, performance first.
Attention:
A system crash may cause data loss. This mode is only suitable for scenarios where data loss is tolerable.
discard
Mount Option
Not enabled
Enable TRIM (SSD/NVMe).

Key sysctl Parameters

Warning:
Some sysctl parameter names may vary depending on the kernel version. You can run sysctl -a to view the parameters available on the current system.
Parameter
Recommended Value
Default Value
Description
vm.dirty_ratio
10-20
20
Dirty page ratio threshold that triggers synchronous writeback
vm.dirty_background_ratio
5-10
10
Dirty page ratio threshold that triggers asynchronous writeback
vm.dirty_expire_centisecs
3000
3000
Dirty page expiration time (unit: 1/100 second, 3000 = 30 seconds)
vm.swappiness
10-20
60
swap tendency. A lower value indicates a stronger tendency to reclaim memory pages.
vm.dirty_writeback_centisecs
500
500
Dirty page writeback interval (unit: 1/100 second, 500 = 5 seconds)

Configuration Example

Viewing and Switching I/O Schedulers

Select a target scheduler based on the device type: For NVMe/SSD devices, the none scheduler is recommended for low latency, or keep mq-deadline for balanced performance by default. For HDDs, the bfq scheduler is recommended for fair scheduling.
View the block devices on the system to determine the device name (NVMe devices are displayed as nvme0n1, and virtio virtual disks are displayed as vda):
lsblk
Output example:
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
vda     253:0   0  50G  0 disk
└─vda1  253:1   0  50G  0 part /
View the current scheduler (the scheduler enclosed in square brackets in the output is the active one, and the rest are available schedulers supported by the system. The list of supported schedulers may vary depending on the device and kernel version):
# View the current scheduler (replace <device_name> with the actual device name obtained in the previous step)
# The scheduler enclosed in square brackets [ ] in the output is the active one, and the rest are available schedulers supported by the system.
# The list of supported schedulers may vary depending on the device and kernel version.
cat /sys/block/<device_name>/queue/scheduler
Output example:
none [mq-deadline] kyber bfq
Switch the scheduler:
Warning:
Switching the IO scheduler takes effect immediately and may affect the IO latency and throughput characteristics of the device. Perform the switch in a maintenance window or non-production environment, and use tools such as fio to verify the impact of different schedulers on business workloads in advance. For NVMe devices, the none scheduler is recommended to achieve optimal latency. For HDDs, the bfq scheduler is recommended to achieve fair scheduling.
echo none > /sys/block/<device_name>/queue/scheduler
Verify the switch result:
cat /sys/block/<device_name>/queue/scheduler
Output example:
none [none] kyber bfq

Persistent I/O Scheduler (udev Rules)

Create udev rules (a file-writing command that produces no output on success):
echo 'ACTION=="add|change", KERNEL=="nvme[0-9]*n[0-9]*", ATTR{queue/scheduler}="none"' > /etc/udev/rules.d/60-io-scheduler.rules
Apply the rules immediately:
Warning:
This command triggers udev change events for all devices in the system, which may cause brief I/O jitter. Verify the impact in a test environment before running this command.
udevadm trigger --type=devices --action=change

Limiting Container I/O Bandwidth

Note:
The following operations require root privileges and the system must be running in cgroup v2 (unified) mode. TencentOS Server 4 uses cgroup v2 by default.
Obtain the major and minor device numbers of the target device:
lsblk -o NAME,MAJ:MIN
Output example:
NAME    MAJ:MIN
nvme0n1 259:0
Create an IO-limited cgroup and set read/write bandwidth limits (example: 10MB/s) and IOPS limits (example: 500). Replace 259:0 with the actual device number obtained in the previous step. No output is returned when the write operations succeed:
Warning:
cgroup IO limits directly affect the read/write performance of target containers or processes. Setting bandwidth or IOPS limits too low may cause service timeouts or exceptions. Set appropriate limit values based on your actual business requirements, and verify the impact in a test environment first.
The cgroup io.max limit takes effect only when the device is busy. Setting it to max means no limit is applied to the corresponding dimension.
mkdir /sys/fs/cgroup/io_limited
echo "259:0 rbps=10485760 wbps=10485760 riops=500 wiops=500" > /sys/fs/cgroup/io_limited/io.max
echo $PID > /sys/fs/cgroup/io_limited/cgroup.procs
After writing, run the following command to confirm that the limits have taken effect:
cat /sys/fs/cgroup/io_limited/io.max
Output example:
259:0 rbps=10485760 wbps=10485760 riops=500 wiops=500

File System Mount Optimization

Remount the root filesystem:
Warning:
noatime disables access time recording. Applications that rely on atime, such as tmpwatch, some backup tools, and monitoring tools, may be affected.
mount -o remount,noatime,nodiratime /
Format and mount the new partition (ext4 + writeback mode, only for scenarios where data loss is tolerable). No output is returned when the format and mount commands succeed:
Warning:
The mkfs.ext4 format operation erases all data on the target device. Before proceeding, use lsblk to confirm that the device name is correct to avoid formatting the system disk by mistake. Performing a remount operation on the root filesystem may affect applications that rely on access time (atime).
The data=writeback mode may cause data loss in the event of a system crash and is only suitable for scenarios where data loss is tolerable. Before proceeding, confirm that your workloads have no atime dependency and no strong consistency requirements.
mkfs.ext4 -O has_journal /dev/vdb1
mount -o noatime,data=writeback,discard /dev/vdb1 /data
After the mount is complete, run the following command to confirm that the mount parameters have taken effect:
mount | grep /data
Output example:
/dev/vdb1 on /data type ext4 (rw,noatime,data=writeback,discard)

io_uring Support Check

Check kernel io_uring support:
grep CONFIG_IO_URING=y /boot/config-$(uname -r)
Output example:
CONFIG_IO_URING=y
Check the sysctl parameters related to io_uring. kernel.io_uring_disabled controls io_uring availability: 0=fully enabled, 1=available only to privileged users, 2=completely disabled:
sysctl -a | grep io_uring
Output example:
kernel.io_uring_disabled = 0
Modify io_uring availability:
Warning:
kernel.io_uring_disabled takes effect immediately after modification:
After it is set to 1, ordinary user processes will be unable to use io_uring;
Setting it to 2 completely disables io_uring, causing applications that depend on io_uring, such as databases and network services, to fail to work properly.
Do not adjust it arbitrarily in production environments. If adjustment is necessary, verify the impact in a test environment first.
Temporary modification (takes effect at runtime and becomes invalid after restart):
sysctl -w kernel.io_uring_disabled=1
Output example:
kernel.io_uring_disabled = 1
Persistent modification:
echo "kernel.io_uring_disabled = 1" > /etc/sysctl.d/99-io-uring.conf
sysctl --system
Output example (sysctl --system lists the loaded configuration files and parameter values):
* Applying /etc/sysctl.d/99-io-uring.conf ...
kernel.io_uring_disabled = 1

Performance Benchmarking (fio)

fio is not installed by default. Install it before use:
# Install fio
dnf install -y fio
Note:
Testing consumes significant IO resources. Perform it in a non-production environment or during a maintenance window.
Before testing, ensure that the target disk has sufficient free space (at least 1 GB).
The --size and --numjobs parameters should be adjusted according to actual hardware specifications. Excessively large values may consume too much memory.
Test files will be written to the target disk. After the test is complete, manually clean them up (rm -f seqread.* randrw_*).
Sequential read/write test:
fio --name=seqread --rw=read --bs=1M --size=1G --numjobs=1 --time_based --runtime=60 --group_reporting
Output example:
seqread: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, io=60.0GiB...
read: IOPS=18.2k, BW=18.2GiB/s (19.6GB/s)(60.0GiB/60001msec)
Random read/write test:
fio --name=randrw --rw=randrw --bs=4k --size=1G --numjobs=4 --time_based --runtime=60 --group_reporting --rwmixread=70
Output example:
randrw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, io=3364MiB...
read: IOPS=71.8k, BW=280MiB/s (294MB/s)(42.0GiB/60001msec)
write: IOPS=30.7k, BW=120MiB/s (126MB/s)(18.0GiB/60001msec)
The read: IOPS=xxx and write: IOPS=xxx lines in the output are key metric lines. You can use them to compare the results before and after tuning.

FAQs

Q1: "Operation not permitted" Error After the I/O Scheduler Is Switched

Cause: The user is not root, or the device does not support dynamic scheduler switching.
Solution: Switch to the root user and try again.
# 1. Confirm root privileges
sudo -i
# 2. View the schedulers supported by the device
cat /sys/block/<device_name>/queue/scheduler
# 3. If the target scheduler is not in the list, the kernel does not have this scheduler compiled.

Q2: No I/O Performance Improvement After nr_requests Is Adjusted

Cause: nr_requests only affects the block-layer queue depth and has limited effect on high-speed devices such as NVMe. The bottleneck may be in the device itself or at the file system layer.
Solution: Use fio to run benchmark comparisons, and check the file system mount parameters and device queue depth:
# View the device queue depth
cat /sys/block/<device_name>/device/queue_depth
# Check file system mount parameters
mount | grep <device_name>

Q3: cgroup io.max Limit Not Taking Effect

Cause: The io controller of cgroup v2 takes effect only when the device is busy, or cgroup v2 is not enabled on the system.
Solution: Confirm that cgroup v2 is enabled, and check whether the limit values have been written correctly:
# 1. Confirm that cgroup v2 is enabled.
mount | grep cgroup2
# 2. Confirm that the io controller is available.
cat /sys/fs/cgroup/cgroup.controllers | grep io
# 3. Confirm that the limit values have been written.
cat /sys/fs/cgroup/<cgroup_name>/io.max

Q4: Data Loss Caused by ext4 data=writeback Mode

Cause: In writeback mode, the file system considers a write complete after data is written to the cache, without enforcing the order in which data is flushed to disk. If the system crashes abnormally, data that has not yet been persisted to disk will be lost, posing a risk to data consistency.
Solution: For databases and other workloads that require high data consistency, keep the default ordered mode and avoid switching to writeback mode.

Q5: Unable to Switch the I/O Scheduler in a Container

Cause: In container environments, the /sys directory is typically mounted read-only, and processes inside the container do not have permission to modify kernel parameters such as block device scheduling policies.
Solution: Perform the relevant operations on the host, or start a privileged container (--privileged) to obtain write access to /sys.
Topik sebelumnya: Network Kernel OptimizationTopik selanjutnya: Memory Management Tuning

Bantuan dan Dukungan

Apakah halaman ini membantu?

masukan

Search
    Enter keywords to search documentation