1. Introduction to TuneD
TuneD is a service that monitors the system and optimizes performance under specific workloads.
The core of TuneD is its configuration files, which can adjust the system for different use cases. The configuration files provided by TuneD are divided into the following categories:
Power-saving configuration files.
Performance-boosting configuration files.
Merging Tuning Configuration Files
TuneD can select multiple configuration files at once and will attempt to merge them during loading. If conflicts exist, settings in the last specified configuration file take precedence.
Prerequisite: TuneD has been installed.
Example: low power consumption in virtual guests.
1. Start TuneD.
Redirecting to /bin/systemctl start tuned.service
2. Optimize the system for running in CVMs to achieve optimal performance, and tune it for low power consumption.
tuned-adm profile virtual-guest powersave
Tuning Policies
balanced: default power-saving configuration file. It aims to strike a balance between performance and power consumption, maximizing the use of autoscaling and autotuning. The only drawback is increased latency.
powersave: a configuration file for maximum power saving. It restricts performance to operate at minimal actual power consumption.
throughput-performance: a server profile optimized for high throughput. It disables power saving mechanisms and enables sysctl settings to improve the throughput performance of disk and network I/O.
accelerator-performance: an accelerator performance configuration file. It incorporates the same tuning adjustments as the throughput-performance configuration file. It can lock the CPU into low C-states, with latency under 100µs. This enhances the performance of certain accelerators, such as GPUs.
virtual-guest: a configuration file based on the throughput-performance. It reduces virtual memory swapping and increases disk readahead, among other tasks.
virtual-host: a configuration file that can reduce virtual memory swapping, increase disk readahead, and enable a more aggressive dirty page writeback value.
optimize-serial-console: a configuration file that tunes I/O activity for serial consoles by reducing printk values.
tuned-adm profile throughput-performance optimize-serial-console
intel-sst: a configuration file that optimizes the system with user-defined Intel Speed Select Technology configurations.
tuned-adm profile cpu-partitioning intel-sst
Tuned CPU Partitioning Configuration Files
In the /etc/tuned/cpu-partitioning-variables.conf file, configure cpu-partitioning configuration files: isolated CPUs with load balancing, isolated CPUs without load balancing, and housekeeping CPUs.
Using a Tuned CPU Partitioning Configuration File to Lower Latency
Prerequisite: The CPU-partitioning TuneD configuration file has been installed using the yum install tuned-profiles-cpu-partitioning command as the root user.
1. Edit the /etc/tuned/cpu-partitioning-variables.conf file and add the following information.
Isolated CPUs with the kernel’s scheduler load balancing:
isolated_cores=2-23
Isolated CPUs without the kernel’s scheduler load balancing:
no_balance_cores=2,3
2. Set the cpu-partitioning TuneD configuration file.
tuned-adm profile cpu-partitioning
3. Restart.
Installing and Starting TuneD
1. Install the TuneD package.
2. Start and enable the TuneD service.
systemctl enable --now tuned
3. (Optional) Install the TuneD configuration file for the real-time system.
yum install tuned-profiles-realtime tuned-profiles-nfv
4. Verify whether the TuneD configuration file is active and applied.
Check whether the current system uses the configuration file using the tuned-adm active command.
Current active profile: virtual-guest
Check whether the current system settings match the preset configuration file using the #tuned-adm verify command.
Verfication succeeded, current system settings match the preset profile.
See TuneD log file ('/var/log/tuned/tuned.log') for details.
Listing Available Tuned Configuration Files
1. List all available TuneD configuration files on the system.
Available profiles:
- accelerator-performance - Throughput performance based tuning with disabled higher latency STOP states
- balanced - General non-specialized tuned profile
- cpu-partitioning - Optimize for CPU partitioning
- desktop - Optimize for the desktop use-case
- hpc-compute - Optimize for HPC compute workloads
- intel-sst - Configure for Intel Speed Select Base Frequency
- latency-performance - Optimize for deterministic performance at the cost of increased power consumption
- my-profile
- network-latency - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance
- network-throughput - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks
- optimize-serial-console - Optimize for serial console use.
- powersave - Optimize for low power consumption
- realtime - Optimize for realtime workloads
- realtime-virtual-guest - Optimize for realtime workloads running within a KVM guest
- realtime-virtual-host - Optimize for KVM guests running realtime workloads
- throughput-performance - Broadly applicable tuning that provides excellent performance across a variety of common server workloads
- virtual-guest - Optimize for running inside a virtual guest
- virtual-host - Optimize for running KVM guests
Current active profile: virtual-guest
2. Show only the currently active configuration file.
Current active profile: virtual-guest
Setting a Tuned Configuration File
The selected TuneD configuration file is activated on your system during this process.
1. You can let TuneD recommend the most suitable configuration file for your system.
tuned-adm recommend
virtual-guest
2. Activate the configuration file.
tuned-adm profile selected-profile
Activate multiple configuration files using the following command.
tuned-adm profile profile1 profile2
3. Restart the system.
4. Verify whether the TuneD configuration file is active and applied.
Verfication succeeded, current system settings match the preset profile.
See TuneD log file ('/var/log/tuned/tuned.log') for details.
Disabling TuneD
1. Temporarily disable all TuneD services.
2. Permanently stop using and disable the TuneD service.
systemctl disable --now tuned
2. Disk Management
Viewing Disks
1. View the disk information. Disks that are not attached are not displayed.
Filesystem Size Used Avail Use% Mounted on
devtmpfs 830M 0 830M 0% /dev
tmpfs 845M 24K 845M 1% /dev/shm
tmpfs 845M 250M 596M 30% /run
tmpfs 845M 0 845M 0% /sys/fs/cgroup
/dev/vda1 50G 4.5G 43G 10% /
tmpfs 169M 0 169M 0% /run/user/0
2. View the list of all disks of the host.
Disk /dev/vda: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disklabel type: dos
Disk identifier: 0xe609e297
Device Boot Start End Sectors Size Id Type
/dev/vda1 2048 104857566 104855519 50G 83 Linux
3. View the disk usage of a specified directory.
Place the directory name directly after the df -h /dev command.
Filesystem Size Used Avail Use% Mounted on
devtmpfs 830M 0 830M 0% /dev
Introduction to the df command:
-a: lists all file systems, including system-specific file systems such as /proc.
-k: displays each file system with the capacity in KB.
-m: displays each file system with the capacity in MB.
-h: displays file systems with capacities in human-readable formats such as GB, MB, KB.
-H: indicates that 1 MB equals 1,000 KB instead of 1,024 KB.
-T: displays the file system type.
-i: displays file systems with the number of inodes instead of the disk capacity.
-l: displays only local file systems.
Introduction to the du command:
Purpose: The du command is used to view the usage of a specified directory.
Option: -h: displays the usage of file system partitions.
-s: displays the size of a file or the entire directory.
Adding a Disk
After adding a hard disk, you must partition, format, and attach it before using it.
1. Partition the disk: There are two partitioning methods.
MBR partitioning, which uses the fdisk command.
GPT partitioning, which uses the gdisk command.
2. Format the disk:
Format a specified partition to the ext4 format: mkfs -t ext4 /dev/sdb1.
Format 2 GB space of a specified disk: mkfs /dev/sdb2 2G.
3. Attach the disk:
Establish a mapping between the device backup partition and the system directory.
mount [device name] [directory for mounting]
Deleting a Disk Partition
1. View the hard disk information to locate the hard disk containing the partition you want to delete.
2. Enter the disk partition.
3. Enter d to delete the partition. If multiple partitions exist, you need to select the partition number.
4. Enter w to save the changes and exit the process.
Available Disk Schedulers
none:
First-In-First-Out (FIFO) scheduling algorithm. It merges requests from the generic block layer through a simple last cache.
mq-deadline:
This scheduler attempts to provide guaranteed latency for requests from the time they arrive at the scheduler. It applies to most use cases, particularly where most write operations are asynchronous.
bfq:
This scheduler is designed for desktop systems and interactive tasks, and it applies to large file replication without causing the system to become unresponsive.
kyber:
This scheduler adjusts itself by calculating the latency of each I/O request submitted to the block I/O layer to meet latency requirements. It applies to fast devices such as non-volatile memory express (NVMe) devices, SSDs, or other low-latency devices.
Default disk scheduler:
For NVMe block devices, the default scheduler is none.
Determining the Active Disk Scheduler
Identify the disk scheduler that is currently active on the given block device.
cat /sys/block/device/queue/scheduler: Replace "device" with the block device name.
[none] mq-deadline kyber bfq
Using TuneD to Configure the Disk Scheduler
1. Identify the configuration set that is currently active.
2. Create a directory to store the TuneD configuration set.
mkdir /etc/tuned/my-profile
3. Find the system unique identifier of the selected block device.
devadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'
[root@VM-6-117-centos ~]
ID_SERIAL=QEMU_DVD-ROM_QM00002
ID_SERIAL_SHORT=QM00002
4. Create the configuration file /etc/tuned/my-profile/tuned.conf. In this file, set the selected disk scheduler for the device matching the World Wide Name (WWN) identifier.
[disk]
devices_udev_regex=IDNAME=device system unique id
elevator=selected-scheduler
5. Enable your configuration set.
tuned-adm profile my-profile
6. Verify whether the TuneD configuration set is active and applied.
Current active profile: my-profile
Verfication succeeded, current system settings match the preset profile.
See TuneD log file ('/var/log/tuned/tuned.log') for details.
7. Read the content of the /sys/block/device/queue/scheduler file.
cat /sys/block/device/queue/scheduler
[root@VM-6-117-centos ~]
mq-deadline kyber [bfq] none
Setting a Disk Scheduler Using udev Rules
1. Find the system unique identifier of the block device.
udevadm info --name=/dev/sr0 | grep -E '(WWN|SERIAL)'
[root@VM-6-117-centos ~]
E: ID_SERIAL=QEMU_DVD-ROM_QM00002
E: ID_SERIAL_SHORT=QM00002
2. Configure udev rules. Create the /etc/udev/rules.d/99-scheduler.rules file using the following content.
Note:
Replace IDNAME with the name of the identifier to be used (for example, ID_WWN).
Replace the system unique identifier of the device with the selected identifier value (for example, 0x5002538d00000000).
ACTION=="add|change", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="QM00002", ATTR{queue/scheduler}="bfq"
3. Reload udev rules.
udevadm control --reload-rules
4. Apply the scheduler configurations.
udevadm trigger --type=devices --action=change
5. Verify whether the scheduler is active.
cat /sys/block/device/queue/scheduler*
[root@VM-6-117-centos ~]
mq-deadline kyber [bfq] none
Setting a Scheduler for a Specific Disk Temporarily
This process configures the given disk scheduler for a specific block device. The settings are not retained after a system restart.
1. Write the name of the selected scheduler to the /sys/block/device/queue/scheduler file.
echo selected-scheduler > /sys/block/device/queue/scheduler
In the file name, replace "device" with the block device name, for example, vda.
2. Verify whether the scheduler is active for the device.
cat /sys/block/device/queue/scheduler
[root@VM-6-117-centos ~]
[none] mq-deadline kyber bfq
3. CPU Utilization and Tuning
Displaying the CPU Architecture Information
1. Collect the CPU architecture information, such as the number of CPUs, threads, kernels, slots, and Non-Uniform Memory Access (NUMA) nodes.
[root@VM-6-117-centos ~]
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 2
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
BIOS Vendor ID: Smdbmds
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8374C CPU @ 2.70GHz
BIOS Model name: 3.0
Stepping: 6
CPU MHz: 2700.000
BogoMIPS: 5400.00
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 48K
L1i cache: 32K
L2 cache: 1280K
L3 cache: 58368K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb ibrs_enhanced fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 wbnoinvd arat avx512vbmi umip avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq arch_capabilities
Viewing CPU Utilization
Display the CPU performance.
%Cpu(s): 0.2 us, 0.2 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
user (usually abbreviated as us): indicates the user-space CPU time. Note: It does not include the nice time below but includes the guest time.
nice (usually abbreviated as ni): indicates the low-priority user-space CPU time, which is the CPU time when the nice value of a process is adjusted between 1 and 19. Note: The nice value ranges from -20 to 19, and a higher value indicates a lower priority.
system (usually abbreviated as sy): indicates the kernel-space CPU time.
idle (usually abbreviated as id): indicates the idle time. Note: It does not include the I/O wait time (iowait).
iowait (usually abbreviated as wa): indicates the CPU time waiting for I/O.
irq (usually abbreviated as hi): indicates the CPU time for handling hard interrupts.
softirq (usually abbreviated as si): indicates the CPU time for handling soft interrupts.
steal (usually abbreviated as st): indicates the CPU time taken by other virtual machines when the system is running in a virtual machine.
guest (usually abbreviated as guest): indicates the time spent on running other OSs through virtualization, which is the CPU time for running virtual machines.
guest_nice (usually abbreviated as gnice): indicates the time spent on running virtual machines with a low priority. CPU utilization is the percentage of time spent on tasks (excluding idle time) to the total CPU time.
Formula: CPU utilization = 1 - idle time / total CPU time.
Viewing CPU Utilization
Install the sysstat tool.
View CPU utilization using the pidstat 1 command.
[root@VM-6-117-centos ~]
Linux 5.4.119-19-0009.1 (VM-6-117-centos) 08/19/2022 _x86_64_ (2 CPU)
03:11:04 PM UID PID %usr %system %guest %wait %CPU CPU Command
03:11:05 PM 0 561098 1.00 0.00 0.00 0.00 1.00 0 barad_agent
03:11:05 PM UID PID %usr %system %guest %wait %CPU CPU Command
03:11:06 PM 0 561098 0.00 1.00 0.00 0.00 1.00 0 barad_agent
perf Tool
1. Install the perf tool. The user-space tool perf is developed based on the Linux subsystem called Performance Counters for Linux (PCL) in the kernel. perf is a powerful tool that uses the performance monitoring unit (PMU) to measure, record, and monitor hardware and software events.
2. Use perf top to analyze CPU utilization.
The perf top command is used for real-time system analysis, and its features are similar to those of the top utility. The top utility typically displays the CPU time used by a given process or thread, while perf top displays the CPU time used by each specific feature.
6.48% perf [.] __symbols__insert
5.21% [kernel] [k] kallsyms_expand_symbol.constprop.1
4.80% perf [.] rb_next
2.48% perf [.] rust_demangle_callback
2.42% libc-2.28.so [.] __GI_____strtoull_l_internal
2.32% [kernel] [k] vsnprintf
2.10% [kernel] [k] number
As shown in the figure, the SLsmg_write_chars feature consumes the most CPU time.
Investigating Busy CPUs via perf
1. Count events with CPU quantity aggregation disabled.
perf stat -a -A sleep seconds
Specify events such as cycles: perf stat -a -A -e cycles sleep seconds.
2. Display CPU samples generated by perf report.
A perf.data file containing perf records is created in the current directory.
Sort by CPU and command: perf report --sort cpu,comm.
Displaying Specific CPUs During Analysis with perf top
Sort by CPU and command: perf top --sort cpu,comm.
Monitoring Specific CPUs with perf record and perf report
1. Record the performance data of specific CPUs, generate a perf.data file, and list CPUs separated by commas.
perf record -C 0,1 sleep seconds
Use a set of CPUs.
perf record -C 0-2 sleep seconds
2. Display the content of the perf.data file for further analysis.
4. Memory Utilization
Viewing Memory Usage
1. Use the free command: free.
[root@VM-6-117-centos ~]
total used free shared buff/cache available
Mem: 1730432 375364 130400 255480 1224668 932144
Swap: 0 0 0
total indicates that the total memory is 1.6 GB.
used indicates the usage of physical memory, which is about 215 MB.
free indicates idle memory.
shared indicates shared memory.
buff/cache indicates the capacity of cache and buffer memory. To improve performance, Linux systems cache a large amount of data, which can be released when necessary and used by other programs.
available indicates available memory.
Swap indicates swap memory.
2. Use vmstat: vmstat -m.
[root@VM-6-117-centos ~]
Cache Num Total Size Pages
rpc_inode_cache 46 46 704 23
rpc_buffers 16 16 2048 16
rpc_tasks 16 16 256 16
ext4_groupinfo_4k 420 420 144 28
fscrypt_info 0 0 104 39
fscrypt_ctx 0 0 64 64
bridge_fdb_cache 0 0 128 32
ip6-frags 0 0 208 19
fib6_nodes 128 128 64 64
ip6_dst_cache 32 32 256 16
Perform statistics on memory usage.
3. View /proc/meminfo: cat /proc/meminfo.
[root@VM-6-117-centos ~]
MemTotal: 1730432 kB
MemFree: 131156 kB
MemAvailable: 933048 kB
Buffers: 80508 kB
Cached: 1070344 kB
SwapCached: 0 kB
Active: 847508 kB
Inactive: 575192 kB
Active(anon): 437152 kB
Inactive(anon): 90176 kB
Active(file): 410356 kB
Inactive(file): 485016 kB
Unevictable: 0 kB
Mlocked: 0 kB
The /proc directory contains virtual files, including dynamic information about the kernel and OS. /proc/meminfo serves as the main entry for understanding memory usage in Linux systems. Commonly used commands such as free and vmstat obtain data through it. The information contained in /proc/meminfo is far more extensive than that obtained by commands such as free.
Viewing the Physical Memory Information
1. Use the dmidecode command.
The displayed information includes the memory size, type, bandwidth, and other information.
Virtual Memory
1. Introduction: Physical memory is finite (even with hot swapping supported) and non-contiguous, with organizational structures varying across CPU architectures. This results in high complexity for the direct use of physical memory. To solve this issue, the virtual memory mechanism is introduced.
2. Usage: The virtual memory mechanism makes each process believe that it occupies the entire memory. When a process accesses memory, the OS translates the virtual memory address provided by the process into a physical address and then obtains data from the corresponding physical address.
3. The virtual memory mechanism not only solves conflicts when multiple processes access memory through address translation but also offers other benefits.
Process memory integrity management: Due to the deception of virtual memory to processes, each process believes that the memory it obtains is a contiguous block of addresses. When developing applications, developers do not need to worry about allocating large blocks of addresses and can simply assume that the system has sufficient large-block memory.
Data sharing: Virtual memory facilitates the sharing of memory and data.
Segmented Paging Mechanism
OSs today employ two memory management mechanisms: segmentation and paging.
Segmentation divides memory into segments, where the starting address of each segment serves as the segment base address. During address mapping, the physical address is obtained by appending the logical address to the segment base address.
Paging divides memory into fixed-length pages. During address mapping, a page table is created first, with each entry in the page table recording the base address of a page. Through the page table, the high-order bits of the logical address are used to obtain the corresponding page base address first. Then, the final physical address is obtained by offsetting this page base address by a specific length, which is determined by the low-order bits of the logical address.
Segmented paging requires the combined use of segment tables and page tables. A segment table contains segment numbers and starting addresses of page tables for each segment. A page table contains page numbers and page offsets. First, the segment number in a virtual address is used to obtain the corresponding segment number in a segment table. Then, the segment number is used to obtain the starting address of the page table for the corresponding segment. Next, the page number within the segment in the virtual address is used to obtain the corresponding page table entry for the physical starting address. Finally, the corresponding physical address is obtained by appending the page offset in the virtual address.
Memory Reclamation
1. Memory reclamation targets: Not all physical memory can be reclaimed by the kernel. For example, if the kernel code segment is reclaimed, the system cannot run properly. Therefore, the kernel code segment, data segment, memory requested by the kernel, and memory occupied by kernel threads generally cannot be reclaimed. All other memory can be reclaimed.
2. Memory reclamation timing:
The kernel needs to provide sufficient memory to handle memory requests at any moment, ensuring that the use of cache and other memory does not cause the remaining memory of the system to remain critically low for extended periods.
If the requested memory exceeds the available free memory, forced memory reclamation will be triggered.
3. Reclamation mechanism:
For the first scenario, Linux systems employ the background program kswapd. When the kernel allocates physical pages but fails to do so under low-watermark conditions due to a system memory shortage, the kernel thread kswapd will be woken up to asynchronously reclaim memory. This is called periodic memory reclamation.
For the second scenario, Linux systems trigger direct memory reclamation. When the kernel calls a page allocation function to allocate physical pages but fails to process the request due to a system memory shortage, it directly triggers the page reclamation mechanism and attempts to solve the issue through memory reclamation. This is called direct memory reclamation.