Overview
In TencentOS Server Ops, a common type of issue is as follows: when system memory utilization remains persistently high, a large amount of disk read IO that is significantly disproportionate to the business request volume occurs, accompanied by a substantial increase in application latency. This type of issue is particularly prevalent in memory-intensive workloads such as databases and big data. This document provides a general troubleshooting methodology to help identify the root cause of the correlation among "high memory + abnormal read IO + application latency" and offers corresponding resolution directions.
Problem Principles
Memory Reclamation Mechanisms: kswapd Asynchronous Reclamation vs. direct reclaim Synchronous Reclamation
When memory is under pressure, the Linux kernel triggers memory reclamation, which is divided into two stages:
kswapd asynchronous reclamation: performed by the kernel thread kswapd in the background to reclaim memory in advance and maintain watermarks. This stage is transparent to applications and does not block memory allocation requests from business threads.
direct reclaim synchronous reclamation: when the kswapd asynchronous reclamation speed cannot keep up with the memory consumption speed and the system watermark falls below the threshold, the thread allocating memory is forced to synchronously reclaim memory in its own context. This stage blocks memory allocation requests from most threads until enough memory is reclaimed.
If direct reclaim occurs frequently, applications perceive the system as "frozen" or "hung", and business latency increases significantly as a result.
Page Cache Composition: file-backed pages vs. anonymous pages
The system's page cache is mainly composed of the following types of memory pages, which can be estimated through /proc/meminfo:
file-backed page: the sum of Active(file) + Inactive(file), which is the in-memory cache of file content and is directly related to file I/O performance. This portion of memory can be reclaimed by the kernel.
Shmem: shared memory implemented based on tmpfs / shared anonymous mmap, which is counted in both Cached and Shmem in statistics.
anonymous page: the sum of Active(anon) + Inactive(anon), which is process-private anonymous memory, such as heap memory allocated by malloc.
The Issue of Unreclaimable Anonymous Memory (Without swap)
The reclamation of anonymous memory depends on the swap mechanism: the kernel reclaims anonymous pages by writing them out (swapping out) to a swap device. When no swap device is configured on the system, anonymous memory is effectively unreclaimable unless OOM is triggered.
This leads to a subtle but serious problem: the buff/cache column in the output of the free command appears to show a large amount of reclaimable cache because the Cached statistic includes Shmem, but most of it may actually be anonymous shared memory, and the file-backed cache that the kernel can actually reclaim is already extremely small.
Application Latency Caused by Direct Reclaim Blocking
After a large amount of file-backed cache is reclaimed, a pagefault is triggered to read from disk whenever an application accesses a page that is not in RAM. The performance gap between disk and memory is huge, with a single read latency of 10ms+ for mechanical disks and millisecond-level latency for SSDs. Even accessing only a small number of pages can increase application latency by hundreds of times. Meanwhile, direct reclaim synchronous reclamation blocks memory allocation requests from all applications, further aggravating the latency.
Diagnostic Steps
Step 1: Checking System Memory Utilization
Use free or /proc/meminfo to check the overall memory usage:
Pay special attention to fields such as MemTotal, MemFree, Cached, Shmem, Active(anon), Inactive(anon), Active(file), and Inactive(file).
If memory utilization approaches 100%, you need to further identify which type of memory is being consumed.
Step 2: Analyzing Page Cache Composition
Estimate the composition of the page cache through /proc/meminfo:
file-backed cache = Active(file) + Inactive(file)
Shared memory = Shmem
Anonymous memory = Active(anon) + Inactive(anon)
Key points for assessment:
If the Cached value is large but Active(file) + Inactive(file) is very low, it indicates that most of the cache is Shmem (shared anonymous memory) and that file-backed cache has been largely reclaimed.
Use the top command to check the SHR and RSS of processes. If a process's SHR is close to its RSS, you can infer that the process mostly uses shared memory, such as anonymous shared memory allocated through mmap(MAP_ANON|MAP_SHARED).
Although anonymous shared memory is counted in both Cached and Shmem in statistics, it cannot be reclaimed when no swap device is available.
Note: The buff/cache column in the free output is calculated from Cached + Buffers + Slab. Relying solely on this column overestimates the amount of reclaimable memory.
Step 3: Checking Whether Direct Reclaim Occurs Frequently
Determine whether synchronous reclamation occurs by checking the allocstall counter in /proc/vmstat:
grep allocstall /proc/vmstat
Take multiple samples during the abnormal period. If allocstall keeps increasing, the system is frequently performing direct reclaim, and application memory allocation requests will be blocked.
Step 4: Checking kswapd CPU Utilization
Observe the CPU utilization of the kswapd kernel thread by using top or ps:
top -p $(pgrep kswapd | tr '\\n' ',' | sed 's/,$//')
If the CPU utilization of threads such as kswapd0 and kswapd1 remains high, it indicates that system memory is under pressure and asynchronous reclamation is running at full speed but still cannot keep up with memory consumption. The system is about to enter or has already entered the direct reclaim phase.
Step 5: Identifying IO Sources with iotop
Use iotop to observe the source of disk I/O:
Focus on the following situations:
Actual DISK READ is significantly greater than the sum of DISK READ across all processes: This is normal because the two metrics in iotop come from different data sources:
Actual DISK READ: This metric is read from /proc/vmstat and represents the actual read/write bandwidth between the block device and the hardware.
Total DISK READ: This metric is read from /proc/<pid>/io and represents the I/O associated with the thread.
Common causes for the difference between the two:
cached IO is recorded in /proc/<pid>/io, but it is typically not recorded in /proc/vmstat at the same time.
cached I/O between processes or threads (such as pipes and sockets) is recorded in /proc/<pid>/io, but it is not recorded in /proc/vmstat.
/proc/<pid>/io records file content I/O, but it does not record file system metadata I/O. For example, when a file is deleted, only metadata changes are involved, so /proc/<pid>/io changes very little, but /proc/vmstat records the changes.
I/O without a corresponding pid. For example, when the system cache buffer is loaded from disk to memory, Actual may also be greater than Total.
When Actual DISK READ is significantly greater than Total DISK READ and no corresponding thread can be found, this usually means that the I/O is triggered by kernel routines (such as page fault disk reads and writebacks during memory reclamation). This I/O does not belong to any user space thread, so iotop cannot associate it with a specific pid.
Root Cause Analysis
Based on the diagnostic steps above, the root cause chain of the issue is as follows:
Anonymous memory keeps growing: The application allocates a large amount of anonymous shared memory through methods such as mmap(MAP_ANON|MAP_SHARED), consuming most of the system memory.
Anonymous memory is unreclaimable: Without a swap device configured, anonymous memory cannot be reclaimed by the kernel.
file-backed cache is squeezed and reclaimed: When the kernel is under memory pressure, it can only reclaim file-backed page cache (including the code segment memory of thread VMAs), causing a sharp decrease in available file cache.
Page faults during disk reads cause a large amount of read I/O: When a page accessed by an application is not in RAM, a page fault is triggered to read from the disk, generating a large amount of disk read I/O. This I/O is triggered by kernel routines, so iotop cannot associate it with a specific process.
direct reclaim blocking increases latency: When kswapd asynchronous reclamation is insufficient to relieve memory pressure, the system enters direct reclaim synchronous reclamation, all application memory allocation requests are blocked, and service latency further increases.
In short: anonymous memory is fully occupied > file-backed cache is reclaimed > pagefault triggers disk reads > massive read I/O + direct reclaim cause blocking > application latency spikes.
Solutions
Based on the root cause, you can resolve the issue from the following aspects:
Adjusting Application Memory Usage
Check whether the application has memory leaks or excessively uses anonymous shared memory.
Optimize the application's memory allocation method to avoid excessive usage of mmap(MAP_ANON|MAP_SHARED).
Set an upper limit on memory usage (such as a cgroup limit) to prevent a single application from exhausting system memory.
Increasing System Memory
If the memory demand of the workload is reasonable but exceeds the current server capacity, consider expanding the physical memory.
Configuring swap Devices
Configure a swap device for the system (either a swap partition or a swap file) to make anonymous memory reclaimable.
Warning: swap introduces disk IO overhead. You need to balance the swap size and the swappiness parameter based on business characteristics. For latency-sensitive workloads, swap is more suitable as a fallback mechanism than as a regular reclamation method.
Other Must-Knows
Do not rely solely on the free command to determine reclaimable memory: the buff/cache column of free includes Shmem, and anonymous shared memory is not reclaimable without swap. Instead, use Active(file) + Inactive(file) from /proc/meminfo to determine the actual reclaimable file-backed cache.
In iotop, Actual DISK READ > Total DISK READ is a normal phenomenon: when I/O is triggered by kernel routines (such as page fault disk reads), there is no corresponding pid, so Actual is greater than Total. This is not a bug in iotop, but is caused by differences in data sources.
The allocstall counter is a key indicator for identifying direct reclaim: sample this counter continuously during the abnormal period. If it keeps increasing, synchronous reclamation is occurring.
High kswapd CPU utilization is an early signal of memory pressure: kswapd is already running at full speed before direct reclaim occurs, so it should be monitored as an early warning indicator.
Too little page cache directly degrades I/O performance: when file-backed cache is too low, even a small number of page accesses trigger disk reads. A single mechanical disk read can take 10ms or more, significantly impacting latency-sensitive workloads.