tencent cloud

TencentOS Server

DocumentaçãoTencentOS ServerUser GuideKernel FeaturesMemory Management Tuning

Memory Management Tuning

Baixar
Modo Foco
Tamanho da Fonte
Última atualização: 2026-10-19 17:22:48
Traduzido por IA

Background Information

Memory tuning refers to systematic adjustments to memory allocation policies, isolation, and protection mechanisms in specific business scenarios (such as memory-intensive scenarios like databases (Redis and MySQL), AI inference, and virtual machines). Through proper tuning, you can significantly improve the performance of memory-intensive applications without modifying application code and prevent memory exceptions in a single process from dragging down the entire machine. Common issues in typical scenarios include:
Low memory access efficiency: The default 4 KB pages cause frequent TLB misses, resulting in high effective memory access latency for memory-intensive applications.
User-space allocation cannot utilize hugepages: glibc malloc does not use hugepages by default, and transparent hugepages (THP) do not take effect for user-space dynamic allocation even when enabled in the system.
Core business processes may be mistakenly killed: When the system runs low on memory, the OOM Killer selects a process to terminate based on its oom_score (usually the process with the highest memory usage), without considering the business importance of the process.
Memory leak in a single container drags down the entire machine: Continuous memory growth in a single container can exhaust the memory of the entire machine, causing other unrelated services to be affected.
Based on the Linux 6.6 kernel, TencentOS Server 4 provides mechanisms such as transparent hugepages (THP), explicit hugepages (HugeTLB), the glibc hugetlb environment variable (process-level hugepage control), cgroup v2 memory limits, and PSI memory pressure metrics.

Must-Knows

Check and confirm the system memory size (view it with free -h). Hugepage pre-allocation must be planned based on the physical memory.
Before tuning in the production environment, verify the configuration in a test environment first. In particular, THP and HugeTLB configurations may affect service latency. Setting transparent hugepages to always may cause occasional latency spikes for latency-sensitive services (such as databases). For container colocation scenarios, madvise is recommended so that applications explicitly request hugepages.
The glibc hugetlb environment variable requires glibc 2.35 or later. TencentOS Server 4 meets this requirement by default. Earlier glibc versions silently ignore this variable.
When using system default hugepages (hugetlb=2) or custom hugepages (hugetlb=<N>), you must pre-allocate nr_hugepages. Otherwise, malloc will fail to obtain hugepages at runtime.
Once hugepage memory is pre-allocated, it is deducted from the regular memory pool. Unused hugepages cannot be used by other regular applications. You need to plan the reserved amount based on the actual memory requirements of your business.
The unit of the custom hugepage value (hugetlb=<N>) is bytes. To set a 1 GB hugepage, write 1073741824, not 1G.
Setting oom_score_adj to -1000 completely prevents the process from being selected by the OOM Killer. However, if the system runs out of memory, the OOM Killer may kill other critical processes instead. It is more recommended to use cgroup memory.max to limit the memory of a single process.
The cgroup memory.high should not be set too low relative to memory.max. Otherwise, it will cause frequent memory reclamation and degrade application performance. It is recommended to set memory.high to 80% to 90% of memory.max.
By default, the OOM Killer kills the process with the highest oom_score in the cgroup (usually the one with the largest memory usage), without considering the business importance of the process. Core business processes should be protected through oom_score_adj or cgroup isolation.
The OOM log examples used in this document may vary slightly with kernel version changes, but the field meanings remain the same.

API and Parameter Description

Transparent Hugepage (THP) Configuration API

Interface File
Value Range
Default Value
Read/Write Description
/sys/kernel/mm/transparent_hugepage/enabled
always/madvise/never
madvise
Read/write: Set the transparent huge page usage policy.
always enables it for all processes;
madvise is enabled only for explicitly requested regions;
never disables it.
/sys/kernel/mm/transparent_hugepage/defrag
always/defer/defer+madvise/madvise/never
madvise
Read/write: Set the memory defragmentation policy.
always continuously defragments;
defer defers defragmentation;
defer+madvise defers defragmentation for madvise regions;
madvise defragments only explicitly requested regions;
never does not defragment.
/sys/kernel/mm/transparent_hugepage/hpage_pmd_size
Read-Only
2097152(2MB)
Read: View the transparent huge page size in bytes.

glibc HugeTLB Environment Variables

Environment Variable
Value
Meaning
GLIBC_TUNABLES=glibc.malloc.hugetlb=0
0
Do not use huge pages.
GLIBC_TUNABLES=glibc.malloc.hugetlb=1
1
Use Transparent Huge Pages (requires THP to be set to madvise).
GLIBC_TUNABLES=glibc.malloc.hugetlb=2
2
Use system default huge pages (requires preallocating nr_hugepages).
GLIBC_TUNABLES=glibc.malloc.hugetlb=<N>
N>2
Use huge pages of a specified size (requires preallocating the corresponding size).
Note:
The unit is bytes, and the value must equal a hugepage size supported by the system, such as 2097152 or 1073741824. Valid values are determined by the specifications available in the /sys/kernel/mm/hugepages directory. If an unsupported value is written, malloc falls back to regular memory allocation.

Explicit Hugepage (HugeTLB) Configuration API

Interface File
Value
Default Value
Description
/proc/meminfo → Hugepagesize
Read-Only
2048 kB(2MB)
System default huge page size
/proc/meminfo → HugePages_Total
Read-Only
0
Total number of allocated huge pages
/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
Positive integer.
0
Read/write: number of reserved 2 MB huge pages
/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
Positive integer.
0
Read/write: number of reserved 1 GB huge pages

cgroup v2 Memory Control API

Interface File
Value Range
Default Value
Read/Write Description
memory.max
Number of bytes or max
max
Read/Write: memory hard limit. OOM is triggered when the limit is exceeded.
memory.high
Number of bytes or max
max
Read/Write: memory soft limit. Reclamation is triggered when the limit is exceeded.
memory.swap.max
Number of bytes or max
max
Read/Write: swap usage limit
memory.reclaim
Number of Bytes
-
Write-only: writing a specified number of bytes triggers proactive reclamation.
memory.stat
Read-Only
-
Read: page-level memory usage statistics
memory.pressure
Read-Only
-
Read: PSI memory pressure metric
memory.oom.control
0/1
0
Read/Write: whether to disable the OOM Killer in a cgroup

Key Parameters of OOM Killer

Parameter
Position
Value
Description
/proc/<pid>/oom_score_adj
procfs
-1000 ~ 1000
Adjusts the probability of a process being selected by the OOM Killer. -1000 means the process is prohibited from being killed.
/proc/<pid>/oom_score
procfs
Read-Only
Current OOM score (a higher score means a higher probability of being killed)
vm.panic_on_oom
sysctl
0/1
0=invokes the OOM Killer (default).
1=panics directly.
vm.oom_kill_allocating_task
sysctl
0/1
0=normal selection process (default). The kernel selects the termination target based on the oom_score, which is typically the process with the largest memory usage, and this target is not necessarily the process that triggered the allocation failure.
1=directly terminates the process that initiated this memory allocation, skipping the oom_score calculation.

Configuration Example

Enabling Transparent Hugepages

Attention:
Changing the THP policy takes effect immediately across the entire system and may affect latency-sensitive workloads (see precautions for details). In production environments, it is recommended to validate the change in a test environment first.
Select a transparent hugepage mode based on your workload type:
# Confirm the enablement status of transparent hugepages
cat /sys/kernel/mm/transparent_hugepage/enabled
# Expected output: always [madvise] never (Brackets indicate the current value)

# Set to always (system-wide priority for transparent hugepages)
echo always > /sys/kernel/mm/transparent_hugepage/enabled

# Set to madvise (used only for explicitly requested memory regions, recommended for container scenarios)
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

Using glibc HugeTLB Environment Variables (Transparent Hugepage Mode)

1. Confirm that THP is set to madvise (glibc hugetlb=1 depends on the madvise mode):
cat /sys/kernel/mm/transparent_hugepage/enabled
# Expected output should include [madvise]
2. Set the glibc hugetlb environment variable:
export GLIBC_TUNABLES=glibc.malloc.hugetlb=1
3. Run the target program:
./your_application
Expected result: On the first malloc call, glibc notifies the kernel via madvise(MADV_HUGEPAGE) to use transparent hugepages for the allocated memory region, reducing TLB misses.

Using glibc HugeTLB Environment Variables (System Default Hugepage Mode)

1. Confirm the system default hugepage size:
cat /proc/meminfo | grep Hugepagesize
# Expected output: Hugepagesize: 2048 kB
2. Pre-allocate hugepages (example: reserve 520 2MB hugepages, totaling approximately 1040MB). Pre-allocated hugepages are deducted from the regular memory pool. Plan the number of hugepages based on the actual memory requirements of your business:
echo 520 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
3. Set the glibc hugetlb environment variable:
export GLIBC_TUNABLES=glibc.malloc.hugetlb=2
4. Run the target program (using 1GB malloc & memset as an example):
./your_application

Using glibc HugeTLB Environment Variables (Custom 1 GB Hugepages)

1. Confirm the hugepage sizes supported by the system:
ls /sys/kernel/mm/hugepages
# Expected output: hugepages-1048576kB  hugepages-2048kB
2. Pre-allocate 1GB hugepages (pre-allocated hugepages are deducted from the regular memory pool, so plan the number based on your actual business requirements):
echo 3 > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
3. Set the glibc hugetlb environment variable (in bytes, 1GB=1073741824):
export GLIBC_TUNABLES=glibc.malloc.hugetlb=1073741824
4. Run the target program:
./your_application

Analyzing OOM Killer Logs

When an OOM event occurs, the kernel log (/var/log/messages or journalctl -k) records detailed OOM information. Key log fields:
Log Fields
Description
invoked oom-killer
Process name that triggered OOM and gfp_mask
cpuset= / mems_allowed=
cgroup and NUMA memory limits of the process
CPU: PID: Comm:
PID and command name of the triggering process
Mem-Info:
System memory usage (in pages; by default, 1 page = 4 KB on x86)
Node X ... free: ... low:
free/low memory of each NUMA node. free < low is the direct cause of OOM.
Killed process ...
Process selected by the OOM Killer and its oom_score
# View recent OOM logs
journalctl -k | grep -i "oom\\|killed process" | tail -20

# Adjust the OOM score of a process (to reduce the probability of core business processes being killed)
echo -500 > /proc/<pid>/oom_score_adj

# Completely prevent a process from being killed by the OOM Killer
echo -1000 > /proc/<pid>/oom_score_adj
Attention:
After oom_score_adj is set to -1000, the OOM Killer may kill other processes instead when the system runs out of memory. It is recommended to use cgroup memory.max to limit the memory of a single process.

Configuring cgroup v2 Memory Limits

# Create a memory-limited cgroup
mkdir /sys/fs/cgroup/myapp

# Set a hard limit (1GB)
echo 1073741824 > /sys/fs/cgroup/myapp/memory.max

# Set a soft limit (512MB, triggers reclamation when exceeded but does not kill the process)
echo 536870912 > /sys/fs/cgroup/myapp/memory.high

# Add the target process to the cgroup
echo $PID > /sys/fs/cgroup/myapp/cgroup.procs

# View cgroup memory statistics
cat /sys/fs/cgroup/myapp/memory.stat
Expected result: When the memory usage of the myapp cgroup exceeds memory.high (512MB), the kernel starts reclaiming memory from this cgroup. When it exceeds memory.max (1GB), the OOM Killer is triggered to kill the process with the highest memory usage in this cgroup, without affecting services in other cgroups.

FAQs

Q1: Increased Database Latency Jitter After Transparent Hugepages Is Set to always

Cause: When THP is set to always, the kernel performs memory defragmentation in the background (khugepaged), which may cause occasional latency.
Solution: For latency-sensitive services, switch to madvise mode so that applications explicitly request hugepages:
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

Q2: Insufficient Normal Memory After HugeTLB Hugepages Are Reserved

Cause: Pre-allocated hugepages are deducted from the regular memory pool. Reserving too many hugepages reduces the memory available to regular applications.
Solution: Adjust nr_hugepages based on actual service requirements, and monitor hugepage utilization by running cat /proc/meminfo | grep HugePages. Idle hugepages can be reclaimed:
# View hugepage usage
grep -i huge /proc/meminfo
# Reduce the number of reserved hugepages
echo 0 > /proc/sys/vm/nr_hugepages

Q3: No Effect After glibc hugetlb Environment Variables Are Set

Cause: When the glibc version is earlier than 2.35, the variable is silently ignored, or sufficient hugepages are not pre-allocated.
Solution: Check whether the glibc version is later than 2.35.
# 1. Check the glibc version
ldd --version
# 2. Confirm that hugepages are pre-allocated
grep HugePages_Total /proc/meminfo
# 3. If hugepages are insufficient, allocate them first (Note: Pre-allocated hugepages are deducted from the regular memory pool)
echo <number_of_pages> > /proc/sys/vm/nr_hugepages

Q4: OOM Killer Terminated Critical Business Processes

Cause: The OOM Killer selects processes based on oom_score without considering business importance. Core business processes are likely to be selected when they have high memory usage.
Solution: Adjust the oom_score value, or use cgroup isolation.
# Method 1: Lower the oom_score of core processes (-1000 means the process is completely protected from being killed. Note that when the system runs out of memory, the OOM Killer will kill other processes instead)
echo -1000 > /proc/<PID>/oom_score_adj

# Method 2: Use cgroup isolation (Recommended. When the memory.max limit is exceeded, only processes within that cgroup are killed, without affecting other services)
echo <limit_bytes> > /sys/fs/cgroup/<cgroup_name>/memory.max

Q5: Application Performance Degradation After cgroup memory.high is set

Cause: memory.high is set too low, causing the kernel to reclaim memory frequently and degrading application performance.
Solution: Set memory.high to 80%–90% of memory.max to avoid frequent memory reclamation.
Anterior: I/O Performance and Storage OptimizationPróximo: System Parameter Configuration Instructions

Ajuda e Suporte

Esta página foi útil?

comentários

Search
    Enter keywords to search documentation