tencent cloud

TencentOS Server

DocumentaçãoTencentOS ServerUser GuideSystem Diagnosis and Observabilitykdump and Kernel Crash Dumps

kdump and Kernel Crash Dumps

Baixar
Modo Foco
Tamanho da Fonte
Última atualização: 2026-10-19 17:22:52
Traduzido por IA

Feature Overview

This document describes how to install, configure, trigger tests for, and analyze dump files from the kdump kernel crash dump mechanism in TencentOS Server 4. It helps you preserve on-site memory images when a kernel panic occurs in production environments, facilitating post-incident root cause analysis of kernel crashes.

Background Information

Although kernel panics are rare in production environments, when one occurs, it often causes the entire machine to reboot and disrupts services. After the reboot, the on-site state is lost, making root cause analysis difficult. kdump uses kexec to quickly boot a preloaded capture kernel when a kernel crash occurs, saving the kernel memory image at the time of the crash as a dump file (vmcore) in a minimal environment for later offline analysis with the crash tool.
TencentOS Server 4 supports kdump by default. Public images of Tencent Cloud CVM come with kexec-tools preinstalled and automatically reserve crashkernel based on memory size (for example, the crashkernel=3G-16G:256M,16G-64G:384M,... tiered policy), and the kdump service starts automatically at boot. You can quickly verify this by running cat /sys/kernel/kexec_crash_loaded (an output of 1 indicates readiness). Self-managed data centers or custom image environments may not have kdump configured. Machines without proper kdump configuration will only reboot after a panic without preserving any on-site information, in which case you need to complete the installation and configuration as described in this document. It is recommended that all production environments verify that kdump is enabled during the initialization phase.

Must-Knows

Before you start configuring kdump, confirm the following items. TencentOS Server 4 supports both x86_64 and aarch64 architectures, which have different crashkernel reservation policies. Run uname -m to confirm your current architecture.
X86_64 architecture: crashkernel=256M is recommended (for memory less than or equal to 16GB) or 256M,high (for memory greater than 16GB). Tencent Cloud CVM public images automatically reserve crashkernel by memory tier by default, so no manual configuration is required.
ARM (aarch64) architecture: crashkernel=512M,high is recommended because ARM capture kernel initialization requires more memory, and insufficient reservation causes kdump loading to fail.
root privileges: All operations in this document must be performed as the root user, or you can switch to root by running sudo -i.
dnf repository availability: Installing kexec-tools and crash requires access to the system default dnf repositories. Installing kernel-debuginfo requires an additional debuginfo repository because the default repositories do not contain this package. For configuration instructions, see Step 1.
Reboot window: After modifying the crashkernel memory reservation parameter, you must reboot the system for the change to take effect. Plan the reboot window in advance.
You can quickly confirm the above by running the following command:
# Confirm the system architecture
uname -m  # Expected output: x86_64 or aarch64

# Confirm that the current user is root
whoami  # Expected output: root
 
# Confirm the disk space (the partition where the dump directory resides, /var/crash by default)
df -h /var/crash  # Expected output: available space is no less than 20% of physical memory
 
# Confirm the kernel version
uname -r  # Expected output: <kernel-version>, for example, 6.6.79-25.8.tl4.x86_64
 
# Confirm that the dnf source is available
dnf repolist  # Expected output: lists the configured software repositories

API and Parameter Description

Core Configuration Parameters for kdump

Parameter
Position
Value
Description
crashkernel
kernel boot parameter
256M / 256M,high / 128M@64M
Reserve memory for the capture kernel.
KDUMP_KERNEL
/etc/sysconfig/kdump
Kernel Image Path
Path to the capture kernel vmlinuz
KDUMP_INITRD
/etc/sysconfig/kdump
initramfs Path
Path to the capture kernel initramfs
KDUMP_COMMANDLINE_APPEND
/etc/sysconfig/kdump
Kernel Parameter String
Append to the capture kernel boot parameters.
path
/etc/kdump.conf
Directory Path
Directory for saving dump files
core_collector
/etc/kdump.conf
makedumpfile Parameters
Policy for filtering and compressing dump files

makedumpfile Filter Levels

Parameter
Description
Dump Size
-d 0
No filtering, full dump
100%
-d 1
Exclude zero pages
~70-80%
-d 31
Exclude zero pages + cache pages + user data pages (recommended)
~10-20%
-d 31 -c
Exclude zero pages + cache pages + user data pages + compress output
Minimum

Detailed Description of makedumpfile Parameters

Parameter
Description
-l
Compress dump files (LZO compression)
--message-level <n>
Controls the verbosity level of output logs. 1 means only progress information is printed.
-d <bitmask>
Filter mask that filters unnecessary pages bit by bit: bit0 = zero page, bit1 = cache page, bit2 = user data page, bit3 = free page; 31 = 0b11111 means excluding all the above.
-c
Compress the filtered data.

Configuration Example

Warning: kdump configuration involves modifying kernel boot parameters and restarting the system. Perform operations during a maintenance window. Triggering a panic test will cause an immediate system restart and is strictly prohibited in production environments.

1. Installing kdump-Related Software Packages

# Install kexec-tools (kdump core) and crash (analysis tool)
dnf install kexec-tools crash

# Install kernel-debuginfo (required for the crash tool to analyze vmcore and provides vmlinux debug symbols)
# Note: kernel-debuginfo is not in the default repository, so add the debuginfo repository first:
cat > /etc/yum.repos.d/tl4-debuginfo.repo << 'EOF'
[tl4-debuginfo]
name=TencentOS Server 4 - Debuginfo
baseurl=https://mirrors.tencent.com/tlinux/4.0/BaseOS/x86_64/debug/tree/
enabled=1
gpgcheck=0
EOF
# The debuginfo package version must exactly match the running kernel (check with uname -r). If the corresponding version is not yet available in the repository, wait for the repository to be updated.
dnf install kernel-debuginfo-$(uname -r)

# Verify the kernel-debuginfo installation result: the vmlinux debug symbol file should exist
ls /usr/lib/debug/lib/modules/$(uname -r)/vmlinux
# Expected output: /usr/lib/debug/lib/modules/<kernel-version>/vmlinux

2. Configuring crashkernel Memory Reservation

# View the current crashkernel configuration
cat /proc/cmdline | grep crashkernel

# Use grubby to configure crashkernel
# x86_64 architecture: 256M is recommended (for memory less than or equal to 16GB) or 256M,high (for memory greater than 16GB)
grubby --update-kernel=ALL --args="crashkernel=256M"

# aarch64 (ARM) architecture: 512M,high is recommended because the ARM capture kernel requires more memory
# grubby --update-kernel=ALL --args="crashkernel=512M,high"

# Restart to make the configuration take effect
reboot
After the reboot, verify whether kdump is loaded:
# Check whether kdump is loaded
cat /sys/kernel/kexec_crash_loaded
# Expected output: 1 (1 indicates that kdump is loaded, and 0 indicates that it is not loaded)

# View the crashkernel reserved information in the kernel log
dmesg | grep -i crash
# Expected output: contains lines such as "Reserving XXXMB of memory for crashkernel"

3. Configuring the kdump Main Configuration File

Run vim /etc/sysconfig/kdump to edit the main configuration file. In the following example, <kernel-version> is a placeholder. Replace it with the output of uname -r (for example, 6.6.79-25.8.tl4.x86_64):
# Capture kernel image path
KDUMP_KERNEL="/boot/vmlinuz-<kernel-version>"
# Capture the kernel initramfs path
KDUMP_INITRD="/boot/initramfs-<kernel-version>kdump.img"
# KDUMP_COMMANDLINE_APPEND is used to append custom parameters (avoiding duplication with default parameters)
KDUMP_COMMANDLINE_APPEND="panic=1 swiotlb=noforce novmcoredd"

4. Configuring the Dump Target

Run vim /etc/kdump.conf to edit the dump target configuration file:
# Local file system dump
path /var/crash
# core_collector uses makedumpfile to filter and compress
# -l: LZO compression; --message-level 1: print progress only; -d 31: exclude zero pages + cache pages + user data pages
core_collector makedumpfile -l --message-level 1 -d 31

# NFS dump (optional)
# nfs my.server.com:/export/crash
# path /var/crash
# core_collector makedumpfile -l --message-level 1 -d 31

5. Starting the kdump Service

# Enable autostart on boot.
systemctl enable kdump

# Start the kdump service.
systemctl start kdump

# Check the service status.
systemctl status kdump
# Expected output: Active: active (exited) with no errors.

6. Verifying kdump Configuration Integrity

# Rebuild the kdump initramfs (run after modifying the configuration or upgrading the kernel).
kdumpctl rebuild
# Expected output: A message indicating that the rebuild was successful with no errors.

# Verify the kdump service status.
kdumpctl status
# Expected output: kdump is operational.

7. Triggering a Kernel Crash Test

Warning: The following command will immediately cause a kernel panic and restart the system. Run it only in a test environment. It is strictly prohibited in production environments, as it will cause service interruption and data loss.
# Enable sysrq.
echo 1 > /proc/sys/kernel/sysrq

# Trigger a kernel crash (test environments only).
echo c > /proc/sysrq-trigger
The system will panic and reboot into the capture kernel, and the dump file will be saved to the /var/crash/ directory.

8. Analyzing Dump Files

# View the dump file.
ls -la /var/crash/
# Expected output: A directory named with the IP address and date-time is listed, for example, 127.0.0.1-2026-08-09-14:30.

ls /var/crash/127.0.0.1-$(date +%Y-%m-%d-%H:%M)/
# Expected output: Contains the vmcore file.

# Analyze with the crash tool (requires installing kernel-debuginfo in advance to provide vmlinux)
# Replace <dump-dir> with the actual dump directory name and <kernel-version> with the output of uname -r.
crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/<dump-dir>/vmcore
Examples of interactive crash commands:
crash> bt              # View the crash backtrace.
crash> ps              # View the process list at the time of the crash.
crash> log             # View the kernel log (dmesg).
crash> sys             # View system information.
crash> quit            # Exit crash.

Expected Results

Operation Steps
Expected Output
Verification Method
Installing a Software Package
dnf install succeeds without errors.
rpm -q kexec-tools crash kernel-debuginfo lists the installed packages.
Configuring crashkernel and Restarting.
cat /sys/kernel/kexec_crash_loaded outputs 1.
dmesg | grep crash displays the reserved memory.
Starting the kdump Service.
Active: active (exited)
systemctl status kdump
kdumpctl rebuild
rebuild success message
kdumpctl status outputs operational.
Triggering a panic (Test).
After the system restarts, a vmcore file is generated under /var/crash/.
ls /var/crash/ shows that a dump directory exists.
Analyzing crash
Enter the crash interactive interface.
After entering the crash interactive interface, you can run commands such as bt, ps, and log normally.

FAQs

Q1: kdump Service Fails to Start with the Error "No kdump initial ramdisk found"

Cause: The kdump initramfs is not generated or the path is configured incorrectly.
Solution: Rebuild the kdump initramfs and verify the path configuration.
# 1. Manually rebuild the kdump initramfs.
kdumpctl rebuild

# 2. Check whether the KDUMP_INITRD path in /etc/sysconfig/kdump is correct.
# Replace <kernel-version> with the output of uname -r.
grep KDUMP_INITRD /etc/sysconfig/kdump
# Expected output: KDUMP_INITRD="/boot/initramfs-<kernel-version>kdump.img"

# 3. Verify that the initramfs file exists.
ls -la /boot/initramfs-$(uname -r)kdump.img

# 4. Restart the service.
systemctl restart kdump

Q2: No Dump File After a panic Restart (/var/crash Is Empty)

Cause: The crashkernel memory reservation failed or is insufficient, causing the capture kernel to fail to start.
Solution: Check the crashkernel reservation status. If the reservation fails, increase the reserved value and then reboot.
# 1. Verify that crashkernel is reserved.
cat /proc/cmdline | grep crashkernel
# Expected output: contains crashkernel=256M or a similar parameter.

# 2. Verify that kdump is loaded.
cat /sys/kernel/kexec_crash_loaded
# Expected output: 1

# 3. Check dmesg to verify whether the reservation is successful.
dmesg | grep -i "crashkernel"
# Expected output: Reserving XXXMB of memory for crashkernel

# 4. If the reservation fails, increase the crashkernel value and then reboot.
# x86_64:grubby --update-kernel=ALL --args="crashkernel=256M,high"
# aarch64:grubby --update-kernel=ALL --args="crashkernel=512M,high"

Q3: Dump File Write Fails with Insufficient Disk Space Error

Cause: The available space in the partition where /var/crash is located is insufficient to accommodate the vmcore.
Solution: Clean up old dump files or reduce the dump size. If the space is still insufficient, change the dump target.
# 1. Check the current available space.
df -h /var/crash

# 2. Clean up old dump files. Before doing so, confirm that the old dumps have been analyzed or archived, because vmcore is a non-reproducible fault scene and cannot be recovered after deletion.
rm -rf /var/crash/127.0.0.1-*

# 3. Increase the makedumpfile filter level to reduce the dump file size.
# Edit /etc/kdump.conf and change core_collector to:
# core_collector makedumpfile -l --message-level 1 -d 31 -c

# 4. Alternatively, change the dump target to a partition with more space or an NFS mount point.

Q4: kdump Stops Working After a Kernel Upgrade

Cause: After the kernel upgrade, the capture kernel path does not match the running kernel, and the kdump initramfs is not updated.
Solution: Update the capture kernel path and rebuild the initramfs to make kdump match the new kernel.
# 1. Update the KDUMP_KERNEL and KDUMP_INITRD paths in /etc/sysconfig/kdump.
# Replace <kernel-version> with the new kernel version (the output of uname -r).
# KDUMP_KERNEL="/boot/vmlinuz-<kernel-version>"
# KDUMP_INITRD="/boot/initramfs-<kernel-version>kdump.img"

# 2. Install the kernel-debuginfo package for the new kernel.
dnf install kernel-debuginfo-$(uname -r)

# 3. Rebuild the kdump initramfs.
kdumpctl rebuild

# 4. Restart the kdump service.
systemctl restart kdump

# 5. Verification
kdumpctl status

Q5: crash Tool Reports "cannot find vmlinux" or "debuginfo not installed"

Cause: kernel-debuginfo is not installed or the version does not match.
Solution: Install the kernel-debuginfo package that matches the currently running kernel version.
# 1. Install the kernel-debuginfo package that matches the current kernel version.
dnf install kernel-debuginfo

# 2. Verify that the vmlinux file exists.
ls /usr/lib/debug/lib/modules/$(uname -r)/vmlinux

# 3. If the versions do not match, install the corresponding version.
dnf install kernel-debuginfo-$(uname -r)

Anterior: OOM Killer Analysis and HandlingPróximo: Disk I/O Issue Analysis

Ajuda e Suporte

Esta página foi útil?

comentários

Search
    Enter keywords to search documentation