Using Single-Line Full-Text Extraction Mode to Collect Logs
Download
포커스 모드
폰트 크기
마지막 업데이트 시간: 2026-08-17 16:39:21
AI 번역
Scenarios
The single-line full-text extraction mode guides you to use the line break \\n as the terminator of a log when collecting, and store each log in a default key value __CONTENT__. The log data itself will not be processed in a structured manner, nor will log fields be extracted. The time item of log attributes is determined by the time of log collection. This document describes how to collect logs using the single-line full-text extraction mode.
Prerequisites
The server where the target file resides has LogListener installed. For details, see:
2. In the left sidebar, select Overview to go to the overview page.
3. In Fast Integration > Server and application, locate and click Single-line full text - File log to enter the LogListener collection configuration process.
4. On the Create Log Topic page, specify the log topic name, configure the log storage duration, select a logset based on your actual requirements, and click Next.
If you want to select an existing log topic, perform the following operations:
2. In the left navigation bar, select Log Topic, then select the log topic to be delivered, click the log topic name to enter the log topic management page.
3. Select the Collection Configuration tab, click Add in the LogListener collection configuration section to go to the collection configuration process page.
4. On the log data source selection page, select Server and application, locate and click Single-line full text - File log to enter the machine group management process.
Step 2: Managing Machine Groups
If the target server from which you want to collect logs does not have LogListener installed, see:
In the machine group management interface, you can choose to use existing machine groups or create machine groups:
Creating a Machine Group
Selecting an Existing Machine Group
If you want to create a machine group, perform the following operations:
1. Click Create Machine Group.
2. Fill in the machine group name, associate the target server with LogListener installed via the machine label (see Machine Group for details), and then click OK
3. After creation is completed, select the system environment of your created machine group from the Tab options, check your target machine group in the list, and click Next.
If you want to select existing machine groups, choose the system environment of your created machine group from the Tab options, check your target machine group in the list, and click Next.
Step 3: Collection Configuration
Configuring the Log File Collection Path
On the Collection Configuration page, enter a collection rule name and specify the Collection Path based on the log collection path format. Example format:
Note:
For Linux systems, the log path must start with /. For Windows systems, the file path must start with a drive letter, such as C:\\.
Log path in a Linux system: /[Directory prefix expression]/**/[File name expression]. Example: /data/log/**/*.log.
Log path in a Windows system: [Drive letter]:\\[Directory prefix expression]\\**\\[File name expression]. Example: C:\\Program Files\\Tencent\\...\\*.log.
After the log collection path is specified, LogListener will match all common prefix paths that meet the rules based on the [directory prefix expression], and listen to all log files under these directories (including subdirectories) that match the [file name expression] rules. The detailed parameter description is as follows:
Field
Description
Directory Prefix
Directory structure prefix of the log file. Only the wildcards * and ? are supported.
* matches multiple arbitrary characters.
? matches a single arbitrary character.
Commas (,) are not supported.
**
Indicates the current directory and all subdirectories.
File Name
Log file name. Only the wildcards * and ? are supported.
* matches multiple arbitrary characters.
? matches a single arbitrary character.
Commas (,) are not supported.
Common configuration patterns are as follows:
/[Common directory prefix]/**/[Common file name prefix]*
/[Common directory prefix]/**/*[Common file name suffix]
/[Common directory prefix]/**/[Common file name prefix]*[Common file name suffix]
/[Common directory prefix]/**/*[Common string]
Example configurations:
No
Directory Prefix Expression
File Name Expression
Description
1
/var/log/nginx
access.log
In this example, the log path is configured as /var/log/NGINX/**/access.log. LogListener will listen to all log files named access.log in the subdirectories under the /var/log/NGINX prefix path.
2
/var/log/nginx
*.log
In this example, the log path is configured as /var/log/NGINX/**/*.log. LogListener will listen to all log files ending with .log in the subdirectories under the /var/log/NGINX prefix path.
3
/var/log/nginx
error*
In this example, the log path is configured as /var/log/NGINX/**/error*. LogListener will listen to all log files starting with error in the subdirectories under the /var/log/NGINX prefix path.
Note:
Windows environments do not support soft link collection.
Only LogListener 2.3.9 and later versions support adding multiple collection paths.
In log rotation scenarios, it is recommended to configure the collection path as log/*.log and rename the old file after log rotation as log/*.log.xxxx.
By default, a log file can only be collected by one log topic. If you need multiple collection configurations for a file and the file resides in a Linux environment, add a soft link to the source file and add it to another set of collection configurations.
Configuring Collection Path Blocklist
After the collection path blocklist is enabled, the specified directory prefixes or complete file paths can be ignored during collection. Directory and file name can be matched exactly or using wildcard patterns.
The collection blocklist supports two filter types, which can be used simultaneously:
File name: In the collection path, the complete file path for the collection needs to be ignored. The wildcard * or ? is supported, and ** path fuzzy matching is supported.
Directory: In the collection path, the directory prefix for the collection needs to be ignored. The wildcard * or ? is supported, and ** path fuzzy matching is supported.
Note:
LogListener 2.3.9 or later is required.
The collection blocklist excludes paths under the collection path. Therefore, in both file name mode and directory mode, the specified path should be a subset of the collection path.
Configuring the Collection Policy
All Collection: When LogListener collects a file, it reads from the beginning of the file.
New Collection: When LogListener collects a file, it collects only the newly added content in the file.
Configuring Backtracking Collection
When Collection Policy is set to New collection, you can further set the starting point for backtracking collection herein, and specify whether to start collecting from the position offset by the specified number of bytes from the latest position when LogListener starts.
Note:
Windows environments currently do not support custom metadata.
Configuring the Encoding Mode
UTF-8: Select this option if your log file is encoded in UTF-8 mode.
GBK: Select this option if your log file is encoded in GBK mode.
Configuring Single-Line Full-Text Format
On the "Collection Configuration" page, set the "Extraction Pattern" to Single line-full-text log. As shown below:
Configuring Custom Metadata
You can configure custom metadata to distinguish logs. The following metadata configurations are supported. For details, see Custom Metadata.
Machine group metadata: Use the machine group metadata.
Collection Path: Extract values from the collection path using regular expressions and set them as metadata.
Custom: Define custom key-value pairs as metadata.
Note:
LogListener 2.8.7 or later is required.
Configuring Filter Conditions
The purpose of the filter is to add log collection filter rules based on business needs, helping you filter out valuable log data.
In single-line full-text mode, __CONTENT__ is used by default as the key name for the full content, and the following filter rules are supported:
Equal to: Collects only logs where the specified field value matches the specified string. Both exact match and regex match are supported.
Not equal to: collects only logs whose specified field values do not match the specified characters. Both exact match and regular expression match are supported.
For example, given a single-line full-text log sample in the format Tue Jan 22 12:08:15 CST 2019 Installed: libjpeg-turbo-static-1.2.90-6.el7.x86_64, if you want to collect all logs from January 22, enter __CONTENT__ in the Key field, select Equal to as the Filter Rule, and enter Tue Jan 22.* in the Value field.
Note:
Windows environments currently do not support custom metadata.
The filtering rules "Not equal to", "Field exists", and "Field does not exist" are only supported in LogListener 2.9.3 and later versions.
Multiple filtering conditions are in an AND relationship. If multiple filtering conditions are configured for the same key name, the rule will be overwritten.
Advanced Configurations
Select the advanced configuration options you need by checking the corresponding items:
In single-line full-text mode, the following advanced configurations are supported:
Name
Description
Configuration Item
Timeout property
This configuration controls the timeout period for the log file. If a log file has no updates within the specified time, it is timed out. LogListener will no longer collect the timed-out log file. When you have a large number of log files, recommend reducing timeout to avoid waste of LogListener performance.
No time out: Log files never time out.
Custom: The timeout for log files can be customized.
Maximum directory levels
The configuration controls the maximum directory depth for log collection. LogListener does not collect log files in directories that exceed the specified maximum directory depth. If your target collection path includes fuzzy matching, it is recommended to configure an appropriate maximum directory depth to avoid LogListener performance waste.
An integer greater than 0. 0 means no drilling down into subdirectories.
Note:
Windows environments currently do not support custom metadata.
Data Processing
If you need to further process the collected CLS logs, such as structuring, masking, or filtering, before writing them into the log topic, you can click Data Processing at the bottom of the Collection Configuration page, add data processing, and then configure the index.
Note:
For operations related to data processing, see the [preprocessing of data] TAB page in the create processing task document.
Data processing will incur fees. For details, see Billing Overview.
Step 4: Configuring Indexes
1. Click Next to go to the index configuration page.
2. On the Index Configuration webpage, configure the following information. For configuration details, please see index configuration.
Note:
Index configuration must be enabled before you can perform searches.
3. Click Submit, go to the edit index configuration confirmation page, and click Confirm.
If you have set the index configuration to take effect only for newly written logs, click Confirm. If you want this configuration to take effect for historical data, after clicking Confirm, for details, see Rebuilds indexes for further settings.
4. Operation succeeded. The collection configuration is complete.
