The single-line - full regular expression mode is a log parsing mode where multiple key-value pairs can be extracted from each log in a log text file in which each line is a raw log based on a regular expression. If you don't need to extract key-value pairs, please configure it as instructed in Collecting Logs with Full Text in a Single Line.
When configuring the single-line - full regular expression mode, you need to enter a sample log first and then customize your regular expression. After the configuration is completed, the system will extract the corresponding key-value pairs according to the capture group in the regular expression.
This document describes how to collect logs in single-line - full regular expression mode.
Suppose your raw log data is:
10.135.46.111 - - [22/Jan/2019:19:19:30 +0800] "GET /my/course/1 HTTP/1.1" 127.0.0.1 200 782 9703 "http://127.0.0.1/course/explore?filter%5Btype%5D=all&filter%5Bprice%5D=all&filter%5BcurrentLevelId%5D=all&orderBy=studentNum" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0" 0.354 0.354
The configured custom regular expression is:
After the system extracts the corresponding key-value pairs according to the
() capture group, you can customize the key name of each group as shown below:
body_bytes_sent: 9703 http_host: 127.0.0.1 http_protocol: HTTP/1.1 http_referer: http://127.0.0.1/course/explore?filter%5Btype%5D=all&filter%5Bprice%5D=all&filter%5BcurrentLevelId%5D=all&orderBy=studentNum http_user_agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0 remote_addr: 10.135.46.111 request_length: 782 request_method: GET request_time: 0.354 request_url: /my/course/1 status: 200 time_local: [22/Jan/2019:19:19:30 +0800] upstream_response_time: 0.354
On the Collection Configuration page, enter the Collection Path according to the log collection path format as shown below:
Log collection path format:
[directory prefix expression]/**/[filename expression].
After the log collection path is entered, LogListener will match all common prefix paths that meet the [directory prefix expression] rule and listen on all log files in the directories (including subdirectories) that meet the [filename expression] rule. The parameters are as detailed below:
|Directory prefix||Directory structure of the log file prefix. Only wildcards
|/**/||It represents the current directory and all subdirectories|
|Filename||Log file name. Only wildcards
Common configuration modes are as follows:
Below is an example:
|No.||Directory Prefix Expression||Filename Expression||Description|
|1.||/var/log/nginx||access.log||In this example, the log path is configured as
|2.||/var/log/nginx||*.log||In this example, the log path is configured as
|3.||/var/log/nginx||error*||In this example, the log path is configured as
- Only LogListener 2.3.9 or above allows adding multiple collection paths.
- By default, a log file can only be collected by one log topic. If you want to have multiple collection configurations for the same file, please add a soft link to the source file and add it to another collection configuration.
On the Collection Configuration page, set the Extraction Mode to Single-line - Full regular expression and enter a sample log in the Log Sample text box as shown below:
Define a regular expression according to the following rules.
The system offers two ways to define a regular expression: manual mode and auto mode. You can manually enter the expression to extract key-value pairs for verification or click Auto-Generate Regular Expression to switch to auto mode. The system will extract key-value pairs to verify the regular expression according to the mode you selected and the regular expression you defined.
No matter whether in auto mode or manual mode, the extraction result will be displayed in the Extraction Result after the regular mode is defined and verified successfully. You only need to define the key name of each key-value pair for use in log search and analysis.
Below are examples of how to enter a time parsing format:
10/Dec/2017:08:00:00.000, then the parsing format is
2017-12-10 08:00:00.000, then the parsing format is
12/10/2017, 08:00:00.000, then the parsing format is
You can set "millisecond" as the log time unit. If you enter a log time with incorrect format, the collection time will be used as the log time.
Filters are designed to help you extract valuable log data by adding log collection filter rules based on your business needs. If the filter rule is a Perl regular expression, the created filter rule will be a hit rule; in other words, only logs that match the regular expression will be collected and reported.
To collect logs in full regular expression mode, you need to configure a filter rule according to the defined custom key-value pair. For example, if you want to collect all log data with a
status field whose value is 400 or 500 after the sample log is parsed in full regular expression mode, you need to configure
status and the filter rule as
The relationship between multiple filter rules is logic "AND". If multiple filter rules are configured for the same key name, previous rules will be overwritten.
Index configuration must be enabled first before you can perform searches.