The single-line - full regular expression mode is a log parsing mode where multiple key-value pairs can be extracted from each log in a log text file in which each line is a raw log based on a regular expression. If you don't need to extract key-value pairs, configure it as instructed in Full Text in a Single Line.
When configuring the single-line - full regular expression mode, you need to enter a sample log first and then customize your regular expression. After the configuration is completed, the system will extract the corresponding key-value pairs according to the capture group in the regular expression.
This document describes how to collect logs in single-line - full regular expression mode.
Suppose your raw log data is:
10.135.46.111 - - [22/Jan/2019:19:19:30 +0800] "GET /my/course/1 HTTP/1.1" 127.0.0.1 200 782 9703 "http://127.0.0.1/course/explore?filter%5Btype%5D=all&filter%5Bprice%5D=all&filter%5BcurrentLevelId%5D=all&orderBy=studentNum" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0" 0.354 0.354
The custom regular expression you configure is:
(\S+)[^\[]+(\[[^:]+:\d+:\d+:\d+\s\S+)\s"(\w+)\s(\S+)\s([^"]+)"\s(\S+)\s(\d+)\s(\d+)\s(\d+)\s"([^"]+)"\s"([^"]+)"\s+(\S+)\s(\S+).*
Then CLS extracts key-value pairs based on the ()
capture groups. You can specify the key name of each group as shown below:
body_bytes_sent: 9703
http_host: 127.0.0.1
http_protocol: HTTP/1.1
http_referer: http://127.0.0.1/course/explore?filter%5Btype%5D=all&filter%5Bprice%5D=all&filter%5BcurrentLevelId%5D=all&orderBy=studentNum
http_user_agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0
remote_addr: 10.135.46.111
request_length: 782
request_method: GET
request_time: 0.354
request_url: /my/course/1
status: 200
time_local: [22/Jan/2019:19:19:30 +0800]
upstream_response_time: 0.354
test-whole
as Log Topic Name and click Confirm.On the Collection Configuration page, enter the collection rule name and enter the Collection Path according to the log collection path format as shown below:
Log collection path format: [directory prefix expression]/**/[filename expression]
.
After the log collection path is entered, LogListener will match all common prefix paths that meet the [directory prefix expression] rule and listen for all log files in the directories (including subdirectories) that meet the [filename expression] rule. The parameters are as detailed below:
Field | Description |
---|---|
Directory Prefix | Directory prefix for log files, which supports only the wildcard characters \* and ? .
|
/**/ | Current directory and all its subdirectories. |
File Name | Log file name, which supports only the wildcard characters \* and ? .
|
Common configuration modes are as follows:
Below are examples:
No. | Directory Prefix Expression | Filename Expression | Description |
---|---|---|---|
1. | /var/log/nginx | access.log | In this example, the log path is configured as /var/log/nginx/**/access.log . LogListener will listen for log files named access.log in all subdirectories in the /var/log/nginx prefix path. |
2. | /var/log/nginx | *.log | In this example, the log path is configured as /var/log/nginx/**/*.log . LogListener will listen for log files suffixed with .log in all subdirectories in the /var/log/nginx prefix path. |
3. | /var/log/nginx | error* | In this example, the log path is configured as /var/log/nginx/**/error* . LogListener will listen for log files prefixed with error in all subdirectories in the /var/log/nginx prefix path. |
Note:
- Only LogListener 2.3.9 or later allows adding multiple collection paths.
- The system does not support uploading logs with contents in multiple text formats, which may cause write failures, such as
key:"{"substream":XXX}"
.- We recommend you configure the collection path as
log/*.log
and rename the old file after log rotationlog/*.log.xxxx
.- By default, a log file can only be collected by one log topic. If you want to have multiple collection configurations for the same file, add a soft link to the source file and add it to another collection configuration.
Manual mode:
Auto Mode (click Auto-Generate Regular Expression to switch):
Note:No matter whether in auto mode or manual mode, the extraction result will be displayed in the Extraction Result after the regular mode is defined and verified successfully. You only need to define the key name of each key-value pair for use in log search and analysis.
Log time is measured in milliseconds.
The time attribute of a log is defined as follows:
Collection time: It is the default time attribute of a log.
Original timestamp: Set Use Collection Time to and enter the time key of the original timestamp and the corresponding time parsing format.
For more information on the time format, see Configuring Time Format.
Collection time: The time attribute of a log is determined by the time when CLS collects the log.
Original timestamp: The time attribute of a log is determined by the timestamp in the raw log.
Below are examples of how to enter a time resolution format:
10/Dec/2017:08:00:00.000
is %d/%b/%Y:%H:%M:%S.%f
.2017-12-10 08:00:00.000
is %Y-%m-%d %H:%M:%S.%f
.12/10/2017, 08:00:00.000
is %m/%d/%Y, %H:%M:%S.%f
.Note:You can set "millisecond" as the log time unit. If you enter a log time with incorrect format, the collection time will be used as the log time.
Filters are designed to help you extract valuable log data by adding log collection filter rules based on your business needs. If the filter rule is a Perl regular expression, the created filter rule will be used for matching; in other words, only logs that match the regular expression will be collected and reported.
To collect logs in full regular expression mode, you need to configure a filter rule according to the defined custom key-value pair. For example, if you want to collect all log data with a status
field whose value is 400 or 500 after the sample log is parsed in full regular expression mode, you need to configure key
as status
and the filter rule as 400|500
.
Note:The relationship between multiple filter rules is logic "AND". If multiple filter rules are configured for the same key name, previous rules will be overwritten.
We recommend you enable Upload Parsing-Failed Logs. After it is enabled, LogListener will upload all types of parsing-failed logs. If it is disabled, such logs will be discarded.
After this feature is enabled, you need to configure the Key
value for parsing failures (which is LogParseFailure
by default). All parsing-failed logs are uploaded with the input content as the key name (Key
) and the raw log content as the key value (Value
).
Note:Index configuration must be enabled first before you can perform searches.
@&()='",;:<>[]{}/ \n\t\r
and can be modified as needed.For more information on log search, see Overview and Syntax Rules.
Was this page helpful?