You can log in to the Cloud Log Service Console and ship data of the separator format to Cloud Object Storage (COS). This document describes how to create a separator-based log shipping task.
The configuration items are described as follows:
| Configuration Item | Description | Rule | Required or Not |
|---|---|---|---|
| Shipping Task Name | Configures the name of a shipping task. | The name can contain letters, numbers, underscores (_), and hyphens (-). | Yes |
| COS Bucket | Target bucket for shipping. The target bucket must be in the same region as the current log topic. | A value selected from the drop-down list. | Yes |
| Directory Prefix | Prefix of the COS bucket directory to which log files are shipped. You can define a directory prefix. By default, log files are directly stored in the bucket, and the file path is {COS bucket}{directory prefix}{partition format}_{random}_{index}.{type}, where {random}_{index} is a random number. |
Not starting with a forward slash (/) | No |
| Partition Format | Directory automatically generated according to the creation time of the shipping task based on the strftime syntax. The slash (/) indicates a level-1 COS directory. | strftime format | Yes |
| File Size | Maximum size of an uncompressed file to be shipped during a shipping interval. A log file larger than this size will be split into multiple files. The value can be from 100 MB to 10,000 MB. | 100 MB to 10,000 MB | Yes |
| Shipping Interval | Specifies the time interval of shipping, which can be from 60 seconds to 3,600 seconds. If you set it to 5 minutes, a log file is generated from your log data every 5 minutes, and multiple log files will be shipped together to your bucket at a regular interval (within half an hour). | 60 seconds to 3,600 seconds | Yes |
Enter partition formats based on the requirements of the strftime format. Different partition formats may affect the paths of files shipped to COS. The following example describes how to use partition formats. For example, if a file is shipped to the bucket_test bucket, the directory prefix is logset/, and the shipping time is 2018/7/31 17:14, the corresponding shipping file path is as follows:
| Bucket Name | Directory Prefix | Partition Format | COS File Path |
|---|---|---|---|
| bucket_test | logset/ | %Y/%m/%d | bucket_test:logset/2018/7/31_{random}_{index} |
| bucket_test | logset/ | %Y%m%d/%H | bucket_test:logset/20180731/14_{random}_{index} |
| bucket_test | logset/ | %Y%m%d/log | bucket_test:logset/20180731/log_{random}_{index} |
| Configuration Item | Description | Rule | Required or Not |
|---|---|---|---|
| Key | Specifies the `Key` field of the written CSV file. (The value must be the key name or reserved field after logs are structured. Otherwise, the key is invalid.) | The value can contain letters, numbers, underscores (_), and hyphens (-). | Yes |
| Separator | Separators between the fields in the CSV file. | A value selected from the drop-down list. | Yes |
| Escape character | If a value field contains the selected separator characters, the separator characters will be enclosed by escape characters to prevent incorrect identification during data reading. | A value selected from the drop-down list. | Yes |
| Invalid Field Filling | If the configured value of the `Key` field is invalid, this field will be set. | A value selected from the drop-down list. | Yes |
| Key in First Line | Adds field name description to the first line of the CSV file. That is, the key value is written into the first line of the CSV file (disabled by default). | Enabled/Disabled | Yes |
| Compressed Delivery | You can determine whether to compress log files before shipping. The size of an uncompressed file to be shipped is limited to 10 GB. Log files can be compressed using gzip, lzop, or Snappy. | Enabled/Disabled | Yes |
(Optional) Advanced Options
You can enable Advanced Options to filter logs based on log content before shipping.
Up to 5 shipping filtering rules are allowed, among rules is “And” logic, i.e. log files are shipped only when all the rules are met.
a. Specify a key and set a filtering rule to perform RegEx extraction on the key.
b. Use "()" to capture objects that need to match the value and enter the value to match. The system performs a match according to the regular expression in the shipping rule, extracts the content of the capture group "()", and compares it with the value. If the captured content is equal to the value, the log data will be shipped.
Sample 1:
Specify a field as status. For example, the key-value pair is status:404. If you want to ship logs with a status field of 404, the filtering rule is (.*).
Sample 2:
Specify a field as http_host. For example, the key-value pair is http_host:172.16.19.20. If you want to ship logs with an http_host field starting with “172.16”, the filtering rule is ^(\d+\.\d+)\..*.
7. Click OK to enable shipping.
Was this page helpful?