tencent cloud

Overview

Through configuration in the CLS console, you can ship log data to COS in the JSON format. This document describes how to create a JSON log shipping task.

Prerequisites

1. You have activated CLS, created a logset and a log topic, and successfully collected log data.
2. You have activated COS and created a bucket in the region of the log topic whose data is to be shipped. For detailed directions, see Creating Buckets.
3. Your account has the permission to configure log shipping. For how to allow sub-accounts/collaborators to configure log shipping, see the “Shipping Authorization” document.

Directions

1. Log in to the CLS console.
2. Click Log Topic in the left sidebar.
3. Click the ID/name of the log topic whose data you want to ship.
4. Click Ship to COS > Add Shipping Configuration, and fill in the information.

The parameters are described as follows:

Configuration Item	Description	Format	Required
Shipping Task Name	Name of a shipping task	Can contain letters, numbers, underscores (_), and hyphens (-)	Yes
COS Bucket	Target bucket for shipping. The target bucket must be in the same region as the current log topic.	Select from drop-down list	Yes
Directory Prefix	Prefix of the directory under the specified COS bucket to which log files are shipped. You can customize a directory prefix. By default, log files are stored directly under the specified bucket as {COS bucket}{directory prefix}{partition format}_{random}_{index}.{type}, where {random}_{index} is a random number.	Cannot start with /	No
Partition Format	A string converted from the creation time of the shipping task using the strftime function. / indicates directory levels.	Must be in strftime format	Yes
File Size	Maximum size of an uncompressed file to be shipped during the specified shipping interval. A log file larger than this size will be split into multiple files. The value can be from 100 MB to 256 MB.	Value range: 100-256 (MB)	Yes
Shipping Interval	Time interval of shipping, which can be 300-900 seconds. If you set it to 5 minutes, a log file will be generated from your log data every 5 minutes, and multiple log files will be shipped to your bucket at a time (with an interval of less than 30 minutes).	Value range: 300-900 (s)	Yes

Make sure the partition format entered is in the strftime format. Partition format affects the path to which log files are saved in COS. For example, if the target COS bucket is `bucket_test`, the directory prefix logset/, and the shipping time `2018/7/31 17:14`, the path to which log files are saved would vary with the partition format entered, as shown below:

Bucket	Directory Prefix	Partition Format	COS File Path
bucket_test	logset/	%Y/%m/%d	bucket_test:logset/2018/7/31_{random}_{index}
bucket_test	logset/	%Y%m%d/%H	bucket_test:logset/20180731/14_{random}_{index}
bucket_test	logset/	%Y%m%d/log	bucket_test:logset/20180731/log_{random}_{index}

5. Click Next to go to the Advanced Configuration page. Select json for Shipping Format and set other parameters.

The parameters are described as follows:

Configuration Item	Description	Format	Required
Compressed Shipping	Whether to compress log files before shipping. The maximum size of an uncompressed file to be shipped is 10 GB. Log files can be compressed in GZip, Lzop, or Snappy format.	Switch	Yes