tencent cloud

Feedback

JSON Shipping

Last updated: 2022-02-22 10:45:27

    Overview

    Through configuration in the CLS console, you can ship log data to COS in the JSON format. This document describes how to create a JSON log shipping task.

    Prerequisites

    1. You have activated CLS, created a logset and a log topic, and successfully collected log data.
    2. You have activated COS and created a bucket in the region of the log topic whose data is to be shipped. For detailed directions, see Creating Buckets.
    3. Your account has the permission to configure log shipping. For how to allow sub-accounts/collaborators to configure log shipping, see the “Shipping Authorization” document.

    Directions

    1. Log in to the CLS console.
    2. Click Log Topic in the left sidebar.
    3. Click the ID/name of the log topic whose data you want to ship.
    4. Click Ship to COS > Add Shipping Configuration, and fill in the information.

    The parameters are described as follows:

    Configuration Item Description Format Required
    Shipping Task Name Name of a shipping task Can contain letters, numbers, underscores (_), and hyphens (-) Yes
    COS Bucket Target bucket for shipping. The target bucket must be in the same region as the current log topic. Select from drop-down list Yes
    Directory Prefix Prefix of the directory under the specified COS bucket to which log files are shipped. You can customize a directory prefix. By default, log files are stored directly under the specified bucket as {COS bucket}{directory prefix}{partition format}_{random}_{index}.{type}, where {random}_{index} is a random number. Cannot start with / No
    Partition Format A string converted from the creation time of the shipping task using the strftime function. / indicates directory levels. Must be in strftime format Yes
    File Size Maximum size of an uncompressed file to be shipped during the specified shipping interval. A log file larger than this size will be split into multiple files. The value can be from 100 MB to 256 MB. Value range: 100-256 (MB) Yes
    Shipping Interval Time interval of shipping, which can be 300-900 seconds. If you set it to 5 minutes, a log file will be generated from your log data every 5 minutes, and multiple log files will be shipped to your bucket at a time (with an interval of less than 30 minutes). Value range: 300-900 (s) Yes
    Make sure the partition format entered is in the strftime format. Partition format affects the path to which log files are saved in COS. For example, if the target COS bucket is `bucket_test`, the directory prefix logset/, and the shipping time `2018/7/31 17:14`, the path to which log files are saved would vary with the partition format entered, as shown below:
    BucketDirectory PrefixPartition FormatCOS File Path
    bucket_test logset/ %Y/%m/%d bucket_test:logset/2018/7/31_{random}_{index}
    bucket_test logset/ %Y%m%d/%H bucket_test:logset/20180731/14_{random}_{index}
    bucket_test logset/ %Y%m%d/log bucket_test:logset/20180731/log_{random}_{index}
    1. Click Next to go to the Advanced Configuration page. Select json for Shipping Format and set other parameters.

    The parameters are described as follows:

    Configuration Item Description Format Required
    Compressed Shipping Whether to compress log files before shipping. The maximum size of an uncompressed file to be shipped is 10 GB. Log files can be compressed in GZip, Lzop, or Snappy format. Switch Yes
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support