tencent cloud


CSV Shipping

Last updated: 2022-02-24 17:04:37


    You can log in to the CLS console and ship Comma Separated Values (CSV)-formatted data to COS. This document describes how to create a CSV-formatted log shipping task.


    1. You have activated CLS, created a logset and a log topic, and successfully collected the log data.
    2. You have activated COS service and created a bucket in the target region for log topic shipping. For more information, see Creating Buckets.
    3. You have ensured that the current account has permission to configure shipping tasks.


    1. Log in to the CLS console.
    2. Click Log Topic in the left sidebar.
    3. Click the desired log topic ID/name to go to the log topic management page.
    4. Click Ship to COS, click Add Shipping Configuration, and then finish the configuration.

    The configuration items are described as follows:

    Configuration Item Description Limit Required
    Shipping Task Name Configures the name of a shipping task The name can contain letters, numbers, underscores (_), and hyphens (-) Yes
    COS Bucket Target bucket for shipping. The target bucket must be in the same region as the current log topic. A value selected from the drop-down list Yes
    Directory Prefix Prefix of the COS bucket directory to which log files are shipped. You can define a directory prefix. By default, log files are directly stored in the bucket, and the file path is {COS bucket}{directory prefix}{partition format}_{random}_{index}.{type}, where {random}_{index} is a random number. Cannot start with / No
    Partition Format The directory will be automatically generated according to the creation time of the shipping task based on the strftime syntax. The forward slash (/) represents a level of COS directory. The value must be in strftime format Yes
    File Size Maximum size of an uncompressed file to be shipped during a shipping interval. A log file larger than this size will be split into multiple files. The value can be from 100 MB to 256 MB. Value range: 100-256 MB Yes
    Shipping Interval Specifies the time interval of shipping, which can be from 300s - 900s. If you set it to 5 minutes, a log file is generated from your log data every 5 minutes, and multiple log files will be shipped together to your bucket at a regular interval (within half an hour). Value range: 300-900s Yes
    Enter partition formats based on the requirements of the [strftime format](http://man7.org/linux/man-pages/man3/strptime.3.html). Different partition formats may affect the paths of files shipped to COS. The following example describes how to use partition formats. For example, if a file is shipped to the `bucket_test` bucket, the directory prefix is `logset/`, and the shipping time is `2018/7/31 17:14`, the corresponding shipping file path is as follows:
    BucketDirectory PrefixPartition FormatCOS File Path
    bucket_test logset/ %Y/%m/%d bucket_test:logset/2018/7/31_{random}_{index}
    bucket_test logset/ %Y%m%d/%H bucket_test:logset/20180731/14_{random}_{index}
    bucket_test logset/ %Y%m%d/log bucket_test:logset/20180731/log_{random}_{index}
    5. Click **Next** to go to the advanced configuration page. Set **Shipping Format** to **CSV** and enter the relevant parameters in order. ![img](https://main.qcloudimg.com/raw/b9b06dedce54cbd960fb4b6b651bdcaf.png) **The configuration items are described as follows:**
    Configuration Item Description Limit Required
    Key Specifies the key name of the written CSV file. The value must be the key name or reserved parameter after logs are structured. Otherwise, the value is invalid. The value can contain letters, numbers, underscores (_), and hyphens (-) Yes
    Separator Separators between the fields in the CSV file A value selected from the drop-down list Yes
    Escape Character If a value field contains the selected separator characters, the separator characters will be enclosed by escape characters to prevent incorrect identification during data reading. A value selected from the drop-down list Yes
    Invalid Field Filling This field will be used if the configured key name is invalid. A value selected from the drop-down list Yes
    Key in First Line Adds field name description to the first line of the CSV file. That is, the key value is written into the first line of the CSV file. This is disabled by default. Enabled/Disabled Yes
    Compressed Delivery You can determine whether to compress log files before shipping. The size of an uncompressed file to be shipped is limited to 10 GB. Log files can be compressed in GZip, Lzop or Snappy format. Enabled/Disabled Yes
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support