CSV Shipping

Last updated: 2020-10-27 11:31:52

    Overview

    You can log in to the Cloud Log Service Console and ship Comma Separated Values (CSV)-formatted data to Cloud Object Storage (COS). This document describes how to create a CSV-formatted log shipping task.

    Prerequisites

    1. You have activated Cloud Log Service (CLS), created a logset and a log topic, and successfully collected the log data.
    2. You have activated COS service and created a bucket in the target region for log topic shipping. For more information, see Creating Buckets.
    3. You have ensured that the current account has permission to configure shipping tasks.

    Directions

    1. Log in to the CLS Console.
    2. Click Logset in the left sidebar.
    3. On the Logset Management page, click the target logset ID/name to go to its details page.
    4. Locate the log topic to be shipped, click Manage, and go to Ship to COS.
    5. Click Add Shipping Task to open the Ship to COS page and complete the basic configurations.

      The parameters are described as follows:
    Parameter Description Limit Required
    Shipping Task Name Configures the name of a shipping task. The name can contain letters, numbers, underscores (_), and hyphens (-). Yes
    COS Bucket Target bucket for shipping. The target bucket must be in the same region as the current log topic. A value selected from the drop-down list. Yes
    Directory Prefix Prefix of the COS bucket directory to which log files are shipped. You can define a directory prefix. By default, log files are directly stored in the bucket, and the file path is {COS bucket}{directory prefix}{partition format}_{random}_{index}.{type}, where {random}_{index} is a random number. Cannot start with / No
    Partition Format Directory automatically generated according to the creation time of the shipping task based on the strftime syntax. The forward slash (/) represents a level of COS directory.. The value must be in strftime format. Yes
    File Size Maximum size of an uncompressed file to be shipped during a shipping interval. A log file larger than this size will be split into multiple files. The value can be from 100 MB to 256 MB. The value must be a number ranging from 100 to 256 in MB. Yes
    Shipping Interval Specifies the time interval of shipping, which can be from 300s - 900s. If you set it to 5 minutes, a log file is generated from your log data every 5 minutes, and multiple log files will be shipped together to your bucket at a regular interval (within half an hour). 300s - 900s Yes

    Enter partition formats based on the requirements of the strftime format. Different partition formats may affect the paths of files shipped to COS. The following example describes how to use partition formats. For example, if a file is shipped to the bucket_test bucket, the directory prefix is logset/, and the shipping time is 2018/7/31 17:14, the corresponding shipping file path is as follows:

    Bucket Name Directory Prefix Partition Format COS File Path
    bucket_test logset/ %Y/%m/%d bucket_test:logset/2018/7/31_{random}_{index}
    bucket_test logset/ %Y%m%d/%H bucket_test:logset/20180731/14_{random}_{index}
    bucket_test logset/ %Y%m%d/log bucket_test:logset/20180731/log_{random}_{index}
    1. Click Next to go to the advanced configuration page. Set Shipping Format to CSV and fill in the configuration information.
      img
      The parameters are described as follows:
    Parameter Description Limit Required
    Key Specifies the `Key` field of the written CSV file. (The value must be the key name or reserved field after logs are structured. Otherwise, the key is invalid.) The value can contain letters, numbers, underscores (_), and hyphens (-). Yes
    Separator Separators between the fields in the CSV file. A value selected from the drop-down list. Yes
    Escape Character If a value field contains the selected separator characters, the separator characters will be enclosed by escape characters to prevent incorrect identification during data reading. A value selected from the drop-down list. Yes
    Invalid Field Filling If the configured value of the `Key` field is invalid, this field will be set. A value selected from the drop-down list. Yes
    Key in First Line Adds field name description to the first line of the CSV file. That is, the key value is written into the first line of the CSV file. This is disabled by default. Enabled/Disabled Yes
    Compressed Delivery You can determine whether to compress log files before shipping. The size of an uncompressed file to be shipped is limited to 10 GB. Log files can be compressed using gzip, lzop or snappy. Enabled/Disabled Yes

    Was this page helpful?

    Was this page helpful?

    • Not at all
    • Not very helpful
    • Somewhat helpful
    • Very helpful
    • Extremely helpful
    Send Feedback
    Help