Storing Tencent Cloud Product Logs in COS

Last updated: 2020-03-25 18:22:16



Tencent Cloud products will generate a large number of logs during operations, which record your business situation for future analysis to facilitate your business development and decision-making. You can leverage the storage capabilities of COS to persistently store Tencent Cloud product logs. Specifically, you can use methods such as APIs, SDKs, and tools to easily and quickly get logs from COS and analyze them.

Storing Tencent Cloud product logs in COS helps meet your following needs:

  • Persistent storage: COS provides stable and persistent storage services. You can persistently store your logs in COS at very low costs. When log-based analysis or decision-making is required for your business, you can get the logs for any time period from COS anywhere, anytime.
  • Data retrieval: COS supports SELECT statements that provide simple retrieval and extraction features for logs stored in COS. You can retrieve desired information only by entering log fields, which can reduce the data download traffic.
  • Data analysis: you can select one or multiple log files stored in COS, use Sparkling to analyze them, and make decisions based on the analysis results.

Log Shipping Methods

You can use either of the following two methods to store Tencent Cloud product logs to COS:

  • Using the log shipping feature built in Tencent Cloud products; for example, for products such as COS, CLB, and CA, you can directly ship their logs to COS.
  • Using the shipping feature of CLS: Tencent Cloud product logs that are shipped to CLS can be further shipped to COS through CLS for persistent storage.

Currently, the support of Tencent Cloud products for the two methods is as follows:

Tencent Cloud Product Name Directly Shipped to COS Shipped to CLS
CloudAudit (CA) Yes No
Cloud Load Balancer (CLB) Yes Yes
Cloud Kafka (CKafka) Yes No
API Gateway No Yes
Serverless Cloud Function (SCF) No Yes
Tencent Kubernetes Engine (TKE) No Yes
Live Video Broadcasting (LVB) No Yes
Tencent Cloud Base (TCB) No Yes; however, shipping from CLS to COS is not supported
Cloud Object Storage (COS) Yes No

Directly shipping logs to COS

The following Tencent Cloud products support directly shipping logs to COS. You can configure log shipping rules as instructed in the product documentation to ship logs to COS.

Tencent Cloud Product Name Log Shipping Document Log Shipping Interval Log Shipping Path
CA Click here 10–15 minutes cloudaudit/customprefix/timestamp
CLB Click here 60 minutes lb-id/timestamp
CKafka Click here 5–60 minutes
, which can be specified
instance id/topic id/timestamp
COS Click here 5 minutes The path prefix can be specified. You are recommended to set a meaningful path, such as cos_bucketname_access_log/timestamp

Message data generated in CKafka can be shipped. If you want to get logs of operations such as CKafka instance creation, you can choose to ship CA logs.

Shipping logs to COS through CLS

Certain Tencent Cloud products support log shipping to CLS for log retrieval and analysis. CLS can also ship logs to COS for persistent storage. For such products, you can enable log shipping to COS in the CLS Console to persistently store your log data. This reduces your storage costs and facilitates further offline analysis. Currently, the following products support log shipping to CLS:

Tencent Cloud Product Name Log Shipping Document
API Gateway Click here
TKE Click here

Logs in CLS can be shipped to COS by using any of the following three formats:

  • Delimited format: you can ship log data in delimited format to COS. For more information, please see CSV Shipping.
  • JSON format: you can ship log data in JSON format to COS. For more information, please see Shipping in JSON Format.
  • Source format: you can ship log data in the source format to COS. Single-line and multi-line full text shipping is supported, and some source text can be shipped in CSV format. For more information, please see Shipping in Source Format.

You need to perform the following steps to ship logs to COS through CLS:

  1. Select the product based on your business needs. Then, configure the logset and log topic as instructed in the product log shipping documentation provided above to import the log data generated by your business to CLS.
  2. Select an appropriate format to ship data to COS based on your business needs. When shipping logs, you are recommended to enter the product name as the path prefix in order to differentiate between logs of different products. For example, you can name TKE logs as TKE_tkeid_log/timestamp.
  3. After configuring a shipping rule, you can configure an event notification for file upload in SCF, so that you can perform further operations based on the event notification after logs are shipped to COS. For more information, please see Event Notifications.

Log Analysis

Downloading logs to local file system for offline analysis

You can download logs to your local file system in various methods such as console, SDKs, APIs, or tools. Documentation for all download methods is provided below for your reference. You can replace the file path in the code with your actual log storage path to download logs.

Download Method Description
Console Click here
COSBrowser Click here
COSCMD Click here
SDK for Android Click here
SDK for C Click here
SDK for C++ Click here
SDK for .NET Click here
SDK for Go Click here
SDK for iOS Click here
SDK for Java Click here
SDK for JavaScript Click here
SDK for Node.js Click here
SDK for PHP Click here
SDK for Python Click here
SDK for WeChat Mini Program Click here
API Click here

Analyzing logs with COS Select

You can use the COS Select feature to directly retrieve and analyze log files in CSV or JSON format stored in COS. With COS Select, you can filter desired log fields, which greatly reduces the data volume of logs transferred by COS and thus reduces your use costs and improves the data acquisition efficiency. For more information on COS Select, please see Select Overview.

Currently, you can use COS Select through the console or API.

Method Description
Console Click here
API Click here