Storing Tencent Cloud Product Logs in COS
Last updated: 2020-03-25 18:22:16PDF
Tencent Cloud products will generate a large number of logs during operations, which record your business situation for future analysis to facilitate your business development and decision-making. You can leverage the storage capabilities of COS to persistently store Tencent Cloud product logs. Specifically, you can use methods such as APIs, SDKs, and tools to easily and quickly get logs from COS and analyze them.
Storing Tencent Cloud product logs in COS helps meet your following needs:
- Persistent storage: COS provides stable and persistent storage services. You can persistently store your logs in COS at very low costs. When log-based analysis or decision-making is required for your business, you can get the logs for any time period from COS anywhere, anytime.
- Data retrieval: COS supports SELECT statements that provide simple retrieval and extraction features for logs stored in COS. You can retrieve desired information only by entering log fields, which can reduce the data download traffic.
- Data analysis: you can select one or multiple log files stored in COS, use Sparkling to analyze them, and make decisions based on the analysis results.
Log Shipping Methods
You can use either of the following two methods to store Tencent Cloud product logs to COS:
- Using the log shipping feature built in Tencent Cloud products; for example, for products such as COS, CLB, and CA, you can directly ship their logs to COS.
- Using the shipping feature of CLS: Tencent Cloud product logs that are shipped to CLS can be further shipped to COS through CLS for persistent storage.
Currently, the support of Tencent Cloud products for the two methods is as follows:
|Tencent Cloud Product Name||Directly Shipped to COS||Shipped to CLS|
|Cloud Load Balancer (CLB)||Yes||Yes|
|Cloud Kafka (CKafka)||Yes||No|
|Serverless Cloud Function (SCF)||No||Yes|
|Tencent Kubernetes Engine (TKE)||No||Yes|
|Live Video Broadcasting (LVB)||No||Yes|
|Tencent Cloud Base (TCB)||No||Yes; however, shipping from CLS to COS is not supported|
|Cloud Object Storage (COS)||Yes||No|
Directly shipping logs to COS
The following Tencent Cloud products support directly shipping logs to COS. You can configure log shipping rules as instructed in the product documentation to ship logs to COS.
|Tencent Cloud Product Name||Log Shipping Document||Log Shipping Interval||Log Shipping Path|
|CA||Click here||10–15 minutes||cloudaudit/customprefix/timestamp|
|CLB||Click here||60 minutes||lb-id/timestamp|
|CKafka||Click here||5–60 minutes
, which can be specified
|instance id/topic id/timestamp|
|COS||Click here||5 minutes||The path prefix can be specified. You are recommended to set a meaningful path, such as cos_bucketname_access_log/timestamp|
Message data generated in CKafka can be shipped. If you want to get logs of operations such as CKafka instance creation, you can choose to ship CA logs.
Shipping logs to COS through CLS
Certain Tencent Cloud products support log shipping to CLS for log retrieval and analysis. CLS can also ship logs to COS for persistent storage. For such products, you can enable log shipping to COS in the CLS Console to persistently store your log data. This reduces your storage costs and facilitates further offline analysis. Currently, the following products support log shipping to CLS:
|Tencent Cloud Product Name||Log Shipping Document|
|API Gateway||Click here|
Logs in CLS can be shipped to COS by using any of the following three formats:
- Delimited format: you can ship log data in delimited format to COS. For more information, please see CSV Shipping.
- JSON format: you can ship log data in JSON format to COS. For more information, please see Shipping in JSON Format.
- Source format: you can ship log data in the source format to COS. Single-line and multi-line full text shipping is supported, and some source text can be shipped in CSV format. For more information, please see Shipping in Source Format.
You need to perform the following steps to ship logs to COS through CLS:
- Select the product based on your business needs. Then, configure the logset and log topic as instructed in the product log shipping documentation provided above to import the log data generated by your business to CLS.
- Select an appropriate format to ship data to COS based on your business needs. When shipping logs, you are recommended to enter the product name as the path prefix in order to differentiate between logs of different products. For example, you can name TKE logs as
- After configuring a shipping rule, you can configure an event notification for file upload in SCF, so that you can perform further operations based on the event notification after logs are shipped to COS. For more information, please see Event Notifications.
Downloading logs to local file system for offline analysis
You can download logs to your local file system in various methods such as console, SDKs, APIs, or tools. Documentation for all download methods is provided below for your reference. You can replace the file path in the code with your actual log storage path to download logs.
|SDK for Android||Click here|
|SDK for C||Click here|
|SDK for C++||Click here|
|SDK for .NET||Click here|
|SDK for Go||Click here|
|SDK for iOS||Click here|
|SDK for Java||Click here|
|SDK for Node.js||Click here|
|SDK for PHP||Click here|
|SDK for Python||Click here|
|SDK for WeChat Mini Program||Click here|
Analyzing logs with COS Select
You can use the COS Select feature to directly retrieve and analyze log files in CSV or JSON format stored in COS. With COS Select, you can filter desired log fields, which greatly reduces the data volume of logs transferred by COS and thus reduces your use costs and improves the data acquisition efficiency. For more information on COS Select, please see Select Overview.
Currently, you can use COS Select through the console or API.