Converting Logs to Metrics

フォーカスモード

フォントサイズ

最終更新日: 2026-02-12 15:13:08

Original logs stored in the log topic can be analyzed using SQL to get metrics. For example, use the following SQL to get the number of error requests, where errorRequestCount is a metric:
status:>=400 | select count(*) as errorRequestCount
This way of obtaining metrics is highly flexible. You can readily get new metrics or change metric calculation rules by adjusting search conditions and SQL statements. However, it depends on the complete storage of original logs in the log topic Standard Storage. If you need to query the trend of metrics in the last 60 days, the original logs should be stored for at least 60 days, which brings a large data storage cost.
When metrics are relatively fixed, you can convert original logs into metrics for storage to reduce the log retention days (or you can use log settlement and shipping logs to Cloud Object Storage (COS) to store historical logs at low cost), thereby lowering overall costs. Besides, since the number of data entries for converted metrics is generally much less than the number of original log entries, and metric topics are optimized for time-series data queries, querying converted metrics typically delivers better performance, enabling faster metric queries and dashboard drawing.
Conversion Logic
Cloud Log Service (CLS) converts original logs into metrics through scheduled SQL analysis. For example, if there are 10 log entries in a minute, CLS converts these 10 entries into metrics through scheduled SQL analysis.
Original logs:
{"requestTime":"2020/12/30 15:35:46","method":"POST","handler":"/messages","timeCost":"45","status":"200"}
{"requestTime":"2020/12/30 15:35:46","method":"POST","handler":"/users","timeCost":"23","status":"200"}
{"requestTime":"2020/12/30 15:35:47","method":"POST","handler":"/messages","timeCost":"57","status":"200"}
{"requestTime":"2020/12/30 15:35:47","method":"POST","handler":"/users","timeCost":"25","status":"200"}
{"requestTime":"2020/12/30 15:35:47","method":"POST","handler":"/users","timeCost":"689","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/users","timeCost":"87","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/users","timeCost":"34","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/users","timeCost":"698","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/messages","timeCost":"25","status":"200"}
{"requestTime":"2020/12/30 15:35:48","method":"POST","handler":"/messages","timeCost":"78","status":"200"}
Scheduled SQL:
* | select count(*) as requests_total,method,handler group by method,handler order by requests_total desc limit 10000
Convert logs to the following metrics:
requests_total{method="POST", handler="/messages"} 4
requests_total{method="POST", handler="/users"} 6
A scheduled SQL task can run periodically, for example, once per minute. Each time, it converts the most recent original logs from the last minute, automatically converting them into minute-level metrics.
Prerequisites
Logs have been reported to the log topic, and standard storage is used. For low-frequency storage, SQL is not supported, so logs cannot be converted to metrics.
The log topic has configured index.
Operation Steps
Step 1: Adding Scheduled SQL Analysis
1. Log in to the CLS console.
2. In the left sidebar, choose Scheduled SQL analysis, and click Create to create a task.
3. In Basic Configuration, fill in the following information:
Configuration Item
Description
Source Log Topic
The input for the scheduled SQL task, namely, the log topic where the original logs to be converted into metrics are located.
Target topic
The output of the scheduled SQL task, select metric topic to store the converted metrics in the specified metric topic. If there is no metric topic currently, you can create a metric topic.
Query Statement
Use SQL statements to statistically analyze original logs to get metrics, for example:
* | select count(*) as requests_total,method,handler group by method,handler order by requests_total desc limit 10000
Metric Name
Select the field that needs to be the metric name from the SQL statement results. For example, in the example above, requests_total is the metric name, and its value is the metric value.
Metric Dimensions
Select the fields that need to be metric dimensions from the SQL statement results. For example, in the example above, method and handler are metric dimensions.
4. Click Next, and fill in the following information in Scheduling configuration:
Configuration Item
Description
Duration
The time range for querying logs. For example, query log data from 2023-01-01 00:00:00 to 2023-03-31 00:00:00. An empty end time means the task will continue to run.
It is recommended to set the end time to empty to continuously convert newly collected logs into metrics.
Scheduling cycle
Periodic query, meaning logs are queried once every X minutes. The value ranges from 1 to 1,440 minutes.
It is recommended to run it every 1 minute to convert logs into metrics more promptly.
Query Time Window
Time window for query statements.
It is recommended to use @m-1m ～ @m, the last minute. With the scheduling cycle of 1 minute, this is equivalent to converting the logs of that minute into metrics every minute.
SQL execution delayed
The delay time for query execution, set in Advanced Settings. The default value to 60 seconds. There is usually a delay in log index generation, and queries cannot be performed before the index is generated. Therefore, it is recommended to set a 60-second delay for querying, by which time the index is already generated (99.9% of the index data will be generated within 5 seconds).
5. Click OK to save the configuration.
Step 2: Querying Generated Metrics
1. In the left sidebar, choose Search and Analysis, select the region where the metric topic is located at the top, select Metric Topic as the subject type, and then select the target metric topic.
2. Enter the metric name in the input box to query metrics.

ヘルプとサポート

この記事はお役に立ちましたか？

営業担当者にお問い合わせいただくかチケットを提出してサポートを求めることができます。

フィードバック

Configuration Item	Description
Source Log Topic	The input for the scheduled SQL task, namely, the log topic where the original logs to be converted into metrics are located.
Target topic	The output of the scheduled SQL task, select metric topic to store the converted metrics in the specified metric topic. If there is no metric topic currently, you can create a metric topic.
Query Statement	Use SQL statements to statistically analyze original logs to get metrics, for example: `* \| select count(*) as requests_total,method,handler group by method,handler order by requests_total desc limit 10000`
Metric Name	Select the field that needs to be the metric name from the SQL statement results. For example, in the example above, `requests_total` is the metric name, and its value is the metric value.
Metric Dimensions	Select the fields that need to be metric dimensions from the SQL statement results. For example, in the example above, `method` and `handler` are metric dimensions.

Configuration Item	Description
Duration	The time range for querying logs. For example, query log data from 2023-01-01 00:00:00 to 2023-03-31 00:00:00. An empty end time means the task will continue to run. It is recommended to set the end time to empty to continuously convert newly collected logs into metrics.
Scheduling cycle	Periodic query, meaning logs are queried once every X minutes. The value ranges from 1 to 1,440 minutes. It is recommended to run it every 1 minute to convert logs into metrics more promptly.
Query Time Window	Time window for query statements. It is recommended to use @m-1m ～ @m, the last minute. With the scheduling cycle of 1 minute, this is equivalent to converting the logs of that minute into metrics every minute.
SQL execution delayed	The delay time for query execution, set in Advanced Settings. The default value to 60 seconds. There is usually a delay in log index generation, and queries cannot be performed before the index is generated. Therefore, it is recommended to set a 60-second delay for querying, by which time the index is already generated (99.9% of the index data will be generated within 5 seconds).

tencent cloud

Cloud Log Service

Converting Logs to Metrics

Conversion Logic

Prerequisites

Operation Steps

Step 1: Adding Scheduled SQL Analysis

Step 2: Querying Generated Metrics

ヘルプとサポート