When configuring notification content, custom webhook configuration, and multi-dimensional analysis in the alarm policy, you can use the alarm notification variables to customize the alarm notification content and multi-dimensional analysis statements. This ensures that the received alarm notifications clearly and accurately describe the causes of the alarms.
Alarm variables can also be delivered to a log topic in the form of logs. For usage instructions, see Manage Notification Groups.
Quick Start
When configuring alarm policies, include the following information in the additional notification content:
Detailed log:
{{.QueryLog[0][0]}}
After the configuration is completed, it will be displayed as shown in the figure below:
When an alarm notification is received, the additional notification content is automatically replaced with the value below, which represents the most recent detailed log that triggered the alarm.
Detailed log:
{"content":{"body_bytes_sent":"33352","http_referer":"-","http_user_agent":"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.17 Safari/537.36","remote_addr":"201.80.83.199","remote_user":"-","request_method":"GET","request_uri":"/content/themes/test-com/images/header_about.jpg","status":"404","time_local":"01/Nov/2018:01:16:31"},"fileName":"/root/testLog/nginx.log","pkg_id":"285A243662909DE3-70A","source":"172.17.0.2","time":1653831150008,"topicId":"a54de372-ffe0-49ae-a12e-c340bb2b03f2"}
Notification Variables
Variable List
Variable
Description
Example Variable Value
Shipped Log
Description
{{.RecordId}}
Alarm notification ID
76afb409-4648-xxxx-b67a-fe350129ceb4
No
Each alarm notification uses a different ID.
{{.RecordGroupId}}
Alarm group ID
52746a79-7681-495f-xxxx-2715593e625a
Yes
From alarm trigger to clearance, the ID remains the same regardless of how many notifications are sent.
When group-based trigger is enabled in an alarm policy, multiple alarms may be generated simultaneously, each corresponding to a different group and ID.
{{.UIN}}
Account ID
100007xxx827
Yes
-
{{.Nickname}}
Account nickname.
xx company
Yes
-
{{.Region}}
Region
Guangzhou
Note:
Logs are delivered in English, for example, ap-guangzhou.
Array structure. {{.QueryParams[0]}} represents the first execution statement, {{.QueryParams[1]}} represents the second execution statement, and so on.
{{.Condition}}
Trigger Conditions
$1.errorLogCount > 1
Yes
-
{{.ConditionGroup}}
Corresponding group information when the alarm is triggered by group.
{"$1.AppName":"userManageService"}
Yes
This variable is available only when group-based trigger is enabled in the alarm policy.
{{.Level}}
Alarm level
Info
Yes
Critical
Warn
Info
{{.Level_zh}}
Alarm level
Info
No
Emergency
Warning
Info
{{.HappenThreshold}}
Number of times that the alarm triggering condition is continuously met.
1
Yes
-
{{.AlertThreshold}}
Alarm interval.
15
Yes
Unit: min.
{{.Label}}
Additional classification field.
{"service":"payService"}
Yes
{{.StartTime}}
First alarm triggering time.
2022-05-28 18:56:37
Yes
Time zone: Asia/Shanghai
{{.StartTimeUnix}}
First alarm triggering timestamp.
1653735397099
Yes
Unix timestamp in milliseconds
{{.NotifyTime}}
Notification time of this alarm.
2022-05-28 19:41:37
Yes
Time zone: Asia/Shanghai
{{.NotifyTimeUnix}}
Notification timestamp for this alarm.
1653738097099
Yes
Unix timestamp in milliseconds
{{.NotifyType}}
Alarm notification type.
1
Yes
1: alarm notification
2: clearance notification
{{.ConsecutiveAlertNums}}
Number of consecutive alarms.
2
Yes
-
{{.Duration}}
Alarm duration.
0
Yes
Unit: min.
{{.DetailUrl}}
Alarm details page URL
https://alarm.cls.tencentcs.com/MDv2xxJh
Yes
No login required
{{.QueryUrl}}
Search and analysis URL for the first execution statement
https://alarm.cls.tencentcs.com/T0pkxxMA
No
To obtain URLs for other execution statements, use {{.QueryParams}}.
{{.SilentUrl}}
Alarm blocking URL
https://alarm.cls.tencentcs.com/K6hyxxyG
Yes
No login required. Available only when {{.CanSilent}} is true, for example:
{{if .CanSilent}}{{.SilentUrl}}{{end}}.
{{.CanSilent}}
Whether Direct Alarm Blocking via {{.SilentUrl}} Is Supported
true
Yes
This value is true when the Alarm Operation Without Login feature is enabled in the notification group; otherwise, it is false.
{{.Message}}
Additional notification content
-
Yes
Indicates the additional notification content specified in the alarm policy configuration. Maximum size: 10 KB. Content exceeding this limit will be automatically truncated.
{{.TriggerParams}}
Parameters upon alarm triggering.
$1.errorLogCount=5;
Yes
When multiple results of an execution statement meet trigger criteria simultaneously, only the first result is recorded.
This variable is available only during alarm trigger instead of alarm clearance.
This variable is available only during alarm trigger instead of alarm clearance.
{{.AnalysisResultFormat}}
Formatted multi-dimensional analysis results
-
No
The content is identical to {{.AnalysisResult}}, but the json is formatted into text, using line breaks, tables, and other forms for presentation to improve readability.
Whether the current alarm notification is silenced
All notifications are blocked: SilentAll.
Some notifications are blocked: SilentPartial.
Not blocked: NotSilent.
Yes
Only supports log delivery.
The relevant variables are described as follows:
{{.QueryResult}}
Meaning: execution results of execution statements (regardless of whether trigger criteria are met).
Description: array structure where {{.QueryResult[0]}} represents the execution result of the first execution statement, {{.QueryResult[1]}} represents the execution result of the second execution statement, and so on. Each statement can contain up to 1,000 entries with a maximum size of 10 KB. Up to 10 statements are supported, with a total maximum size of 100 KB. Content exceeding these limits will be automatically truncated.
Variable Value Example:
Assume the alarm policy has two execution statements:
First execution statement: status:>=400 | select count(*) as errorLogCount.
Second execution statement: status:>=400 | select count(*) as errorLogCount,request_uri group by request_uri order by count(*) desc.
Meaning: execution results of execution statements (trigger criteria met).
Description: array structure where {{.TriggerResult[0]}} represents the execution result of the first execution statement, {{.TriggerResult[1]}} represents the execution result of the second execution statement, and so on. Each statement can contain up to 1,000 entries with a maximum size of 10 KB. Up to 10 statements are supported, with a total maximum size of 100 KB. Content exceeding these limits will be automatically truncated.
Variable Value Example:
Based on the {{.QueryResult}} variable value example above, if the trigger condition is $1.errorLogCount > 1, the variable values are as follows:
[
[{
"errorLogCount":7
}],
[{
"errorLogCount":3,
"request_uri":"/apple-touch-icon-144x144.png"
}, {
"errorLogCount":3,
"request_uri":"/feed"
}]
]
{{.QueryLog}}
Description: Detailed logs matching the search conditions in the execution statement (excluding filter conditions in SQL).
Description: array structure where {{.QueryLog[0]}} represents the detailed log of the first execution statement, {{.QueryLog[1]}} represents the detailed log of the second execution statement, and so on. Each statement can contain up to 10 entries with a maximum size of 50 KB. Up to 10 statements are supported, with a total maximum size of 150 KB. Content exceeding these limits will be automatically truncated.
Example Variable Values:
[
[{
"content":{
"__TAG__":{
"pod":"nginxPod",
"cluster":"testCluster"
},
"body_bytes_sent":"32847",
"http_referer":"-",
"http_user_agent":"Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01",
"remote_addr":"105.86.148.186",
"remote_user":"-",
"request_method":"GET",
"request_uri":"/apple-touch-icon-144x144.png",
"status":"404",
"time_local":"01/Nov/2018:00:55:14"
},
"fileName":"/root/testLog/nginx.log",
"pkg_id":"285A243662909DE3-5CD",
"source":"172.17.0.2",
"time":1653739000013,
"topicId":"a54de372-ffe0-49ae-a12e-c340bb2b03f2"
}, {
"content":{
"__TAG__":{
"pod":"nginxPod",
"cluster":"testCluster"
},
"body_bytes_sent":"33496",
"http_referer":"-",
"http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36",
Description: An object structure where each top-level object corresponds to the result of a multi-dimensional analysis. The key of a top-level object is the name of the multi-dimensional analysis, and its value is the result of that analysis. This variable is available only when an alarm is triggered (that is, {{.NotifyType}}=1) and is not available when the alarm recovers (that is, {{.NotifyType}}=2). Each multi-dimensional analysis is limited to a maximum of 10 KB. Up to 5 multi-dimensional analyses are supported, with a total maximum size of 50 KB. Content exceeding these limits will be automatically truncated.
Related Raw Logs: up to 10 entries per statement, subject to user configuration.
Top 5 Field Values and Percentage Statistics: up to 5 entries per statement
Custom Search and Analysis (SQL): up to 1,000 entries per statement.
Custom Search and Analysis (search): up to 10 entries per statement.
Example Variable Values:
Depending on the multi-dimensional analysis types, variable value formats vary. Examples are as follows.
Here, Error Log URL Distribution is the name of the multi-dimensional analysis configured in the alarm policy.
Variable Syntax
The variable syntax is similar to the Go Template syntax, allowing for the extraction and formatting of alarm notification variables. This enables them to be presented more clearly in the alarm notification content. All variables and variable syntax are enclosed within {{ }}, and text outside these delimiters will not be processed.
Variable Extraction
Syntax Format
{{.variable[x]}} or {{index .variable x}}
{{.variable.childNodeName}} or {{index .variable "childNodeName"}}
Syntax Description
For array variables, use {{.variable[x]}} to extract the corresponding array element by its index, where x is an integer greater than or equal to 0. This syntax is equivalent to {{index .variable x}}.
For object variables, use {{.variable.childNodeKey}} to extract the corresponding child object value by its key. This syntax is equivalent to {{index .variable "childNodeName"}}.
Note:
When the child object name contains special characters such as spaces, $, -, or consists of pure numbers, use the {{index .variable "childNodeName"}} syntax. Examples include {{index .AnalysisResult "Top URL"}}, {{index .ConditionGroup "$1.AppName"}}, and {{index .variable "123"}}.
Do not exceed the array length when using array indices to extract elements. If necessary, you can use the len function with conditional judgment syntax to determine whether the array length is exceeded.
Usage Example
The value of the {{.QueryResult}} variable is:
When the variable is an array or an object containing multiple child objects, this syntax can be used to display each element/object in a specified format.
Note:
Inside the range syntax, elements/objects from within the .variable are obtained by default. You cannot directly use the {{.UIN}} format to obtain external variables. To obtain them, create an independent variable outside the range syntax in advance, for example:
Execute the corresponding expressions respectively according to the conditional judgment results. The conditional judgment supports logical operations using AND, OR, and NOT, allowing for value comparisons.
eq arg1 arg2: returns trueif arg1 == arg2.
ne arg1 arg2: returns trueif arg1 != arg2.
lt arg1 arg2: returns trueif arg1 < arg2.
le arg1 arg2: returns trueif arg1 <= arg2.
gt arg1 arg2: returns trueif arg1 > arg2.
ge arg1 arg2: returns trueif arg1 >= arg2.
Use Case
The variable values of {{.QueryResult}} are as follows:
To display the request_uri and its corresponding errorLogCount from the second array, where errorLogCount is greater than or equal to 2 and less than or equal to 100, use the following expression:
{{range .QueryResult[1]}}
{{if and (ge .errorLogCount 2)(le .errorLogCount 100)}}
The if statement can also be used to determine whether a field value exists. If the field value is an empty string or does not exist, it evaluates to false. For example:
{{if .QueryLog[0][0].apple}}
apple exist, value is :{{.QueryLog[0][0].apple}}
{{else}}
apple is not exist
{{end}}
Removing Whitespace Characters
Syntax Format
{{- xxx}} or {{xxx -}}
Syntax Description
The variable syntax preserves all whitespace characters (including spaces, indents, and line breaks) during execution. For example, loop iterations and conditional judgment may return results with excessive blank lines that clutter the output. In this case, you can use - at the beginning or end of {{ }} to remove leading or trailing whitespace, respectively.
Use Case
Building on the example in conditional judgment, the expression has been revised as follows:
{{- range .QueryResult[1]}}
{{- if and (ge .errorLogCount 2)(le .errorLogCount 100)}}
Escapes special characters such as " so that variables can be placed in JSON to maintain correct JSON formatting.
Variable value:
The variable value of {{.ExecuteQuery[0]}} is status:>=400 | select count(*) as "error log count".
Function example:
{{escape .ExecuteQuery[0]}}
Returned results:
status:>=400 | select count(*) as \\"error log count\\"
{{escape_markdown .variable}}
Escapes special characters such as * and # so that variables can be placed in Markdown to maintain correct Markdown formatting.
Applicable to the WeCom bot notification channel.
{{escape_markdown_html .variable}}
Escapes special characters such as * and # so that variables can be placed in Markdown to maintain correct Markdown formatting.
Applicable to the DingTalk and Feishu bot notification channels.
Extracting strings
{{substr .variable start}}
Extracts a string from the specified start position with an optional length. Both the start position and length are calculated in bytes. Generally, each letter or digit occupies 1 byte.
Variable value:
The variable value of {{.QueryLog[0][0].fileName}} is /root/testLog/nginx.log.
Function example:
{{substr .QueryLog[0][0].fileName 6 7 }}
Returned results:
testLog
{{substr .variable start length}}
{{extract .variable startStringRegex}}
Extracts a string based on the specified start string and optional end string. Both the start and end strings are specified using regular expressions.
Variable value:
The variable value of {{.QueryLog[0][0].fileName}} is /root/testLog/nginx.log.
Obtains the number of elements in arrays or maps, and the length of strings. Does not support retrieving the length of numeric values.
Variable value:
The variable value of {{.QueryResult}} is
[[{"number":19778,"text":"hello"}],[]]
Function example:
{{len.QueryResult}}
{{len.QueryResult[0]}}
{{len.QueryResult[0][0]}}
{{len.QueryResult[0][0].text}}
Returned results:
2
1
2
5
Mathematical Calculation Functions
Function
Note:
Example
Addition
{{add .variable1 .variable2 ...}}
Integer (int64) addition operation
Function example:
{{ add 5 3 }}
Returned results:
8
{{addf .variable1 .variable2 ...}}
Floating-point number (float64) addition operation
Subtraction
{{sub .variable1 .variable2}}
Integer (int64) subtraction operation
Function example:
{{ sub 10 3 }}
Returned results:
7
{{subf .variable1 .variable2}}
Floating-point number (float64) subtraction operation
Division
{{div .variable1 .variable2}}
Integer (int64) division operation
Function example:
{{ div 10 2 }}
Returned results:
5
{{divf .variable1 .variable2}}
Floating-point number (float64) division operation
Modulo
{{mod .variable1 .variable2}}
Integer (int64) modulo operation
Function example:
{{ mod 10 3 }}
Returned results:
1
Multiplication
{{mul .variable1 .variable2}}
Integer (int64) multiplication operation
Function example:
{{ mul 4 5 }}
Returned results:
20
{{mulf .variable1 .variable2}}
Floating-point number (float64) multiplication operation
Minimum value
{{min .variable1 .variable2 ...}}
Minimum value from integer (int64)
Function example:
{{ min 3 7 }}
Returned results:
3
{{minf .variable1 .variable2 ...}}
Minimum value from floating-point number (float64)
Maximum Value
{{max .variable1 .variable2 ...}}
Maximum value from integer (int64)
Function example:
{{ max 3 7 }}
Returned results:
7
{{maxf .variable1 .variable2 ...}}
Maximum value from floating-point number (float64)
Rounding
{{ceil .variable}}
Rounding up
Function example:
{{ ceil 5.3 }}
Returned results:
6
{{floor .variable}}
Rounding down
Rounding
{{round .variable decimal}}
Rounds the value variable to the specified decimal place decimal.
Function example:
{{ round 5.6748 2 }}
Returned results:
5.67
Type Conversion Functions
Function
Note:
Example
{{atoi .variable}}
Converts a string to an integer.
Function example:
{{ atoi "123"}}
{{float6412}}
{{int"42"}}
{{int641234}}
{{ toString 123}}
Returned results:
123
12
42
1234
123
{{float64 .variable}}
Converts the value to the float64 type.
{{int .variable}}
Converts the value to the int type matching the system width.
{{int64 .variable}}
Converts the value to the int64 type.
{{toString .variable}}
Converts the value to a string.
Time Functions
Function
Note:
Example
Converting a UNIX timestamp
{{fromUnixTime .variable}}
Converts UNIX timestamp (support millisecond and second timestamp) to a readable date and time. The time zone is optional, with the default set to Asia/Shanghai.
Variable value:
The variable value of {{.QueryLog[0][0].time}} is 1653893435008.
Converts the specified timestamp to a string in the specified format. The first parameter specifies the time format, based on 2006-01-02 15:04:05, which can be adjusted.
Encrypts the specified variable or string using a specific encryption algorithm. The default returned result is encoded as a hexadecimal string, which can be modified as needed to Base64 or Base64URL encoding.
Encrypts the specified variable or string using a specific encryption algorithm. The default returned result is encoded as a hexadecimal string, which can be modified as needed to Base64 or Base64URL encoding. Secretkey is the key for the HMAC encryption algorithm and can be modified as needed.
Case 1: Displaying the Most Recent Detailed Log in the Alarm Notification
Scenarios
Add the most recent detailed log that conforms to the execution statement search conditions to the alarm notification. Present it in key:value format, with each key on a separate line, excluding any CLS preset fields and metadata fields.
Notification Content Configuration
{{range $key,$value := .QueryLog[0][0].content}}
{{if not (containstr $key"__TAG__")}}
{{- $key}}:{{$value}}
{{- end}}
{{- end}}
Here, .QueryLog[0][0] represents the most recent detailed log that matches the search conditions of the first execution statement in the alarm policy. Its value is as follows:
{
"content":{
"__TAG__":{
"a":"b12fgfe",
"c":"fgerhcdhgj"
},
"body_bytes_sent":"33704",
"http_referer":"-",
"http_user_agent":"Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.3319.102 Safari/537.36",
"remote_addr":"247.0.249.191",
"remote_user":"-",
"request_method":"GET",
"request_uri":"/products/hadoop)",
"status":"404",
"time_local":"01/Nov/2018:07:54:08"
},
"fileName":"/root/testLog/nginx.log",
"pkg_id":"285A243662909DE3-210B",
"source":"172.17.0.2",
"time":1653908859008,
"topicId":"a54de372-ffe0-49ae-a12e-c340bb2b03f2"
}
Alarm Notification Content
remote_addr:247.0.249.191
time_local:01/Nov/2018:07:54:08
http_user_agent:Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.3319.102 Safari/537.36
remote_user:-
http_referer:-
body_bytes_sent:33704
request_method:GET
request_uri:/products/hadoop)
status:404
Case 2: Displaying Detailed Logs for Multiple Execution Statements in Alarm Notifications
Scenarios
When an alarm policy contains multiple execution statements, the detailed logs corresponding to each statement are added to the alarm notification.
Notification Content Configuration
{{- range .QueryLog}}
{{- range .}}
{{.content}}
{{- end}}
{{- end}}
Here, .QueryLog represents the detailed logs that match the search conditions of all execution statements in the alarm policy (up to a maximum of 10 logs). Its value is as follows:
{
"QueryLog":[
[{
"content":{
"__TAG__":{
"pod":"nginxPod",
"cluster":"testCluster"
},
"body_bytes_sent":"32847",
"http_referer":"-",
"http_user_agent":"Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01",
"remote_addr":"105.86.148.186",
"remote_user":"-",
"request_method":"GET",
"request_uri":"/apple-touch-icon-144x144.png",
"status":"404",
"time_local":"01/Nov/2018:00:55:14"
},
"fileName":"/root/testLog/nginx.log",
"pkg_id":"285A243662909DE3-5CD",
"source":"172.17.0.2",
"time":1653739000013,
"topicId":"a54de372-ffe0-49ae-a12e-c340bb2b03f2"
},{
"content":{
"__TAG__":{
"pod":"nginxPod",
"cluster":"testCluster"
},
"body_bytes_sent":"33496",
"http_referer":"-",
"http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36",
{"__TAG__":{"pod":"nginxPod","cluster":"testCluster"},"body_bytes_sent":"32847","http_referer":"-","http_user_agent":"Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01","remote_addr":"105.86.148.186","remote_user":"-","request_method":"GET","request_uri":"/apple-touch-icon-144x144.png","status":"404","time_local":"01/Nov/2018:00:55:14"},
{"__TAG__":{"pod":"nginxPod","cluster":"testCluster"},"body_bytes_sent":"33496","http_referer":"-","http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36","remote_addr":"222.18.168.242","remote_user":"-","request_method":"GET","request_uri":"/opt/node_apps/test-v5/app/themes/basic/public/static/404.html","status":"404","time_local":"01/Nov/2018:00:54:37"},
{"__TAG__":{"pod":"nginxPod","cluster":"testCluster"},"body_bytes_sent":"32847","http_referer":"-","http_user_agent":"Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01","remote_addr":"105.86.148.186","remote_user":"-","request_method":"GET","request_uri":"/apple-touch-icon-144x144.png","status":"404","time_local":"01/Nov/2018:00:55:14"},
{"__TAG__":{"pod":"nginxPod","cluster":"testCluster"},"body_bytes_sent":"33496","http_referer":"-","http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36","remote_addr":"222.18.168.242","remote_user":"-","request_method":"GET","request_uri":"/opt/node_apps/test-v5/app/themes/basic/public/static/404.html","status":"404","time_local":"01/Nov/2018:00:54:37"}
Case 3: Displaying Execution Statement Results in Alarm Notifications
Scenarios
Add the statement execution results that meet the trigger conditions to the alarm notification, formatted as a list.
Alarm policy execution statement: status:>=400 | select count(*) as errorLogCount,request_uri group by request_uri order by count(*) desc
Trigger condition: $1.errorLogCount > 10
Case 4: Using Execution Statement Results in Multidimensional Analysis
Scenarios
When configuring multi-dimensional analysis in an alarm policy, use the value of a specific field from the statement execution results as the filter condition for the analysis.
Alarm Policy Configuration
Here, {{.QueryResult[0][0].host}} represents the host field in the first row of the execution result for the first execution statement.
Note:
When the variable value is a string, please enclose it in quotes to comply with the rules for search and analysis statements. Use double quotation marks for search criteria, for example, "{{.QueryResult[0][0].host}}". Use single quotation marks for SQL statements, for example, '{{.QueryResult[0][0].host}}'.
