Process a log entry using functions such as filtering, distribution, and splitting.
log_output Function
Function Definition
Output to a specified target topic. It can be used together with branch conditions or independently.
Syntax Description
log_output (Alias)
The alias is defined when a processing task is configured, as shown in the following figure:
Parameter Description
Parameter Name
Parameter Description
Parameter Type
Required
Default Value
Parameter Value Range
alias
Alias name of the target topic
string
Yes
-
-
Example
The loglevel field values are distributed to three different log topics as waring/info/error.
Raw logs:
[
{
"loglevel":"warning"
},
{
"loglevel":"info"
},
{
"loglevel":"error"
}
]
Processing rules:
//The loglevel field has three values including waring, info, and error, and therefore the log is distributed to three different log topics accordingly.
Processing result: As shown in the following figure
log_auto_output Function:
Function Definition
Output logs to dynamic target topics. For example, if you need to dynamically create multiple target log topics based on the value of the log field "pd" and distribute the corresponding logs to those topics, you can use this function. Assuming the values of "pd" are "CLB", "Ckafka", "COS", and "CDN", using this function will dynamically create target log topics named "CLB", "Ckafka", "COS", and "CDN", and write the relevant logs to the corresponding topics. Additionally, you can configure the index type and storage period for these newly created topics.
If the parameter `topic_name` contains a vertical bar ("|"), the vertical bar is removed from the generated topic name.
If the length of the parameter `topic_name` exceeds 250 characters, the generated log topic name contains only the first 250 characters, and any excess characters are truncated.
logset_name
Logset Name
string
y
-
-
index_options
all_index: Enables key-value and full-text indexing.
no_index: Does not enable indexing.
content_index: Enables full-text indexing.
key_index: Enables key-value indexing.
string
n
all_index
If storage_type=cold (that is, STANDARD_IA storage), then all_index and key_index do not take effect, meaning STANDARD_IA storage does not support key-value indexing.
period
Storage period, typically ranging from 1 - 3600 days
3640 indicates permanent storage.
number
n
3
1 - 3600 days.
storage_type
Storage type of the log topic, with optional values
hot: STANDARD storage
cold: STANDARD_IA storage
string
n
hot
When the value is cold, the minimum value for period is 7 days.
hot_period
0: Disable log archiving
Non-zero: The number of days for STANDARD storage after log archiving is enabled.
HotPeriod must be greater than or equal to 7 and less than Period, and this rule only takes effect when StorageType is hot.
number
n
0
-
tag_dynamic
Adds dynamic Tags to a log topic. It works with the extract_tag() function to extract Tag key-value pairs from log fields. For example: tag_dynamic=extract_tag(v("pd"),v("env"),v("team"), v("person")).
string
n
-
The total number of Tags for tag_dynamic and tag_static combined must not exceed 10 pairs.
tag_static
Adds static Tags to a log topic. For example: tag_static="Ckafka:test_env,developer_team:MikeWang".
string
n
-
Example of not exceeding 10 pairs of Tags in combination with tag_dynamic.
Note:
After you modify the parameters in the log_auto_output() function and restart the task, the changes take effect only for newly generated topics. Existing topics are not affected. If you need to modify already generated logsets or log topics, go to the Log Topic List or call the API to make the modifications.
Without modifying the log_auto_output() function, if you change the names of topics dynamically created by the data processing task, the fallback logset name, or the fallback log topic name, the data distribution logic remains unaffected. For example, if a processing task has dynamically created a log topic named "Nginx" and you rename it to "Nginx001", data will still be written to this "Nginx001" topic, and a new log topic named "Nginx" will not be created.
Deleting a data processing task (which dynamically creates topics) does not delete the already created logsets and log topics.
Example
Raw logs:
[
{
"pd": "CLB",
"dateTime": "2023-05-25T00:00:26.579"
},
{
"pd": "Ckafka",
"time": "2023-05-25T18:00:55.350+08:00"
},
{
"pd": "COS",
"time": "2023-05-25T00:06:20.314+08:00"
},
{
"pd": "CDN",
"time": "2023-05-25T00:03:52.051+08:00"
}
]
Processing rules:
log_auto_output(v("pd"), "My Logset", index_options="content_index", period=3)
Processing result: Four log topics are automatically generated, namely "CLB", "Ckafka", "COS", and "CDN". The logset name is "My Logset", as shown in the following figure:
Note:
Creating a Log Topic
Special Scenario
How It Works
If the parameter topic_name fails to be obtained or is "" (empty string), the log entry will go to the log topic named "null".
For example, if the app field is specified as topicname (that is, the log topic name), but a log entry does not contain the app field or its value is " ", then the log entry will be written to the log topic named "null". The logset to which the "null" topic belongs is the logset specified in the logset parameter of this function.
Note:
If you use string concatenation to specify a log topic name, for example log_auto_output( str_join("-", "myPrefix", v("app")), "myLogSet",period=14,storage_type="hot",hot_period=10), then when the app does not exist or its value is " ", logs are written to the log topic named "myprefix".
Duplicate topic name (within the same logset)
If a log topic named "Nginx" already exists, when the dynamic creation task, according to the rules, also needs to create a log topic named "Nginx", it will add the CreateByCLSETL suffix, and the created log topic will be named "Nginx-CreateByETL".
Creating a Logset
Special Scenario
How It Works
Duplicate logset name (within the same region)
If you manually create a logset named "myLogSet" and a dynamically created task also needs to create a logset named "myLogSet" according to the function rules, the suffix "CreateByCLSETL" is automatically appended.
If Auto-Create Task 1 creates a logset named "myLogSet", and Auto-Create Task 2 also defines a logset name as "myLogSet", then Auto-Create Task 2 uses the same logset name "myLogSet" as the target logset. This means a new logset with the suffix "CreateByCLSETL" is not automatically created.
Note:
Logset and log topic names are case-insensitive. On the CLS side, "abc" and "Abc" represent the same resource.
Adding a Tag to a Log Topic
Special Scenario
How It Works
Failure to Obtain KV Values of Dynamic Tags.
For example, if field A is specified as the Tag Key and field B as the Tag Value, but a log entry lacks field A or field B, or the values of fields A and B are left empty, the Tag addition fails.
Leading or Trailing Spaces in Tag KV Values.
Tag addition fails.
Total Number of Dynamic and Static Tags Exceeds 10 Pairs.
Dynamic Tags have higher priority. If the total number is still less than 10 pairs after Dynamic Tags are added, Static Tags are supplemented, but the total number does not exceed 10 pairs.
Tag KV Length Exceeds Limit.
If the Tag key length exceeds 127 or the value length exceeds 255, the Tag addition fails.
Note:
If you need to repeatedly rebuild a data processing task with the same name during a POC, the procedure is as follows:
1. First, delete the old data processing task. Otherwise, it will continue to generate log topics.
2. Delete the fallback topic auto_undertake_topic_${etl-task-name}. If a task with the same name is created three consecutive times, the over-limit fallback topic fails to be created, which in turn blocks the normal startup of the task.
If the POC uses the same dataset and it is not real-time data, we recommend deleting the logsets and log topics generated by the old task.
log_split Function
Function Definition
Split specific fields into multiple rows of logs based on the value of a specified field by using a delimiter and JMES expression.
