tencent cloud

Feedback

Function Overview

Last updated: 2022-04-19 16:22:14

CLS's data processing functions can be flexibly combined and used to cleanse, structure, filter, distribute, mask log data and more.
The figure below shows how a log containing JSON data is processed: the log data is processed and transformed into structured data, and then a field in the log is masked before the processed log is distributed.

The following is an overview of data processing functions.

  • Key-value extraction functions: extracting field-value pairs from log text
    FunctionDescriptionSyntax DescriptionReturn Value Type
    ext_sepExtracts field value content based on a separator.ext_sep("Source field name", "Target field 1,Target field 2,Target field...", sep="Separator", quote="Non-segmentation part"", restrict=False, mode="overwrite")Log after extraction (LOG)
    ext_sepstrExtracts field value content based on specified characters (string).ext_sepstr("Source field name","Target field 1,Target field 2,Target field...", sep="abc", restrict=False, mode="overwrite")Log after extraction (LOG)
    ext_jsonExtracts field values from JSON data.Log after extraction (LOG) ext_json("Source field name",prefix="",suffix="",format="full",exclude_node="JSON nodes not to expand")Log after extraction (LOG)
    ext_json_jmesExtracts a field value based on a JMES expression.ext_json_jmes("Source field name", jmes= "JSON extraction expression", output="Target field", ignore_null=True, mode="overwrite")Log after extraction (LOG)
    ext_kvExtracts key-value pairs by using two levels of separators.ext_kv("Source field name", pair_sep=r"\s", kv_sep="=", prefix="", suffix="", mode="fill-auto")Log after extraction (LOG)
    ext_regexExtracts the value of a field by using a regular expression.ext_regex("Source field name", regex="Regular expression", output="Target field 1,Target field 2,Target field.......", mode="overwrite")Log after extraction (LOG)
  • Enrichment functions: adding fields to existing fields according to rules
    FunctionDescriptionSyntax DescriptionReturn Value Type
    enrich_tableUses CSV structure data to match fields in logs and, when matched fields are found, the function adds other fields and values in the CSV data to the source logs.enrich_table("CSV source data", "CSV enrichment field", output="Target field 1,Target field 2,Target field....", mode="overwrite")Mapped log (LOG)
    enrich_dictUses dict structure data to match a field value in a log. If the specified field and value match a key in the dict structure data, the function assigns the value of the key to another field in the log.enrich_dict("JSON dictionary", "Source field name", output=Target field name, mode="overwrite")Mapped log (LOG)
  • Flow control functions: condition determination
    FunctionDescriptionSyntax DescriptionReturn Value Type
    composeCombines multiple operation functions. Providing combination capabilities similar to those of branch code blocks, this function can combine multiple operation functions and execute them in sequence. It can be used in combination with branches and output functions.compose("Function 1","Function 2", ...)Log (LOG)
    t_ifExecutes a corresponding function if a condition is met and does not perform any processing if the condition is not met.t_if("Condition", Function)Log (LOG)
    t_if_notExecutes a corresponding function if a condition is not met and does not perform any processing if the condition is met.t_if_not("Condition",Function)Log (LOG)
    t_if_elseExecutes a function based on the evaluation result of a condition.t_if_else("Condition", Function 1, Function 2)Log (LOG)
    t_switchExecutes different functions depending on whether branch conditions are met. If all conditions are not met, the data is deleted.t_switch("Condition 1", Function 1, "Condition 2", Function 2, ...)Log (LOG)
  • Row processing functions: log distribution, deletion, and splitting
    FunctionDescriptionSyntax DescriptionReturn Value Type
    log_outputOutputs a row of log to a specified log topic. This function can be used independently or together with branch conditions.log_output(Log topic alias) (The alias here is the target log topic alias specified when the data processing task is created.)No return value
    log_splitSplits a row of log into multiple rows of logs based on the value of a specified field by using a separator and JMES expression.log_split(Field name, sep=",", quote="\"", jmes="", output="")Log (LOG)
    log_dropDeletes logs that meet a specified condition.log_drop(Condition 1)Log (LOG)
    log_keepRetains logs that meet a specified condition.log_keep(Condition 1)Log (LOG)
  • Field processing functions: field Create/Read/Update/Delete/Rename
    FunctionDescriptionSyntax DescriptionReturn Value Type
    fields_dropDeletes the fields that meet a specified condition.fields_drop(Field name 1, Field name 2, ..., regex=False,nest=False)Log (LOG)
    fields_keepRetains the fields that meet a specified condition.fields_keep(Field name 1, Field name 2, ..., regex=False)Log (LOG)
    fields_packMatches field names based on a regular expression and encapsulates the matched fields into a new field whose value is in JSON format.fields_pack(Target field name, include=".*", exclude="", drop_packed=False)Log (LOG)
    fields_setSets field values or adds fields.fields_set(Field name 1, Field value 1, Field name 2, Field value 2, mode="overwrite")Log (LOG)
    fields_renameRenames fields.fields_rename(Field name 1, New field name 1, Field name 2, New field name 2, regex=False)Log (LOG)
    has_fieldIf the specified field exists, returns `True`. Otherwise, returns `False`.has_field(Field name)Condition value (BOOL)
    not_has_fieldIf the specified field does not exist, returns `True`. Otherwise, returns `False`.not_has_field(Field name)Condition value (BOOL)
    vGets the value of a specified field and returns the corresponding string.v(Field name)Value string type (STRING)
  • Value structuring functions: JSON data processing
    FunctionDescriptionSyntax DescriptionReturn Value Type
    json_selectExtracts a JSON field value with a JMES expression and returns the JSON string of the extraction result.json_select(v(Field name), jmes="")Value string type (STRING)
    xml_to_jsonParses and converts an XML-formatted value to a JSON string. The input value must be an XML string. Otherwise, a conversion exception will occur.xml_to_json(Field value)Value string type (STRING)
    json_to_xmlParses and converts a JSON string value to an XML string.json_to_xml(Field value)Value string type (STRING)
    if_jsonChecks whether a value is a JSON string.if_json(Field value)Condition value (BOOL)
  • Regular expression functions: matching and replacing characters in text by using regular expressions
    FunctionDescriptionSyntax DescriptionReturn Value Type
    regex_matchMatches data in full or partial match mode based on a regular expression and returns whether the match is successful.regex_match(Field value, regex="", full=True)Condition value (BOOL)
    regex_selectMatches data based on a regular expression and returns the corresponding partial match result. You can specify the sequence number of the matched expression and the sequence number of the group to return (partial match + sequence number of the specified matched group). If no data is matched, an empty string is returned.regex_select(Field value, regex="", index=1, group=1)Value string type (STRING)
    regex_splitSplits a string and returns a JSON array of the split strings (partial match).regex_split(Field value, regex=\"\", limit=100)Value string type (STRING)
    regex_replaceMatches data based on a regular expression and replaces the matched data (partial match).regex_replace(Field value, regex="", replace="", count=0)Value string type (STRING)
    regex_findallMatches data based on a regular expression and returns a JSON array of the matched data (partial match).regex_findall(Field value, regex="")Value string type (STRING)
  • Date value processing functions
    FunctionDescriptionSyntax DescriptionReturn Value Type
    dt_strConverts a time field value (a date string in a specific format or timestamp) to a target date string of a specified time zone and format.dt_str(Value, format="Formatted string", zone="")STRING
    dt_to_timestampConverts a time field value (a date string in a specified format; time zone specified) to a UTC timestamp.dt_to_timestamp(Value, zone="")STRING
    dt_from_timestampConverts a timestamp field value to a time string in the specified time zone.dt_from_timestamp(Value, zone="")STRING
    dt_nowObtains the current datetime of the processing calculation.dt_now(format="Formatted string", zone="")STRING
  • String processing functions
    FunctionDescriptionSyntax DescriptionReturn Value Type
    str_countSearches for a substring in a specified range of a value and returns the number of occurrences of the substring.str_count(Value, sub="", start=0, end=-1)INT
    str_lenReturns the length of a string.str_len(Value)INT
    str_uppercaseConverts a string to uppercase.str_uppercase(Value)STRING
    str_lowercaseConverts a string to lowercase.str_lowercase(Value)STRING
    str_joinConcatenates input values by using a concatenation string.str_join(Concatenation string 1, Value 1, Value 2, ...)STRING
    str_replaceReplaces an old string with a new string.str_replace(Value, old="", new="", count=0)STRING
    str_formatFormats strings.str_format(Formatting string, Value 1, Value 2, ...)STRING
    str_stripDeletes specified characters from a string concurrently from the start and end of the string, and returns the remaining part.str_strip(Value, chars="\t\r\n")STRING
    str_lstripDeletes specified characters from a string from the start of the string, and returns the remaining part.str_strip(Value, chars="\t\r\n")STRING
    str_rstripDeletes specified characters from a string from the end of the string, and returns the remaining part.str_strip(Value, chars="\t\r\n")STRING
    str_findChecks whether a string contains a specified substring and returns the position of the first occurrence of the substring in the string.str_find(Value, sub="", start=0, end=-1)INT
    str_start_withChecks whether a string starts with a specified prefix.str_start_with(Value, sub="", start=0, end=-1)BOOL
    str_end_withChecks whether a string ends with a specified prefix.str_end_with(Value, sub="", start=0, end=-1)BOOL
  • Logical expression functions
    FunctionDescriptionSyntax DescriptionReturn Value Type
    op_ifReturns a value based on a specified condition.op_if(Condition 1, Value 1, Value 2)If the condition is met, value 1 is returned. Otherwise, value 2 is returned.
    op_ifnullReturns the first non-null and non-empty result value.op_ifnull(Value 1, Value 2, ...)First non-null result value in the parameter
    op_andPerforms the AND operation on values. If all the specified parameter values are evaluated to true, `True` is returned. Otherwise, `False` is returned. op_and(Value 1, Value 2, ...)BOOL
    op_orPerforms the OR operation on values. If one or more of the specified parameter values are evaluated to false, `False` is returned. Otherwise, `True` is returned.op_or(Value 1, Value 2, ...)BOOL
    op_notPerforms the NOT operation on values.op_not(Value)BOOL
    op_eqCompares two values. If the values are equal, `True` is returned.op_eq(Value 1, Value 2)BOOL
    op_geCompares two values. If `Value 1` is greater than or equal to `Value 2`, `True` is returned.op_ge(Value 1, Value 2)BOOL
    op_gtCompares two values. If `Value 1` is greater than `Value 2`, `True` is returned.op_gt(Value 1, Value 2)BOOL
    op_leCompares two values. If `Value 1` is less than or equal to `Value 2`, `True` is returned.op_le(Value 1, Value 2)BOOL
    op_ltCompares two values. If `Value 1` is less than `Value 2`, `True` is returned.op_lt(Value 1, Value 2)BOOL
    op_addReturns the sum of two specified values.op_add(Value 1, Value 2)Calculation result
    op_subReturns the difference between two specified values.op_sub(Value 1, Value 2)Calculation result
    op_mulReturns the product of two specified values.op_mul(Value 1, Value 2)Calculation result
    op_divReturns the quotient of two specified values.op_div(Value 1, Value 2)Calculation result
    op_sumReturns the sum of multiple specified values.op_sum(Value 1, Value 2, ...)Calculation result
    op_modReturns the remainder of a specified value divided by the other specified value.op_mod(Value 1, Value 2)Calculation result
  • Type conversion functions
    FunctionDescriptionSyntax DescriptionReturn Value Type
    ct_intConverts a value (whose base can be specified) to a decimal integer.ct_int(Value 1, base=10)Calculation result
    ct_floatConverts a value to a floating-point number.ct_float(Value)Calculation result
    ct_strConverts a value to a string.ct_str(Value)Calculation result
    ct_boolConverts a value to a Boolean value.ct_bool(Value)Calculation result



Contact Us

Contact our sales team or business advisors to help your business.

Technical Support

Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

7x24 Phone Support