- Release Notes and Announcements
- User Guide
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection Overview
- Collecting Logs in Self-Built Kubernetes Cluster
- Collecting Syslog
- Collection by LogListener
- Collecting Text Log
- Uploading Log over Kafka
- Uploading Logs via Anonymous Write
- Uploading Logs via Logback Appender
- Uploading Logs via Log4j Appender
- Uploading Log via SDK
- Uploading Log via API
- Importing Data
- Tencent Cloud Service Log Access
- Log Storage
- Search and Analysis
- Syntax and Rules
- Statistical Analysis (SQL)
- Quick Analysis
- SQL Syntax
- SQL Functions
- String Function
- Date and Time Functions
- IP Geographic Function
- URL Function
- Mathematical Calculation Functions
- Mathematical Statistical Function
- General Aggregate Function
- Geospatial Function
- Binary String Function
- Estimation Function
- Type Conversion Function
- Logical Function
- Operators
- Bitwise Operation
- Regular Expression Function
- Lambda Function
- Conditional Expressions
- Array Functions
- Interval-Valued Comparison and Periodicity-Valued Comparison Functions
- JSON Functions
- Window Functions
- Sampling Analysis
- Configuring Indexes
- Reindexing
- Context Search and Analysis
- Downloading Log
- Dashboard
- Data Processing documents
- Data Processing
- Creating Processing Task
- Viewing Data Processing Details
- Data Processing Functions
- Function Overview
- Key-Value Extraction Functions
- Enrichment Functions
- Flow Control
- Row Processing Functions
- Field Processing Functions
- Value Structuring Functions
- Regular Expression Processing Functions
- Time Value Processing Functions
- String Processing Functions
- Type Conversion Functions
- Logical and Mathematical Functions
- Encoding and Decoding Functions
- IP Parsing Functions
- Processing Cases
- Scheduled SQL Analysis
- SCF
- Data Processing
- Shipping and Consumption
- Log Shipping
- Monitoring Alarm
- Historical Documentation
- Best Practices
- Developer Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Index APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- Data Types
- Error Codes
- FAQs
- CLS Service Level Agreement
- CLS Policy
- Contact Us
- Glossary
- Release Notes and Announcements
- User Guide
- Product Introduction
- Purchase Guide
- Getting Started
- Operation Guide
- Resource Management
- Permission Management
- Log Collection
- Collection Overview
- Collecting Logs in Self-Built Kubernetes Cluster
- Collecting Syslog
- Collection by LogListener
- Collecting Text Log
- Uploading Log over Kafka
- Uploading Logs via Anonymous Write
- Uploading Logs via Logback Appender
- Uploading Logs via Log4j Appender
- Uploading Log via SDK
- Uploading Log via API
- Importing Data
- Tencent Cloud Service Log Access
- Log Storage
- Search and Analysis
- Syntax and Rules
- Statistical Analysis (SQL)
- Quick Analysis
- SQL Syntax
- SQL Functions
- String Function
- Date and Time Functions
- IP Geographic Function
- URL Function
- Mathematical Calculation Functions
- Mathematical Statistical Function
- General Aggregate Function
- Geospatial Function
- Binary String Function
- Estimation Function
- Type Conversion Function
- Logical Function
- Operators
- Bitwise Operation
- Regular Expression Function
- Lambda Function
- Conditional Expressions
- Array Functions
- Interval-Valued Comparison and Periodicity-Valued Comparison Functions
- JSON Functions
- Window Functions
- Sampling Analysis
- Configuring Indexes
- Reindexing
- Context Search and Analysis
- Downloading Log
- Dashboard
- Data Processing documents
- Data Processing
- Creating Processing Task
- Viewing Data Processing Details
- Data Processing Functions
- Function Overview
- Key-Value Extraction Functions
- Enrichment Functions
- Flow Control
- Row Processing Functions
- Field Processing Functions
- Value Structuring Functions
- Regular Expression Processing Functions
- Time Value Processing Functions
- String Processing Functions
- Type Conversion Functions
- Logical and Mathematical Functions
- Encoding and Decoding Functions
- IP Parsing Functions
- Processing Cases
- Scheduled SQL Analysis
- SCF
- Data Processing
- Shipping and Consumption
- Log Shipping
- Monitoring Alarm
- Historical Documentation
- Best Practices
- Developer Guide
- API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Topic Management APIs
- Log Set Management APIs
- Index APIs
- Topic Partition APIs
- Machine Group APIs
- Collection Configuration APIs
- Log APIs
- Metric APIs
- Alarm Policy APIs
- Data Processing APIs
- Kafka Protocol Consumption APIs
- CKafka Shipping Task APIs
- Kafka Data Subscription APIs
- COS Shipping Task APIs
- SCF Delivery Task APIs
- Scheduled SQL Analysis APIs
- COS Data Import Task APIs
- Data Types
- Error Codes
- FAQs
- CLS Service Level Agreement
- CLS Policy
- Contact Us
- Glossary
Overview
CLS provides log collection clients that allow you to collect your application logs and import them into CLS through APIs or SDKs. Currently, CLS requires logs to be uploaded in a structured manner as key-value pairs.
Log Structuring
The structuring of logs is to store your log data on the CLS platform in key-value format. Structured logs can be searched for, analyzed, and shipped based on specified keys. CLS allows you to report structured data directly. For more information, see the following:
For example, a local raw log is as follows:
10.20.20.10;[Tue Jan 22 14:49:45 CST 2019 +0800];GET /online/sample HTTP/1.1;127.0.0.1;200;647;35;http://127.0.0.1/
Specify that the log is parsed by separator and select semicolon (;) as the separator. Then the log can be parsed into multiple field groups and each group is organized in key-value pairs. A key name is defined for each key as follows:
IP: 10.20.20.10time: [Tue Jan 22 14:49:45 CST 2019 +0800]request: GET /online/sample HTTP/1.1host: 127.0.0.1status: 200length: 647bytes: 35referer: http://127.0.0.1/
LogListener provides various parsing modes, in which reported logs with full text in a single line and full text in multi lines are both structured. LogListener adds
__CONTENT__ as a key and uses the original text as the value by default. The key-value pair is as follows:LogListener Parsing Mode | Key | Description |
Full text in a single line/Full text in multi lines | __CONTENT__ | __CONTENT__ is the key name by default. |
JSON format | Name in the JSON file | The name and value in the original text of the JSON log are used as a key-value pair. |
Separator format | Custom | After dividing fields using separators, you need to define a key name for each field group. |
Full regular expression | Custom | After extracting fields based on a regular expression, you need to define a key name for each field group. |
Collection Methods
CLS provides multiple methods for data collection:
Collection Method | Description |
API | You can call CLS APIs to upload structured logs to CLS. For more information, see Uploading Log via API. |
SDK | |
LogListener client | LogListener is a log collection client provided by CLS. You can quickly access CLS by simply configuring LogListener in the console. For more information, see LogListener Use Process. |
A comparison of the collection methods is as follows:
Category | Collection via LogListener | Collection via API |
Code modification | Provides a non-intrusive collection method for applications, without code modification. | Reports logs only after modifying application code. |
Resumable upload | Supports resumable upload of logs. | Automatically implemented by code. |
Retransmission upon failure | Provides an inherent retry mechanism. | Automatically implemented by code. |
Local cache | Supports local cache, ensuring data integrity during peak hours. | Automatically implemented by code. |
Resource occupation | Occupies resources such as memory and CPU resources. | Occupies no additional resources. |
Accessing Log Sources
You can access different log sources in different ways. For more information, see the following tables:
Log source environment
System Environment | Recommended Access Method |
Linux/Unix | |
Windows | |
iOS/Android/Web |
Tencent Cloud service logs
Yes
No
Was this page helpful?