In large-scale distributed applications, or when the workloads of a service are distributed across multiple servers, you can use Linux commands to view application logs one by one. However, this method is highly inefficient, and it is even more complicated when using container deployment services rather than normal distributed applications. Because containers are managed by an upper-level orchestration system, their associated connections with the server are variable. Without a single service to collect and display the application logs, exploring the logs becomes even more difficult.
That’s why TKE is integrated with Tencent Cloud Log Service (CLS), which supports the configuration of collection rules to automatically collect and report logs. It also supports using the log collection and visualization service ELK (Elasticsearch + Logstash + Kibana). This document introduces the application of basic templates provided by Tencent Cloud TKE to create an ELK deployment that uses log data read from Kafka.
Elasticsearch is an open source, distributed search engine based on Apache Lucene (TM) that provides a RESTful API. Within the framework of ELK, Elasticsearch provides quick data storage and query capabilities. Kibana is an open source data analysis and visualization platform for Elasticsearch. It is used to search and display data stored in the Elasticsearch search engine.
The following steps use the ELK basic template provided by TKE to build an Elasticsearch cluster and Kibana.
If the node does not have Git, first executes the
yum install gitcommand to install Git.
Execute the following command to download the required YAML files to nodes in a TKE cluster.
git clone https://github.com/tencentyun/ccs-elasticsearch-template.git /tmp/kubernetes-elasticsearch
Execute the following commands to create an Elasticsearch deployment.
cd /tmp/kubernetes-elasticsearch kubectl create -f es-svc.yaml kubectl create -f es-client.yaml kubectl create -f es-data.yaml kubectl create -f es-discovery-svc.yaml kubectl create -f es-master.yaml
Execute the following commands to create a Kibana deployment.
cd /tmp/kubernetes-elasticsearch kubectl create -f kibana-svc.yaml kubectl create -f kibana.yaml
Logstash is an open source log analysis and processing program that can collect and transform data from a variety of sources, including Syslog, Filebeat, Kafka, etc. It also supports sending collected data to Elasticsearch.
The Logstash deployed in this example reads data from the configured Kafka by default, and then sends it to the deployed Elasticsearch service.
cd /tmp/kubernetes-elasticsearch vim logstash-config.yaml
kubectl create -f logstash-config.yaml kubectl create -f logstash-consumer.yaml
This text takes ELK deployment in a TKE cluster and log data read from Kafka as an example. For more instructions on using and troubleshooting ELK, check online.