Logstash, an open-source log processing tool, is used to collect data from multiple sources, filter collected data, and store data for other uses.

Logstash is highly flexible and has powerful syntax analysis capabilities. With a variety of plugins, it supports multiple types of inputs and outputs. In addition, as a horizontally scalable data pipeline, it has powerful log collection and retrieval features that work with Elasticsearch and Kibana.

How Logstash Works

The Logstash data processing pipeline can be divided into 3 stages: inputs → filters → outputs.

1. Inputs: generate data. Inputs are data sources such as file, syslog, redis, and beats.
2. Filters: modify and filter data. Filters are intermediate processing components in the Logstash data pipeline. They can modify events based on specific conditions. Some commonly used filters are grok, mutate, drop, and clone.
3. Outputs: transfer data to other locations. An event can be transferred to multiple outputs, and the event ends when the transfer is completed. Elasticsearch is the most commonly used output.

In addition, Logstash supports encoding and decoding data, so you can specify data formats on the input and output ends.

Advantages of Connecting Logstash to Kafka

• Data can be asynchronously processed to prevent traffic spikes.
• Components are decoupled, so when an exception occurs in Elasticsearch, the upstream work will not be affected.

Note：

Logstash consumes resources when processing data. If you deploy Logstash on a production server, the performance of the server may be affected.

Directions

Preparations

• Download and install Logstash. For more information, please see Installing Logstash.
• Download and install JDK 8. For more information, please see Java SE Development Kit 8 Downloads.
• You have created a CKafka instance.

Step 1. Get the CKafka instance access address

1. Log in to the CKafka console.
2. Select Instance List on the left sidebar and click the ID of an instance to enter the instance basic information page.
3. On the instance basic information page, get the instance access address in the Access Mode module.
   

Step 2. Create a topic

1. On the instance basic information page, select the Topic Management tab on the top.
2. On the topic management page, click Create to create a topic named logstash_test.
   

Step 3. Connect to CKafka

Note：

You can click the following tabs to view the detailed directions for using CKafka as inputs or outputs.