Logstash, an open-source log processing tool, is used to collect data from multiple sources, filter collected data, and store data for other uses.
Logstash is highly flexible and has powerful syntax analysis capabilities. With a variety of plugins, it supports multiple types of inputs and outputs. In addition, as a horizontally scalable data pipeline, it has powerful log collection and retrieval features that work with Elasticsearch and Kibana.
The Logstash data processing pipeline can be divided into 3 stages: inputs → filters → outputs.
In addition, Logstash supports encoding and decoding data, so you can specify data formats on the input and output ends.
The compatibility of official versions is described in the following table:
Kafka Client Version | Logstash Version | Plugin Version |
---|---|---|
0.8 | 2.0.0 - 2.x.x | < 3.0.0 |
0.9 | 2.0.0 - 2.3.x | 3.x.x |
0.9 | 2.4.x - 5.x.x | 4.x.x |
0.10.0.x | 2.4.x - 5.x.x | 5.x.x |
The latest version is V5.1.8, which uses the Consumer API V0.10 to read data.
For information about parameter configuration, see Kafka input plugin.
The compatibility of official versions is described in the following table:
Kafka Client Version | Logstash Version | Plugin Version |
---|---|---|
0.8 | 2.0.0 - 2.x.x | < 3.0.0 |
0.9 | 2.0.0 - 2.3.x | 3.x.x |
0.9 | 2.4.x - 5.x.x | 4.x.x |
0.10.0.x | 2.4.x - 5.x.x | 5.x.x |
The latest version is V5.1.7, which uses the Producer API V0.10 to produce data.
For information about parameter configuration, see Kafka output plugin.
Run bin/logstash-plugin list
to check whether logstash-input-kafka is included in the supported plugin list.
Write a configuration file input.conf
.
In the following example, Kafka is used as the data source, and the standard output is taken as the data destination.
input {
Kafka {
bootstrap_servers => "172.16.16.12:9092" // VIP address of the CKafka instance
group_id => "logstash_group" // CKafka group ID
topics => ["logstash_test"] // CKafka topic name
consumer_threads => 3 // Number of consumer threads, which is generally the same as the number of CKafka partitions
auto_offset_reset => "earliest"
}
}
output {
stdout{codec=>rubydebug}
}
Launch Logstash to consume messages.
In the proceeding figure, you can see that messages in the topic have been consumed.
Run bin/logstash-plugin list
to check whether logstash-output-kafka is included in the supported plugin list.
Write a configuration file output.conf
.
In the following example, the standard input is taken as the data source, and Kafka is used as the data destination.
Launch Logstash to produce messages.
Verify the production data from the previous step.
Was this page helpful?