Kafka and Spark Versions in Each EMR Version

Last updated: 2020-02-28 20:02:30

PDF

Dependencies

EMR Version Spark Version Kafka Version Maven Example
v1.3.1 2.0.2 0.8.2.1 or higher spark-streaming-kafka-0-8_2.11
v2.0.1 2.2.1 0.8.2.1 or higher spark-streaming-kafka-0-8_2.11
v2.1.0 2.3.2 0.10.0 or higher spark-streaming-kafka-0-10_2.11
v3.0.0 2.4.3 0.10.0 or higher spark-streaming-kafka-0-10_2.12

Spark 2.3 or above no longer supports Kafka 0.8.2.

Query Method

  1. Access the official website URL and enter the version number link template:
    https://spark.apache.org/docs/{spark.version}/streaming-kafka-integration.html
    Replace {spark.version} with the corresponding Spark version. For example, if you want to query the dependencies of Spark 2.0.2, the access link will be as follows:
    https://spark.apache.org/docs/2.0.2/streaming-kafka-integration.html
  2. View the dependencies and click the corresponding link to view detailed descriptions.