Component Version

Last updated: 2021-11-17 16:31:02

    Elastic MapReduce (EMR) consists of a series of open-source applications in the big data ecosystem. EMR supports five types of clusters: Hadoop, Druid, ClickHouse, Kafka, and Doris. Each EMR version contains a specific set of open-source applications. When you create a cluster, you can choose the most appropriate EMR version based on your actual needs.

    EMR comes with two component editions. One is the Standard edition, which is based on the stable open source community component edition to provide services. To keep your EMR clusters consistent with the evolution of the community, you are advised to choose the Standard edition. The other one is the TianQiong edition, which has incorporated Tencent Cloud's proprietary, mature, and stable features on the basis of the stable open source community component edition for better performance and stability. Please choose an edition that suits your business when you purchase a cluster. It is not recommended to switch between the two editions after the cluster is launched.

    EMR version numbers are in the format of EMR va.b.c as detailed below:

    • The meanings of a for different clusters are as follows:
      -For Hadoop clusters, a indicates the Hadoop versions supported by the current version. When a is 1 or 2, Hadoop v2.X is supported; when a is 3, Hadoop v3.X is supported.
      • For Druid clusters, a indicates the Druid versions supported by the current version. When a is 1, Druid v0.17.X is supported.
      • For ClickHouse clusters, a indicates the ClickHouse versions supported by the current version. When a is 1, ClickHouse v19.X and v20.X are supported.
      • For Kafka clusters, a indicates the Kafka versions supported by the current version. When a is 1, Kafka v1.X is supported.
      • For Doris clusters, a indicates the Doris versions supported by the current version. When a is 1, Doris v0.13X is supported.
    • b indicates that the version has new components or supports component version upgrade.
    • c indicates feature optimization.
    Note:

    • The components and their versions bundled with each EMR version are fixed. Currently, neither selecting multiple versions of a component nor changing a component version in one EMR version is supported. For example, Hadoop v2.7.3 and Spark v2.2.1 are built into EMR v2.0.1.
    • Once a version of EMR is selected for cluster creation, the EMR and component version used by the cluster will not be automatically upgraded. For example, if EMR v2.0.1 is selected, then Hadoop will always be v2.7.3, and Spark will always be v2.2.1. Even if EMR is upgraded to v2.1.0, Hadoop is upgraded to v2.8.4, and Spark is upgraded to v2.3.2 afterward, the previously created cluster will not be affected, and only new clusters will use the new versions.
    • When you upgrade the cluster through data migration, for example, from EMR v2.0.1 to EMR v2.1.0, in order to avoid issues such as version incompatibility or environment changes, be sure to test the tasks to be migrated and ensure that they can work properly in the new software environment.
    • EMR v2.4.0 comes with Kona (based on OpenJDK8). We have developed and improved Kona based on the characteristics of cloud scenarios.

    Hadoop cluster is available in Standard Edition and TianQiong Edition. For more information on the differences between the two editions, please see EMR TianQiong Introduction.

    EMR Standard

    Currently, EMR Standard supports the Hadoop, Druid, ClickHouse, Kafka, and Doris clusters.

    Hadoop2.X

    展开&收起
    ComponentEMR v1.3.1EMR v2.0.1EMR v2.1.0EMR v2.2.0EMR v2.3.0EMR v2.4.0EMR v2.5.0EMR v2.5.1EMR v2.6.0
    Release Date--May 2019March 2020May 2020August 2020September 2020April 2021July 2021
    Hadoop (required)2.7.32.7.32.8.42.8.52.8.52.8.52.8.52.8.52.8.5
    Spark_Hadoop2.7 2.0.22.7 2.2.12.8 2.3.22.8 2.4.32.8 2.4.32.8 3.0.02.8 3.0.0--
    Spark-------3.0.03.0.2
    Hive2.1.12.3.22.3.32.3.52.3.52.3.72.3.72.3.72.3.7
    Tez0.8.50.8.50.8.50.9.20.9.20.9.20.9.20.9.20.9.2
    Presto0.1610.1880.2150.2280.228----
    PrestoSQL-----332332332332
    Storm1.1.01.1.01.1.01.2.31.2.31.2.31.2.31.2.31.2.3
    Flink1.2.01.2.01.4.21.9.21.9.21.10.01.10.01.10.01.12.1
    HBase1.2.41.3.11.3.11.4.91.4.91.4.91.4.91.4.91.4.9
    Phoenix (integrated in HBase)4.8.14.11.04.13.04.13.04.13.04.13.04.13.04.13.04.14.0
    Ganglia3.7.23.7.23.7.23.7.23.7.23.7.23.7.23.7.23.7.2
    Hue3.12.03.12.04.4.04.6.04.6.04.6.04.6.04.6.04.6.0
    Sqoop1.4.61.4.61.4.71.4.71.4.71.4.71.4.71.4.71.4.7
    Oozie4.3.14.3.14.3.15.1.05.1.05.1.05.1.05.1.05.1.0
    Ranger-0.7.10.7.11.2.01.2.01.2.01.2.01.2.01.2.0
    ZooKeeper (required)3.4.93.4.93.4.93.5.53.5.53.6.13.6.13.6.13.6.1
    Flume--1.8.01.9.01.9.01.9.01.9.01.9.01.9.0
    Impala---2.10.02.10.02.10.02.10.02.10.03.4.0
    Kylin---2.5.22.5.22.5.22.5.22.5.22.5.2
    Zeppelin---0.8.20.8.20.8.20.8.20.8.20.9.1
    Alluxio--1.8.11.8.11.8.11.8.12.3.02.5.02.5.0
    Knox (required)1.2.01.2.01.2.01.2.01.2.01.2.01.2.01.2.01.2.0
    Kerberos--1.15.01.15.01.15.01.15.01.15.01.15.01.15.0
    Hudi---0.5.10.5.1---0.7.0
    Superset---0.35.20.35.20.35.20.35.20.35.20.35.2
    Livy---0.7.00.7.00.7.00.7.00.7.00.8.0
    TensorFlowSpark----1.4.41.4.41.4.41.4.41.4.4
    Jupyter----4.6.34.6.34.6.34.6.34.6.3
    Kudu-----1.12.01.12.01.12.01.12.0
    OpenLDAP (required)--------2.4.44

    Hadoop3.X

    展开&收起
    ComponentEMR v3.0.0EMR v3.1.0EMR v3.2.0EMR v3.2.1EMR v3.3.0
    Release DateNovember 2019December 2020April 2021July 2021September 2021
    Hadoop (required)3.1.2---3.2.2
    HDFS (required)-3.1.23.2.23.2.23.2.2
    Yarn (required)-3.1.23.2.23.2.23.2.2
    Spark_ Hadoop3.12.4.3----
    Spark-2.4.33.0.23.0.23.0.2
    Hive3.1.13.1.13.1.23.1.23.1.2
    Tez0.9.20.9.20.10.00.10.00.10.1
    Presto0.222----
    PrestoSQL-332350350350
    Flink1.8.11.10.01.12.11.12.11.12.1
    HBase2.2.02.3.32.3.32.3.32.3.5
    Phoenix (integrated in HBase)-5.0.05.0.05.0.05.1.2
    Hue4.4.04.4.04.4.04.4.04.10.0
    Sqoop1.4.71.4.71.4.71.4.71.4.7
    Oozie5.1.05.1.05.1.05.1.05.1.0
    Ranger2.0.02.0.02.1.02.1.02.1.0
    ZooKeeper (required)3.4.93.6.13.6.13.6.13.6.1
    Flume1.9.01.9.01.9.01.9.01.9.0
    Impala2.10.03.4.03.4.03.4.03.4.0
    Alluxio1.8.12.3.02.5.02.5.02.5.0
    Knox (required)1.2.01.2.01.2.01.2.01.2.0
    Kudu-1.13.01.13.01.13.01.15.0
    Kerberos1.15.11.15.11.51.11.51.11.15.1
    Zeppelin-0.8.20.9.10.9.10.9.1
    Iceberg--0.11.00.11.00.11.0
    OpenLDAP (required)---2.4.442.4.44
    Hudi----0.8.0
    Kyuubi-----1.1.0
    Livy----0.8.0
    Ganglia----3.7.2
    ComponentDruid v1.0.0
    Release DateApril 2020
    Hadoop (required)2.8.5
    Druid (required)0.17.0
    ZooKeeper (required)3.5.5
    Knox (required)1.2.0
    Superset0.35.2
    Ganglia3.7.2

    ClickHouse

    展开&收起
    ComponentClickHouse v1.0.0ClickHouse v1.1.0
    Release DateApril 2020May 2020
    ClickHouse (required)19.16.12.4920.3.10.75
    ZooKeeper (required)3.4.93.4.9
    Superset-0.35.2
    ComponentKafka v1.0.0
    Release DateMay 2021
    Kafka (required)1.1.1
    KafkaManager (required)2.0.0.2
    Knox (required)1.2.0
    ZooKeeper (required)3.6.1
    ComponentDoris v1.0.0
    Release DateMay 2021
    Doris (required)0.13.0
    Knox (required)1.2.0

    EMR TianQiong

    Currently, EMR TianQiong supports Hadoop clusters only. It has integrated the enhanced edition of Spark and Tencent's proprietary JDK Kona.

    Hadoop

    展开&收起
    ComponentEMR TianQiong v1.0.0
    Release DateNovember 2020
    Hadoop (required)2.8.5
    Spark3.0.1 (Enhanced)
    Hive2.3.7
    Tez0.9.2
    PrestoSQL332
    Storm1.2.3
    Flink1.10.0
    HBase1.4.9
    Phoenix4.13.0
    Ganglia3.7.2
    Hue4.6.0
    Sqoop1.4.7
    Oozie5.1.0
    Ranger1.2.0
    ZooKeeper (required)3.6.1
    Flume1.9.0
    Impala2.10.0
    Kylin2.5.2
    Alluxio2.3.0
    Knox (required)1.2.0
    Kerberos1.15.0
    Hudi0.5.1
    Superset0.35.2
    Livy0.7.0
    TensorFlowSpark1.4.4
    Jupyter4.6.3
    Kudu1.12.0