Deploying with Tencent Cloud EMR

Last updated: 2021-07-30 11:53:42

    The GooseFS-integrated Tencent Cloud EMR will be released in the latest version, which allows you to use GooseFS as an EMR component without having to deploy it for the EMR environment separately.

    For Tencent Cloud EMR clusters that have not integrated with GooseFS, you can refer to this document to deploy the EMR environment for GooseFS.

    First, select a framework that suits the production environment by referring to Cluster Deployment and Running and then deploy the cluster.
    After that, configure according to the GooseFS-supportive EMR component. This document describes the configuration for Hadoop MapReduce, Spark, and Flink supports.

    Hadoop MapReduce Support

    To allow Hadoop MapReduce jobs to read and write GooseFS data, add the GooseFS client’s dependent path to HADOOP_CLASSPATH in hadoop-env.sh. This can be performed in the EMR console as follows:

    Then, configure the HCFS implementation of GooseFS in core-site.xml in the EMR console.

    Configure fs.AbstractFileSystem.gfs.impl as follows:

    com.qcloud.cos.goosefs.hadoop.GooseFileSystem
    

    Configure fs.gfs.impl as follows:

    com.qcloud.cos.goosefs.hadoop.FileSystem
    

    After the configurations are delivered, you can restart YARN-related components for the configurations to take effect.

    Spark Support

    To allow Spark to access GooseFS, you also need to set spark.executor.extraClassPath to the GooseFS client’s dependency package in the spark-defaults.conf file as follows:

    ...
    spark.driver.extraClassPath ${GOOSEFS_HOME}/client/goosefs-x.x.x-client.jar
    spark.executor.extraClassPath ${GOOSEFS_HOME}/client/goosefs-x.x.x-client.jar
    spark.hadoop.fs.gfs.impl com.qcloud.cos.goosefs.hadoop.FileSystem
    spark.hadoop.fs.AbstractFileSystem.gfs.impl com.qcloud.cos.goosefs.hadoop.GooseFileSystem
    ...
    

    This configuration can also be done and delivered on the Spark component configuration page in the EMR console

    EMR’s Flink adopts the Flink on YARN deployment mode. Therefore, you need to ensure that fs.hdfs.hadoopconf is set to the Hadoop configuration path in ${FLINK_HOME}/flink-conf.yaml. For Tencent Cloud EMR clusters, the value is /usr/local/service/hadoop/etc/hadoop in most cases.

    You don’t need to configure other configuration items. You can use Flink on YARN to submit the Flink jobs, which can access GooseFS through gfs://master:port/<path>.

    Note:

    If Flink needs to access GooseFS, the master and port must be specified.

    Hive, Impala, HBase, Sqoop, and Oozie Support

    If the environment support for Hadoop MapReduce has been configured, components such as Hive, Impala, and HBase can be used directly without requiring separate configuration.