Help & DocumentationElastic MapReduceEMR Development GuideSpark Development GuideSpark Environment Info

Spark Environment Info

Last updated: 2020-03-25 10:53:56

PDF

Currently, the Spark component in EMR supports Spark 2.0.2, 2.2.1, and 2.3.2. The software environment information is as follows:

  • By default, Spark is installed on a master node.
  • After the log-in, run the su hadoop command then you can switch to the Hadoop user.
  • The Spark software path is /usr/local/service/spark.
  • The relevant logs are stored in /data/emr.

For more information, please see the community documentation. Here is mainly about how to access COS through Spark.