Spark Environment Info

Last updated: 2019-07-26 17:45:47

PDF

Currently, EMR supports Spark version 2.1.0. The software environment information is as follows:

  • By default, Spark is installed on a master node.
  • After the log-in, run su hadoop command then you can switch to the Hadoop user.
  • The Spark software path is /usr/local/service/spark
  • The relevant logs are stored in /data/emr

For more information, see the community documentation. Here is mainly about how to access COS using Spark.