By default, Hadoop stores YARN job logs in HDFS. Tencent Cloud EMR also provides the ability to store YARN job logs in external storage (COS).
The EMR cluster needs to support COS. For more information, please see Analyzing Data in HDFS/COS with API.
yarn-site.xml
and deliver the configuration to all nodes.yarn.nodemanager.remote-app-log-dir=cosn://[bucket_name]/[logs_dirs]
core-site.xml
and deliver it to all nodes.fs.AbstractFileSystem.cosn.impl=org.apache.hadoop.fs.cosnative.COS
nodemanager/datanode
services in the cluster.hive/spark
job to view the job logs stored in COS.hdfs dfs -ls cosn://[bucket_name]/[logs_dirs]
Was this page helpful?