Integrating Knox with Tez

Last updated: 2021-09-03 17:16:56

    This document describes how to integrate Knox with Tez, such as installing Tomcat and Tez UI, creating roles, configuring Timeline Server, configuring Tez, and starting services. 172.**.**.9 is the private IP of the master node, 159.**.**.70 is the public IP of the master node, and the Tez version is v0.9.2.

    Installing Tomcat and Tez UI

    cd  /usr/local/service
    tar -zxvf apache-tomcat-9.0.46.tar.gz
    mv /usr/local/service/apache-tomcat-9.0.46 /usr/local/service/tomcat

    Modify Tomcat port numbers:

    vim /usr/local/service/tomcat/conf/server.xml
    1. Change 8005 to 2020.
    2. Change 8080 to 2000.
    mkdir -p /usr/local/service/tomcat/webapps/tez-ui
    cp /usr/local/service/tez/tez-ui-0.9.2.war /usr/local/service/tomcat/webapps/tez-ui/
    cd /usr/local/service/tomcat/webapps/tez-ui
    unzip tez-ui-0.9.2.war
    vim ./config/configs.env

    Change localhost to the private IP of the current server.

    Creating roles

    vim /usr/local/service/knox/conf/topologies/emr.xml 

    Modify the emr.xml file:

    1. Add the following content:

    2. Add the following content:


    Configuring YARN Timeline Server

    Modify the yarn-site.xml file in configuration management, save the changes, and restart the components whose configuration has been changed.

    Parameter Value
    yarn.timeline-service.enabled true
    yarn.timeline-service.hostname 172.**.**.9 (replace it with your IP)
    yarn.timeline-service.http-cross-origin.enabled true
    yarn.resourcemanager.system-metrics-publisher.enabled true
    yarn.timeline-service.address 172.**.**.9:10201 (replace it with your IP)
    yarn.timeline-service.webapp.address 172.**.**.9:8188 (replace it with your IP.)
    yarn.timeline-service.webapp.https.address 172.**.**.9:2191 (replace it with your IP)
    yarn.timeline-service.generic-application-history.enabled true
    yarn.timeline-service.handler-thread-count 24

    Modifying Tez Configuration

    In the tez-site.xml file, add the following parameters, save the changes, and restart the components whose configuration has been changed.

    Parameter Value
    tez.tez-ui.history-url.base http://172.**.**.9:2000/tez-ui/ (replace it with your IP)
    tez.history.logging.service.class org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService

    Starting Services

    1. Start Timeline Server.
      /usr/local/service/hadoop/sbin/  start timelineserver  
    2. Start Tomcat.
    3. Restart Tez.
      su hadoop
      rm -rf  /usr/local/service/knox/data/deployments/*
      /usr/local/service/knox/bin/ stop
      /usr/local/service/knox/bin/ start
      /usr/local/service/knox/bin/ stop
      /usr/local/service/knox/bin/ start
    4. Access Tez UI at the following address.

      The account and password are the same as the server login account and password.

      https://{public IP of the cluster}:30002/gateway/emr/tez/