tencent cloud

COS Ranger Plugin extends the service types of the Ranger Admin console. Users can configure the COS-related permissions in the Ranger console.

Source code download

You can go to GitHub > ranger-plugin to obtain the source code.

Version

v1.1 or later

Deployment directions

1. Create a COS directory in the service definition directory in Ranger. Note that you should at least have execute and read permissions on the directory.
   a. In the Tencent Cloud EMR environment, the path is ranger/ews/webapp/WEB-INF/classes/ranger-plugins.
   b. In the self-built Hadoop environment, you can search the path of the Ranger-integrated components (such as HDFS) to find the path.
   
2. Place cos-chdfs-ranger-plugin-xxx.jar (with at least the r permission) and cos-ranger.json files in the COS root directory. You can get them from GitHub.
3. Restart Ranger.
4. Register COS service in Ranger. A command sample is as follows:
   
   ## Create the service. The Ranger admin account and password, as well as the Ranger service address should be specified.
   ## For an EMR cluster, the admin user is root, and the password is the root password set when the EMR cluster is created. Replace the IP of the Ranger service with the master node IP of EMR.
   adminUser=root
   ## Password set during EMR cluster creation, which is also the login password of the Ranger web service.
   adminPasswd=xxxxxx
   ## If the Ranger service has multiple master nodes, select any of them.
   rangerServerAddr=10.0.0.1:6080
   ## Specify the .json file in step 2 as -d in the command.
   curl -v -u${adminUser}:${adminPasswd} -X POST -H "Accept:application/json" -H "Content-Type:application/json" -d @./cos-ranger.json http://${rangerServerAddr}/service/plugins/definitions
   ## To delete a defined service, for example, the service created above, pass the service ID that is returned when you created the service.
   serviceId=102
   curl -v -u${adminUser}:${adminPasswd} -X DELETE -H "Accept:application/json" -H "Content-Type:application/json" http://${rangerServerAddr}/service/plugins/definitions/${serviceId}

5. When the service is created successfully, you can view the COS service in the Ranger console, as shown in the following figure:
   
6. Click the + icon on the right of the COS service to define the service instance. On the Edit Service page, customize the Service Name, for example, cos or cos_test, as shown below:
   
   Where, policy.grantrevoke.auth.users needs to be set to the name of the user that is used to launch COS Ranger Service (i.e., the user that is allowed to pull permission policies). We recommend you set it to hadoop, which can be used as the username to launch COS Ranger Service in subsequent operations.
7. Click the newly created COS service instance.
   
   Add a policy as shown below:
   
8. On the page that is displayed, configure the following parameters:

• Bucket: Bucket name, for example, examplebucket-1250000000. It can be queried in the COS console.
• Path: Path of the COS object. Note that it does not start with a slash (/).
  • include: Indicates whether the permission applies to the specified path, or other paths except for the specified path.
  • recursive: Indicates the permission applies to the specified path as well as the subpaths (i.e., the recursive subpaths) under it. It is usually used when the path is set to a directory.
• Select Group/Select User: Username and user group in logical OR relationship; that is, the operation is authorized as long as the username or user group condition is met.
• Permissions:
  • Read: Permission on operations such as downloading objects and querying the object metadata. It corresponds to the GET and HEAD operations in COS.
  • Write: Permissions on operations such as uploading objects. It corresponds to the PUT operation in COS.
  • Delete: Delete permission. It corresponds to the delete object operation in COS. To rename a path in Hadoop, you need to have delete permission on the original path and write permission on the new path.
  • List: Traverse permission. It corresponds to the List Object operation in COS.
    

COS Ranger Service is the core of the permission system. It integrates with the Ranger client to receive the authentication requests, token generation and lease renewal requests, and temp key generation requests of the Ranger client. It is also where the sensitive information (Tencent Cloud keys) are stored. Usually, it is deployed on the jump server, and only the cluster admin can perform operations and query the configurations.

COS Ranger Service supports the one-master, multiple-slave HA deployment. DelegationToken can be stored to HDFS, and the master and slave nodes can be determined by ZooKeeper lock obtaining. Then, the master node will write the address to Zookeeper so that COS Ranger Client can perform address routing.

Source code download

You can go to GitHub > cos-ranger-server directory to obtain the source code.

Version

v5.0.6 or later

Deployment directions

1. Copy the code of COS Ranger Service to several nodes of the cluster. In the production environment, the code should be copied to at least two nodes (one master node, and one slave node). As sensitive information is involved, we recommend you use jump servers or nodes with tight permission control.
2. Modify the relevant configuration items in the cos-ranger.xml file. Those that must be modified are as follows. For the description of the configuration items, see the comments in the file. You can get the configuration file in the cos-ranger-service/conf directory at GitHub.

• qcloud.object.storage.rpc.address
• qcloud.object.storage.status.port
• qcloud.object.storage.enable.cos.ranger
• qcloud.object.storage.zk.address (ZooKeeper address. After COS Ranger Service starts, it will be registered in ZooKeeper)
• qcloud.object.storage.cos.secret.id
• qcloud.object.storage.cos.secret.key

3. Modify the relevant configuration items in the ranger-cos-security.xml file. Those that must be modified are as follows. For the description of the configuration items, see the comments in the file. You can get the configuration file in the cos-ranger-service/conf directory at GitHub.

• ranger.plugin.cos.policy.cache.dir
• ranger.plugin.cos.policy.rest.url
• ranger.plugin.cos.service.name

4. In the start_rpc_server.sh file, modify the configuration of hadoop_conf_path and java.library.path, which corresponds to the directory of the Hadoop configuration files (for example, core-site.xml and hdfs-site.xml) and hadoop native libraries, respectively.
5. Run the following command to launch the service:
   

   chmod +x start_rpc_server.sh
   nohup ./start_rpc_server.sh &> nohup.txt &
   

6. If the launch failed, check whether an error message is contained in the error log.
7. COS Ranger Service supports displaying the HTTP port status (port name: qcloud.object.storage.status.port. Default value: 9998). You can run the following command to obtain the status information, such as whether the leader is contained, and the authentication statistics:
   

   # Replace `10.xx.xx.xxx` with the IP address of the device deployed with the ranger service.
   # Replace `9998` in the command with the value of `qcloud.object.storage.status.port`.in the configuration file.
   curl -v http://10.xx.xx.xxx:9998/status
   

• If only one COS Ranger Service node is deployed, you can see that the current node becomes the leader in the above API response.
• If multiple COS Ranger Service nodes are deployed, you can see that another node becomes the leader in the above API response. After all nodes are restarted, you can see that the first restarted node becomes the leader.

COS Ranger Client is dynamically loaded by the Hadoop COSN plugin. It is a proxy that encapsulates all COS Ranger Service access requests, such as obtaining temp keys, obtaining tokens, and performing authentication.

Source code download

You can go to GitHub > cos-ranger-client to obtain the source code.

Version

v3.8 or later

Deployment directions

1. Copy the JAR packages of COS Ranger Client and COS Ranger Interface to the same directory as COSN (in the /usr/local/service/hadoop/share/hadoop/common/lib/ directory generally). Be sure to select JAR packages whose major version is the same as that of Hadoop and make sure that them have the read permission.
2. Add the following configuration in core-site.xml:
   
   <configuration>
             <!--*****Required Configuration********-->
             <!-- ZooKeeper address, from which the client can obtain the ranger-service address -->
             <property>
                 <name>qcloud.object.storage.zk.address</name>
                 <value>10.0.0.8:2181,10.0.0.9:2181,10.0.0.10:2181</value>
             </property>
              <!--***Optional Configuration****-->           
             <!-- Set the Kerberos credential used on the COS Ranger Service side. It should be the same as that configured on the COS Ranger Service side. If verification is not involved, you can skip this configuration. -->
             <property>                
                    <name>qcloud.object.storage.kerberos.principal</name>
                    <value>hadoop/_HOST@EMR-XXXX</value>
             </property>
            <!--***Optional Configuration****-->  
           <!-- IP address path of the Ranger server recorded in ZooKeeper. The default value is used herein. The value must the same as that configured in COS Ranger Service. -->
            <property>              
                   <name>qcloud.object.storage.zk.leader.ip.path</name> 
                   <value>/ranger_qcloud_object_storage_leader_ip</value>
            </property>
           <!-- IP address path of the COS Ranger Service follower recorded in ZooKeeper. The default value is used here. The value must the same as that configured in COS Ranger Service. The master and slave nodes provide the service at the same time. -->
            <property>
                      <name>qcloud.object.storage.zk.follower.ip.path</name>
                      <value>/ranger_qcloud_object_storage_follower_ip</value>
            </property>
   </configuration>
   

Version

v8.0.1 or later

Deployment directions

For detailed directions on the deployment of COSN, see Hadoop Tool. Note the following:

1. If Ranger is used, the key information of fs.cosn.userinfo.secretId and fs.cosn.userinfo.secretKey does not need to be configured. COSN will obtain the temp key via COS Ranger Service.
2. fs.cosn.credentials.provider needs to be set to org.apache.hadoop.fs.auth.RangerCredentialsProvider so that the verification and authentication can be performed via Ranger. The following is an example:
   
   <property>
          <name>fs.cosn.credentials.provider</name>
          <value>org.apache.hadoop.fs.auth.RangerCredentialsProvider</value>
   </property>