tencent cloud

Taking the essence of the deep learning task into account, Job node deployment is used as an example in this document. For more information on how to deploy a Job, please see Job Management.
The following is the example of deployment in the console:

1. In the Volume (optional) configuration item, select Using NFS disk and enter the name and IPv4 address of the CFS file system created previously as shown below:
   
2. In the Mount Target configuration item in Containers in the Pod, select the volume and configure the mount target as shown below:
   
   
   >!
   >- As the dataset may need to be downloaded online, you need to configure the public network access for the cluster. For more information, please see Public Network Access.
   >- After selecting a GPU model, when setting the request and limit, you need to assign the container CPU and memory resources meeting the resource specifications. The actual values do not need to be accurate down to the ones place.
   When configuring in the console, you can also delete the default configuration and leave it empty to configure "unlimited" resources, which also have the corresponding billing specifications. This approach is recommended.
   >- The container running command is inherited from Docker's CMD field, whose preferred form is exec. If you do not call the shell command, there will be no normal shell processing. Therefore, if you want to run a command in the shell form, you need to add "sh" and "-c" at the beginning.
   When you enter multiple commands and parameters in the console, each command should take a line (subject to the line break)

You can also use a YAML file to create a task.

1. Prepare a YAML file. Below is the sample file gpu_pod.yaml:
   
   apiVersion: v1
   kind: Pod
   metadata: 
       name: tf-cnn
       annotations: 
       #eks.tke.cloud.tencent.com/cpu: "8"
       #eks.tke.cloud.tencent.com/gpu-count: "1"
       eks.tke.cloud.tencent.com/gpu-type: T4
       #eks.tke.cloud.tencent.com/mem: 32Gi
   spec: 
       containers: 
     - name: tf-cnn
       image: hkccr.ccs.tencentyun.com/carltk/tensorflow-model:latest # Training task image
       env:  
       - name: MODEL_DIR
         value: /tf/model
       - name: DATA_DIR
         value: /tf/data
       command: 
         - "sh"
         - "-c"
         # Script that triggers the training task
         - "python3 official/vision/image_classification/mnist_main.py \ 
           --model_dir=$MODEL_DIR <br>        --data_dir=$DATA_DIR <br>        --train_epochs=5 <br>        --distribution_strategy=one_device <br>        --num_gpus=1 <br>        --download"
       resources: 
         limits: 
           #cpu: "8" 
           #memory: 32Gi
           nvidia.com/gpu: "1" 
         requests: 
           #cpu: "8"
           #memory: 32Gi
           nvidia.com/gpu: "1" 
       volumeMounts: 
       - name: tf-model-cfs
         mountPath: /tf
       volumes:  
     - name: tf-model-cfs   # Persistently store the training results to CFS
       nfs:  
         path: /            # Enter the root directory of the CFS file system here
         server: 10.0.1.8   # Enter the IPv4 address of the created CFS file system
       restartPolicy: OnFailure

2. Run the following command to complete deployment:
   

   kubectl create -f [yaml_name]
   

   
   

   Note

   In addition to the precautions mentioned above for directions in the console, you also need to pay attention to the following:

   • You need to use annotations to declare resource assignment in the YAML file. For more information, please see Annotation. You also should note that different GPU models correspond to different CPU and memory options. We recommend you enter the values as needed.
   • Here, NFS is used as the data volume. If you want to use other data volumes for persistent storage, please see Instructions for Other Storage Volumes.
   • You can reserve eks.tke.cloud.tencent.com/gpu-type only with no other items needed in annotations. If /gpu-count is specified, then cpu and mem must also be specified. (In this document, we recommend you not add other items, which will not affect the actual effect. If you enter other items without following the specifications, OOM errors may occur.)
   • For nvidia.com/gpu in GPU scheduling, only limits is required. If only annotations is specified, an error will be reported that no cards are found. If only limits is specified, its values will be considered as the request. If request is also specified, its value must be the same as that of limits. For more information, please see Schedule GPUs (here, adding the cpu and memory settings in request and limits is also not recommended as detailed above).

After creating a Job, you will be redirected to the Job management page by default. You can also enter the page as follows:

1. Log in to the TKE console and click Elastic Container > Elastic Cluster on the left sidebar.
2. In the elastic cluster list, click the ID of the cluster whose events you want to view to enter the cluster management page.
3. Select Workload > Job and click the newly created Job in the Job list.
   • Select the Event tab to view events as shown below:
     
   • Select the Log tab to view logs as shown below:
     
     

You can run commands to view events or logs:

• Run the following command to view events:
  

  kubectl describe pod [name]
  

  
  See the figure below:
  
• Run the following command to output logs continuously:
  

  kubectl logs -f [pod_name]
  

  
  See the figure below:
  
  
  As EKS containers will be terminated after use, you can view logs only when the Pod is in Running status. For the solution, please see Log Collection.

Viewing storage

If you have configured NFS as instructed above, you can go to the mount target to view NFS storage:

1. Run the following command to enter the relevant mount directory to check whether it exists:
   

   cd /mound_data
   

   
   See the figure below:
   
2. Enter the model directory and view whether there is relevant data in it as shown below:
   
3. Enter the data directory and view whether there is relevant data in it as shown below: