GPU Usage Shows 100%

Last updated: 2018-11-26 18:05:50

PDF

Problem Description

When using a GPU computing instance, if you use nvidia-smi to view the GPU status in the system, the GPU usage may be displayed as 100% while no processes are using GPU.

Possible Cause

This may be caused by the ECC Memory Scrubbing mechanism used when the instance loads the NVIDIA driver.

Solution

Run the nvidia-smi -pm 1 command in the instance system to get the GPU Driver into the Persistence mode.

Procedure

  1. Log in to the GPU computing instance and run the following command:
    nvidia-smi -pm 1
  2. Run the following command to check GPU usage:
    nvidia-smi
    Here's a screen indicating that the GPU usage is normal: