When using a GPU instance, if you use
nvidia-smi to view the GPU status in the system, the GPU usage may be displayed as 100% while no processes are using GPU, as shown below:
This may be caused by the ECC Memory Scrubbing mechanism used when the instance loads the NVIDIA driver.
nvidia-smi -pm 1 command in the instance system to get the GPU Driver into the Persistence mode.
nvidia-smi -pm 1
You will see the GPU usage is normal, as shown below: