I. `nvidia-smi` — NVIDIA System Management Interface#

Overview: nvidia-smi is the NVIDIA System Management Interface (系统管理接口). It is used to monitor and manage GPU device status, including GPU memory usage, GPU utilization, temperature, and power consumption.

1. Show Overall GPU Status#

1
nvidia-smi

Most commonly used to quickly check whether GPUs are idle or busy.

2. Monitor in Real Time#

Refresh every second:

1
nvidia-smi -l 1

3. List All GPUs#

1
nvidia-smi -L

4. Show Processes Using GPUs#

pmon = process monitor

1
nvidia-smi pmon

Refresh every 2 seconds:

1
nvidia-smi pmon -d 2

1) `pmon` Column Reference#

Column	Full Name	Meaning
pid	Process ID	The Linux process ID using the GPU
type	Process type	GPU workload type: C = Compute (CUDA), G = Graphics, C+G = Both
sm	Streaming Multiprocessor utilization	Percentage of GPU compute cores being used by the process
mem	Memory controller utilization	Percentage of GPU memory bandwidth used by the process
enc	Encoder utilization	Usage of the NVENC video encoder
dec	Decoder utilization	Usage of the NVDEC video decoder
jpg	JPEG engine utilization	Usage of the hardware JPEG decoder/encoder
ofa	Optical Flow Accelerator utilization	Usage of the hardware optical-flow engine (video/vision tasks)
fb	Frame Buffer memory	Amount of GPU VRAM used by the process (in MB)
ccpm	Compute & Copy Engine / Protected Memory	Internal GPU engine / protection state info; often 0 on most systems

Key columns to watch: pid — shows which process is using the GPU. sm — indicates whether GPU cores are actively computing. fb — VRAM usage in MB; shows how much memory is consumed. mem — memory bandwidth utilization; indicates I/O pressure on GPU memory.

2) Example Output#

1
[xli49@ghpc008 ~]$ nvidia-smi pmon -i 0 -s um
2
# gpu         pid   type     sm    mem    enc    dec    jpg    ofa     fb   ccpm    command
3
# Idx           #    C/G      %      %      %      %      %      %     MB     MB    name
4
    0          -     -      -      -      -      -      -      -      -      -    -
5
    0          -     -      -      -      -      -      -      -      -      -    -
6

7
[xli49@ghpc008 ~]$ nvidia-smi pmon -i 0
8
# gpu         pid   type     sm    mem    enc    dec    jpg    ofa    command
9
# Idx           #    C/G      %      %      %      %      %      %    name
10
    0          -     -      -      -      -      -      -      -    -
11
    0          -     -      -      -      -      -      -      -    -

All dashes (-) indicate GPU 0 is currently idle — no processes are running on it.

5. Custom Query of GPU Information#

1
nvidia-smi --query-gpu=name,memory.used,utilization.gpu --format=csv

Commonly used for scripts, logging, and automated monitoring.

6. Log GPU Status to a File#

1
nvidia-smi -l 5 -f gpu.log

Records GPU information every 5 seconds and appends it to gpu.log.

💡 One-line Takeaway
Use nvidia-smi for a quick snapshot, nvidia-smi pmon to watch per-process GPU activity in real time, and --query-gpu for scriptable, structured output — focus on pid, sm, fb, and mem for the most actionable signals.

I. nvidia-smi — NVIDIA System Management Interface#