NVIDIA GPU Fan - Pwr:Usage - ERR!1. NVIDIA GPU Fan - Pwr:Usage - ERR!2. NVIDIA GPU 硬件故障2.1. 检查物理供电线缆2.2. 重新插拔显卡2.3. 重启服务器2.4. 调换插槽与线缆 (交叉测试)References1. NVIDIA GPU Fan - Pwr:Usage - ERR!rootyongqiang:~# nvidia-smi Mon Nov 25 17:38:06 2019 ----------------------------------------------------------------------------- | NVIDIA-SMI 418.43 Driver Version: 418.43 CUDA Version: 10.1 | |--------------------------------------------------------------------------- | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | || | 0 GeForce RTX 208... On | 00000000:1B:00.0 Off | N/A | |ERR! 39C P0 ERR! / 250W | 0MiB / 10989MiB | 0% Default | --------------------------------------------------------------------------- | 1 GeForce RTX 208... On | 00000000:1C:00.0 Off | N/A | | 22% 31C P8 24W / 250W | 34MiB / 10989MiB | 0% Default | --------------------------------------------------------------------------- | 2 GeForce RTX 208... On | 00000000:1D:00.0 Off | N/A | | 22% 29C P8 3W / 250W | 0MiB / 10989MiB | 0% Default | --------------------------------------------------------------------------- | 3 GeForce RTX 208... On | 00000000:1E:00.0 Off | N/A | | 22% 29C P8 11W / 250W | 2279MiB / 10989MiB | 0% Default | --------------------------------------------------------------------------- | 4 GeForce RTX 208... On | 00000000:89:00.0 Off | N/A | | 22% 29C P8 21W / 250W | 2387MiB / 10989MiB | 0% Default | --------------------------------------------------------------------------- | 5 GeForce RTX 208... On | 00000000:8A:00.0 Off | N/A | | 22% 28C P8 21W / 250W | 537MiB / 10989MiB | 0% Default | --------------------------------------------------------------------------- | 6 GeForce RTX 208... On | 00000000:8B:00.0 Off | N/A | | 22% 25C P8 1W / 250W | 209MiB / 10989MiB | 0% Default | --------------------------------------------------------------------------- | 7 GeForce RTX 208... On | 00000000:8C:00.0 Off | N/A | | 22% 24C P8 18W / 250W | 10MiB / 10989MiB | 0% Default | --------------------------------------------------------------------------- ----------------------------------------------------------------------------- | Processes: GPU Memory | | GPU PID Type Process name Usage | || | 1 55182 C detection_worker_gpu1 12MiB | | 1 55203 C detection_worker_gpu1 12MiB | | 3 54733 C detection_worker_gpu3 1136MiB| | 3 54752 C detection_worker_gpu3 1132MiB | | 4 54823 C detection_worker_gpu4 1188MiB | | 4 54842 C detection_worker_gpu4 1188MiB | | 5 55057 C detection_worker_gpu5 514MiB | | 5 55282 C detection_worker_gpu5 12MiB | | 6 55082 C detection_worker_gpu6 199MiB | ----------------------------------------------------------------------------- rootyongqiang:~#2. NVIDIA GPU 硬件故障GPU 0 风扇转速 (Fan) 和功耗 (Pwr:Usage) 均显示为ERR!。2.1. 检查物理供电线缆关机并断开电源。打开机箱拔掉 GPU 0 上的 PCIe 供电线。检查接口是否有烧焦、熔化或变色迹象。重新用力插紧供电线确保卡扣锁死。2.2. 重新插拔显卡PCIe 插槽接触不良会导致数据传输错误。将 GPU 0 从主板上拆下。清理 PCIe 金手指和主板插槽的灰尘重新紧固地插回插槽。2.3. 重启服务器重启服务器直接运行sudo reboot。2.4. 调换插槽与线缆 (交叉测试)将 GPU 0 和 GPU 1 的供电线互换将 GPU 0 和 GPU 1 的 PCIe 插槽互换如果错误跟随 GPU 0 移动说明是显卡硬件损坏。如果错误留在原插槽或原线缆说明是主板插槽或电源线材故障。请联系供货方技术支持。对于 AMAX 供货方来说是已知问题换卡是目前已知的方案。References[1] Yongqiang Cheng (程永强), https://yongqiang.blog.csdn.net/[2] NVIDIA GPU Fan - Pwr:Usage - ERR!, https://yongqiang.blog.csdn.net/article/details/103244347
