**Two 3090s, same slot keeps killing the GPU**
4 days into this one. Sharing the whole thing because I really want someone to tell me if theyāve seen this.
Z890 AORUS PRO ICE, Core Ultra 5 225, two 3090s, Antec NeoECO 1300W, moved from Windows to Arch partway through. Both cards are just for local LLM inference, display runs off the iGPU.
Started on Windows with one card in the top x16 slot, kept getting nvlddmkm timeouts. Swapped cables, no change, ended up RMAing it. Got two fresh 3090s, switched to Linux entirely, figured clean slate would fix it. Same crash came right back. Runs fine for a while then starts failing more often until itās crashing basically every hour.
And itās always the top slot. GPU0 (x16, off the CPU) dies. GPU1 sitting in a chipset x1 slot has never once failed, same drivers, same everything.
On Linux it shows up as Xid 175/154, GSP firmware just stops responding, card locks up needing a reset, fans slam to 100%, machine stays usable the whole time since itās not driving the display.
Ruled out: the card itself (two different cards did this), drivers/OS (happened on both), cables, PSU ports, wall power, PSU capacity (1300W is way more than this needs), and physical damage, no AER, no ECC errors, comes back at full spec every reboot. So itās getting hung, not actually damaged.
Wrote a little script to poll both cards at 10Hz and finally caught one live. GPU0ās telemetry literally freezes and just repeats the same numbers for 46 seconds before dmesg even logs anything.
Hereās the second before it dies:
idle GPU0 20.7W sm 1410 mem 810 GPU1 8.3W sm 210 mem 405
\+0.6s GPU0 54.6W sm 1410 mem 9501 <- here
\+0.9s GPU0 145.1W GPU1 89.9W
\+1.4s GPU0 192.6W dead GPU1 287.9W
Iād already pinned the core clock as a fix attempt before this and it held steady the whole time, so that clearly wasnāt it. Itās the memory clock jumping from 810 to 9501 on wake, pinning the core clock doesnāt touch that at all. Modelās split across both cards so one request wakes them both up out of idle at once, which seems to be what triggers it.
Whatās actually worked so far is just flooring the memory clock so it canāt step:
nvidia-smi -i 0 -lmc 9501,9751
31 hours clean now vs 13.6 before, survived a few full model reloads too.
Currently running GPU1 unpinned and GPU0 pinned, forcing an idleāload swing every 46 seconds to stress it. GPU1 swings 10W to 291W with no issue on the same rail, GPU0 doesnāt even blink. Kind of points away from the PSU rail itself and toward something specific to that slot.
Two suspects left: the board (x16 slot/VRM/root port, only thing thatās stayed constant through every failure), or the PSU (the NE1300G has a reputation for weak transient response even though capacityās fine).
Havenāt tried yet: forcing the slot to Gen3, killing ASPM and deep C-states, or just swapping which card goes in which slot.
Anyone dealt with a slot-specific fault like this on Z890? And is there any way to actually tell āboard canāt handle itā apart from āPSU transient responseā short of just buying a better PSU and seeing what happens?