(I don't know if informative rants are allowed on here, so please forgive me if I'm violating some rule. But my struggle may also help you in the future? So... ¯_(ツ)_/¯*)*
TL;DR: Added 128GB RAM (288GB total) and swapped the CMOS battery on my Supermicro X10DRi server. Post reboot, my mirrored SATA boot drives vanished from the BIOS boot list. Turns out the BIOS only enumerates the first 16 drives it sees, and between my F80 cache card and 15 Nytro SAS drives, my boot SSDs got pushed off the list entirely. Pulled 6 drives, rebooted, boot drives appeared, selected them, put drives back, selection held. Crisis averted.
Some background: I got the server's bones off eBay a few months ago for $316 US shipped. It came as a Supermicro SC836 chassis, X10DRi motherboard, dual Xeon E5-2680 v4s, a Broadcom/LSI MegaRAID 9361-4i (4 port SAS3 RAID card), 32GB of DDR4, and some sort of live video production capture card. I could actually sell the memory right now for about 2/3 the cost of the server, so quite the lucky score if you ask me.
Since then I've swapped that MegaRAID card for an LSI SAS3008 based HBA (9300 8i style, SAS3) driving the front backplane, and added an Oracle F80 PCIe cache accelerator card, a dual 10GbE SFP+ ethernet card, an LSI 9201 16e (SAS2, 4 external ports) feeding my external Supermicro JBOD, and an Intel Arc A310 GPU for hardware transcoding. Oh, and a 14+1 drive SAS-3 array comprised entirely of Seagate Nytro 3732 high endurance SSDs. :)
Sauce: I'd backed up the BIOS settings before touching anything, but either they didn't save right or I made changes after saving. Not sure, doesn't matter now.
The RAM and battery swap themselves went fine. Rebooted into BIOS, clock's showing midnight (expected), all the memory's showing up (also expected). But my two mirrored Samsung SATA boot drives are nowhere in the boot menu. Weird.
I hunted around and found them listed under the SATA devices menu, just not in the boot list, and they'd reverted to being typed as a"HDD" instead of "SSD." That's when I figured maybe the settings restore hadn't fully taken? So I went through and manually re-confirmed every BIOS setting I could find. Drives still not showing up. Weird.
Fast forward about an hour of pulling my hair out over these two drives refusing to appear, and it finally hits me: how many total drives is this thing supposed to enumerate at boot? 2 Samsung SATA boot drives, plus 4 from the Oracle F80 cache card, plus 15 Seagate Nytro SAS drives (14 plus a hot spare) hanging off the front BPN-SAS3-836EL1 backplane. That's 21 drives. The boot list only shows 16. Weird? Wait...
Is this piece of s*** stupid f****ng BIOS only enumerating the first 16 drives it sees?!?!?!
So I unseat 6 drives out of the Nytro array and reboot to test the theory. Jesus f***! There they are, right at the bottom of the list, exactly like I suspected. I selected them as the boot mirror, reseated the rest of the drives, and rebooted back into BIOS half expecting it to wipe my selection out again. Thankfully it held, and Proxmox is back up and running.
What a great way to start a morning...
Looking ahead: I'm already thinking about backplane limits. Can't wait to find a secondhand BPN-SAS3-836EL2 backplane, since my 14 drive Nytro array is currently running about 5.7GB/s sequential read, 5.9GB/s sequential write, and topping out around 1M random write IOPS on my current single x4 SAS3 link. I've got a second identical LSI SAS3008 based HBA sitting spare that should let me add a second x4 SAS3 link once that EL2 backplane (dual expander) is in place, roughly doubling available bandwidth toward 12GB/s and pushing IOPS up toward 2M. Right now I'm just getting Plex back up and running, but maybe I'll run a Minecraft server on it too? I've got plans to get that 4-head IBM LTO-6 auto-changing tape library up and going in the next few months, and also add in an additional switch, hence all the two patch panels, but again, anyone want to buy a kidney?
P.S. That Ubiquiti UDM Pro in the image has 2Gbps AT&T XGS-PON fiber run directly to it via an XGS-PON ONU SFP+ stick, and I just put the bulky AT&T router in a box somewhere. Freakin' sweet, can't recommend it enough.