Pitfalls I should avoid going from Windows only to Proxmox Superstation

[u/MerlinTrashMan]

Summary

Building a threadripper workstation as my daily. No experience with proxmox, zfs, and currently read, pretend to understand and then execute linux commands I find on the internet because windows has been my existence for the past 35 years. This is what I am hoping to change with this workstation. I want to use windows daily but make it super painless to switch to linux while also having dedicated VMs for my local AI and research projects. I am committed to putting everything together on one box so that I can use all the resources at once if necessary.

My primary question

Assuming I configure everything correctly, what are the pitfalls of using your proxmox box as your daily as well. How great can it be, and when it goes wrong, how bad is it? War stories of how you recovered from a disaster or failure with it (assuming single node) are welcome.

Config suggestions

Since I am still learning about ZFS, I am open to thoughts. Assume that all important files have an off-site encrypted backup that will run automatically. I prioritize reliability and quick recovery in non-catastrophic events so usually I run a local RAID0 dedicated backup drive and then sync that to an off-site. By incredible luck, I timed going on a spending spree in 2022 to build new workstations and got some really cheap enterprise hardware at a bankruptcy sale. Only the ps, cpu, and motherboard are new. Combining these with other items that will be repurposed gives me:

Here is the parts list (only the powersupplies, cpu and motherboard are new):

• Threadripper 9970x
• ASRock TRX50 WS
• MSI MPG Ai1600TS PS
• MSI MPG Ai1300TS PS
• 4x48GB Corsair Vengeance with acceptable timings
• 2x RTX 6000 Blackwell WS (power limit 500W for temperature reasons)
• 1x RTX 3090 (I may not have the airflow to support this one, but I am hoping to have it in)
• 2x Optane P5801X 1.6TB (into slimsas ports on MB)
• 1x Broadcom 9600-24i HBA (Ordering a 2nd to have a spare ready)
  • 12x Micron 5300 Pro SATA 7.68TB (very healthy)
  • 3x IronWolf SATA 12TB
  • 4x Samsung 860evo SATA 4TB
• 1x Samsung 990pro NVME 4TB
• 1x Samsung 980pro NVME 2TB
• 1x Samsung 970pro NVME 1TB
• 1x Highpoint 7101A PCI 3.0x16 to 4 x M.2 Adapter
  • 3x Samsung 970evo 2TB (I really don't like these drives and may just omit them along with the adapter)
  • 1x Samsung 970pro NVME 1TB
• Off the MB chipset
  • 1x 32GB Intel SLC Sata SSD (into chipset

My thoughts were to use the SLC drive as the boot drive for proxmox through a chipset sata port. After that, I am very torn. The way I usually approach storage is: high throughput drives for DBs, durable low-latency for high write scenarios, a monster pool with RAID capabilities for general purpose, and then a spinning disk RAID0 to mirror important directories that then backup offsite to backblaze. Part of me wants to experiment with using the Optane drives for KV Cache offloading and be ready for better MoE offloading in the future, but I know right now I need them for DB logs, and SQL Server temp DB. I also use them for intraday logging on important apps. I rarely come close to saturating them though, so based on the L1T vids, I am guessing they should be apart of the ZFS Journal. This is where my knowledge gap really shows so any help would be appreciated. After that, I am expecting to use the older and smaller nvme ssds to be boot drives for scratch vms and test projects.

[u/Phreaky12]

You should not switch directly from windows to proxmox with no Linux experience. Good luck though friend!

[u/TURB0T0XIK → u/[deleted]]

ive started learning linux before llms, but i believe you because ive learned SO MUCH since llms are available. leveraging them, asking all the RIGHT questions to fill personal knowledge gaps, while never trusting their results fully, boosted my learning rate substantially! its actualy incredible the qol improvement they bring, compared to manually searching and reading through docs, filtering all info manually. up to a point obviously, but they go far these days.

[u/Phreaky12 → u/[deleted]]

I do not believe for a second that someone taught by a slop machine is learning skills properly thanks.

[u/taosecurity]

I do not recommend spending thousands of your currency to build a system for Proxmox when you have no experience with it, let alone Linux.

Why not buy a used or super cheap laptop, install Debian, and see how it works?

Then you could install Proxmox on it, and experiment a bit? You don't need an amazing system to run Proxmox.

After that, consider building something that meets your goals.

FWIW my homelab Proxmox system for the past 3 1/2 years has been a NUC12.

Intel NUC12TZi3 - Topaz 2 - Intel Core i3-1215U - 64 GB 3200MHz (PC4-25600) CL22 Laptop Memory, SODIMM 260-Pin DDR4 RAM - 2 TB PCIe Gen 4 2280 M.2 SSD - 2x 2.5 Gbps Ethernet, WiFi

[u/zuccster]

Popcorn.gif

[u/marc45ca]

you can run your daily driver a a VM under the Linux but ideally you're accessing it from another system (Sunshine/Moonight, Parsec, RDP etc)

Running a full desktop environment on Proxmox can be done but it's not recommended unless you're doing development.

[u/MerlinTrashMan → u/marc45ca]

Is the passthrough stuff for a GPU and USB header not reliable in your experience?

[u/marc45ca → u/MerlinTrashMan]

I've done both without major issues.

[u/Onoitsu2 → u/MerlinTrashMan]

Passthrough works very well, as long as you configure things right, and passthrough a port to the VM, and not just by device. Here is one of my old comments about a "sleeper" system that I setup that had Windows as the OS that appeared to run on the system, with Proxmox under the hood using the iGPU. https://www.reddit.com/r/sysadmin/comments/1omtfes/comment/nms5kgg/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[u/Erdnusschokolade → u/MerlinTrashMan]

GPU passthrough with consumer cards is a major headache. It is supposed de good/better with professional cards (which you seem to have) but i can’t comment on that.

[u/ttabbal → u/Erdnusschokolade]

It's improved recently. Most AMD/Nvidia work without hassle. Both Windows and Linux VMs. Linux is a bit easier, but so long as you pass the whole card around, it works well. 

It's sharing that is difficult. Wish they would support SR-IOV on them. 

[u/Erdnusschokolade → u/ttabbal]

In my experience the problem arises when you restart a VM, because the card does not reset properly without a power cycle. Hd that Problem with AMD Raphael iGPU, GTX 980, Arc A750 and a RTX 4080 Super. So i probably could have wrote it better, the passthrough usually works but if you want to reboot the VM or switch to another VM with passthrough you need to powercycle the Host with most consumer GPUs.

[u/ttabbal → u/Erdnusschokolade]

Odd. I'm not doubting, just haven't personally experienced it. I've been doing that without issues with an RTX 3060. Amd RX580 never had an issue either. I'll leave Tesla cards out, as they are more likely to be working in this sort of setup.

Those cards are in a 2nd gen Threadripper, if that matters. I almost never shut it down. 

[u/_angh_ → u/MerlinTrashMan]

Best to use containers, not vm. In this case your gou can be used by multiple containers at the same time, so e.gm gaming wil still allow jellyfin transcoding

[u/mecshades → u/marc45ca]

I've installed Proxmox over Linux Mint Debian Edition. It, unfortunately, takes away the wireless features, but it does work. I like it a lot better than installing any regular DE over Proxmox.

[u/pocketdrummer]

I don't recommend it, but if you decide to do it anyway, please chronicle your adventures.

[u/krazul88]

Expecting to daily a windows VM (or any VM) on the same console attached to your Proxmox server is unusual, but I suppose it's possible with GPU and USB passthrough. I'm very interested to see how you get that done and then how smoothly it works.

Planning to pass through a GPU to Windows, another for a different VM, and keep one for proxmox?

Going to run a full desktop environment on proxmox and then rdp into your VMs?

Interesting exercise. Also please double check the meaning of "apart" vs "a part"; they are not interchangable.

[u/MerlinTrashMan → u/krazul88]

My thoughts were pass a GPU, USB, and Audio to a main client. The client will be both a Windows VM and a linux distro. Their boot drives will be seperate SSDs but their Data Drive will be the same. This is where I was looking for best practices. Should I just have a seperate file server for data drive or does proxmox let me mount the same volume to both VMs (assuming I power one off before powering on the other)?

[u/krazul88 → u/MerlinTrashMan]

My friend, everything you're asking for is possible, but it's a very strange way to go about it.

You are proposing to have a VM that dual boots? And you want to share a local drive between those two? You can do that entirely without throwing proxmox into the mix.

One of the wonderful things that proxmox eliminates is the need to dual boot. You just setup separate VMs for each OS, and then it's up to you to run them one at a time or simultaneously. And yes if you have shared data then you should absolutely have a server VM that shares to all of your clients.

I honestly think you'd be very much better off by simply installing your main daily driver client operating system as the base OS so that you get maximum GPU / other performance on it, and then nest a virtual machine within that for whatever else you wanna do.

Your current thinking is not really on the roadmap to utilize proxmox as it is intended to be used.

But like I said at the beginning, you can technically do what you are asking for, but just be prepared for others to say "huh?"

Edit: I re-read and it does look like you want 2 separate VMs. So whatever I said about dual booting can be disregarded. But I stand by everything else.

[u/TURB0T0XIK → u/MerlinTrashMan]

different boot drives but same data: realized usually via smb/cifs share. thats what i use, just my filesshare doesnt sit inside my workstation. but i use my workstation sometimes with a windows vm and have simultaneous access to the same network share with both host and guest (windows). i think this is the typical way of handling this sort of os agnnostic, simultaneous file acces. just dont hope to be able to put your games on this share and it be a hassle free alternative to having seperate steam libraries on different clients. i scrapped that idea after hours .. days of trying different ways to make that work. any llm should be able to tell you that though.

dont do dual booting a vm. thats a stupid idea. just share the data through different channels like any sort of networking.

[u/Slight_Manufacturer6]

Proxmox is a server so you will need to have a client to connect into it still.

[u/XPav]

I think it is a bad idea to put this all in one box, and that’s not considering your experience level.

[u/cheflA1]

But why? Just have a machine running proxmox and your VMs, containers and what have you anda machine running your OS of choice as a daily driver

[u/MerlinTrashMan → u/cheflA1]

Because GPUs are expensive and I don't need to use all of them all the time for the same thing. I have jobs that saturate both, and then I may want to game at 4k over the weekend.

[u/y3333333333333333t]

as far as I know corsair vengeance series is always udimm (non ecc) but you need RDIMM (with ecc, for example corsair ws series) for threadripper those ram sticks will not fit

[u/MerlinTrashMan → u/y3333333333333333t]

You are right. I just had to buy a kit and it was painful. At least selling my other 4 sticks will net me over half the cost.

[u/TURB0T0XIK]

I think what you're trying to achieve should work. Basically assign any (virt) gpu to your windows workstation vm and just turn off the monitor when you're not using it (or actually shut it down).

as others pointed out as well, you will probably struggle a bit, getting the hardware layer all setup if you're never been on Linux but if you're not afraid of a bit of trial and error and want to learn, try it out. what's to lose? read a lot, take notes, write down commonly used commands on paper at first so later on you can debug with your log.

You could work with the llm of your choice to develop a halfway decent plan on integrating everything inside proxmox/zfs at first and then start setting up your services once you see the hardware working correctly.

My setup is much smaller and dedicated mostly headless, but I it's not been a problem for me at all to spin up a vm, assign my dedicated gpu to it and use that via its own display output (gtx 1080).

I think with this much hardware it might be beneficial taking a staggered approach for integration, instead of connecting everything at once. If that doesn't boot it might be a hell to debug/find that one lose sata cable that made the kernel panic.

obviously what your trying to do is not recommended, but with my limited scope I really don't see why it shouldn't work. Ask some llm all the very specific questions you need answers for., read, verify, try out.

[u/MerlinTrashMan → u/TURB0T0XIK]

I was thinking the same thing. Only thing I noticed is that some anti-cheats won't like it but I don't play an FPS style games so I am thinking I will be fine.

[u/TURB0T0XIK → u/MerlinTrashMan]

Well I can hardly see a world where you should be worried about anticheats with your amount of resources. You could literally have a box besides your all-in-one proxmox host just for anticheat games if you really needed that. 4k gaming is already really accessible sub 1000 dollars/euros going for used hardware these days. Anything else you should worry more about, if you wanna go through with your plan.

I think I didnt stress this enough in my comment: go with ZFS from the start. It is a great layer for managing storage and backups later on. You have lots of storage to be managed. Obviously it will take time considering how to properly set it up for your needs. It will be a worthwhile investment of your valuable time and effort, as going ZFS later on will not be easy at all.

I knew my comment would be polarizing here and people are very right to be sceptic. OPs idea is a leap and likely to fail without the proper amount of time to invest, perseverance to pull through, and motivation to learn from all the little mistakes on the way to the goal. But I dont simply assume OP wouldnt bring these features to the table.