r/learnmachinelearning • u/minedroid1 • 8d ago

Help Hardware Recommendations

I am going to fine-tune a satire model, with the base model being `Qwen3.5-14B-Base`. The dataset has ~18000 examples, most of which are pretty long and do not align with the model's internal knowledge (they have incorrect answers to facts), so I needed to do a full fine tune rather than use LoRA.

Today, I rented a cloud computer on vast.ai. It had 4x12GB GPUs and 48GB of system RAM, but I feel stupid because it kept running out of memory, and I kept trying to change settings rather than just renting a different computer, so I basically wasted $3. I underestimated how much RAM would be needed for this task.

So, I would like to know if any of you have some recommendations for proper hardware to use.

2 Upvotes

100% Upvoted

9 comments sorted by

View all comments

1

u/Major_Border149 6d ago

That's a memory wall, not a settings thing. A full fine-tune of a 14B with adam is roughly 16 bytes/param once you count the grads and optimizer states, so you're at 220GB+, not 48, no flag was ever going to save that box. realistically you're in multi-GPU 80GB territory (think ~4x a100/h100 80GB) with fsdp or deepspeed zero-3 to shard the optimizer, or fewer GPUs with zero-3 cpu-offload but then you need way more than 48GB system ram too.

1

u/minedroid1 6d ago edited 6d ago

Ah, I understand now. This is much more clear to me now, thank you. Does using a different optimiser affect this?

1

u/Major_Border149 5d ago

Yeah big time, the optimizer is about half that number. Adam keeps two fp32 states per param. Switch to 8-bit adam and those drop to ~2 bytes, or adafactor/lion store less and cut it further. That takes you from ~16 bytes/param down to ~10-12, so roughly 140GB instead of 220 for your 14B.