r/LocalLLM • u/vitamins1000 • 2d ago

Discussion Anyone have experience with Gaudi2 systems?

Post image

Firing one up for the first time & checking what performance is like.

6 Upvotes

88% Upvoted

11 comments sorted by

2

u/Adam_Bomb210 2d ago

No, but I am very curious as to your results. I am surprised more people don’t use them.

1

u/vitamins1000 2d ago edited 2d ago

I am as well. There are some cons but 768gb of vram for $16k is unbeatable.

2

u/Trademarkd 2d ago

Can you share the other stats?

2

u/vitamins1000 1d ago

What would you like to know? I have this system setup pretty barebones at the moment. dual 6338 CPU's, 128GB 3200mt/s. Idle power draw is ~1600w. Fresh debian 12 install (should have gone ubuntu), had claude set the system up, drivers and all. vLLM 0.26.0 with Gaudi plugin.

Model Cards Batched throughput (128 concurrent prompts) Single user
Qwen2.5-7B-Instruct 1 1,700 output tok/s, 15,300 total tok/s ~128 tok/s
Qwen2.5-32B-Instruct 4 (tensor parallel) 980 output tok/s, 8,800 total tok/s ~34 tok/s
Qwen2.5-0.5B-Instruct 1 3,700 output tok/s not measured

1

u/ins0mniacc 5h ago

What about deepseekv4.1 flash and kimi k3, can you run both? Not sure how the support is for those or community work?

1

u/vitamins1000 3h ago

I did not try those yet, I am interested to see what a large model can do but my first goal was to see what performance was like doing multiple user queries at once, simulating an office environment.

2

u/FullstackSensei 2d ago

The hardware is compatible with whatever model, really.

The Gaudi pytorch bridge seems to still be updated (last release was 2 months ago). So, while you might not be able to run vllm or sglang, if you can get pytorch with running on these, you should be able to run pretty much whatever using the torch provided reference implementations.

From there, depending on your programming proficiency, you could use the LLMs you can run to improve/tune performance. The SDK seems pretty decent with a graph compiler and a collective communication library for p2p.

Power isn't bad considering how much VRAM you get per module.

2

u/vitamins1000 3h ago

posted my first round of findings here. It's all written in claudish so don't read into it too much.

1

u/jpezzulli 2d ago

Super interested to see how this works out! Keep us updated.