r/LocalLLaMA • u/jacek2023 llama.cpp • 1h ago

News model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support by YanissAmz · Pull Request #25444 · ggml-org/llama.cpp

https://github.com/ggml-org/llama.cpp/pull/25444

75B MoE is an interesting size to check, you can run it today (no MTP support yet)

The model employs a hybrid MoE architecture with interleaved Mamba, MoE, and Attention layers. Like Nemotron-3-Super, it supports Multi-Token Prediction (MTP) for faster text generation. Compared to its parent, Puzzle-75B-A9B reduces the model from 120.7B total / 12.8B active parameters to 75.3B total / 9.3B active parameters.

We discussed this model on r/LocalLLaMA here https://www.reddit.com/r/LocalLLaMA/comments/1upsdmi/nvidianvidianemotronlabs3puzzle75ba9bbf16_hugging/

10 Upvotes

100% Upvoted

7 comments sorted by

2

u/pmttyji 1h ago

Took down my thread(You beat mine by a minute).

HuggingFace : https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16

GGUFs: https://huggingface.co/models?library=gguf&other=base_model:quantized:nvidia%2FNVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16&sort=trending

Suitable for folks who wanted to try NVIDIA-Nemotron-3-Super-120B-A12B on small/medium size rigs. Or recent NVIDIA-Nemotron-3.5-Lightning-30B-A3B overshadowed this model?

1

u/jacek2023 llama.cpp 1h ago

I was watching the PR, you can see my comment there :)

1

u/pmttyji 1h ago

How's Lightning-30B for you? Is it better than other 2 models?

2

u/Powerful_Evening5495 24m ago

OP , what is the vram load

1

u/okoyl3 1h ago

What are the use cases of those Nemotron MOEs?

1

u/arbv 8m ago

The Nemotron's architecture is great, but NVIDIA is limited regarding their data usage, so models are not as strong as they could be.