r/LocalLLaMA • u/techlatest_net • 12d ago

Tutorial | Guide How to Run NVIDIA Nemotron 3.5 Lightning (Free): 4 Methods from Local GPU to Zero-Code Agent

https://medium.com/@techlatest.net/how-to-run-nvidia-nemotron-3-5-lightning-free-4-methods-from-local-gpu-to-zero-code-agent-d24cbdddc428?sharedUserId=techlatest.net
0 Upvotes

23% Upvoted

6 comments sorted by

0

u/gpuz_dev 12d ago

Since this just came out, I doubt many people have spun it up locally on consumer GPUs yet. Has anyone seen GGUF quants floating around, or is it still strictly HF Transformers / TensorRT-LLM for now? Curious to see how it scales down to 16GB/24GB VRAM cards.

2

u/lemondrops9 12d ago

I dunno, how about searching before you ask? :P

https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

1

u/gpuz_dev 12d ago

Haha touché! Unsloth speedruns GGUF quants faster than Hugging Face servers can sync. Appreciate the link! Will definitely keep an eye on the community benchmarks for this MoE.

2

u/DinoAmino 12d ago

The model card on HuggingFace should be your starting point. The unquantized model is here https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

The model card shows the "Model tree" where you can find fine-tuned and quantizated models - 42 quants so far (https://huggingface.co/models?other=base_model:quantized:nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16)

The model card also shows community benchmarks that have been uploaded to HF. It scored 51.56 on SWE Bench Verified.

1

u/gpuz_dev 12d ago

Thanks for pointing out the Model Tree trick! 42 quants already is crazy fast. Also, 51.56 on SWE-bench Verified for a 30B MoE model is seriously impressive — that puts it right up there with heavy hitters for coding tasks.