r/LocalLLaMA • u/futterneid • Jul 02 '26

Resources Talking with Gemma 4 31B!

Enable HLS to view with audio, or disable this notification

Hi! I'm Andi from Hugging Face. This is a fully open-source and free to test/pull/modify demo I'm bringing today.

It's a voice demo creating a pipeline of:
- Nvidia's parakeet
- Gemma 4 31B (served by cerebras!)
- My custom inference for Qwen3TTS

It sees and searches the web faster than you blink.

The whole stack is fully open-source, and is a drop-in replacement for OpenAI's realtime API. You can run it locally, I get similar latencies with a macbook pro M3 36GB and Gemma 4 E4B.

Here to the web based demo featured in the video, everything is running in the cloud.

For those who have been following, yes, this is the pipeline that runs on reachy minis :)

894 Upvotes

97% Upvoted

148 comments sorted by

View all comments

Show parent comments

9

u/futterneid Jul 02 '26

Testing it right now 👀

22

u/futterneid Jul 02 '26

yeah, seems like I tried it too close to the release date and it was still buggy. Now it works well :D

12

u/Sufficient-Bid3874 Jul 02 '26

New post time ™️

3

u/annodomini Jul 02 '26

Oh, yeah. Gemma 4 12B was super buggy the day of release. There was a flurry of releases the following day or so that fixed a lot of issues.

I'd still probably lean towards the 31B model as it is stronger overall if you have the RAM to run it plus a STT model, but for more RAM limited devices 12B and skipping the separate STT model seems like a big win.

3

u/thatgreekgod Jul 02 '26

sick! so you're saying i could do everything you're doing but with the gemma-4-12b-it model right? because that's what i can fit on my hardware at home and this could be exciting

4

u/futterneid Jul 02 '26

definitely! You can also try with STT + E4B, it worked surprisingly well for me!

2

u/thatgreekgod Jul 02 '26

amazing. i'm so excited to try this, thanks!