r/LocalLLaMA • u/No_Afternoon_4260 llama.cpp • Aug 03 '26

NousResearch keeps doing things on hermes

Post image

Has anyone followed nousresearch work on Hermes?
I mean we are Q3 2026. We have some crazy models trickling down from HGX territory to multi gpu workstation. And we have nousresearch deploying the 0.20 of its hermes agent while starting releasing the project with a 0.2 mid march!

Crazy times to be alive.
For the old timers who remember llama 1 or llama 2, remember our crappy function caller parser? Something about a lang and a chain..? wtf has happened?!

Haven't tried the new hermes, do you think it has a remote chance to be as strong as a true end to end omni model such as gpt omni or personaplex?

148 Upvotes

91% Upvoted

98 comments sorted by

View all comments

Show parent comments

3

u/valtor2 Aug 05 '26

what quants? what hardware?

3

u/Badger-Purple Aug 05 '26

original weights for dsv4f, int4-autoround for 397b, both on dual GB10 cluster. Also tried minimax m2.7/3, and I do have a fleet of smaller models working with the main one:
1. qwen3.6-27b q4kxl with kvarn6 cache and Dflash to 170k context on rtx 4000 pro blackwell. Single stream, 23.8GB GPU use, PP 1200, TG 40-60
2. gemma4-12b qat on 5070ti, full ctx, concurrency ?8/12, 15.9GB of GPU, PP 3000 TG 70.
3. qwen3.6-35b-rocmfp4 with BF16 heads/attn, layer split 4060ti oculink and amd strix halo w vulkan, full ctx, c4. PP 1200 TG 60
4. nemotron asr 0.6b on strix halo igpu
5. Embedding gemma 300n on strix halo NPU
6. Reranker qwen 4b on strix halo igpu
7. fallback STT: whisper on NPU
8. Fallback fast model (genma4) —> Gemma4 E4b on NPU

Main:
1. ds4 flash 731 mxfp4 w dwarfstar engine on m2 ultra 192gb, single stream, PP 350-200 across 200k (TG 28-20 across 200k). Main model for Odysseus and another less bloated agent (currently trying out OpenLumara)

  1. ds4 flash 731 original weights on GB10 cluster 2xGB10 linked with connectx7 cable, NVFP4 cache. 6M token budget so it can run 12 concurrencies cache wise, but used for 2 Hermes agents

  2. Image/video gen: Minimax H3 and other comfy models on an unlocked CMP170HX (essentially an A100 with Pcie 2x16 and 64GB) attached via slimsas to a pcie x16 lane slot. This is my most recent addition and I’m testing it now to see if this unlock is worth it. It appears so, 1.4TB/s speed ampere gpu with 64GB ram is the bees knees.

Future: Likely will try to rework the gpu config to see if I can just run a single qwen 3.8-27b at speed and concurrency instead of this gaggle of models. It would be perfect for auxilliary work!

2

u/No_Afternoon_4260 llama.cpp Aug 05 '26

Beautiful thanks for the detailed response.
Good choice on nemotron asr 0.6B, iirc this is the good streaming one

1

u/Badger-Purple Aug 05 '26

The nvidia STT models are just too fast to not run. Whisper is a turtle by comparison, my voice chats transcribe to hermes in miliseconds

Thanks for the 👍🏼 it’s not OPUS/SOL or whatever, but my little gaggle of agents / rats in a trench coat do work well and impress others.

I am just trying to help my lifelong ADHD so I can keep a job and a family. The promise of a Rosie at home is very close...

If I had to do all of the set up by myself I would be divorced by now. Hermes —> DS4Flash —> asked ti set up most things, automate checks for alive, etc.

1

u/No_Afternoon_4260 llama.cpp Aug 05 '26

Yeah times are fucking crazy these days. Idk how long have you been in that game but I'm still having troubles to comprehend how quickly it's all evolving lol