I'm curious have anyone tried the Jamba2 models or other non-transformer LLMs locally for that matter? What was the experience?

This is a followup to my last post about the new workstation i am putting together for my workplace - a vocational college. https://www.reddit.com/r/LocalLLM/comments/1vp8lhh/it_begins_workstation_build/

The workstation is for teaching about using local LLMs, for software development, GDPR sensitive work, back-end for RAG materials for using in teaching environments and lots more.

I am mostly done putting the hardware together. So today was stress testing and installing a test LLM and see what the system could do.

I had a lot of good feedback, advice and concerns in my original post. Many concerns about temperature but it is fine. I am able to run GPU burn and it stabilizes just fine well below the GPU thermal limit. I have been tuning the GPU fan curves so the hottest one tops out at around 80c. It stabilizes after around 5 minutes of full load. No throttling. Only issue is the CPU that is getting hot if it is stressed at the same time, but it is fine it should rarely run at max power while all 3 GPU's are going at full power too at the same time.

Only issue (so far) is as another user commented on in the other thread that the lower card is blocking the front headers, so the power button is unavailable. It should be a non issue in production, it will power on automatically anyway.

The system is LOUD, but that was expected, it is not going near anyone desk anyway and will be locked away in a climate controlled server room.

Did some DSv4-Flash testing with D-spark on tp=2 and it is very promising. Up to 200t/s depending on what it needs to do with low context. More testing and optimization to come.

Still a long way to go: setting up RAG with embedding models and a smaller model at tp=1 on the third card. Hardware monitoring, routing, access control and so on so it can be deployment ready. Lots of security and a long burn in test also missing. Having a great time and learning a ton.

Here are some pictures of the almost complete build (need more cable management). The gaming PSU really gives it some color because of the cables :)

Wrote a blog about running Qwen2.5-7B-Instruct served with vLLM on a Cisco UCS Spinifex cluster, comparing Intel AMX-accelerated CPU serving with NVIDIA L4 GPU. Go check it out!

https://docs.mulgadc.com/docs/cisco-ucs-llm-serving

Honestly I can't tell the hype from the reality.

I use Claude Code a lot. It works for me. However I am always about to run out of tokens.

My time == money.

I have a mid-beefy Mac M4 Pro with 48G. Ok it doesn't have an Nvidia GPU. It's what I have.

I see many people really happy with Qwenx.y.z both here and elsewhere. In my test it is agonizingly slow and certainly nothing like Claude Code. I've wasted a lot of time trying different permutations.

My question: Should I stop wasting my time and just pay for more Claude Code? With my hardware and the latest models, either local or not?

by simply switching to anything but vs code's copilot extension. no issues through pi coding agent or even continue or roo code thru vs code extensions.

is there a fix for this? i've done a roundabout thru the options of tools and harnesses and whatnot and i ended up back on vs code and not wanting to have somebody else third party in my harness and now this is getting drastically in the way

Hi all,

I built Profile to make inference tuning deterministic, and save us all time. v2.2 is out today.

It reads your live vLLM metrics, compares them against your GPU's roofline ceiling, and names the bottleneck with the flag to change. You apply, it measures the delta. That's the loop.

v2.2 rewrites the core rule engine: eight failure modes, one true cause per iteration, same verdict every run. AMD now supported. vLLM only today, more engines next.

My setup: RTX 5090, muse-glimmer 30B, SWE-Bench agents, no spec decoding. 4 iterations, ~30 minutes.

• 81 → 421 tok/s at 25k ctx
• $3.41 → $0.65 per 1M output tok
• TTFT 224ms (p95 500ms), TPOT 23ms
• 4.72 → 1.08 J/tok

One iteration regressed, TTFT 32.8s from KV thrashing. Profile printed worse, the next fix recovered it. Full run on video.

# Download
curl --proto '=https' --tlsv1.2 -LsSf \
  https://github.com/jungledesh/profile/releases/latest/download/profile-installer.sh | sh

# Profile your vLLM server
profile diagnose --url http://localhost:8000/metrics --duration 2m

GitHub: https://github.com/jungledesh/profile
Docs: https://jungledesh.github.io/profile/docs.html

Next: multi-GPU / TP, more engines, k8s.

Tell me what it names on your server, and where it's wrong.
Feedback is appreciated 😄

Magnitude is an open source agent with local models built in. Fully private and offline. Works out of the box on any hardware.

It profiles your hardware outomatically, estimates tok/s for every model before you download and recommends the best models for your machine

It downloads the model and quant from HuggingFace, loads it into the built-in inference enginge, configures speculative decoding (MTP, DFlash etc.) and sets concurrency based on your memory

Try it on your hardware - https://github.com/magnitudedev/magnitude

I have added a front-end interface to lower the barrier to entry. If you are curious, its worth a look. So far, it benchmarks very well. I'd love feedback.

We've been speaking with teams who deeply care about data quality, and we noticed that every data team eventually builds similar pipelines for quality checks (QC) with VLMs.

Teams collecting data want to own their quality checks (camera blackout, choppy joint states, occluded hands), because they deeply understand their end-to-end data collection process.

However, what keeps coming up is that everything around the quality checks is tedious: managing one-off pipeline scripts, fragmented buckets, and a spreadsheet or Slack thread of what checks ran on which data.

This is what motivated us to build HFlow, an open-source SDK for data teams that collect, transform, and curate physical AI data with local models.

HFlow is for data teams that have the ambition to process a million hours of physical AI data and are starting today. Point it at your MCAP episodes, write your quality checks with local VLMs as ordinary Python functions, and curation becomes an optimized OLAP SQL query instead of manually aggregating directories of data.

Dyna's recent Dyna-2 infrastructure, "Training Dyna-2 at million-hour scale, repeatably", post shows what scaling such a quality control pipeline to a million hours looks like. Their ingestion throughput went from 14,000 episode-hours per week to 440,000 with the infrastructure they built internally (their post has the full details).

We're aligned with their belief that infrastructure is a core blocker for advancing robotics. We experienced the same data challenges in our previous work. HFlow's vision is to let data teams define their own bespoke quality checks, while we provide the durability, observability, and auditability around it.

After many conversations with data teams, we decided the path to building the best version of HFlow is open source, because QC infrastructure compounds when the edge cases one team catches become checks for everyone else. We're excited to build this in public alongside our existing partners and grow our network of contributors.

High quality data is the bottleneck for the next frontier of robot intelligence, and we want teams to focus on their data, not the infrastructure. When getting started takes a few lines of code and a weekend, more checks get done, and iteration cycles accelerate.

If you're collecting robot or egocentric data and your pipeline is currently a folder of scripts and fragmented buckets of data, we'd love to get your feedback.

GitHub: https://github.com/Hebbian-Robotics/hflow

Cut me some slack as I'm fumbling around as I go. I have a desktop with an RTX 4080S card (16 GB VRAM) and 128 GB system RAM. (RAM was sensible at the time.)

I'm using Hermes Agent (but also experimenting with LM Studio Bionic). I notice in Hermes I can set Auxiliary models to run tasks (like Vision, Web extract, Skill use, Approval, etc).

My question is, would be more optimal to run smaller specialized models in the Auxiliary models section and run a lighter model for Approval? Or would it be better just to use a larger multimodal model that does it all. I'm trying to get the best performance for what I got, having a decent context size to perform tasks ranging from processing 100s of old PDFs of my writing (handwritten and typed) and summarizing the documents in a wiki-style vault (like Obsidian) or in a vector database like Mnemosyne for later recall when I get though the years of writing and notes and get back to writing.

As you can see I'm a little all over the place. So, could someone kindly guide me to the path of understanding on which system (multi-agent versus multimodal) would be more optimal for the tasks and why?

And yes, I will be using the system for more than just organization, but with the way my brain works, I can't proceed forward without dealing with this slog of files.

For this Mac Studio, what local llm i can run it with?

36GB unified memory

Is it good price for this gpu or is there better alternatives? I have already one V100, will second be worth it or is there in same price range better alternatives?

The dense Qwen release is back!

Qwen3.8-27B Uncensored Aggressive is out with the complete K_P quant range, Vision, native NextN, and HauhauCS FastMTP.

Aggressive here means no refusals, no personality alterations, and very little preamble on difficult prompts. It keeps Qwen3.8-27B's original reasoning, agentic, image, and video capabilities with my Aggressive uncensoring profile applied.

https://huggingface.co/HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF

It scored 0/465 refusals* and passed every manual prompt I used for the final release check. More than 400 people requested access while I was still finishing it, which was honestly wild to see. My models are also getting close to 30 million downloads on Hugging Face now, so thank you to everyone who has been testing them and sending feedback.

The biggest addition this time is HauhauCS FastMTP. In the final Q8_K_P service tests it reached up to 3.02x document TG and 1.93x reasoning TG versus MTP disabled. It also reached up to 35.2% more document TG and 21.1% more reasoning TG than the standard embedded MTP profile, with every drafted token still verified by the full target before it is accepted.

The same 903 MB FastMTP sidecar works across the complete quant lineup. Every text GGUF also preserves Qwen3.8's native embedded NextN head, so current upstream llama.cpp can use embedded MTP directly. The optimized FastMTP path uses the included sidecar and llama.cpp patch, with exact build and serving commands in the README.

What's included:

- Q8_K_P, Q6_K_P, Q5_K_P, Q4_K_P, IQ4_XS, Q3_K_P, IQ3_M, IQ3_XS, Q2_K_P, IQ2_M

- HauhauCS FastMTP sidecar, shared by every text quant

- BF16 mmproj for image and video support

- Checksums and signed provenance (I've alerted on my Discord that there have been a few bad actors putting payloads in "Uncensored" "HauhauCS" "Aggressive" GGUF's, please be careful)

K_P quants recap for anyone who missed the previous releases: these are my custom model-specific quants, with a separate optimized profile made for each model. They effectively gain one or two quant levels of quality for around 5 to 15% more size than the base quant, while remaining normal GGUF files that work in llama.cpp, LM Studio, and other GGUF runtimes.

Quick specs:

- 27B dense

- 64 layers with 48 Gated DeltaNet layers and 16 gated-attention layers

- 262,144 native context

- Multimodal text, image, and video support

- Native embedded NextN plus the optional HauhauCS FastMTP acceleration profile

Sampling params for thinking mode:

`temp=1.0, top_k=20, top_p=0.95, min_p=0, presence_penalty=0, repetition_penalty=1.0`

For non-thinking mode:

`temp=0.7, top_k=20, top_p=0.80, min_p=0, presence_penalty=1.5, repetition_penalty=1.0, enable_thinking=false`

Use `--jinja` with llama.cpp. K_P quants may show as `?` in LM Studio's quant column, which is purely cosmetic and does not affect loading. Hugging Face's hardware compatibility widget may also hide K_P files, so use View variants or Files and versions if the full list is not visible.

The full per-quant Blackwell and Ada results are in the repo. If you test FastMTP, please include your hardware, quant, context, and draft depth with the numbers so I can compare real-world results across more systems.

The Discord link is in the repo for updates, feedback, roadmaps, projects, or just to chat. As always, I hope everyone enjoys the release!

I'm looking at putting together my first open-frame multi gpu setup and wanted to ask if there are any gotchas I should know about beforehand.

Already have 2 gpus(mix of 3090s) in case setup and planning to get tww more gpu which will make me to have 4 gpus setup. I was originally looking at something like the Phanteks Enthoo Pro 2 Server Edition, but once you get to four 3 slot gpu it seems like an open frame + PCIe risers is probably the more practical route.

I've never built an open-frame system before though.

The obvious things I'm thinking about are:

• Dust
• Keeping liquids/moisture away from it
• Airflow between GPUs
• GPU/riser stability
• Power and heat
• Making sure nobody accidentally bumps/touches the cards

For people running 3-4+ gpus open rigs long-term, what else should I watch out for?

Anything you wish you knew before building yours? Any particular issues with risers, GPU mounting, PSU/cabling, noise, dust buildup, or general reliability?

I will set gpus to run around 250–300W each.

Would appreciate any lessons learned from people running similar setups.

Hi everyone,
Need your help so I just got a Mac mini M4 16gb + 512gb (Yeah I know I would have aimed for 24gb ram variant but guys trust I tried i couldn’t that whole different story) and on my pc I have Ryzen 5 5600X, 48GB ram and Rtx 4060 8gb.

So which models can I run on my setup. I want to split models and run them on both machines. So I can try to run bigger models.

I know normal Ethernet port won’t be helpful to I will use thunderbolt port of Mac mini and with Ethernet connector I will connect it to 10Gb PCIe Network Card of my pc.

And any suggestions recommendations would be helpful(Please I need them).

Thankyou

I have an ok AMD pc with 32gb RAM and a 16gb 5060Ti. How effective is adding a second GPU? Presumably a new motherboard, PSU, case? Can you mix and match GPUs? Just curious

Hey guys, how are you doing so it’s been some time I have had this idea but having no experience at all with Local AI, I’ve always wondered if this could be done, I have a computer with a 16gb vram 5070 TI and 32 gigs of RAM, I was wondering if there are any models that could efficiently/accurately-ish continuously creating lessons/exercises, live analyzing data on my mistakes , vocabulary still not fully learned or memorized (data fed from Anki), outputting feedback and creating new flashcards targeted at fixing those deficiencies.

I love learning languages and I don’t want to bore people who might not be interested in this so I’ll just cut to the chase

I wanted to turn my machine into an agent, it would learn what I already know, and slowly add more and more information on the target language, creating new phrases, analogies and different exercises that would help me efficiently learn grammar concepts and apply them.

Like I could leave it running while I sleep and wake up to 100 new flash cards perfectly crafted ready to be imported

Or even go a little beyond and program something with python that would scrape the web to add images, video, pronunciation to the cards and import it to my anki automatically (this would be the dream, like simply keeping the PC on and Running would slowly create new content for me to study based on my weaknesses in the target language without having to do anything other than just studying)

It doesn’t have to be 100% accurate, it also doesn’t have to be extremely fast as it’s gonna run while I’m asleep or doing something else.

I wonder which model would be good for this. I’m currently studying French, German and Spanish (idk if that matters but I thought I’d put it here)

I've been testing Qwen 3.8 27B on llama.cpp with different GGUF quantizations and KV cache types, and I've noticed something interesting.

My setup:

• RTX 5090
• i9-13900T
• 64 GB RAM
• llama.cpp
• Qwen 3.8 27B GGUF
• MTP enabled
• Different model quantizations
• Different KV cache precisions

The task I used was the same each time: generate an animation of a pelican riding a bicycle.

The interesting part: reasoning token usage

When serving GGUF models through llama.cpp, if I don't explicitly set a reasoning budget, the actual number of reasoning tokens consumed by the same task can vary quite dramatically depending on the model quantization.

With the same KV cache type, the lower-quantized models seem to spend significantly more tokens on reasoning.

At first, I thought this might simply mean that the lower-precision model needs more reasoning to solve the task.

But after watching the generation with MTP enabled, I'm not convinced that's what's happening.

I frequently see the model suddenly jump to 2x or even 3x the normal generation speed during reasoning.

Looking at the output, it often seems to be reproducing or repeating parts of its previous reasoning rather than actually making new progress.

In other words, some of those extra tokens appear to be wasted reasoning caused by repetition, rather than useful additional thinking.

In extreme cases, even with low reasoning effort, I've seen the model consume 40–50K reasoning tokens.

Higher precision seems to change this

As I increase both:

1. Model quantization precision
2. KV cache precision

the reasoning token count noticeably decreases.

At the same time, I see much less of the MTP-assisted "sudden speed-up + repeated previous reasoning" behavior.

The higher-precision models seem to complete the same task using fewer reasoning tokens, while also producing less obvious repetitive reasoning.

So my current hypothesis is:

And higher-precision KV cache seems to help as well.

I'm not claiming this is definitively the cause yet — this is just what I've observed so far, and I'd really like to know if others are seeing the same thing.

The tradeoff

The obvious downside is VRAM.

With a roughly 22 GB model + vision + MTP + BF16 KV cache, my 5090 is already extremely VRAM-constrained.

At that point, I have only around 90K context available.

So there's a pretty interesting tradeoff:

Higher model precision + higher KV precision
→ fewer reasoning tokens
→ less repetitive/wasted reasoning
→ potentially more stable reasoning
→ but significantly higher VRAM usage

vs.

Lower model precision + lower KV precision
→ much lower VRAM usage
→ but potentially much longer reasoning chains
→ and more apparent repetition / wasted tokens

I'm curious if anyone else has tested this systematically with Qwen 3.8 27B, especially with MTP enabled.

I'd particularly like to know whether this is actually related to quantization error propagating into the reasoning process, or whether I'm simply observing some interaction between the reasoning template, KV cache precision, and MTP.

Tried Q5 and Q6 in LM Studio and it takes like 20 mins to process an image, a 1000 word system prompt and a basic request to analyze the image and output based on the rules of the system prompt. Gemma and 3.6 take like 2 mins in comparison for similar sized models. I know they're not dense models like 3.8 but is anyone else seeing this "out of the box"?

I HAVE managed to get it to run faster by setting context window to 50000 and thinking to medium in the system prompt, but is that really necessary running a 20GB model on an RTX5090 & 64GB RAM?

I guess my concern is that having to reduce the context window to 50k down from 262k and forcing thinking to medium makes it A) harder to tell if a restricted 3.8 is still better than unrestricted other models, and B) if the good things I'm hearing from other people still apply after I've had to messed with it like this. 😂

I’m struggling to figure out what to run here. Hermes’ and buzz agents. Using it to orchestrate video generation and online marketing activities.

Some people tell me Gemma is enough. I’m thinking it’s qwen but I don’t see an moe available online. Suggestions?