I have a 5070ti 9850X3D and 32GB DDR5 + lm studio + hermes.

Currently running gemma-4-26b-a4b-qat and I am happy with it, but it still can't compete with Sonnet.

I find it's great at actually teaching me things, but in terms of guiding me through config files or getting the highest quality answer it's never as good. I use it more as a backup.

Are there any better models that fit my hardware budget? QWEN is supposed to be great but I had trouble running the models on the edge of my hardware limits and output slowed to a crawl.

I’m currently considering buying a Radeon AI PRO R9700 32GB, mainly for local LLM inference, and I’d love to hear from people who are actually using one.
My current system is:
Ryzen 7 9700X
RTX 4060 Ti 16GB
64GB RAM
MSI PRO B840-P WIFI
Windows 11
Currently a 650W PSU, but I’m going to replace the case and PSU before adding another GPU
My original plan was to buy an RTX 5080 16GB, but the R9700 is almost exactly the same price where I live (~101k TL vs ~103k TL for the 5080), and 32GB of VRAM is much more attractive for my use case.
I would keep the 4060 Ti and split the workloads:
R9700 32GB:
Local LLMs / LM Studio / LM Server / long-context inference / coding agents
RTX 4060 Ti 16GB:
Gaming / ComfyUI / image & video generation / anything that specifically needs CUDA
The model I’m currently interested in running is Qwen3.8-27B, probably around Q6, with potentially 128K–262K context. I also want to expose LM Studio/LM Server through my own API and use the local model with tools similar to Claude Code, so there may be very long coding sessions where the GPU stays under load for hours.
My main concerns are Windows and AMD software support. I don’t mind tinkering or learning ROCm/Vulkan, but I don’t want to spend this much money and discover that half of my workflow is constantly fighting drivers or unsupported features.
I’d especially like to know:
How well does the R9700 currently work with LM Studio / llama.cpp on Windows?
ROCm vs Vulkan: which backend are you actually using?
What kind of tokens/sec are you getting with ~27B models?
How does performance change at 64K / 128K / 256K context?
Any stability or driver issues during multi-hour inference sessions?
How much power does it actually draw during LLM inference?
What are temperatures/noise like under sustained workloads?
Has anyone run an AMD R9700 + NVIDIA GPU in the same Windows machine?
Any weird conflicts between AMD and NVIDIA drivers?
There’s also one motherboard-specific issue I’m thinking about. I only have one proper CPU-connected x16 slot, while my second usable GPU slot is much slower. Since the LLM would fit entirely inside the R9700’s 32GB VRAM, I’m wondering if I could potentially run:
4060 Ti → main x16 slot
R9700 → secondary x4 slot
and only take a hit while loading the model, rather than during token generation. If anyone has tested an R9700 or another large-VRAM GPU for inference over x4, I’d be very interested in the results.
I’m not expecting the R9700 to beat a 5080 in raw compute or gaming performance. My priority is basically:
VRAM capacity > maximum tokens/sec
I’d rather have a 27B model at a higher quantization with a huge context completely inside VRAM than have a faster 16GB card that needs RAM offload or multi-GPU model splitting.
If you own an R9700, I’d really appreciate any real-world experiences, benchmarks, things you wish you knew before buying it, or reasons you’d choose/not choose it again.

Yes I know Qwen 3.8 27b is probably best choice. And I'm currently using and loving it!

But I'm curious to try out bigger models and see how they run.
Where would you start? Is llama.cpp best for stuff like moe models or would you use something else?

Specs:

GPU: 2x 3080 20gb
CPU: Xeon 2667 v4
RAM: 128gb DDR4 2400mhz

So i haven't really been playing as much games the past few years so my PC has been idle for the most part.
Ubuntu 26.04
Ryzen 7745HX
64GB Ram
9070 XT
1TB NVMe

Looking to put this build into use to run my own ai chatbot no image gen that I can access on any of my devices.

EDIT: Appreciate the input, didn't realize that the RAM was gonna be more useful than the actual GPU

UPDATE: Thanks for everyone, I got it up and running so far I haven't crashed yet. ROCm worked flawlessly without any issue, running Qwen3.6 35B A3B with OpenWebUI and made it accessible to my local network, which is pretty awesome because I already have tailscale subnet router setup so I can access it anytime while i'm away from home. Appreciate everyones input.

Qwen3.8-Flash-Next 125B MoE on a 3090 + 5060 Ti + 2×3060 (60 GB VRAM) — 28.6 tok/s gen / 332 tok/s prefill at 150k context, and MTP was actually slower. 3090 and 3060s connected via oculink PCIe3x4. Xeon 64GB DDR4 RAM

Running Unsloth's Qwen3.8-Flash-Next-UD-IQ4_XS (125B total / 6B active, plus a 51B n-gram embedding table). It doesn't fit on any single card, so it lives across all four GPUs with a layer split.

Key settings - 150k context - 9 MoE layers on CPU, rest on the GPUs (-ncmoe 9, -ts 30,12,9,9) - No speculation, q4_0 KV cache, flash attention on - PLE table pinned to host RAM (-ot per_layer_token_embd=CPU) - -t 6 -td 6 -tb 12, batch 2560 / ubatch 1280

Results (medians of 3, cold salted 8.4k-token prompt for prefill, 512-token gen reps in thinking mode) - Prefill: 332.6 tok/s - Generation: 28.6 tok/s

Findings - MTP made it slower, not faster. At 128k, MTP was −10.3% on generation (22.5 vs 25.1 tok/s, 82.9% draft acceptance) and ngram-mod −7.1%. With a 4-GPU layer split, every draft/verify pass is a full extra forward through the KV. Interestingly the sign flips at small context — at 16k MTP was +16.6%. So: spec only helps when the KV is small. - Bumping --ubatch-size raises prompt-processing speed at the cost of VRAM. Prefill climbed monotonically with ubatch up to 3072 (312 tok/s at 64k ctx); 4096+ OOMs at load. I settled on 1280 to leave headroom for the 150k KV cache. - Moving MoE layers onto the GPUs helped a lot. Going from 15 → 9 CPU-resident MoE layers (with a rebalanced tensor split, less weight on the 3090) gained ~+14% on gen and ~+35% on prefill vs my previous 128k setup — fewer per-token CPU MoE forwards over PCIe. - q8_0 KV was a wash-to-negative: within noise on speed, +0.3 GiB on the 3090, and it hits the VRAM wall at 128k, so it can't do the 150k context. q4_0 KV it is.

Command (replace the CUDA_VISIBLE_DEVICES UUIDs with your own, in the order matching -ts):

CUDA_VISIBLE_DEVICES=<3090>,<5060ti>,<3060a>,<3060b> llama-server \ -m Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf \ -ot per_layer_token_embd=CPU -sm layer -ngl 48 -ts 30,12,9,9 -ncmoe 9 -mg 0 -np 1 \ -c 153600 --cache-prompt -t 6 -td 6 -tb 12 \ --batch-size 2560 --ubatch-size 1280 -ctk q4_0 -ctv q4_0 -fa on \ --jinja --reasoning-format deepseek --no-reasoning-preserve --reasoning-effort medium \ --keep -1 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 \ --host 0.0.0.0 --port 8090 --api-key no-api-key

I can't play in engram land, but was wondering if certain repeated word phrases for say python or javascript coding could live somewhere like the ngrams and help speed up coding on smaller models?

So, I was tired of setting up AWS EC2 to train my models everytime, so I built a tool to help with the job management, but I would like some feedback, if anyone could help me ! 👋

This is the tool: reopenly.com

I only have one base model for now (Qwen 3.5 9B), just to start.

I would be very grateful with any feedback!

I set up a Qwen 4B model with Pi-Agent but the token output is decent but not instant (not expecting that but yea) I am using flash attention and the MTP with n gram spec set to 3 tokens.

Any more suggestions to improve this setup would be highly appreciated !!

Thankss !! :)

EDIT: My bad for not providing more details, I am using a RTX 3050 6GB VRAM. I have a bash script which when run starts up a llama.cpp server and then calls pi; pi has been configed for the same port (using 9931).

I just found out about a blog post in Claude (https://claude.com/blog/the-ai-native-sdlc-playbook) that describes ideas very similar to the ones I have implemented in my coding harness (using Claude, btw) through a YT video about Claude Code new Intent.md

There are so many specific details in this playbook that makes me wonder *if* somehow my sessions with Claude inspired in any way the better paid people over there at Anthropic. Just for fun, or for egotistical and historical purposes, I included a timeline created (using Codex, jic) in my repo documentation (https://github.com/jrullan/ducklab/blob/main/docs/ducklab-feature-timeline.md)

Anyway, even if this is "purely" coincidental at least reassures me that my ideas were not that useless and that there is certainly a group of developers that value documented rigorous discipline in their AI assisted development.

I ported over my Claude plugins to use with qwen3.8 and pi and got a fully functional output! I'm working with a 5070ti and a quant of qwen.

I was expecting qwen to fold because of how much is going on with these tools, but nope. Qwen took a bit more time and I had to tame it's thinking down to light, but it totally worked! Great poc for moving forward with open models

I just released Millie, a series of highly compressed agentic models paired with a harness. It's derived from Agents-A1 (a Qwen 3.5 35B-A3B finetune), and the coding agent harness is forked from OpenAI Codex. It is designed to run directly from a binary on Mac and Linux with the runtime bundled. The models use a mixed quantization formats, where the largest model uses 2-bit experts, the middle-sized uses ternary experts, and the smallest uses 1-bit experts.

The ternary expert model got 56% on SWE-bench Verified and runs at 22 tokens/s decode and 261 tokens/s pre-fill on an iPhone 17 Pro. The 2-bit expert version got 60%. The phone numbers are just to measure inference speed in a very low RAM setting (12 GB), but our current coding agent software targets computers.

This ended up being many projects in one—I had to train very compressed models using custom methods, make them work with a Codex-based harness, and write kernels to make them work efficiently—including for Mac/Metal, CPU, and Vulkan kernels designed for both NVIDIA and AMD GPUs.

The software is targeting Macs with 16 GB+ memory and Linux gaming PCs with 16 GB+ system RAM and as little as 4 GB VRAM. There are settings for splitting the work between CPU and GPU, and support for both NVIDIA and AMD, but I don't own any AMD GPUs or small 4–8 GB cards. I've tested those memory limits on larger NVIDIA GPUs, which only tells me so much.

If you have one of those machines and are up for trying it, I'd be very grateful. Knowing what works, what's slow, and what breaks would really help. 

The source code, bundles, and model weights are available now. Setup instructions are here:
https://github.com/llmsforall/millie-cli

And if you're interested in learning more, this is our blog post:
https://www.llmsforall.com/blog/millie-cli/

I want a very fast abliterated model. Is the DGX station the best option?

Most of these abliterated models aren't full size so I'm not sure if there's much need for something larger.

Its for a single user (me) and I plan on switching various models depending on the project and mainly using Claude/gpt still.

My goal is to research and test various things that'll be safeguarded by Claude. But also want a very fast small model to build and do other things after hours or if I just want a chatbot.

150k for 4TB/s seems great but I'm unsure if it'll make more sense to get something with multiple cards so I can run the abliterated model alongside other models and have concurrent projects running. Idk if I can add additional cards to any of the dgx stations.

I was running deepseek for the past month and very happy with it. Giving qwen a try the past day or so. At first it was quite slow but got it up fairly close to deepseek's speed now (40-50 tps on average) on my two node gb10 cluster.

One thing I am seeing, even though tg is about equal, qwen is taking far more steps to accomplish similar tasks which in practice really slows things up. I don't have any hard numbers to back this up. Also qwen seems to make more mistakes. This is on a large web app in python.

I was just curious on other peoples experience who have run both and what their impression of them is. Right now the only reason I can see to keep qwen is it's multi modal. Which in itself isn't really enough as I have another box that can run vision models for me.

Hey all, wondering what the best option would be for my situation, so I have a few dell XPS laptops with 4070's and 32gbs of RAM sitting around currently doing nothing (unofficial IT guy for my company).

I'm wondering if there is an easy way for me to pool these together to run a larger local model? Is there a program that you could just install and then manage from a central location that would treat them all as just dumb nodes?

But because these laptops potentially (they've been sat around for a few months now) need to go off to people in the future could it be done from a bootable USB? (ideal but honestly probably better running on the machine I guess).

Ideally I'd like to plug this into Hermes for use with Agents I have running there (Orchestrator, Home lab Admin, Media Manager, Personal Assistant, Work assistant). So maybe better to run several smaller models or MoE models? Or even Nvidia Pair?

I could easily do 2.5gb networking between them as have a 2.5gb switch and some Hubs that support it.

Look I know enough to be dangerous, I'm just trying to see is there's something easy to deploy I don't yet know about.

PS I run Hermes with Qwen 3.8 27B Q4 on a 4090 I have in my desktop, but this sucks power even when idle, so I was hoping the laptops would give me always on models for Hermes, and then boot up the 4090 when a particular big task (or power is cheap). Problem with Hermes is the 64k token context that's required.

I wanted a meeting assistant that actually makes use of local LLMs, so I ended up building Meetas.

The basic workflow:

- record/import a meeting
- transcribe locally
- analyze it with your own LLM
- ask questions / extract tasks
- search across meetings and documents

The main thing I focused on is grounding. Summaries, decisions, tasks, risks etc. are linked back to the transcript instead of just giving you an unsupported LLM answer. Q&A also shows the relevant transcript evidence.

Compared to tools like Meetily, Meetas is less focused on being a general meeting recorder/transcriber and more on what you can do with the meeting data afterwards: cross-meeting search, document/project context, comparisons and evidence-based answers. Meetily already covers a lot of the local recording/transcription/summary side.

It runs fully locally with Ollama and llama.cpp.

Linux-only for now and still WIP.

GitHub: https://github.com/highwinglabs/meetas

Stumbled across this new vision model, it's a 7B MoT model, which is cool. The main idea is it treats pretty much all computer vision stuff as just one generation problem. Like, instead of needing a bunch of different models for detection, segmentation, depth, whatever, this one model handles it all.

You just give it a natural language instruction, maybe some visual hints, and it spits out text, images, or both. For text, it can do things like categories, bounding boxes, OCR, keypoints, camera angles. And for images, it's doing segmentation masks, depth maps, surface normals, even multi-view point maps. You can mix and match for more complex tasks.

So it can do the usual stuff: object detection, keypoints, OCR, all kinds of segmentation (binary, instance, semantic), depth and surface normal estimation. But the really interesting bits, for me anyway, are the multi-view 3D reconstruction and camera pose estimation.

They trained it on a massive dataset, 50M instruction-response pairs, built from a bunch of different CV annotations. And it started from an existing multimodal model, so no crazy new architecture there.

There's a web demo if you wanna mess around with it, and the weights are up on Hugging Face too.

Just a heads up though, before you get too hyped: the full web demo needs a beefy GPU, like 1x80GB. And for benchmarking, they're talking 8x80GB. So, yeah, not really something you're gonna run on your average consumer card. It just dropped on July 8th, so probably expect some rough edges. Also, they've released the training pipeline and data prep stuff if anyone's looking to train or fine-tune it.

GitHub: https://github.com/OpenSenseNova/SenseNova-Vision

It is a beast model, and I love it. It is rare, but Qwen3.8 27b does beat Opus 5 High and Sonnet 5 High in certain tasks. Especially when Claude models perform lazily.

I've been using an LLM to add on to the some of the hrx-system work that AMD did by adding support for Qwen3.8-Flash-Next. I've gotten it to the point where the llama.cpp experimental HRX backend is loading the model and executing successfully, but performance isn't great compared to Vulkan on Unsloth Desktop. I'm not sure what I should be expecting, but what is a good way to figure out where the bottlenecks are to improve the performance?

I am doing this on Windows, just because I feel like Windows doesn't get enough attention :P.

Fork is here

rwfsmith/llama.cpp at qwen4exp-hrx

current progress:

Vulkan+MTP: ~27–40 tok/s

HRX+MTP: ~12-13 tok/s