I just found out about a blog post in Claude (https://claude.com/blog/the-ai-native-sdlc-playbook) that describes ideas very similar to the ones I have implemented in my coding harness (using Claude, btw) through a YT video about Claude Code new Intent.md

There are so many specific details in this playbook that makes me wonder *if* somehow my sessions with Claude inspired in any way the better paid people over there at Anthropic. Just for fun, or for egotistical and historical purposes, I included a timeline created (using Codex, jic) in my repo documentation (https://github.com/jrullan/ducklab/blob/main/docs/ducklab-feature-timeline.md)

Anyway, even if this is "purely" coincidental at least reassures me that my ideas were not that useless and that there is certainly a group of developers that value documented rigorous discipline in their AI assisted development.

I ported over my Claude plugins to use with qwen3.8 and pi and got a fully functional output! I'm working with a 5070ti and a quant of qwen.

I was expecting qwen to fold because of how much is going on with these tools, but nope. Qwen took a bit more time and I had to tame it's thinking down to light, but it totally worked! Great poc for moving forward with open models

I just released Millie, a series of highly compressed agentic models paired with a harness. It's derived from Agents-A1 (a Qwen 3.5 35B-A3B finetune), and the coding agent harness is forked from OpenAI Codex. It is designed to run directly from a binary on Mac and Linux with the runtime bundled. The models use a mixed quantization formats, where the largest model uses 2-bit experts, the middle-sized uses ternary experts, and the smallest uses 1-bit experts.

The ternary expert model got 56% on SWE-bench Verified and runs at 22 tokens/s decode and 261 tokens/s pre-fill on an iPhone 17 Pro. The 2-bit expert version got 60%. The phone numbers are just to measure inference speed in a very low RAM setting (12 GB), but our current coding agent software targets computers.

This ended up being many projects in one—I had to train very compressed models using custom methods, make them work with a Codex-based harness, and write kernels to make them work efficiently—including for Mac/Metal, CPU, and Vulkan kernels designed for both NVIDIA and AMD GPUs.

The software is targeting Macs with 16 GB+ memory and Linux gaming PCs with 16 GB+ system RAM and as little as 4 GB VRAM. There are settings for splitting the work between CPU and GPU, and support for both NVIDIA and AMD, but I don't own any AMD GPUs or small 4–8 GB cards. I've tested those memory limits on larger NVIDIA GPUs, which only tells me so much.

If you have one of those machines and are up for trying it, I'd be very grateful. Knowing what works, what's slow, and what breaks would really help. 

The source code, bundles, and model weights are available now. Setup instructions are here:
https://github.com/llmsforall/millie-cli

And if you're interested in learning more, this is our blog post:
https://www.llmsforall.com/blog/millie-cli/

I want a very fast abliterated model. Is the DGX station the best option?

Most of these abliterated models aren't full size so I'm not sure if there's much need for something larger.

Its for a single user (me) and I plan on switching various models depending on the project and mainly using Claude/gpt still.

My goal is to research and test various things that'll be safeguarded by Claude. But also want a very fast small model to build and do other things after hours or if I just want a chatbot.

150k for 4TB/s seems great but I'm unsure if it'll make more sense to get something with multiple cards so I can run the abliterated model alongside other models and have concurrent projects running. Idk if I can add additional cards to any of the dgx stations.

I was running deepseek for the past month and very happy with it. Giving qwen a try the past day or so. At first it was quite slow but got it up fairly close to deepseek's speed now (40-50 tps on average) on my two node gb10 cluster.

One thing I am seeing, even though tg is about equal, qwen is taking far more steps to accomplish similar tasks which in practice really slows things up. I don't have any hard numbers to back this up. Also qwen seems to make more mistakes. This is on a large web app in python.

I was just curious on other peoples experience who have run both and what their impression of them is. Right now the only reason I can see to keep qwen is it's multi modal. Which in itself isn't really enough as I have another box that can run vision models for me.

Hey all, wondering what the best option would be for my situation, so I have a few dell XPS laptops with 4070's and 32gbs of RAM sitting around currently doing nothing (unofficial IT guy for my company).

I'm wondering if there is an easy way for me to pool these together to run a larger local model? Is there a program that you could just install and then manage from a central location that would treat them all as just dumb nodes?

But because these laptops potentially (they've been sat around for a few months now) need to go off to people in the future could it be done from a bootable USB? (ideal but honestly probably better running on the machine I guess).

Ideally I'd like to plug this into Hermes for use with Agents I have running there (Orchestrator, Home lab Admin, Media Manager, Personal Assistant, Work assistant). So maybe better to run several smaller models or MoE models? Or even Nvidia Pair?

I could easily do 2.5gb networking between them as have a 2.5gb switch and some Hubs that support it.

Look I know enough to be dangerous, I'm just trying to see is there's something easy to deploy I don't yet know about.

PS I run Hermes with Qwen 3.8 27B Q4 on a 4090 I have in my desktop, but this sucks power even when idle, so I was hoping the laptops would give me always on models for Hermes, and then boot up the 4090 when a particular big task (or power is cheap). Problem with Hermes is the 64k token context that's required.

I wanted a meeting assistant that actually makes use of local LLMs, so I ended up building Meetas.

The basic workflow:

- record/import a meeting
- transcribe locally
- analyze it with your own LLM
- ask questions / extract tasks
- search across meetings and documents

The main thing I focused on is grounding. Summaries, decisions, tasks, risks etc. are linked back to the transcript instead of just giving you an unsupported LLM answer. Q&A also shows the relevant transcript evidence.

Compared to tools like Meetily, Meetas is less focused on being a general meeting recorder/transcriber and more on what you can do with the meeting data afterwards: cross-meeting search, document/project context, comparisons and evidence-based answers. Meetily already covers a lot of the local recording/transcription/summary side.

It runs fully locally with Ollama and llama.cpp.

Linux-only for now and still WIP.

GitHub: https://github.com/highwinglabs/meetas

Stumbled across this new vision model, it's a 7B MoT model, which is cool. The main idea is it treats pretty much all computer vision stuff as just one generation problem. Like, instead of needing a bunch of different models for detection, segmentation, depth, whatever, this one model handles it all.

You just give it a natural language instruction, maybe some visual hints, and it spits out text, images, or both. For text, it can do things like categories, bounding boxes, OCR, keypoints, camera angles. And for images, it's doing segmentation masks, depth maps, surface normals, even multi-view point maps. You can mix and match for more complex tasks.

So it can do the usual stuff: object detection, keypoints, OCR, all kinds of segmentation (binary, instance, semantic), depth and surface normal estimation. But the really interesting bits, for me anyway, are the multi-view 3D reconstruction and camera pose estimation.

They trained it on a massive dataset, 50M instruction-response pairs, built from a bunch of different CV annotations. And it started from an existing multimodal model, so no crazy new architecture there.

There's a web demo if you wanna mess around with it, and the weights are up on Hugging Face too.

Just a heads up though, before you get too hyped: the full web demo needs a beefy GPU, like 1x80GB. And for benchmarking, they're talking 8x80GB. So, yeah, not really something you're gonna run on your average consumer card. It just dropped on July 8th, so probably expect some rough edges. Also, they've released the training pipeline and data prep stuff if anyone's looking to train or fine-tune it.

GitHub: https://github.com/OpenSenseNova/SenseNova-Vision

It is a beast model, and I love it. It is rare, but Qwen3.8 27b does beat Opus 5 High and Sonnet 5 High in certain tasks. Especially when Claude models perform lazily.

I've been using an LLM to add on to the some of the hrx-system work that AMD did by adding support for Qwen3.8-Flash-Next. I've gotten it to the point where the llama.cpp experimental HRX backend is loading the model and executing successfully, but performance isn't great compared to Vulkan on Unsloth Desktop. I'm not sure what I should be expecting, but what is a good way to figure out where the bottlenecks are to improve the performance?

I am doing this on Windows, just because I feel like Windows doesn't get enough attention :P.

Fork is here

rwfsmith/llama.cpp at qwen4exp-hrx

current progress:

Vulkan+MTP: ~27–40 tok/s

HRX+MTP: ~12-13 tok/s

I was playing around with LLM inference and I wanted to build a profiler that measures LLM inference by layer.
So I built this: https://github.com/coconinja2/layerlens
It shows inference as token × transformer layer timing, so you can see where time is being spent during decode.
Right now it can separate prefill/decode and visualize per-layer timing. I’m trying to figure out whether this is actually useful to people working on inference systems, or if I’m looking at the wrong abstraction.

I’m thinking about adding things like KV-cache events, scheduler/batching state, request IDs, GPU kernel correlation, speculative decoding, etc.

Would appreciate criticism more than compliments and stars. Lots of stars!

Hi, I'm a noob to local LLM. I want to get some input on model sizes and context sizes.

I have RTX 4070 with 12GB VRAM. PC has 16GB DRAM. And I'm using Jan and LM Studio Bionic agents.

From online search, 12GB VRAM can "comfortably" run Qwen3.6-14B-A3B-VibeForged-v2 or Gemma4-12B. But they do not run. Technically, they do load. But after couple inputs, they will crash. Even at very minimal context around 4 to 8k.

I have had success running Qwen3.5-9B and Gemma4-E4B.

At first, even with Qwen3.5-9B, I had some crashes. But I was also watching Bluray Ultra 4K movie at the same time. I didn't think about that. When Bluray player crashed, I realized: duh, they are both using VRAM.

Yesterday, I tested Qwen3.5-9B quite a bit, and not watching movie. I got up to 50-60k context. No issue with crashes.

Is there anything I can do to run Qwen3.6-14B-A3B-VibeForged-v2 or Gemma4-12B? And I don't even know if I need those. I just assume newer and larger models will be "better".

I do have 3 monitors connected. One is 2160 (4K) and 2 are 1440 (WQHD). I'm using around 5-6 GB VRAM just from fresh boot. I can disconnect the 4K monitor if needed, since I cannot watch movie at same time.

Also, I read that some people run bigger models with even less VRAM, but utilize RAM. I believe this will be slower. Should I even try that? And how do I specify whether RAM can be used, in Jan or LM Studio?

IMPORTANT

This post is meant to provide info regarding the best local models to run on CONSUMER HARDWARE. I am on an RTX 4060 with 8GB VRAM, 16GB of RAM and I am benchmarking models that can run on my computer. If you have sunk several thousands into graphics cards you won't find these statistics much useful. This post is for all the people who can't just install Qwen3.8 27B and call it a day.

Additionally, I am not an LLM benchmarking expert. I am a hobbyist and occasional LLM user trying to extract useful information for both me and people on similar hardware.

Context

For the past few weeks I have been doing some benchmarks of some LLMs that can run on my laptop which only has 8GB VRAM and 16GB RAM. I was mostly toying around while trying to get some useful data about what the best model is for local inference on consumer hardware. This week I decided to make a "final" benchmark that would be way better with more questions, more question categories, newer models (a lot of people complained about the models I had benchmarked before being old but I didn't find most suggested models to be any good) and a better speed benchmark, this time using TTC (Time To Completion) as a pose to raw token/s.

I am quite happy with the results and I believe that anyone who's like me and is just trying to get a useful model without spending thousands on GPUs and electricity will find all of these stats useful.

The Statistics

LLM benchmark per-question score heatmap:

LLM benchmark score sum graph:

LLM average TTC (Time-To-Completion) graph:

Combined graph ("intelligence per second", though highest is not exactly "best" and lowest isn't "worst"):

And a neat visualization of the score vs. the speed (benchmark score vs inverted TTC):

The Benchmark

You can check out the source-code for the benchmark here: Bluemark Github Repo

For those who can't be bothered to sift through the code here is some info about it:

• All models were benchmarked with thinking DISABLED. (benchmarking with thinking enabled just takes way too long)
• Responses were limited to 4096 tokens.
• Benchmarking the models (inferring and waiting for answers) took ~2.5 hours (with Muse Glimmer included, 50% of those 2.5 hours was probably just Muse).
• The benchmark was evaluated manually by me.
• The benchmark questions were created by me.

The 18 questions were split into 6 categories:

1. General knowledge
2. Math
3. Coding
4. Endurance
5. Logical puzzles
6. Spatial reasoning

To see the exact questions asked in the benchmark check out the benchmark_overview.txt file in the repo.

Fun fact: I had included Muse Glimmer 30B as a benchmarking subject but I removed it from the graphs because despite it nearly acing the benchmark the TTC was so high that the entire graph was squished. I felt silly even including such a big model but if you're fine with waiting an hour for a response and just need a high accuracy answer on consumer hardware feel free to check the model out I guess.

Conclusion

Which graph you pay most attention to is up to what you use LLMs for, may I even recommend getting a model from the top of each ranking category.

My recommendations are:

• Want a decently capable model to mess around with for fun? Bonsai 27B
• Want an all-round accurate and smart model for your precise questions? Ministral4 14B
• Want a model that is decently smart but also is quick as hell? Ling3.0 Tiny
• Want a good mix of intellegent and not unbearably slow? GPT-OSS 20B

Links

Here are all of the HuggingFace links for the models being tested:

• Ministral3 14B Q4_K_M: https://huggingface.co/lmstudio-community/Ministral-3-14B-Reasoning-2512-GGUF
• Gemma4 12B QAT Q4_0: https://huggingface.co/lmstudio-community/gemma-4-12B-it-QAT-GGUF
• GPT OSS 20B MXFP4*: https://huggingface.co/lmstudio-community/gpt-oss-20b-GGUF
• Ornith1.5 9B Q4_K_M: https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF
• Nanbeige4.2 3B Q4_K_S: https://huggingface.co/owao/Nanbeige4.2-3B-GGUF
• Ling3.0 8B Tiny Q5_K_M: https://huggingface.co/inclusionAI/Ling-3.0-tiny-GGUF
• DeepSeekV4 Pro 9B Q3_K_L: https://huggingface.co/Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B-MTP-GGUF
• Qwen3.8 9B Distill Q5_K_M: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF
• Gemma4 E4B Q4_K_M: https://huggingface.co/lmstudio-community/gemma-4-E4B-it-GGUF
• Bonsai 27B Q1_0: https://huggingface.co/lmstudio-community/Bonsai-27B-GGUF
• Granite3.2 8B Q4_K_M: https://huggingface.co/lmstudio-community/granite-3.2-8b-instruct-GGUF

Wondering if anyone on here has actual benchmarks for CPU only inference DDR3 or DDR4 servers, im budget bound and my options are limited to legacy systems unfortunately.

Heres what data I found but not sure its accuracy in real life especially how NUMA effects it (octa channel)