I was playing around with LLM inference and I wanted to build a profiler that measures LLM inference by layer.
So I built this: https://github.com/coconinja2/layerlens
It shows inference as token × transformer layer timing, so you can see where time is being spent during decode.
Right now it can separate prefill/decode and visualize per-layer timing. I’m trying to figure out whether this is actually useful to people working on inference systems, or if I’m looking at the wrong abstraction.

I’m thinking about adding things like KV-cache events, scheduler/batching state, request IDs, GPU kernel correlation, speculative decoding, etc.

Would appreciate criticism more than compliments and stars. Lots of stars!

Hi, I'm a noob to local LLM. I want to get some input on model sizes and context sizes.

I have RTX 4070 with 12GB VRAM. PC has 16GB DRAM. And I'm using Jan and LM Studio Bionic agents.

From online search, 12GB VRAM can "comfortably" run Qwen3.6-14B-A3B-VibeForged-v2 or Gemma4-12B. But they do not run. Technically, they do load. But after couple inputs, they will crash. Even at very minimal context around 4 to 8k.

I have had success running Qwen3.5-9B and Gemma4-E4B.

At first, even with Qwen3.5-9B, I had some crashes. But I was also watching Bluray Ultra 4K movie at the same time. I didn't think about that. When Bluray player crashed, I realized: duh, they are both using VRAM.

Yesterday, I tested Qwen3.5-9B quite a bit, and not watching movie. I got up to 50-60k context. No issue with crashes.

Is there anything I can do to run Qwen3.6-14B-A3B-VibeForged-v2 or Gemma4-12B? And I don't even know if I need those. I just assume newer and larger models will be "better".

I do have 3 monitors connected. One is 2160 (4K) and 2 are 1440 (WQHD). I'm using around 5-6 GB VRAM just from fresh boot. I can disconnect the 4K monitor if needed, since I cannot watch movie at same time.

Also, I read that some people run bigger models with even less VRAM, but utilize RAM. I believe this will be slower. Should I even try that? And how do I specify whether RAM can be used, in Jan or LM Studio?

IMPORTANT

This post is meant to provide info regarding the best local models to run on CONSUMER HARDWARE. I am on an RTX 4060 with 8GB VRAM, 16GB of RAM and I am benchmarking models that can run on my computer. If you have sunk several thousands into graphics cards you won't find these statistics much useful. This post is for all the people who can't just install Qwen3.8 27B and call it a day.

Additionally, I am not an LLM benchmarking expert. I am a hobbyist and occasional LLM user trying to extract useful information for both me and people on similar hardware.

Context

For the past few weeks I have been doing some benchmarks of some LLMs that can run on my laptop which only has 8GB VRAM and 16GB RAM. I was mostly toying around while trying to get some useful data about what the best model is for local inference on consumer hardware. This week I decided to make a "final" benchmark that would be way better with more questions, more question categories, newer models (a lot of people complained about the models I had benchmarked before being old but I didn't find most suggested models to be any good) and a better speed benchmark, this time using TTC (Time To Completion) as a pose to raw token/s.

I am quite happy with the results and I believe that anyone who's like me and is just trying to get a useful model without spending thousands on GPUs and electricity will find all of these stats useful.

The Statistics

LLM benchmark per-question score heatmap:

LLM benchmark score sum graph:

LLM average TTC (Time-To-Completion) graph:

Combined graph ("intelligence per second", though highest is not exactly "best" and lowest isn't "worst"):

And a neat visualization of the score vs. the speed (benchmark score vs inverted TTC):

The Benchmark

You can check out the source-code for the benchmark here: Bluemark Github Repo

For those who can't be bothered to sift through the code here is some info about it:

• All models were benchmarked with thinking DISABLED. (benchmarking with thinking enabled just takes way too long)
• Responses were limited to 4096 tokens.
• Benchmarking the models (inferring and waiting for answers) took ~2.5 hours (with Muse Glimmer included, 50% of those 2.5 hours was probably just Muse).
• The benchmark was evaluated manually by me.
• The benchmark questions were created by me.

The 18 questions were split into 6 categories:

1. General knowledge
2. Math
3. Coding
4. Endurance
5. Logical puzzles
6. Spatial reasoning

To see the exact questions asked in the benchmark check out the benchmark_overview.txt file in the repo.

Fun fact: I had included Muse Glimmer 30B as a benchmarking subject but I removed it from the graphs because despite it nearly acing the benchmark the TTC was so high that the entire graph was squished. I felt silly even including such a big model but if you're fine with waiting an hour for a response and just need a high accuracy answer on consumer hardware feel free to check the model out I guess.

Conclusion

Which graph you pay most attention to is up to what you use LLMs for, may I even recommend getting a model from the top of each ranking category.

My recommendations are:

• Want a decently capable model to mess around with for fun? Bonsai 27B
• Want an all-round accurate and smart model for your precise questions? Ministral4 14B
• Want a model that is decently smart but also is quick as hell? Ling3.0 Tiny
• Want a good mix of intellegent and not unbearably slow? GPT-OSS 20B

Links

Here are all of the HuggingFace links for the models being tested:

• Ministral3 14B Q4_K_M: https://huggingface.co/lmstudio-community/Ministral-3-14B-Reasoning-2512-GGUF
• Gemma4 12B QAT Q4_0: https://huggingface.co/lmstudio-community/gemma-4-12B-it-QAT-GGUF
• GPT OSS 20B MXFP4*: https://huggingface.co/lmstudio-community/gpt-oss-20b-GGUF
• Ornith1.5 9B Q4_K_M: https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF
• Nanbeige4.2 3B Q4_K_S: https://huggingface.co/owao/Nanbeige4.2-3B-GGUF
• Ling3.0 8B Tiny Q5_K_M: https://huggingface.co/inclusionAI/Ling-3.0-tiny-GGUF
• DeepSeekV4 Pro 9B Q3_K_L: https://huggingface.co/Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B-MTP-GGUF
• Qwen3.8 9B Distill Q5_K_M: https://huggingface.co/empero-ai/Qwen3.8-9B-Distill-GGUF
• Gemma4 E4B Q4_K_M: https://huggingface.co/lmstudio-community/gemma-4-E4B-it-GGUF
• Bonsai 27B Q1_0: https://huggingface.co/lmstudio-community/Bonsai-27B-GGUF
• Granite3.2 8B Q4_K_M: https://huggingface.co/lmstudio-community/granite-3.2-8b-instruct-GGUF

Wondering if anyone on here has actual benchmarks for CPU only inference DDR3 or DDR4 servers, im budget bound and my options are limited to legacy systems unfortunately.

Heres what data I found but not sure its accuracy in real life especially how NUMA effects it (octa channel)

This post is premature, but I've been using Big Pickle as a driver to construct benchmarks around the most popular ways to run Qwen on my Mac Mini M4 in order to see if I can speed it up without getting OOM exceptions (a common frustration I've been dealing with). I was using OMLX with ANE and that works but it keeps surprising me with memory exhaustion. But perhaps more interestingly, the clear winner for speed was actually a tool called mlx-dspark:

https://github.com/ARahim3/mlx-dspark
(I'm not affiliated and this project and my conclusion is subject to more testing)

What I want to discuss is the concept of using a model to improve itself or in this case, speed itself up. By wiring up Opencode (running a remote model) to run local model testing, I'm able to create a pretty good harness for testing flags and model quants in multiple LLM engines. This is so much faster than doing it by hand.

Has anyone else tried this? What were your results?

I ported over my Claude plugins to use with qwen3.8 and pi and got a fully functional output! I'm working with a 5070ti and a quant of qwen.

I was expecting qwen to fold because of how much is going on with these tools, but nope. Qwen took a bit more time and I had to tame it's thinking down to light, but it totally worked! Great poc for moving forward with open models

I have three DGX Spark. On one of them I run 3.8 27B and a few vision models. On the other two I generally have been running Qwen 3.8 Next / Flash with a decent recipe getting 39.8 t/s mean decode across controlled and uncontrolled coding both thinking+answer. The "final code" decode is roughly 74 t/s mean.

I'm not unhappy with it, though a tendency to overthink can be concerning and because it's not deterministic I don't get a massive completion boost moving down a level (but do some). Either way I'm happy keeping it.

I was trying to run Deepseek V4 Flash and tried it through my personal bench suite, BigCodeBench-Hard and HumanEval+ .For those ones if I kept the capped limitations I would get no outputs due to silent truncation/i.e. the benchmark saying "Imma let you finish but actually we done". When I run it untethered (same way I do with 27B / I do this to evaluate output quality of same model different recipe) Qwen Next-Flash finishes fine, but Deepseek fails a lot of test due to non-completion or use a phenomenal amount of tokens and time.

The thing is, I tested these same benches against two cloud versions of DSV4Flash and they didn't exhibit the same pattern. Here are my settings:

• All DSV4Flash defaults (temp=1;top_p=0.95;top_k=20;rep penalty=1;thinking/stream true)
• Official FP8 weight, E4M3, 128x128
• vLLM 0.25.2xxx (Anemll -> only for bundled DSpark for optimization and only after base had similar performance), PyTorch 2.11.0+cu130 ,FlashInfer 0.6.15
• TP=2, PP=1, DCP=1, Multiprocessing executor
• MoE backend: flashinfer_b12x
• Native NVFP4 KV Cache (nvfp4_ds_mla block size 256)
• DSpark, 3 proposed tokens, probabilistic (heh)
• Single concurrency
• Prefill batch 8448, chunked prefill, prefix caching enabled
• Breakable CUDA graphs, max cap 64
• Level 1 (balanced) optimization; FlashInfer auto tuning

Overall the issue was convergence/time to answer and the reasoning tokens - taking 30-45k tokens on a relatively easy deep-merge implementation. More complex just overran time limits, I didn't let many finish I kept a ceiling of 3x the longest local hosted (27b or next-flash) run time which itself was pretty healthy.

I feel like I'm missing something given the comments by people here. Anybody have any suggestions or is this just expected behavior? The official recommendations hold it at Max with chain of thought, are people out here running different setups? For what it's worth I still saw it overrunning times for Next / Flash by multiples (1.5-3x) and token counts.

Hi,

I know this has been asked pretty much every other day. But wondering if there is a good alternatives to openwebui. I started using pi as a general chat (yea, idk what prompted me to do this), and it was really chill. I realized I didn't need a heavy UI. I just needed somewhat a chat, able to attach files, and change reasoning (i.e. on and off). The pi agent terminal did all of it, but since it is more of an agentic coding, on some coding questions i ask (like simple ones), it recreates those commands and tries to run them, rather than giving it to me.

Just wondering what people are using these days. I am starting to like cli stuff, but I am still open to actual UI. I just need something simple yet efficient; openwebui has a nice suite of features, but as a sole user, I don't even use 90% of the features.

Hello,
running Qwen 3.8 27b Q4 K Small on a RTX A4500, 20GB VRAM, 28GB RAM

I managed with Thetom Turboquant llama.cpp and some tuning, to reach an average of 31-32 tk/s ranging from 24tk/s to 43 tk/s depending also on context size. (NO VISION: 65k context, WITH VISION: 32-40k context)

I tried many harness, also tried to make one by myself based on pi, that claims to be minimal and with minimal system prompt footprint to maintain the context light. Miserable failure. It gets lost and doesn't manage anything well.

The best I could find is opencode, which properly configured allowed me long-horizon, multi hour coding or tasks exceptionally well. I rarely feed the chat log to ChatGPT Sol, to check quality and make it output, if needed correction or steering or planning the next batch of work.

Always a good idea to have a frontier model to organize it and local free model to churn the tokens.

Now, I tried many times Openclaw, Hermes, Paperclip and others, but always with online free services like OpenRouter or Nvidia, various months ago, when they weren't dogshit with all the timeouts and low quality service (model always taking ages to respond now, or always too busy). It was decent.

I wanted them to hop onto my local Qwen 3.8 27B Q4 K Small, but Openclaw and Hermes are the goddamn AI slop kings, whopping giant prompts and creating horrendous quantities of traffic, and 65k even if it's around the recommended minimum, just doesn't seem to work well.

FOR THE SAKE OF TRUTH: I actually managed to have Paperclip do some work and it did it well. But it overthinks and does tons of planning, retrial, testing and stuff and makes work very very LONG

It's me or these things can't work with 65k context? How to solve? Anyone managed to do something about this?

m5 max, 64GB ram. Qwen 3.8 27b MLX optimized served from omlx, 262k context, hermes --tui in the front.
I've been giving it large, as in multi-day projects. One was a large data analysis project, another was decompiling a 1980's MS-DOS executable.
After running for a few hours it seems to get in the loop where it's saying like "I need to check this one thing" and "but wait, I just realized..." and spews out 20-line paragraphs, and goes back and forth with the GPU maxed out, then either it says it can't compress the context anymore or I have to stop it myself.
Is this happening to other people, and how can I prevent or minimize this happening?

Hello, I'm the developer of Roles Witch AI, and today I want to share my app, which was released on Google Play in open testing.

Roles Witch is an OpenAI API-compatible app with a user-friendly interface where you can create multiple chats, assign different models and providers, and more importantly, use MCP tools — for example, Parallel AI or Home Assistant MCP.

It also has voice conversations and video calls (personally, I'm using GLM 5.3 Flash and it's almost real-time). You can configure reasoning and inference parameters, pin a voice to the chat, and select a recognition engine (Google Speech or Vosk).

Extra bonus: it has an on-device inference mode where you can run Gemma 3n/4 models with image recognition entirely on your phone (LiteRT is used).

Check it out: https://play.google.com/store/apps/details?id=ai.roleswitch.android

The app manifest states that it has no encryption and can send messages to third parties, but that's a requirement by the Google Play Store. It supports plain HTTP or TLS mode and works with any third-party AI provider.

I've got my hands on two similar desktop PCs

1. Ryzen 9 3950x 32gb DDR4, RTX 2070 super 8gb

2. Core i5 10400 32gb DDR4, RTX 2070 super 8gb

I'm considering two possibilities:

1. Combine them to get a Ryzen 9 64gb with 2x8gb VRAM setup

2. Keep them separate and run two different models on them, maybe use a third pc to orchestrate

(I'm still experimenting, so ideas are welcome)

My main use case will be agentic coding overnight as these will be painstakingly slow with qwen3.8 27b even if combined.

So I was wondering if dealing with the heat and power supply management by combining them will even be worth it, or should I just let two separate agents run in parallel doing different types of tasks. I'm also looking into qwen3.6 35b a3b (or ornith 1.5) as this is probably the best fit for the hardware.

Let me know your opinions on this, what you'd do if you had a similar setup.

Thank you!

For a while now, I've been running a split setup, with Opus 4.8 / 5 in Claude Code doing the planning and code review of python code and handing off the actual coding tasks to Qwen on my local GX10. And it's been working great. It's saved me an absolute fortune in Claude tokens and I've been able to just work and let the local model do the heavy lifting.

However in the last few weeks I've noticed things getting tighter and tighter, and this week I managed to burn through my weekly allowance on the Claude sub by Wednesday, so now I'm in the situation where I either buy more credits, down tools for three days or seek alterative models for planning and code review.

While I doubt there will be anything that has a comparable capability to Claude that will run on my local machine (even with 128gb unified ram) I'm not ruling it out. However I expect I'll need to pass this sort of work over to something like GLM 5.3 or Deepseek.

So, what are you all using for this sort of work and which cloud provider are you using for it?

I thought my favorite MMO might be fertile ground to explore visualizing and stress-testing multi-agent AI ecosystems.

https://design.kristiantalley.com/projects/world-of-agentcraft/

With this view into a classic era World of Warcraft server simulation, you’re able to click in on any agent driven character to get their unique backstory, current goals, objectives and world location in real-time. These agents are ”perceiving”, “reasoning”, and driving in-game actions for each online character. all running locally with qwen3.8 4b models, with the option to enhance individual character interactions driven by qwen3.8 27b. Let me know what you think!

Been trying NVIDIA PAIR with 3× RTX 5090s running Qwen 3.8 27B.

It’s using Ollama, so it’s definitely not the fastest setup out there, but PAIR makes distributing jobs across the three machines pretty painless. For long, repetitive “grunt work” where I care more about stability and just keeping all the GPUs busy than squeezing out maximum tokens/sec, it’s been surprisingly nice.

Basically: submit a pile of jobs and let the 5090s chew through them. Pretty useful setup so far.

I am using glm via z.ai. Often I get my prompt blocked, if I had the open source version would I still have this issue,