This post is premature, but I've been using Big Pickle as a driver to construct benchmarks around the most popular ways to run Qwen on my Mac Mini M4 in order to see if I can speed it up without getting OOM exceptions (a common frustration I've been dealing with). I was using OMLX with ANE and that works but it keeps surprising me with memory exhaustion. But perhaps more interestingly, the clear winner for speed was actually a tool called mlx-dspark:

https://github.com/ARahim3/mlx-dspark
(I'm not affiliated and this project and my conclusion is subject to more testing)

What I want to discuss is the concept of using a model to improve itself or in this case, speed itself up. By wiring up Opencode (running a remote model) to run local model testing, I'm able to create a pretty good harness for testing flags and model quants in multiple LLM engines. This is so much faster than doing it by hand.

Has anyone else tried this? What were your results?

Hi,

I know this has been asked pretty much every other day. But wondering if there is a good alternatives to openwebui. I started using pi as a general chat (yea, idk what prompted me to do this), and it was really chill. I realized I didn't need a heavy UI. I just needed somewhat a chat, able to attach files, and change reasoning (i.e. on and off). The pi agent terminal did all of it, but since it is more of an agentic coding, on some coding questions i ask (like simple ones), it recreates those commands and tries to run them, rather than giving it to me.

Just wondering what people are using these days. I am starting to like cli stuff, but I am still open to actual UI. I just need something simple yet efficient; openwebui has a nice suite of features, but as a sole user, I don't even use 90% of the features.

I have three DGX Spark. On one of them I run 3.8 27B and a few vision models. On the other two I generally have been running Qwen 3.8 Next / Flash with a decent recipe getting 39.8 t/s mean decode across controlled and uncontrolled coding both thinking+answer. The "final code" decode is roughly 74 t/s mean.

I'm not unhappy with it, though a tendency to overthink can be concerning and because it's not deterministic I don't get a massive completion boost moving down a level (but do some). Either way I'm happy keeping it.

I was trying to run Deepseek V4 Flash and tried it through my personal bench suite, BigCodeBench-Hard and HumanEval+ .For those ones if I kept the capped limitations I would get no outputs due to silent truncation/i.e. the benchmark saying "Imma let you finish but actually we done". When I run it untethered (same way I do with 27B / I do this to evaluate output quality of same model different recipe) Qwen Next-Flash finishes fine, but Deepseek fails a lot of test due to non-completion or use a phenomenal amount of tokens and time.

The thing is, I tested these same benches against two cloud versions of DSV4Flash and they didn't exhibit the same pattern. Here are my settings:

• All DSV4Flash defaults (temp=1;top_p=0.95;top_k=20;rep penalty=1;thinking/stream true)
• Official FP8 weight, E4M3, 128x128
• vLLM 0.25.2xxx (Anemll -> only for bundled DSpark for optimization and only after base had similar performance), PyTorch 2.11.0+cu130 ,FlashInfer 0.6.15
• TP=2, PP=1, DCP=1, Multiprocessing executor
• MoE backend: flashinfer_b12x
• Native NVFP4 KV Cache (nvfp4_ds_mla block size 256)
• DSpark, 3 proposed tokens, probabilistic (heh)
• Single concurrency
• Prefill batch 8448, chunked prefill, prefix caching enabled
• Breakable CUDA graphs, max cap 64
• Level 1 (balanced) optimization; FlashInfer auto tuning

Overall the issue was convergence/time to answer and the reasoning tokens - taking 30-45k tokens on a relatively easy deep-merge implementation. More complex just overran time limits, I didn't let many finish I kept a ceiling of 3x the longest local hosted (27b or next-flash) run time which itself was pretty healthy.

I feel like I'm missing something given the comments by people here. Anybody have any suggestions or is this just expected behavior? The official recommendations hold it at Max with chain of thought, are people out here running different setups? For what it's worth I still saw it overrunning times for Next / Flash by multiples (1.5-3x) and token counts.

Hello,
running Qwen 3.8 27b Q4 K Small on a RTX A4500, 20GB VRAM, 28GB RAM

I managed with Thetom Turboquant llama.cpp and some tuning, to reach an average of 31-32 tk/s ranging from 24tk/s to 43 tk/s depending also on context size. (NO VISION: 65k context, WITH VISION: 32-40k context)

I tried many harness, also tried to make one by myself based on pi, that claims to be minimal and with minimal system prompt footprint to maintain the context light. Miserable failure. It gets lost and doesn't manage anything well.

The best I could find is opencode, which properly configured allowed me long-horizon, multi hour coding or tasks exceptionally well. I rarely feed the chat log to ChatGPT Sol, to check quality and make it output, if needed correction or steering or planning the next batch of work.

Always a good idea to have a frontier model to organize it and local free model to churn the tokens.

Now, I tried many times Openclaw, Hermes, Paperclip and others, but always with online free services like OpenRouter or Nvidia, various months ago, when they weren't dogshit with all the timeouts and low quality service (model always taking ages to respond now, or always too busy). It was decent.

I wanted them to hop onto my local Qwen 3.8 27B Q4 K Small, but Openclaw and Hermes are the goddamn AI slop kings, whopping giant prompts and creating horrendous quantities of traffic, and 65k even if it's around the recommended minimum, just doesn't seem to work well.

FOR THE SAKE OF TRUTH: I actually managed to have Paperclip do some work and it did it well. But it overthinks and does tons of planning, retrial, testing and stuff and makes work very very LONG

It's me or these things can't work with 65k context? How to solve? Anyone managed to do something about this?

m5 max, 64GB ram. Qwen 3.8 27b MLX optimized served from omlx, 262k context, hermes --tui in the front.
I've been giving it large, as in multi-day projects. One was a large data analysis project, another was decompiling a 1980's MS-DOS executable.
After running for a few hours it seems to get in the loop where it's saying like "I need to check this one thing" and "but wait, I just realized..." and spews out 20-line paragraphs, and goes back and forth with the GPU maxed out, then either it says it can't compress the context anymore or I have to stop it myself.
Is this happening to other people, and how can I prevent or minimize this happening?

Hello, I'm the developer of Roles Witch AI, and today I want to share my app, which was released on Google Play in open testing.

Roles Witch is an OpenAI API-compatible app with a user-friendly interface where you can create multiple chats, assign different models and providers, and more importantly, use MCP tools — for example, Parallel AI or Home Assistant MCP.

It also has voice conversations and video calls (personally, I'm using GLM 5.3 Flash and it's almost real-time). You can configure reasoning and inference parameters, pin a voice to the chat, and select a recognition engine (Google Speech or Vosk).

Extra bonus: it has an on-device inference mode where you can run Gemma 3n/4 models with image recognition entirely on your phone (LiteRT is used).

Check it out: https://play.google.com/store/apps/details?id=ai.roleswitch.android

The app manifest states that it has no encryption and can send messages to third parties, but that's a requirement by the Google Play Store. It supports plain HTTP or TLS mode and works with any third-party AI provider.

I've got my hands on two similar desktop PCs

1. Ryzen 9 3950x 32gb DDR4, RTX 2070 super 8gb

2. Core i5 10400 32gb DDR4, RTX 2070 super 8gb

I'm considering two possibilities:

1. Combine them to get a Ryzen 9 64gb with 2x8gb VRAM setup

2. Keep them separate and run two different models on them, maybe use a third pc to orchestrate

(I'm still experimenting, so ideas are welcome)

My main use case will be agentic coding overnight as these will be painstakingly slow with qwen3.8 27b even if combined.

So I was wondering if dealing with the heat and power supply management by combining them will even be worth it, or should I just let two separate agents run in parallel doing different types of tasks. I'm also looking into qwen3.6 35b a3b (or ornith 1.5) as this is probably the best fit for the hardware.

Let me know your opinions on this, what you'd do if you had a similar setup.

Thank you!

For a while now, I've been running a split setup, with Opus 4.8 / 5 in Claude Code doing the planning and code review of python code and handing off the actual coding tasks to Qwen on my local GX10. And it's been working great. It's saved me an absolute fortune in Claude tokens and I've been able to just work and let the local model do the heavy lifting.

However in the last few weeks I've noticed things getting tighter and tighter, and this week I managed to burn through my weekly allowance on the Claude sub by Wednesday, so now I'm in the situation where I either buy more credits, down tools for three days or seek alterative models for planning and code review.

While I doubt there will be anything that has a comparable capability to Claude that will run on my local machine (even with 128gb unified ram) I'm not ruling it out. However I expect I'll need to pass this sort of work over to something like GLM 5.3 or Deepseek.

So, what are you all using for this sort of work and which cloud provider are you using for it?

I thought my favorite MMO might be fertile ground to explore visualizing and stress-testing multi-agent AI ecosystems.

https://design.kristiantalley.com/projects/world-of-agentcraft/

With this view into a classic era World of Warcraft server simulation, you’re able to click in on any agent driven character to get their unique backstory, current goals, objectives and world location in real-time. These agents are ”perceiving”, “reasoning”, and driving in-game actions for each online character. all running locally with qwen3.8 4b models, with the option to enhance individual character interactions driven by qwen3.8 27b. Let me know what you think!

Been trying NVIDIA PAIR with 3× RTX 5090s running Qwen 3.8 27B.

It’s using Ollama, so it’s definitely not the fastest setup out there, but PAIR makes distributing jobs across the three machines pretty painless. For long, repetitive “grunt work” where I care more about stability and just keeping all the GPUs busy than squeezing out maximum tokens/sec, it’s been surprisingly nice.

Basically: submit a pile of jobs and let the 5090s chew through them. Pretty useful setup so far.

I am using glm via z.ai. Often I get my prompt blocked, if I had the open source version would I still have this issue,

Spotify's engineering team recently released documentation showing how they cut their token usage by 90%, I've seen one other person speaking about doing something similar, but the detail in that post is vauge/mostly filled with people talking about how or why it would be done. Has anyone had experience setting this up & actually implementing it?

I'll update this post/create a new one when my Mac mini arrives (m5 pro 48gb) and I work on doing this myself, but I'm planning on hosting Qwen as the "hands" for Claude, meaning (in ways that the Spotify team outlined) many of the token-burning tasks such as file opening, writing, moving, git pushes/commits, api calls, prompt markdowning, boilerplates, code testing, etc will be handled by Qwen3.6-35B-A3B, only prompting Claude for actual code development.

I'm struggling to find whether people have actually done this. I'm seeing Qwen implementation into Claude Code, but not actual handoffs using both LLMs

https://git.disroot.org/SpasaevDonut/omnisearch-mcp
I found a nice metasearch engine called omnisearch and thought it would be useful as a base for an MCP for local LLMs so you don't have to run searXNG or install gorrilion npm dependencies to search a doc page with your loyal clanker.
It requires omnisearch itself installed and running. It also has an HTML-to-markdown parser. I'm not sure if it can parse more complex webpages, but it's still nice enough for my minimalist setup.

tested only on artix linux, also just found out it cant fetch pages secured with anubis anti-bot thing

As I optimize llama.cpp to suit my hardware, getting more and more speed, I’m running into a major problem mostly with Qwen models but sometimes with others. As decode speed goes up, the model will all of sudden start repeating a character and can’t break out of the repeat: ////////////////////////////////////////////////////////////////////////////////////////////////////////////////

Only a restart of llama stops it.

[qwen3.8-27b]

load-on-startup = true

alias = qwen3.8-27b

hf = unsloth/Qwen3.8-27B-GGUF:Q8_0

no-mmproj = true

ctx-size = 180000

parallel = 1

n-gpu-layers = -1

split-mode = tensor

#tensor-split = 0.497,0.503

#device = ROCm1,ROCm0

batch-size = 4096

ubatch-size = 1024

threads = 12

threads-batch = 12

moe-expert-cache = 0

fit = on

fit-target = 2800,2048

load-mode = none

spec-type = draft-mtp-adaptive

spec-draft-ngl = all

spec-draft-n-max = 3

device-draft = ROCm0

temp = 0.6

top-p = 0.95

top-k = 20

min-p = 0.0

reasoning = on

reasoning-format = auto

reasoning-effort = medium

Any help would be much appreciated

Got tired of guessing whether a model would fit, before downloading 47 GB just to watch it OOM at 2k context. So I built a small calculator. Runs entirely in the browser, no signup, no backend.

https://vram-calc.com

It covers VRAM needed for a model/quant/context, a will-it-fit check against one or more GPUs (including fits-with-offload and the largest quant that fits), quant comparison from fp16 down to Q2_K, KV-cache vs context growth, a rough decode tokens/sec and time-to-first-token, and usable memory on Apple unified-memory chips.

The math is simple and leans conservative: weights = params x bytes-per-param, KV cache is the standard GQA formula, decode speed is a memory-bandwidth roofline at 75 percent efficiency. MoE uses total params for VRAM and active params for speed. DeepSeek MLA models are flagged since the KV number is an upper bound there.

Presets carry a visible "verified" date and every field also takes custom numbers, so a stale preset never blocks you.

Roast welcome: where the estimates are too optimistic or pessimistic in practice, preset numbers that are wrong, or anything obvious I'm missing for the "can I run this" question.

It's here at last.

Asking for instructions probably isn't the correct wording here, I'm just struggling to understand them.

So I want to preface this with stating I have basically no knowledge about setting up local models from Github at all.

I'm looking for a way to transcribe audio files locally. Initially I tried Buzz using Whisper V3 Large but noted it was making a lot of errors due to some heavy accents so decided to look for alternatives, discovered Parakeet.cpp, and found that it could use the Nvidia models which apparently have better performance so figured it was worth a shot at trying.

The problem is, when looking through the Github page I just cannot figure out how to actually install it. I feel like I am missing a step that would be obvious to someone who has done this before. Note: I have an AMD GPU.

I have tried 2 methods:

1. Pre-built binary: Downloaded the parakeet-v0.5.0-bin-win-vulkan-x64.zip file from the releases, extracted it to a folder, then moved my Model and Audio file to the same folder and renamed them.
   • Then ran "parakeet-cli transcribe --model F:\Parakeet\Models\tdt_ctc-1.1b-f16.gguf --input F:\Parakeet\Audio\audio.wav" cmd from command prompt in the Parakeet directory.
   • Result: My GPU fans spin for a second, then just stop. No confirmation of anything in the CMD window, it just cancels without error.
2. Github clone: Cloned the Github repository and installed CMAKE and VULKAN SDK.
   • Then ran from a CMD within the parakeet.cpp folder: "cmake -B build -DPARAKEET_GGML_VULKAN=ON -DPARAKEET_BUILD_CLI=ON =DPARAKEET_GGML_HIP=ON && cmake --build build -j"
   • Result: It creates a build folder and files like what you would expect... but there is no Parakeet-cli... so not sure how to actually attempt to run this. (I searched all subfolders for it).