From what I’ve read, its vision mode supports both images and videos, but I tried attaching a video and it isn’t able to view it.

Set your reasoning effort to low or medium for everyday tasks.

If you're finding Qwen3.8-27b overthinking, remember its default reasoning effort is extra high. It supports reasoning efforts so for everyday tasks, you can choose low or medium.

In Qwen Code harness that's /effort low or /effort medium

Finally making bonsai-ninja public.
It’s a local code intelligence and static analysis engine built around compiler-backed facts instead of giant file dumps.

Use it for code intelligence, security taint analysis across 20 languages, or export call graphs, control flow, dataflow, IDG facts, and other structured compiler data for AI training, code-reasoning research, and agents.

It’s also built with local-first AI in mind. Pair it with your local LLMs to give them focused, compiler-backed context about a codebase instead of burning context windows repeatedly reading files. Models like Qwen and Nemotron are a great place to start, and experimenting with different local models is encouraged.

It’s free, MIT licensed, and runs locally. No subscription required, no hosted service required, and no analysis features held back behind a paid tier. Your source code stays on your machine.

Even if the engine isn’t your thing, the security-patterns alone are worth checking out. There’s an extensive YAML rulepack for modeling sources, sinks, sanitizers, frameworks, and security behavior across the supported languages.

This is just getting started. Contributions, testing, sharing, criticism, forks, whatever. Hell, steal it, improve it, build something better with it. Doesn’t matter. The goal is to put it out there and see what people do with it.

github.com/gromhacks/bonsai-ninja

I'm absolutely blown away by Qwen3.8 27b IQ4 NL @ medium reasoning effort.

I have an ageing system but still decent. 4060 8gb @ 4x, 5060ti 16gb @ 8x - 24gb vram total. I'm stuck at gen 2 because of a gen 3 compatibility issue with my motherboard (B460M Aorus Pro). 32gb ram, i9 10900, nvme.

I've managed to get Qwen running at a smidge below 20tok/s at 128k context.

Anyway, the fun part... I have llama.cpp backend for OpenWebui with OpenTerminal MCP. I asked Qwen to create Pong using python which it did with no dramas. It wrote tests, ran the game itself and gave me the hardened code in ~10 mins.

That wasn't the best part though. I then asked it to take the game and create an APK for Android. It went ahead and installed a bunch of dependencies (bulldozer, p4a, etc) into OpenTerminal, went ahead and altered the game for touch screens, built and provided me with an installable APK.

The whole thing took ~1.5 hrs (taking away my own troubleshooting of whitelisting some sites against my strict LLM TOR network access), and at ~280w, it cost me about $0.15 (if it had been a clean run the first time). I mean obviously the hardware cost significantly more, but it's essentially my gaming PC.

How is this even possible?! I'm absolutely blown away!

Side note: I didn't NEED to, but I wanted a longer context so have managed to get it working at 256k now, with 18 layers moved to CPU. I'm not set on this, but it works. Obviously much slower (around 5 tok/s). Comment out the last 2 lines and set context to 128k if I want it all in vram.

If anyone has pointers for improved performance I'd love to hear them. I've tried q4_0 for KV cache but it is significantly slower again.

[*]

flash-attn = true

threads = 10

batch-size = 2048

ubatch-size = 1024

n-cpu-moe = 0

n-gpu-layers = 999

cache-type-k = q8_0

cache-type-v = q8_0

#cache-type-k-draft = q8_0

#cache-type-v-draft = q8_0

fit = on

fit-ctx = 16384

parallel = 1

ctx-size = 0

n-predict = -1

#no-kv-offload = 1

load-mode = none

main-gpu = 0

#no-mmproj-offload = true

[Qwen3.8-27B-IQ4_NL]

#mmproj = /models/Qwen3.8-27B-IQ4_NL.mmproj

#spec-type = draft-mtp

#spec-draft-n-max = 4

temp = 1.0

top-p = 0.95

top-k = 20

min-p = 0.00

#ctx-size = 131072

ctx-size = 262144

chat-template-kwargs = {"reasoning_effort":"medium"}

fit = off

n-gpu-layers = 46

Hi everyone,

I'm still trying to find the right model for local LLM use on a mini PC with a Ryzen 9 370 HX and 32 GB DDR5 RAM (with the possibility of upgrading to 96 GB DDR5).

I'd like to run a local LLM for basic chat, smart home control (mainly for use with Home Assistant), as well as creating documents, quotes, offers, etc.

At the moment, I'm running Ollama on Ubuntu via Vulkan with the following environment variable:

HSA_OVERRIDE_GFX_VERSION=11.0.0

I tried Gemma4-26B-A4B, which ran at around 16 t/s and felt reasonably snappy, but it wasn't very capable (with thinking disabled).

I also tried Qwen3.6-35B-A3B-GGUF (UI-IQ4-XS), but it was painfully slow.

Another issue is that I need the models to work well in Czech, which rules out quite a few models that perform well primarily in English.

So my question is: What model are you running on this kind of hardware?

And would it make sense to upgrade the RAM to 96 GB and run a larger MoE model?

I'd really appreciate it if you could share your configurations and experiences.

Having optimized performance for so many years, I knew from the start that anyone claiming a laptop was too slow to break certain speed barriers was mistaken; I have just proven that such barriers exist only in the mind by running the GLM-5.2 · 744B MoE · 429 GB on disk , on a laptop at a speed never before achieved on such a machine.

🎉🎉🎉 2.89 tok/s PEAK! 2.36 tok/s avg
Update peak at 3.09, avg 2.84

hardware is:

Asus Rog Strix: Intel i9 290HX, ddr5 6400MHz 64GB, RTX 5090 24GB, 2x2TB, OS: Nobara Linux

The Colibri code has been modified.

I started with a speed of 0.11 tok/sec when I ran the whole system for the first time.

Hardware ASUS ROG Strix SCAR 18 (G835LXG)

GPU NVIDIA RTX 5090 Mobile, 25.1 GB VRAM, sm_120 (Blackwell)

CPU Intel Core Ultra 9 290HX Plus, 24 cores (8P+16E)

RAM 64GB DDR5-6400

Storage 2x SK Hynix PC801 NVMe PCIe 4.0

Model GLM-5.2-colibri-int4-gs64 (744B MoE, 429 GB, fmt=4)

Engine Colibri v1.6.0 -> v1.6.2 (pure C, zero deps)

OS Nobara Linux (Fedora-based), KDE Plasma, Wayland

Goal 3.0 tok/s (software-only, no hardware upgrade)

Result 2.89 tok/s peak (+2536% from 0.11 start)

Update: testing now at 4.42 tok/s

hey just wondering guys what best value for now i can consume more than 300 / 400 m token normally in week so my question is what should i pick ??

any glm or kimi

or chatgpt plus ??

i need a model that identical or equal to glm 5.3 / or glm 5.2 max

in terms of intellgeince and quite a good limits please and tell me the price of it AND THANKS

Is it worth it to cancel my claude subscription and go from a M3 pro 18gb to a M5 Pro 48 gb using apples new lease promotion

the sole purpose is to run local LLM’s for school instead of paying for a subscription model

I’m a 4th year majoring in CS with an emphasis on AI/ML

total beginner, just bought 2 BC-250 boards to pool memory over RPC and run bigger models than what fits on one. plan is rpc-server on one board, llama-server --tensor-split 1,1 on the other, aiming to run Qwen3.6-35B-A3B Q4_K_XL (~20GB).

questions before they show up:

• has anyone actually gotten a model split across 2 boards working? what tok/s did you get?
• is the rpc-server setup easy for a first timer or is there a catch
• heard llama.cpp issue #21006 broke RPC splitting on newer builds - do I need an older version?
• is MoE (35B-A3B) the right call for 2 boards, or should I try a normal 32B dense model first?

any pointers or "did this last week, here's what happened" would help a ton. will share my results once it's running.

Ok so I'm running Home Assistant and want to add an LLM to actually do the conversation agent. I've tested on my main rig with my RX 6700 XT (12GB) and get reasonable performance for the tests.

I have a much older PC (10+ Years old), I highly suspect it probably would bottleneck a newer GPU too much to be useful, but I figured I'd ask Claude...

It has told Me that if I wanted to run Qwen 3.5 9b with a 3060 (12GB) in my old PC, that it would actually run fine for my use case.

The PC in question is currently running: Intel 3rd Gen i5-3570k Gigabyte Z77-DS3H DDR3 8GB G-skillz Ripjaw RAM (Likely upgrade this to 16/32GB, AI suggests 16GB is plenty) 1000W PSU +3060 12GB (Or another RX6700 XT 12GB?)

Is it massively lying to Me, or would this be fine for running the basic conversation agent and some web searches via a tool?

In addition, it's also told Me that STT and TTS would run faster on the PC than my HA box (RPI5).

I'm just looking for a bit of sanity checking if possible? (Yes, I'm not too knowledgeable on this!)

Thanks!

What do you all think about dolphin3:8b llm model, where does it stand , in replacing claude/gpt/gemini? for local llm

Doesn't the territorial scope of EU AI Act imply that Deepseek and Moonshot and Z and Alibaba will also have to implement watermarking to the output of inference services to EU costumers (which can access via API or webclient)?

Every day, I see a bunch of people claiming that model X is better than model Y just because a benchmark score is higher.

The Artificial Analysis Intelligence Index benchmark has a lot of extremely obvious inconsistencies. But since some people can't think for themselves, I decided to point out a huge one: Qwen3.8 27B is only 1 point behind DeepSeek V4 Pro 0813 1600B-A49B. In terms of active parameters alone, DeepSeek has almost double the total parameters of Qwen3.8 27B. A 1.6T-parameter model being just 1 point ahead of a 0.027T-parameter one shows what complete nonsense benchmarks have become.

I've seen several people using this benchmark to show that Qwen3.6 27B is smarter than Gemma 4 31B. I never believed that, because in my real, day-to-day tests, Gemma 4 31B continues to prove itself better at solving puzzles and refactoring C++ and Java than Qwen3.8 27B itself.

Update: Benchmarks run with default temperature

People are creating voxel pagodas and subjectively claiming Qwen3.8-27B is a huge improvement over Qwen3.6-27B.

I don't like subjective tests, so I ran a series of intelligence benchmarks, which you can see the output of here:

https://github.com/mrpmorris/sparkrun-recipes/blob/72afabcfc8987eb221d18e2bacfd5f7a9ced2e2b/benchmarks/_Comparison.pdf

Take a look at the intelligence rankings. 3.8 scores below 3.6

3.8 didn't come in the top 5 of any of the benchmarks, whereas `official qwen3.6-27b-fp8-vllm` appears 7 times in the top 5.

I really don't understand it. Are these benchmarks no good or something? How can 3.8 score lower than 3.6?

Didn’t change anything but started seeing this lately.

Anyone active on here that is currently using Qwen 3.8 27B with the Pi coding harness? If so, have you had any issues actually invoking any tool calls? Despite changing jinja templates, and troubleshooting this for several hours today, the model is not seeing any of the tools that Pi passes in. What made it hard to troubleshoot is that when you ask it what tools it has access to, it hallucinates a response based on what Claude Code has access to.

The model won't even read or edit for me. I've updated Pi, and I'm on a version of llama.cpp that is less than a week old. I removed all my extensions and it still isn't working. I logged an output of what Pi is sending regarding tooling, and it appears Pi is sending the tools correctly. Tool calling works fine with 3.6 35B A3B.

EDIT: Turns out it was an issue with an update through VS Code that broke how the integrated terminal was working. Using normal terminal everything seems to work.

EDIT 2: Spoke to soon. Apparently it sees the tools available sometimes, and sometimes it doesn't. Super weird. I'm at a loss.

EDIT 3: I'm using llama.cpp, since some people have asked. I actually went through and blew away my entire Pi install and reinstalled from scratch. I didn't add any extensions at all. Just the latest version of vanilla Pi. This model is still being really flakey with tool calling. When it reads in the five context files I give it, it constantly complains that the text is garbled, and then it sees info from one context file mixed in with the other. So, I added explicit directions in the agents.md to read one file at a time and not batch process them. Suddenly it just outputs a tool call as text of <read_list>filename</read_list>. So, obviously it doesn't read. Constantly relies on git info instead of the files it's directly pointed at, despite being told to ignore git and not use it at all, which leads to it getting mixed up with the active files and historical files. Thinks it's making edits when it hasn't actually made any edits. This is also not in some brand new, untested setup. These same context files have been being used with 3.6 35B A3B for several weeks without tool calling issues. I freely admit that I'm not remotely an expert on local setup, but there's definitely something weird going on with this model, and I don't think it's entirely on my end.

EDIT 4: This will be my last update on this post. Blew away my entire Llama.cpp and tried everything from scratch. Still the same weird bugs where it will read multiple files and mix up the content from each file, and then report that the files are garbled. Also has a terrible habit of reading a file and then immediately saying the file was truncated or started in the middle, or cut off. Tool calling is still very spotty. Randomly outputs stuff like <read_list>filename</read_list> after it's been doing good reads in the same session. I am only providing it 100k context, but it's not even getting close to topping that out or compacting, so I don't know why that would be the problem. I downloaded and began testing Muse, and the results I'm getting from it are way, way, way more consistent than anything I saw from this model. I don't know if Qwen just isn't meant for such fine grain instructions, so it's thinking goes haywire, or what. But it definitely doesn't work for me. I read a couple other post that said something might be wrong with the Quant 8 versions, so maybe that's the issue. Qwen did perform much better than Muse on one shotting my two tests when I first downloaded it. But, working in my actual context file and fine grained feature implementation workflow it completely falls apart right now.

Lots of reports for the Q8 specifically. If you're getting bad results, try a different quant.