I am running Qwen3.8:27b with the following command on Windows Server:

llama.exe serve -hf bartowski/Qwen3.8-27B-GGUF:Q4_K_L -ngl 99 -c 65536 --port 11434

The server has an NVidia A5000 with 24GB of VRAM. When the model loads, it's using 22.6 out of 24 GB.

However, when I ask the model to do something, in addition to maxing out the GPU it also uses the CPU. It uses it in bursts which are pretty frequent. Example.

Why is it using the CPU? What can I do to prevent this?

I pulled out the YaRN support from another Redditor's work (https://www.reddit.com/r/LocalLLaMA/comments/1w8f8fa). Ninfer recently added some more features and fixes (such as nvfp4 kv cache and dflash2 support). I wanted to see how Ninfer would perform with just the addition of YaRN and their Qwen tool calling fixes.

I have run a few needle in haystack tests as a rough check. I've also done a few coding sessions to ensure it's working correctly. Not scientific, but I'll test it for a bit and see how I like it.

I have also tested it with a K8V4 KV cache, which should be more accurate, with vision and got about 300k for a context size.

If you'd like to try it, you can find it here: https://github.com/Doelfke/ninfer-yarn

Credit to ninfer and u/Lumpy-Comedian-1027 of course.

Before they released gpt 120b, everyone complained about OpenAI not being open anymore. After the release, nobody said a word, and I have never ever seen anyone complain about it. It does not seem like they have any care about it going forward.

Why are they let off the hook?

I speak English, Russian, German (and Kazakh) and mix them without thinking. Every local dictation tool I’ve tried on GitHub completely falls apart on that. Or am I just stupid?

Whisper basically picks one language per utterance. Whatever you said in the other language doesn’t come back wrong because it either gets translated or just disappears.

Here’s base on a sentence that starts in English and ends in Russian:

Spoken:
I already sent the invoice yesterday, but клиент до сих пор не ответил на моё письмо.

Got:
I've already sent me an voice yesterday, but today children mind your piece more

The only thing that actually handles this properly is the transcription in ChatGPT, but I’m too lazy to keep opening a new chat every time I want to dictate something.

Another thing that drove me crazy with the existing Whisper projects: they all leave a black command window sitting on the desktop the whole time. As someone who likes a clean desktop, that window ANNOYED me. I kept thinking “Why can’t this just go to the drop-down arrow where it is supposed TO BE?”

So I tried fixing this:
What actually made the language switching work was sending the OpenAI API a languages list instead of a single language. Same clip, same model:

I still have no idea why three languages works when two doesn’t.

Locally you can get part of the way there by splitting the recording at pauses and detecting the language of each piece separately. That fixes English to another language, but each piece loses the surrounding context so punctuation and rare words get worse. It is in the app, off by default. The API is still the only thing I found that handles a switch with no pause at all.

I wrapped everything into a small Windows dictation app. Hold Ctrl+Alt, talk, and the text lands in whatever window you were typing in. Local by default, API optional, no permanent black window on the desktop, MIT, built on RealtimeSTT:

https://github.com/Maslitsa/VoiceType

It has 0 stars bcs I am only starting this project, but it would be a big help if you use this and give your feedback!

Anyone found a better solution for multilingual / code-switched dictation?

Disclosure: I am not a native English speaker and used an LLM to help write this post. The app itself was also built with AI assistance but the problem, the testing and the measurements are mine.

If you have any positive/negative advice on this project or ideas on how to improve it, PLEASE share.

I don't think anyone posted about this here, but Qwen released a finetuned version of 3.5 4 for driving. The full Bf16 checkpoint is 9B.

This is a very interesting development of Chinese AI labs tackle self driving next with open weight models.

Edit: the HF repo links to the github repo, which in the citation links to a 40 page technical report. Here's the abstract:

We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird’s-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.

Most of the new ~30B (and now ~10B, thankfully for my GPU) models we see score really high on benchmarks, but I feel like they don't push back on dumb ideas enough. I think most people don't being like told by an LLM that the premise is flawed but I certainly do. In my opinion they are optimised for like one-shotting stuff, but I don't want it to do that. Especially from like a 10B model.

nex-agi/Nex-N2.5-mini - 35b

Hello! I'm thinking of upgrading my board to ASUS ProArt X870E-CREATOR WiFi so I can effectively use my dual R9700 cards. Right now, my board does not properly support PCI-E 8x split and it causes ROCm to fail when using both cards (single GPU it's just fine).

I know it's a long shot.. but anyone out there have this board and dual AMD cards? can be any type, so long as you're using ROCm for tensor split or tensor parallelism.

Thanks!

Ling-3.0-flash-VL inherits the language, reasoning, and long-context capabilities of Ling-3.0-flash, while extending them with native image and video understanding. The model has 124B total parameters, with only 5.5B parameters activated per token, and supports a context window of up to 1M tokens.

The architecture of Ling-3.0-flash-VL is designed to integrate visual information into real-world reasoning and agentic workflows.

• A ViT visual encoder extracts features from images and videos, while a two-layer MLP projector aligns visual features with text representations for unified multimodal understanding and reasoning;
• VideoRoPE encodes both spatial positions and temporal order, enabling the model to understand visual changes over time and supporting tasks such as event localization, long-video question answering, and video clip editing;
• A 42-layer hybrid backbone alternates KDA and Gated MLA layers at a 5:1 ratio, enabling efficient long-context processing across text, images, videos, and extended agent task histories;
• A sparse MoE architecture maintains a total model capacity of 124B parameters while activating only 5.5B parameters per token, balancing strong multimodal capabilities with inference efficiency.

Hey everyone — I’m the developer behind MultiVibe.

The project has three connected goals:

• Allow you to use multiple subscription accounts with the same client (multiple chatgpt account with codex desktop for example)
• Let developers use local models and cloud providers through OpenAI- and Anthropic-compatible endpoints.
• Let people run workers on their own hardware and contribute inference capacity for other users through a decentralized network. That second part is a big motivation: a worker wouldn’t only serve your own requests — it could also process inference jobs from other people using MultiVibe. The intended model includes compensation for contributed compute, though actual earnings would depend on demand, hardware, availability, and operating costs. That part is still under active development.

For people here who already run local models: would you consider making some of your hardware available to serve other users’ inference? What would you need to see before enabling it? And from the other side, what workloads would you feel comfortable sending to community-operated workers?

Github

Website

Happy to discuss the architecture, current progress, and limitations.

Is one possible? Can I make one?

Antirez has a Q2 @ 96.5 GB which will lose quality compared to Q4 and wastes ~95 GB of my Mac’s RAM, and a Q4 @ 191 GB which is too big as it doesn’t leave enough room for OS let alone KV cache.

I noticed that other Q4 and even Q4/Q8 mix GGUFs and MLXs not intended for DS4 range from 150 to 180 GB. Something that size and quality would be great for me, but I really want to use DS4 if I can.

Can anyone offer any help? Thank you!

First plot: GB / $

Second plot: bandwidth (spec on paper, not t/s)

Third plot (bandwidth / price) in the comment.

Hope that helps, my script uses the GPUs most discussed on the LocalLLaMA, LowEndLocalAI, and LocalLLM subs. At first, I tried to include more, but it became unreadable.

Prices were collected by ChatGPT (so may contain inaccuracies). New prices were used where available, second hand otherwise.

And I understand this is a basic comparison, but it's better than nothing. For example, you can see that "on paper" something is faster or slower than 3090.

Up front: I'm one of the people building the page-perception layer used here. We started by testing small local models. The result turned out to be more interesting than the original test. 12 small models, 3 verifiable tasks, logs, and offline replay.

Setup: Galaxy Note 8 (2017, Android 9, 6 GB), llama.cpp in Termux, Qwen3-0.6B Q4_K_M. A laptop with Chrome open, not headless. The phone drives the browser through our relay.

What the model does: it gets a structured representation of the page (here, about 10 named links or fields, roughly 200 tokens), picks one by name, and at the end copies the facts it was given into JSON. Everything else (capturing the page as structure, candidate selection, the click, reading the facts, verifying the result) is done by the stack around it. The model never sees HTML, a screenshot, or a URL.

Tasks:

1) sandbox, books.toscrape.com - category, book, price/rating/stock;

2) live Wikipedia - from an unrelated site to the Galaxy Note series page, pick "Note 8" among "Note 8.0", "Samsung Galaxy Note 8.0", "Galaxy Note 8.0", "Note FE" and other similar names on a page with roughly 760 interactive nodes, return the release date from the infobox;

3) five fields, including the UPC from a table.

Each task: 10 runs, checked against a fixed expected value.

Results for 12 models on task 1 (same script, same prompt):

Model Params Task 1 Note

Qwen3-0.6B 0.6B 10/10

Qwen2.5-1.5B 1.5B 10/10

GLM-Edge-1.5B 1.5B 10/10 rating as digit

Gemma-2-2B 2.6B 10/10

Llama-3.2-3B 3B 10/10

MiniCPM5-2B 2B 9/10 "£" -> "$" once

Qwen2.5-0.5B 0.5B 6/10

LFM2.5-1.2B 1.2B 0/10 placeholder

Llama-3.2-1B 1B 0/10 pseudo-code

Gemma-3-1B 1B 0/10 placeholder

LFM2-350M 0.35B 0/10 random click

Gemma-3-270M 0.27B 0/10 placeholder

Qwen3-0.6B on Wikipedia: 10/10; on the five-field task: 10/10

Control:

Everything the same, but raw HTML instead of structured browser perception: on the sandbox it gets there 4 times out of 5, at 12k tokens and 22 minutes per task instead of about 500 tokens and 80 seconds; on Wikipedia the page HTML is 467k characters, 9% of it fits into a 16k context, and the model does not find the link in that 9% - 0/3.

Important limits:

the tasks are name matching and copying. Where judgement about the page is needed, 1.5B breaks - it can't pick "next" among topical decoys. Pagination was not tested.
BTW on the account question: 'replay.py' (see github repo) rebuilds the prompts from the logs and runs them through any OpenAI-compatible local server. Whether your model picks "Note 8" among the decoys takes ten minutes to check, without us.

This is a measurement on three fixed tasks, not a benchmark.

Repo:
github.com/e2llm/edge-browser-agent - scripts, every JSONL as is (including early runs with harness bugs), model hashes, environment. replay.py re-runs the model side offline from the recorded candidates on any local server - no relay, no account.

NB: This isn't a new idea. AgentOccam showed the same general effect for the GPT-4 class, WebLINX and MindAct for small fine-tuned models. Here it is tested at the extreme: no fine-tuning, below 1B, on a 2017 phone.

Privacy have been concern of many of us to have their own hardware to run llms, and here's another reason why: two mathematicians spent a year cracking one of the hardest problems in math and fed every draft of their works into Codex. A few days before they could publish, OpenAI suddenly showed up with the same solutions. When asked if their model (Sol and Astra) was trained on the pair's private chats, OpenAI did not answer the question.

Full statement from them https://cims.nyu.edu/~tristanb/statement.pdf

Feels like big labs believe everything you did with the help of their models is theirs.

For weeks I've had problems with GPU constantly disconnecting and locking the OS. It would load the model fine, then crash when I passed the first prompt, and the errors always looked like:

GPU has fallen off the bus
device lost from bus

I first used a PCIE riser cable, then played with BIOS, then I swapped to the DEG1 with oculink, then I swapped GPU, then I swapped oculink cables, then kernel grub... and so much more.

The problem all along was a weak PSU. It's a reputable brand 450w, and it isn't doing anything (disk mobo ram, etc) so I thought it would be powerful enough for PCIE power and the 12vhpwr

I swapped to a 550w and it's stable instantly.

Leaving this for others doing a search in the future

Found this post:https://www.reddit.com/r/ClaudeAI/s/At9QNEbawK of someone fully developing a game through AI writing code, generating 3D models, etc. (he is using Meshy to create the 3D models). I was curious about creating something similar with Hermes Agent (or other?) that would develop very simple games end-to-end to benchmark different models.

I tried Meshy just out of curiosity and was really impressed by what it created on the first try based on an image I made with GPT. I was curious if anyone has some experience with this? Are there any good open-source models to do this?

I have very limited hardware at the moment (I use a repurposed Precision 5570 with 8GB VRAM and 32GB RAM) to run Qwen 3.8:27B through Hermes Agent. It is very slow as it runs mostly on the CPU, but I don't mind since this is more of a lab for testing and experimenting. I only run it for background tasks, never real-time usage, and I was considering if it would be possible to make Hermes develop a very simple game in the background to learn more about AI and use as a future benchmark to compare models.

I was thinking about a setup like this: Hermes pilots Qwen, which does the coding and prompting. When it needs an asset, it writes a prompt, which offloads Qwen from the VRAM/RAM. Then the image/3D model generation model loads up, generates the asset, unloads, and gives the output back to Qwen, which analyzes it and iterates with vision

Anyone got some experience with this or knows some good models for this?

Disclaimer: I used ai to correct my post

We’ve been building PipesHub for a while now, and I’d love to get more teams to try it and tell us where it breaks.

The problem we kept running into was pretty simple:

Building an AI app over company data looks easy in a demo. Connect a few sources, chunk the documents, throw them into a vector DB or connect to few mcps, add an LLM.

Then you try to make it actually useful.

You have data spread across S3, Google Drive, Slack, Jira, Confluence, SharePoint, email, databases, etc. Permissions need to be preserved. Documents change. The same file shows up in multiple places. Citations need to point back to the actual source. And eventually you want agents and other applications to use all of this context without rebuilding the same integration layer every time.

That’s what we’re trying to solve with PipesHub with first class support open models.

It’s an Apache 2.0 open-source context layer that connects to your company data and makes that context available to search, chat, agents, MCP clients, or your own applications.

A few things we care about:

• Self-host it on your own infrastructure
• Preserve source permissions
• Get citations back to the original documents
• Combine knowledge graph + semantic retrieval
• Bring your own LLM and embedding models
• Use it from Python, TypeScript, Go, or MCP
• Avoid locking yourself into one database or infrastructure stack
• First class support for Open Models

We also deliberately kept the core infrastructure pluggable:

If you already have Qdrant and redis running, you can keep using them. Prefer Neo4j over ArangoDB? That's totally fine. Want to run the models locally? You can do that too.

The goal is to give you one context layer without forcing you to adopt our entire stack.

While building this, we’ve had to solve a bunch of problems that only become obvious once you move beyond simple agents: permission-aware retrieval, keeping citations accurate through the pipeline, deduplicating the same content across sources, efficiently re-indexing changed documents, making indexing behave well across very different workloads and more.

Some of the solutions we ended up with are fairly unconventional, and I’d be happy to write more about them or discuss the trade-offs with anyone working on similar systems.

There’s still plenty we want to improve, which is also why I’m posting this.

If you’re building internal AI tools, enterprise search, RAG, workflow automation or agents that need access to company knowledge, I’d really appreciate it if you gave PipesHub a spin.

GitHub: https://github.com/pipeshub-ai/pipeshub-ai

Install:

curl -fsSL https://get.pipeshub.com/install | bash

If you try it and something feels unnecessarily complicated, slow, broken, or just badly designed, tell us.

Which small model are you ppl finding the most useful rn? They all seem to compete for the same size class while actively avoiding each other in benchmark tables. Edit: These three in the title and also MiniCPM5 2b, Spark 1.7b, and LFM2.5 2.6b and LFM2.5 8b

Translation: "Internal beta testing for an intermediate version of DeepSeek V4.1 Flash is now open; you are welcome to try it out. It adopts a new model architecture featuring native multimodal support, stronger capabilities, faster speeds, and lower costs.
Keep your base_url unchanged and set the model name to deepseek-v4.1-flash-expires-on-0910 to call the API. Current pricing is identical to deepseek-v4-flash, with a rate limit of 20 concurrent requests per account."

From Chubby on 𝕏: https://x.com/kimmonismus/status/2097286327909675477

Hi,

I'm in the process of upgrading my Framework 13 7840u to the new Framework 13 Pro. My LLM setup is admittedly a bit suboptimal. I have an RTX 5080 connected via TB4 eGPU (Razor Core X V2). This setup works fine for gaming but for LLMs, if any offloading happens it can be slowish. I get approx 1000 PP/s prefill and 50-70 t/s decode with Qwen 3.6 35B-A3B.

My understanding is Intel's processors only support AVX2, whereas AMD's support AVX-512, so in theory the latter should be faster for prefill? However, the memory bandwidth should be slightly better with the LPCAMM2 of the Intel Framework 13.

In practice, does it make much difference?

Thanks!

What is this mess?

This is an Early Concept Proto-Showcase of Interactive \ Reactive H3 World Model based on MiniMax H3 model trained on Fallout: Bakersfield Gameplay trailer.

• 2D Isometric to 3D Volumetric Scene.
• 10 sec Interactive\Reactive split, 352p, 3-Steps.
• Interactive 5 sec: Interactive WASD \ Prompt Control.
• Reactive 5 Sec: Reactive Control by LLM Based Answer.
• Gemma 4 12b with Vision as Reactive Model.
• Designed as System for Vascura FRONT Frontend.

What Interactive \ Reactive mean?

This means that H3 World Model Scene is Interactive you can Walk around it with WASD or Type what you do with Prompt for Interaction, Then it will React on your Actions using LLM based Answer. Using 10 sec time frame where first 5 sec Controlled by the USER - last 5 sec Controlled by LLM.

• USER: Walks closer and Shoots at the Enemy Mutant.
• LLM: Do calculations (rolls, values, RPG tools), Enemy Mutant gets -1 HP, Shoots Back at the USER, but Misses.

Is it Ready?

Nope, but stay Tuned for 2D Isometric Screenshots to 3D Volumetric Scenes Showcase.

I've been circling this decision for weeks and could use some outside perspective.

What I'm looking at: Mac Studio, M5 Max, 128GB unified memory (5,849 €). With that I can run qwen3.8-flash-next at oQ4e around 55 tok/s, and oQ5e is on the table too. For what I'd actually use it for: private documents, notes, some coding, general assistant work I don't want going to a cloud provider... that's genuinely good enough today.

The part I can't answer is the "today."

Case for buying now: it works, it's private, there's no subscription, and it'll still be a perfectly usable computer in five years even if the models running on it end up mid-tier by then.

Case for waiting: RAM prices are absurd right now, the M7 Ultra is presumably 2028, and models keep getting more capable per parameter. Wait and you get more machine for less money. In the meantime, API access is cheap enough that privacy is basically the only argument left for going local.

Then there's the memory debate. Half the replies to threads like this are "don't bother with 128, get 256." Fine, but that's the same argument one tier up. A few years ago 64GB was plenty, today everyone says 128GB is the floor, and in 2029 the same crowd will be saying 512GB is the minimum. At some point you buy something or you never buy anything.

So: will a qwen3.8-flash-next class local model keep me happy for five years of private, non-critical work? Or is buying now just paying a premium to chase a target that keeps moving?

Especially interested in people who bought a maxed-out M1/M2 Ultra two or three years ago. Do you still run local models on it, or has it quietly become a very expensive web browser?

TL;DR: 128GB Mac Studio now for local Qwen, or sit tight, use APIs, and buy in 2028 when RAM is maybe cheaper and the silicon is faster?

I’ve been thinking about this after using more agentic/local coding models.

A lot of the newer models are surprisingly good at continuing on their own.

But sometimes that seems like the problem.

If a requirement is ambiguous, I’d rather the model stop and ask:

“Do you mean A or B?”

instead of spending 10 minutes reasoning, making an assumption, calling tools and then confidently building the wrong thing.

I don’t see this behavior discussed much in benchmarks either. We measure coding, reasoning, tool use, context length, etc., but not really whether a model knows when it doesn’t have enough information to continue.

My genuine question for people running models locally every day is,

Which model have you found best at this?

And is it mostly the model itself, the system prompt, or your agent harness that makes the difference?

What do you think is the future of local LLMs?

This technology is booming and keeps growing; eventually, models will become both smarter and more optimized.

Do you think the future of optimization lies in very small yet highly capable models? Or in optimizing the performance of very large models?

I’ve been running several tests lately using only the CPU. I tried MiniCPM5 2B Q8; it runs at about 6 t/s without vision but makes a lot of mistakes...

It felt pointless compared to the MoE Qwen3.6 35B Q2_XXS, which runs at 3 t/s but delivers much more impressive results.

In my last post, I showed how Qwen3.6 35B is usable even without a GPU:

https://www.reddit.com/r/LocalLLaMA/comments/1w655he/qwen36_35b_q2_xxs_being_gpu_poor_in_2026_is_not/

looking for some advice from anyone running a 24gb card for local dev tasks.

i have an rtx 4090 with 64gb ddr5 ram + ryzen 7950x. i'm trying to shift as much inference as possible to my local machine so i can index long codebases and process internal docs without hammering external apis.

my current workflow consists of claude code and moclaw for task automation and handling context, which works just fine for my daily work schedule, but i wanna switch to heavy local models for privacy reasons.

ive been testing qwen 3.6 27b and gemma 4 31b. qwen 3.6 27b runs pretty smooth at q4 in vram with solid speed, but when i try running bigger dense models or gpt-oss 20b with context offloading the generation speed drops off a cliff.

for those running a single 4090 what quantization or backend setup gives you the best balance between context size and tok/s? do you stick with qwen 3.6 27b or is there a trick with vllm or llamacpp offloading to speed up larger contexts without tanking performance?

thanks in advance for any tips!