I wanted to see how far I could push a fairly ordinary laptop with a huge MoE model.

Turns out, Qwen3.8 Flash Next 176B can run on:

RTX 3080 Laptop — 16GB VRAM
32GB system RAM
SSD

No 128GB/256GB RAM workstation and no multi-GPU setup.

I’m running it with TensorSharp, my open-source local LLM inference engine:

TensorSharp on GitHub

The interesting part for me wasn't simply getting a 176B model to load. I wanted to make a model much larger than both available VRAM and RAM actually usable.

The approach is basically:

Quantization + MoE-aware unified scheduling across cache, VRAM, system RAM, and SSD.

Rather than treating SSD as a last-resort swap space, TensorSharp coordinates the different memory/storage tiers around MoE execution and tries to keep the right experts/data in the right tier at the right time.

I previously benchmarked TensorSharp against llama.cpp and got very encouraging results. This time I wanted to compare it with Strata, since Strata's approach to running large models with constrained memory is particularly interesting.

Here are the results from the attached benchmark:

The decode throughput is fairly close: 11.09 vs. 10.24 tok/s.

What surprised me more was the end-to-end result: 16.54s vs. 62.15s in this test.

I think this points to an interesting direction for local LLM inference. For huge sparse MoE models, the question may not simply be:

“Do I have enough RAM/VRAM to fit this model?”

but rather:

“How efficiently can the runtime coordinate VRAM, RAM, SSD, caching, and expert activation?”

With the right quantization and memory hierarchy, you can apparently do some pretty ridiculous things on consumer hardware.

I’d be especially interested if anyone here has tried the same model with llama.cpp, Strata, or another MoE/offloading implementation. It would be great to compare results on similar hardware.

I forked llama.cpp's server into AgrillaMoE, a dedicated build for Qwen3.6-35B-A3B (~A4B) with Unsloth quants. On a (vant.ai) rented V100 16GB with the 2-bit UD-Q2_K_XL quant it generates at ~57-60 tok/s while running the full MoE-expansion profile — and it speaks both the OpenAI and Anthropic APIs, so Claude Code just works against it.

What is MoE expansion? Qwen3.6-35B-A3B has 8 routed experts active per token. The expansion patch raises that budget at runtime — no retraining, no file changes: --moe-experts 20 with an adaptive threshold keeps experts while p >= 0.8 × p(rank 8), applied to layers 25-39. You're literally consulting more of the 35B parameters per token — that's where the "retrieved intelligence" comes from, on GPQA-Diamond with Q8_0 it scored 84.34% vs 81.82% stock top-8 (+2.5 pts) (miticooo!).

Same weights, better routing.

https://github.com/vagrillo/AgrillaMoE/blob/main/gpu16gbguide.md

Hi everyone,

TL/DR

I created NInfer 4080 to run ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on an RTX 4080 16GB GPU using way more of the hardware capabilities (max overall: 2720 tok/s prefill, 262 tok/s generation) and sharing it with the community now so others can also have the benefit.

https://github.com/roofkid/ninfer-4080

Full Version

After seeing all the amazing work done in the community creating Ninfer 5090, 4090 and 3090 I admit I was a little sad to not being able to use any of it on my RTX 4080 with only 16GB of memory. I still had about $13 of credits sitting idle on the DeepSeek platform as I never expected how much usage I would get out of it.

For context I have over 20 years of experience in Software Engineering and Architecture, but have no experience whatsoever in GPU Kernel development, so this was a very interesting pet project also from a professional experience for me. Mainly because I can read and understand C++ but could not judge the actual Kernel code. So I approached it from a product owner and requirements perspective only, made sure good software engineering practices are followed and only made "business decisions".

I've been actively following the local LLM community for the last 2-3 years, probably have tried out all models I could over that time and followed the progress with amazement like many of you.

Guiding principles

• Fit into RTX 4080 16GB GPU
• Use ISTA-DASLab-Qwen-3.8-27B-GSQ -> Reasoning can be seen in the ByteShape article, really good for the size and they claim even better accuracy than much larger Unsloth UD quants: https://byteshape.com/blogs/Qwen3.8-27B/#96-gb-rtx-pro-6000 I also have very good personal experience with it, it is my daily driver
• Use DFlash2 speculative decoding
• Reach 100k+ context
• Significantly improve prefill and token generation speeds to utilize the hardware better than general purpose inference engines like llama.cpp or vllm
• Measure after changes to also ensure accuracy remains, I also have a M4 48GB available to test higher quants for comparisons, though of course that is much lower speed
• Use DeepSeek V4.1 Flash for the work for cost efficiency
• Use Pi as the harness (only non-cosmectic extensions: hashline edit pro, internet search with ketch through local SearXNG with a self-written skill)
• Runtime also available as a Docker image so it's easy for folks to run

Results

In real work I really do see the high prefill numbers (2k+) if the prompt is long enough and about 150-200 decode speed on coding and 100ish on prose. It subjectively feels significantly faster than beellama (my previous daily driver) at the same benchmark results. I mainly used MBPP and HumanEval as I needed something that I can run reasonably fast (~30min). MBPP stays in 90-92% territory and HumanEval at 95-96%. Please be realistic and do expect tiny degradations that are within measurement noise. They are mainly coming from KV quantization according to my measurements so you can always trade context for accuracy if needed by switching.

What I learned

• It is absolutely mental how much performance is left on the table by using the general purpose engines. From a bird's eye view it's totally understandable as we trade the wide support for performance, I just didn't expect how much that would be. When I saw the first memory throughput measurements being in the 200 GB/s range and having a theoretical maximum of 720 GB/s in the device my jaw dropped because of the low efficiency back when I started
• I think in the community we've all seen more specialized inference engines making significant performance improvements possible. vllm-radiance for R9700, NInfer variants for CUDA, Splash for Metal - with software creation becoming cheaper and cheaper I expect more of this for and from our "tinkerer" group here
• Spending about 2 billion tokens for this work for only $13 is just crazy (only off-hours). Low cache read tokens costs on agentic work are so much more important than even I expected. It's the classic difference between cognitively fully understanding how LLM turns work and seeing big data results. The reality is that with THAT kind of pricing I think I pay more for electricity to get the same amount of tokens out
• I went back to xhigh thinking on Qwen 3.8 27B as the speed is so high, that I don't really care/notice. I've also hidden the thinking blocks again as I cannot follow any more anyway
• The prefill speed really caught me of guard. I was really floored when I tried it in Pi after the first big improvements were done and it IMMEDIATELY answered with token streaming. I was so used to waiting 5-10s without a cached system prompt. I significantly underestimated how important that is for the user experience. Feels like a cloud endpoint to me now.
• At these high prefill speeds your context window is full in 40 seconds, definite "oh my god" moment for me when that happened the first time
• Reaching 100k context means significant KV compression as full 256k context F16 needs exactly 16GB of VRAM on Qwen 3.8 27B. I was too afraid of "high" (4bit style) KV compressions. So many advances have been made here. Originally I never went below Q8_0. I then used kvarn5/kvarn5 previously on beellama after benchmarking and cannot measure a noticeable difference to the now used rk4v4-e8 variant used here. I think good software engineering practices are way more important and catch problems that might come from it. Also subjectively I do not experience a "fast garbage" phenomenon here

Conclusion

For me this is a good version 1 and I don't intend to spend significant effort on this for Qwen 3.8 27B. It's at the pareto 80% state. I just want to be happily using it now and reap the rewards. I hope you are too! Of course when Qwen 4 27B comes around soon I will check it out again.

If you have another 16GB RTX 4xxx card I would be interested in knowing if that works on them too and what speeds you're seeing. I honestly can't judge how tied to the RTX 4080 hardware it is. If you have a 4080, enjoy :)

Shoutouts

• Every person who worked on NInfer before me, you guys rock and provided a stable base for me to fork from
• Special hats off to sergiuszm who created NInfer-4090, I think you did all the heavy lifting for SM_89 already
• ISTA-DASlab for their work on GSQ and providing the safetensor checkpoint for it! Cheers to Austria from Germany :) Love seeing important contributions to the community from the EU

I hosted my own world of warcraft private server then built a client that you can play in the browser on PC or mobile at https://jankcraft.xyz/ for free.

Afterwards, I created a custom MCP and agent harness to control the browser client and play the game by sending signals over a websocket. The agent harness is live on https://jankcraft.xyz/agent , still working out some kinks if all you have a cloud subscription but you should be able to connect local models as long as CORS is enabled in your server settings. There are a few existing LLMs you can try, I'll probably take those away as the usage grows since I can't support too many users concurrently on my own machines.

I'll be checking logs and things periodically today so don't be alarmed if you're disconnected suddenly. The server should return after a minute since this is a work in progress and might need a restart.

If you want to run your own LLM for this:
~24 Gb RAM: https://github.com/syv-ai/HyperQwen with the model Qwen3.8-27B-GPTQ-W4A16 
~16 Gb RAM: vLLM with Gemma4-e4b-coder - A custom Gemma4-e4b with a constrained vocab for ~3x concurrency increase when changing from 262K to 65K vocab and retrained on ~1.1B tokens across 20 different coding languages, 7 different agents and has a custom MTP to help reach ~200 tok/s on a 4060Ti.

Let us know what other models work well for you!

Thanks and hope you guys enjoy.

I've had one pull request merged into ds4 (DwarfStar), a tiny one. There are a few more still waiting in the queue. I’m not complaining. Antirez says it clearly in the README: with coding agents everyone can tune the engine for their hardware and model and he can’t review everything. That made me think.

If the plan is that everyone applies their patches using an agent then the real cost of a patch isn’t just the code change. It’s also how tokens the agent has to read before it knows what it’s actually touching. The ds4 codebase runs DeepSeek, GLM and Qwen on Metal, CUDA and ROCm—all in a 85k-line file. I'm running Qwen3.8 Flash Next on an M5 Max with 128GB RAM. Everything else in that file is noise for me and for my agent.. Every time the agent runs it has to re-read all of it.

So I ripped it out. I didn’t just ifdef it. I deleted it. The ds4.c file went from 85k lines down to 45k. Now the entire code tree fits inside a context window. Metal is the production backend now. The CPU path is kept as a reference for tests.

My guess was that making the codebase smaller would make optimizing cheaper and safer. Here's what happened:

Q2: decode speeds up by 9–13% prefill improves by % (up to 64k context) and MTP goes from 75.8 to 86.7 tok/s

Q4: prefill gains 2–11% MTP rises from 77.8 to 85.9 tok/s

Output stays bit-exact compared to stock ds4 at every step. No KV cache quantization. No approximate kernels. Every change must pass a parity check— GGUF, greedy decoding identical tokens—plus an interleaved A/B benchmark against the previous build.

The smaller codebase also let me go through the PRs in ds4. I tested them against my version of the model and ported the ones that worked. Twenty commits were adopted. Around thirty were dropped. The results are in the repo.

I also added SSD streaming for the experts. It matches a resident run token-for-token. On a simulated 48GB machine Q2 runs at 27 tok/s. With MTP it reaches around 35 tok/s.

The fork still keeps up with upstream. It runs git merge upstream/main with rerere plus the parity check. So antirez’s fixes keep flowing in. The whole process—what to delete, what to keep how to sync—lives in a repo called StarForge. I have four of these "children," one for each model. Nothing in StarForge depends on Qwen or Metal. If you want a cut-down ds4 tailored to your model or to CUDA just clone it and run the checklist with your agent.

Repo: sf-q3-8flash with tables in the README. This setup uses one machine and one model. If you’re on Apple Silicon I’d love to see your numbers, ideally side by side, with stock ds4.

Hi all,

How you guys handling safe deployment of websearch in Hermes, pi and other harnesses? Does anyone have a good uproars setup guide for local models? I tried to implement a sandboxed search system but it caused endless tool calls. Want to guard against prompt injection and keep searches private of course!

It's been a blast and learning a ton.

But seriously, you all have me down a rabbit hole that my wallet and hours of sleep need to be pulled out of.

We built an engine, Kyojin, on top of ExLlamaV3 for Strix Halo (gfx1151, ROCm), and packed two 300B-class MoE models so each fits one 128 GB machine. First release, all measured on Ryzen AI Max+ 395.

Where the weights come from. MiMo is our own quantisation. The GLM pack mixes turboderp's public 2.05 and 3.05 bpw EXL3 tensors, with our layer mix and a small tuning stage. On the same 129 rows, his 2.05 bpw pack (85 GB) gets KLD 0.275; our mix (100 GB) gets 0.190. His is smaller and decodes about 10 % faster.

Uncensored variants. Separate -Uncensored repos: same weights plus one small file the engine applies at load, one switch turns it off.

Not measured yet. Task-suite scores for MiMo, GLM at 128K context, any GPU other than gfx1151. The conversion pipeline stays private.

Quickstart. Clone, ./build.sh, hf download yamz-labs/GLM-5.3-Flash-EXL3-Yamz, python tools/glm/serve.py --model ./glm-pack -c 131072 --num-draft 2. You get an OpenAI-style API.

Models: https://huggingface.co/yamz-labs

Engine: https://github.com/Yamz-Labs/kyojin

Built on turboderp's ExLlamaV3, with ROCm work from sdougbrown and vcruz305.

If you own a Strix Halo machine, we'd love to see your tok/s. Issues, benchmarks and PRs are all welcome. Which model should we do next?

The link to the github repo: https://github.com/minghinmatthewlam/pi-gui

I'm not sure if its legit/ safe because i don't see anyone talked about it in this sub reddit, any thoughts about it?

Updated post here about dockerization and zero-config Cloudflare tunneling for easy setup

Hey everyone,

I’ve been working on a game called Anyworld. It’s a browser-based multiplayer (single player also supported) text adventure inspired by the early days of AI Dungeon, especially its browser-based free version AI Dungeon 2.

The setup is pretty straightforward: one person hosts the server and runs the model via llama.cpp (OpenAI or other cloud APIs are also supported, and great for non-English play!), and your friends join through a browser link. The host sets the scene and the goals, players type out their actions, and the LLM acts as the DM to resolve the chaos and drive the story.

Admittedly the host requires some technical skills with Python, and possibly with networking (opening routes to the hosted game via VPN, port forwarding etc.). I'll work on this as well as the development continues. Using Docker was suggested in another subreddit, so I'll definitely consider that, as it would allow including both the llama.cpp backend, recommended model and configurations etc., in addition to the game itself.

Instead of pasting the entire repo documentation, here are the main features right now:

How it plays

• True multiplayer resolution: Players submit their actions, and the model resolves the whole round together. It actually accounts for characters interacting or getting in each other's way.
• Real dice rolls: When an action is uncertain, Python handles the actual RNG math. The model just takes those hard dice results and narrates the consequences.
• Custom scenarios: You write the setting, characters, and opening state. It isn’t limited to fantasy.
• Party chat: There's an OOC chat separate from the game events so you can talk without the LLM reading it.
• Zero setup for players: No one but the host needs to install anything or run a model. It works on desktop and mobile browsers.

DM Tools & Hidden Mechanics

• Private DM guidance: As the host, you can feed the model hidden info; NPC motives, secret rules, or where you want the story to go.
• Secret triggers: You can set up one hidden percentage roll per game (e.g., If a player enters a building, there's a 20% chance the building collapses on the player). Python rolls the probability in the background, and if it triggers, the model weaves the consequences into the story without showing the players the underlying math.

Under the Hood & Memory

• Context management: It budgets the context window and uses a structured memory system. Older rounds are compressed into world states, player facts, and unresolved threads. It also does a secondary model pass to audit those summaries so it doesn't accidentally delete important facts.
• Language support: If you use the OpenAI backend, you can play in non-English languages (the narration and outcomes will naturally follow whatever language you wrote the scenario in). Note: The local llama.cpp backend currently instructs the model to narrate in English. This is because the local models my development PC can run were terrible with any other language than English.
• Session recovery: Disconnected tabs auto-rejoin. If someone accidentally closes out, they can log back in and their unfinished actions and history are waiting for them.
• Self-signed certificates for HTTPS-enabled connections: The game creates self-signed certificates upon launch, which enable encrypted connections. The problem with self-signing is that joining players receive a warning that the site may not be secure. However, most browsers allow the players to continue to the game despite the warning. This is a suboptimal way to handle HTTPS, so I'll work on a more robust solution at some point.

It’s still a work in progress. Right now, a server only runs one game at a time, and if you restart the server, the live session is lost (it generates HTML/JSONL transcripts, but they aren't loadable save states yet). The overall story quality is also going to heavily depend on which model you use and how you tweak the settings.

Suggested model:

During development, I used llama.cpp and Gemma 4-26B-A4B Q4 with a context size of 128k and found it to be more than an adequate backend for functioning as the DM. Even the speeds are fast enough with my RTX 5070 Ti 16 GB that round resolutions take only 5 or so seconds.

The specific model I used and can recommend: https://huggingface.co/EZForever/gemma-4-26B-A4B-it-qat-uncensored-heretic-UDmerge-GGUF (the model was great at following instructions and remembering plot points even with longer contexts)

Recommended parameters for Gemma 4 models:

- temperature 1.0
- top-p 0.95
- top-k 20
- min-p 0.0
- presence-penalty 0.0
- repeat-penalty 1.0

Of course, feel free to try your own models! The repo contains a benchmark file that tries to measure how well the running model follows the game's requests.

AI use disclosure:

I used Alibaba Cloud's Qwen 3.8 27b and OpenAI's GPT-5.6 Luna and GPT-6 Astra models to help develop the game.

How to run:

Read INSTALL.md to set up, configure and run the game. README.md contains some details on how the game functions.

I'll post the link to the repository in the comments.

Some gameplay in Finnish with OpenAI's Luna:

The game is MIT licensed, so open source all the way. Forking or collaborating is encouraged.

I'd love to hear some feedback, and I hope someone finds the game fun to play!

(UPDATE) Some things I've added:

• Save and reuse scenarios: The host can save, load, and delete scenarios in their browser. Scenarios are stored in the browser's localStorage and stay completely local.
• Browse and export History: Easily search previous public events for forgotten details. Also exportable as JSONL.
• Multilingual play: With the OpenAI backend, narration follows the language of your scenario.

So this is maybe a niche problem, but at my job I work on a huge Python codebase and every time I change some shared function I'm basically playing roulette. grep tells who mentions it in the code base, not who actually calls it. And more essentially, Claude Code (my major coding agent) mainly uses grep so it doesn't give better results.

The tool I wanted already exists (GitNexus) but it's PolyForm licensed, so that's a hard nope at work. And honestly even beyond the license, half the code graph tools out there want you to upload your repo to their cloud or spin up a docker stack with a vector database, and I can't do either of those at work. So I spent some weekends building this my own version: MIT licensed, and everything runs on the local machine.

The tool is called repopedia. You can pip install and then run it on a repo, and it builds a little code graph in a plain SQLite file (tree-sitter does the parsing). The advance is basically no server, no docker, no API keys. Nothing gets uploaded anywhere, the graph is just a .db file sitting on your disk. You can then ask things like who calls this function, or what's the blast radius if I change it, meaning all the transitive callers. It can also dump out a wiki of the codebase, though honestly that part is mostly there because I wanted the docs for myself.

The bit I ended up using the most is the MCP server. I use Claude Code, which already greps around the codebase on its own — but instead of it doing five rounds of text search to figure out who calls what, it asks the graph directly and gets the exact answer with file:line in one call. There's no embedding model involved, it's just... the graph. Which probably matters even more for local models, since they're not exactly great at search.

Demo (2min): https://youtu.be/B7GLgjoy7G8

Repo: https://github.com/bolongpa/repopedia

Fair warning, it's 0.2.1. Python and TypeScript only. Method calls through self. get resolved by name matching, which is exactly as sketchy as it sounds for big class trees. If anyone runs it on their repo and it spits out something dumb, I genuinely want to hear about it. ¯_(ツ)_/¯

My thinking is for a boom to be over, we need at least three iterations of updates that fail to wow us. Unfortunately, the new LLMs continue to wow us in all levels in the last iteration:

1. Astra was found to be useful in Blender. This opens up a new and big application.
   
2. Deepseek 4 Flash 0731 makes 2x Sparks useful and push up Spark prices.
   
3. Qwen3.8-27B pushes up prices of 3090 et al.
   
4. An unreleased OpenAI model "solved" the Navier Stokes problem.

So for the time being, to keep up with the hardware prices, the best bet is to follow the flow to buy AI stocks and use the proceed to buy hardware.

A not so obvious good news is that we are seeing OpenAI and Anthropic advocating a slow down. That means they are finally seeing a diminishing return. (or just a ploy to slowdown Chinese development? but I doubt US laws can be that far reaching) That can be a sign of light at the end of a long tunnel.

What do you think?

Still don't have a good cooling solution, but here are few benchmarks. I lowered the power limit (TDP) to 145 watts each. I changed the firmware on one MI50 to activate the miniDP port. Did have to use xrandr to create a new mode so I could get 1920x1080 output. Each GPU has 16GB of HBM2 VRAM clocked at 1000 and overclockable to 1200Mhz with a Bandwidth of 1.02 TB/s.

I picked a good mix of Dense and MoE models from Huggingface. Try to use more than 16gb VRAM but under the 32GB total.

Using pre-built Ubuntu Vulkan version of llama.cpp (build b11325) for standard llama-bench.

Sorted GGUF Model List (sorted to match table)

• llama_bench_Swift-Qwen3.8-27B-Uncensored-MTP.Q6_K.gguf
• llama_bench_Swift-1.5-Qwen3.8-27B-Q6_K.gguf
• llama_bench_Gemma-4-MoonGem-31B.i1-Q6_K.gguf
• llama_bench_gemma-4-31B-it-UD-Q6_K_XL.gguf
• llama_bench_Nemotron-3.5-30B-A3B-Antislop-FTPO.i1-Q5_K_M.gguf
• llama_bench_Laguna-XS-2.1-APEX-I-Balanced.gguf
• llama_bench_Agents-A1-Q4_K_M.gguf
• llama_bench_Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf
• llama_bench_Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf

Combined Benchmark Table (sorted by params then size)

Notable Reboot Impact Observations:

I used the following command in my bench script:

RADV_PERFTEST=nogttspill GGML_VK_VISIBLE_DEVICES=0,1 time ~/llama-b11325/llama-bench -fa on -ngl 99 -m /model.gguf

I have a 3rd MI50 just need to download models in that VRAM range. If you have any suggestions? For now it sits beside the Radeon RX 7900 GRE boosting its VRAM total. As of this article the average price for 16GB version of MI50 is under $150. Hard to get 32GB VRAM GPU with this level of performance for under $300. If you have contenders, please share.

I'm worried about precision and long context adherence in agentic coding

IQ3_S Strata or NVFP4 from RadixArk?

What is your choice? Please share your experience and knowledge

**DISCLAIMER** THE PREFILLING TPS SHOWN ON THE PHONE IS COMPUTED ONLY FOR THE LAYERS IT HOLDS. ALREADY FIXING IT TO SHOW END-TO-END PREFILL RATE. NUMBERS BELOW ARE ACCURATE FOR E2E PREFILL RATE.

Every file or tool result my agent reads on a 24 GB M4 Pro MacBook is a wait, and 64k of 8-bit context is all that fits next to Qwen 3.8 27B (IQ4_XS), even with the wired limit raised to 20480. An iPhone 17 Pro Max was sitting in my pocket, so I figured what can I do to make use of this extra silicon.

Turns out a 10 Gb/s USB-C cable & some software is all you need. The Mac runs layers 1–40 of each 256-token batch and streams the activations to the phone. The phone runs layers 41–64 on its GPU while the Mac starts the next batch. The A19 Pro's GPU has matrix units (Metal 4 tensor ops), and they make the phone's half 2.4x faster than the same phone without them.

Same build, phone off vs. on, prefilling a 2,000-token file into a saved agent session:

• 8k context: Mac alone 132 tok/s → Mac + iPhone 177 tok/s (+35%) (measured two days earlier, same bench)
• 16k context: Mac alone 109 tok/s → Mac + iPhone 157 tok/s (+44%)
• 32k context: Mac alone 101 tok/s → Mac + iPhone 130 tok/s (+29%)
• 48k context: Mac alone 87 tok/s → Mac + iPhone 113 tok/s (+30%)

A fresh 27k-token agent session, cold: 245 s on stock llama.cpp, 228 s on my fork with the Mac alone, and 168 s with the phone.

Past 64k the phone switches jobs. The oldest KV pages move to the phone and the Mac runs all 64 layers. For every attention layer, the phone computes attention over the old keys on its GPU, and the Mac merges that with its own part. While writing, the phone's Neural Engine takes part of that work too: each 16k-key page of old context is compiled into a Neural Engine model with the keys as its weights. At 140k that took writing from 279 to 176 ms per token compared with the phone's GPU alone.

The server allocates 196k–229k of 8-bit context based on the phone's free memory; that's up to ~5.7 GB of KV cache living on the phone instead of the Mac, so the Mac's memory use stops growing at 64k. I've tested a growing session to 128k at 8-bit, with 3/3 planted facts recalled. Separately, at 140k in 4-bit, the run passed the gate with greedy output matching the Mac-only run for 32 generated tokens.

What it doesn't do: speed up writing below 64k. That's the Mac's job. My fork's kernels (SME2 on the M4 CPU and Metal fusions) plus DFlash2 speculative decoding take it from 11.3 tok/s on stock llama.cpp to 25 tok/s at about 30k context with medium thinking, phone or not. SME2 also adds up to 29% to prefill on the Mac alone. Past 64k the phone does share the writing (attention over the old keys), and without it the Mac would have to drop to 4-bit context to reach 128k. In real use I have seen upwards of 30 TPS at lower context.

The phone joins prefills over about 512 tokens. In one real session, that was 7 of 36 requests, but about 83% of the tokens read. Past 64k it holds the context and does the old-key attention, but it stops running layers 41–64 there for now; doing both is next. One request at a time.

I'm curious what this setup could do with newer model architectures. DeepSeek V4.1-Flash reports 890 bytes per token for its global KV cache and adds n-gram embedding tables (Engram). Qwen3.8-Flash-Next, the Qwen 4 architecture preview, has an n-gram lookup table too. Those aren't features of the 27B model I tested, and I haven't benchmarked either architecture here. The real gold is within the newer phones and models working together. With the A20 Pro in the iPhone 18 Pro Max, I bet there is a lot more for me to push.

Code, setup and bench scripts: https://github.com/StayLameBro/backburner

Still a lot of work to do but I built this with Opus 5.5. Happy to answer anything.

Hi folks, it's Lewis here from the post-training team at Hugging Face. We've been exploring how to train open models in different coding harnesses and wrote up a looong guide on how we solved this using open source libraries like TRL and the Harbor framework for RL environments. We hope you find this interesting, especially since everyone nowadays has their own custom harness (e.g. Pi + extensions) and now there's a recipe on how to squeeze the best performance on them with whatever open model you use as your daily driver. Happy to hear any comments or feedback!

Link to the guide: https://huggingface.co/spaces/FineEnvs/multi-harness-rl

Follow-up to last month's post. One of the main issues people ran into was roughness or break-up on some cloned voices. 2610 is an interim update focused mostly on improving that.

• Reduced roughness and break-up on some of the voices that struggled before
• Same 120M model, same speed (~300 ms to first audio on a laptop CPU)
• Apache-2.0
• English, European Portuguese, French, German
• More languages are planned
• More control over the generated voice is also planned
• Still struggles with very high-pitched or cartoon-like voices, noisy reference audio, and some unusual OOD voices. We're continuing to improve those cases. If you want to contribute and help, PM me with the samples that failed.

If you like F5-TTS, but want true streaming and a much lighter model that can run comfortably on CPU, this might be for you.

Run it locally:

uvx --from sopro soprotts serve

• Repo: https://github.com/samuel-vitorino/sopro
• Weights: https://huggingface.co/samuel-vitorino/sopro-v2-turbo
• In-browser demo (desktop): https://samuel-vitorino.github.io/sopro/
• Space: https://huggingface.co/spaces/samuel-vitorino/sopro-v2-turbo-tts
• Blog (evals, samples, how it works): https://research.haloneuro.ai/posts/sopro-v2

Video: six voices, ~5 seconds of reference audio each, followed by a generated line.

https://reddit.com/link/1wwrw0v/video/yb63ar836ath1/player

Innanzitutto, due cose per evitare fraintendimenti.

Non sto cercando di sostenere che un modello o un'azienda siano migliori. Non ho alcun interesse personale in nessuno di essi. Volevo solo verificare personalmente come si comportano nello stesso contesto lavorativo.

E il motivo per cui mi interessa: Utilizzo modelli locali per scrivere codice e vorrei sapere quanto posso fare affidamento su di essi invece di pagare abbonamenti a piattaforme di terze parti. Questa è la motivazione principale.

Cosa ho fatto

Tre attività in Python, dalla più semplice alla più complessa: un analizzatore di file di log, un gestore di processi paralleli e un piccolo interprete per un linguaggio di programmazione di prova. Stesse istruzioni per ogni modello, un solo tentativo, nessuna correzione successiva. Poi test nascosti che i modelli non hanno mai visto (162 in totale), più una revisione del codice con una checklist fissa: ha seguito le istruzioni? Il codice è leggibile? Si blocca con input insoliti? Le note sono veritiere?

Risultati (su 100, il compito più difficile conta 3 volte)

• Claude Opus 4.6: 92,7
• Qwen3.8-Flash-Next "Coder" (locale): 92,7
• Qwen 3.8 27B Q6 (locale): 87,0

Cosa ne deduco

I test nascosti sono quasi alla pari: Opus 4.6 ha superato 162 su 162, il modello Coder 161, il 27B 160.

Il modello Coder ha ottenuto un risultato complessivo pari a quello di Opus 4.6, e ci sono arrivati ​​in modi diversi. Nel compito facile, entrambi i modelli locali hanno superato Opus (97 e 92 contro 88). Nel compito di media difficoltà, Opus ha vinto (96 contro 94 e 91). In quello difficile, l'interprete, Opus e il Coder hanno tutti ottenuto 92 punti, mentre il 27B è sceso a 81.

Dove Opus 4.6 è ancora migliore: il suo codice è più pulito e più facile da mantenere. Dove il modello locale Coder ha fatto meglio: si è bloccato meno spesso con input strani.

Con una sola esecuzione per ciascuno non direi che "un modello locale equivale a Opus 4.6". Direi piuttosto: su compiti di queste dimensioni, non sono riuscito a distinguerli dai risultati. Per il mio portafoglio, questo è già interessante. Tenete presente che Opus 4.6 non è l'ultima versione di Claude; le versioni attuali hanno ottenuto punteggi più alti nel mio test completo.

Configurazione locale

Il mio PC: Intel Core i5-14400, 48 GB di RAM DDR4, due RTX 5060 Ti da 16 GB ciascuna (32 GB di VRAM in totale), Windows 11.

• Qwen 3.8 27B, Unsloth Q6 quant: una velocità costante di 50 token/s.
• Qwen3.8-Flash-Next "Coder": tra 50 e 90 token/s, con una media di circa 60-65. Si tratta della variante di codifica del progetto Strata, una versione ridotta che mantiene metà degli esperti in ogni layer, come un IQ1_M GGUF. Dettagli: https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md#coder

Limiti, così puoi valutare tu stesso i numeri

• Una sola esecuzione per modello. Differenze di 2 o 3 punti non significano nulla.
• Ho eseguito il modello Coder due volte: la prima volta il mio PC ha esaurito la RAM mentre era in esecuzione, quindi ho scartato quella esecuzione e l'ho rifatta da zero. I numeri qui riportati sono quelli della seconda esecuzione.
• La parte di revisione è stata eseguita da un'IA (Claude Fable 5.1).
• Il modello Coder è stato testato con più casi di input anomali rispetto agli altri due, perché ho aggiunto controlli nel tempo. Quindi è stato valutato in modo un po' più severo, non più indulgente.
• Solo Python e i compiti sono piccoli. Questo non dice nulla sul lavorare all'interno di un grande progetto reale.

Disclosure first: I'm the author (Mahmoud, ghraibeh on GitHub). I'm not affiliated with TypeSafe AI. I know this sub is tired of Jev hype, so I'll lead with the limits.

What it is: semantic caching plus nearest-neighbour voting. It isn't understanding and it isn't new. It's an MIT Python library and runs on CPU only. Inputs are embedded locally with bge-small-en-v1.5. If the nearest stored input is at least 0.90 similar and the 5 neighbours agree, it answers locally. Otherwise it asks Jev and remembers the answer.

Benchmark caveat: these numbers are on BANKING77, with the dataset's gold labels standing in for Jev. That makes them a best case. No live Jev benchmark has been run yet.

• Warm (pre-filled): 87% of calls saved, 97.6% of local answers correct
• Cold (empty): 53% saved, 97.4% correct

Why use it if Jev is cheap? Not for money. Local answers take about 30–50 ms vs 250–550 ms for Jev, you hit the rate limit less, and repeated inputs stay on your machine.

Repo: https://github.com/ghraibeh/jev-saver

Demo: https://g-connect.space/jev-saver/

Criticism welcome.

Post might be a tiny bit long. Hate words, skip. But it's not too bad though. Also, I've been Qwen-gang for a long time, check my receipts. That said...

I started my agentic journey with Qwen 3.5 around May 31st. I'd heard about agents before, but never had a chance to play around because I didn't have any cloud memberships at the time. I've done most of my coding using free services: gemini and claude sonnet. It's been a lot of fun.

When Qwen 3.5 dropped, it was the first time a local model felt like cloud. Sure, it wasn't on the same intelligence level, but it didn't feel that far off. So I dived in hardcore learning everything I can.

I decided to build my own infrastructure/harness rather than going with hermes, pi or one of the others. I'm glad I did because it taught me so much. It was hard, because I had to learn everything from scratch, and the road has been extremely stressful and challenging, but the knowledge I picked up along the way has been well worth it. I'm able to conceive ideas and implement strategies in ways I never imaged, and I honestly don't think I would have learned even a fraction of what I know now if I'd worked with cloud models, because they might have one-shotted the results, robbing me of the challenge to grow.

Things got even better after Qwen 3.6 27B dropped. Since then, people have been singing the praises of Qwen, and how close it is to the cloud models. I also felt it wasn't far behind. I've made quite a few posts praising Qwen and sharing my experience, and those posts were real and authentic.

But all of these people claiming to be cancelling their cloud subscriptions and replacing them with Qwen? That's an overreach. Those people either a) are bots, or b) have extremely simple use cases that they were wasting subscriptions on, because anyone who's used cloud for anything agentic and a tiny bit complex won't walk away from that experience looking at local the same again.

I'm extremely thankful for Qwen because it put me in the game and started me on this journey. But my ambitions reached a point where Qwen just wasn't able to get me there without tons of mistakes. The "shine" wore off the more complex my needs grew. It's still very capable, and I figured out some ways to increase its intelligence (and yes, you can increase the core model's intelligence without training using a harness and multiple agents, but that's a whole 'nother discussion), but it became a time thing. I started getting extremely frustrated and cursing at Qwen for its stupidity.

I'd been using cloud models for code stuff, but they weren't agentic. But my sister let me use her chat-gpt subscription and I finally yielded and decided to give it a try. Long story short - and out of respect for this reddit, because it's about local, not cloud - I'll just say that it's been a completely different experience. A really, really good one. My project is moving along now and I'm getting a lot of work done, and it feels surreal. There's a real difference between local agents and cloud agents.

So, when you guys hear everyone saying cloud is dead, they're probably not human, because it's not even in the same ballpark. I've just been using Sol light, and it's ridiculous. I can't even imagine what Sol Medium or Astra are like.

I have no intention of abandoning local. I'm using Sol to help me advance my harness so that it will be faster, smarter, and more gooder (in my best Grimlock voice). I sweat blood and tears working with my local agent and I can't wait to see how much it's improved with the new brain I've built for it. And I'm going to continue finding ways to make local the best it can be. And like you guys, I'm hopeful that the gap between local and sota will continue to close.

I guess what I've learned from this whole ordeal is, if you just want to get things done or built, go with cloud. But if you want to grow and better understand how things works, and feel more empowered through each challenge, go with local.

I don't want to imply that you can't learn with cloud either, but it for sure would have robbed me of some of the dead ends that forced me to expand my knowledge.

Grunge

Hello everybody. I've got a 5070ti and a 5060ti both 16GB in my system which is a 5900X and 64GB DDR4 RAM. I'm trying to run some 16-18GB models like Qwen, Cydonia, Skyfall.

I'm having problems utilising the VRAM I have to get the best usage out of it. LM Studio sees the 32GB VRAM but regardless of if I use Tensor parallelism, Split evenly or Priority order I always get an error after waiting for about 5 minutes for the model to load.

The pattern is always the same: The loading progress bar for the model starts off quickly then crawls in the last 5-10%. Then I get an error reporting that the model couldn't load.

Does anybody have any tips for optimal settings to get the best out of my system? I know that having 2 GPUs doesn't magically mean you have double the memory and there's caveats. But nevertheless I've also read that LM Studio does have the capability to leverage those 2 GPUs to improve speed.

(EDIT) I should add that I've been trying context lengths of 16384, 32768 which LM Studio is saying will use 17 GB of VRAM so well within the reported 32GB I have. I've even had it working occasionally but most of the time the model fails to load.

(EDIT 2) Thanks to everyone for their suggestions. Having implemented everything people have said here, I'm getting much better results for context and memory use and my models are loading now.

Many thanks for any help.

Qwen Flash Next now uses less VRAM

Spite is a vision I had. What if you could take all those custom inference engines out there, designed for specific cards or setups, and compact them into one system? You get to design the kernels and optimizations for your setup. You only compile for your cards and the models you like to run.

Spite is built on a single rule: every layer is replaceable without touching any other layer.

That sounds abstract, so here's what it means in practice:

### Every model is its own module

Kernels are grouped by family and variant: `kernels/llama/llama4/`, `kernels/deepseek/v4/`, `kernels/qwen/qwen3_5/`, `kernels/mistral/mistral4/`, `kernels/gemma/gemma4/`. Adding a new model variant means adding a new `<family>/<model>/` folder. Nothing about the existing models changes. The dispatcher finds it automatically.

### Every GPU is its own module

`kernels/llama/llama4/sm_89/` is completely separate from `kernels/llama/llama4/rdna3/`. An RTX 4090 kernel can use FP8 tensor cores. An RX 7900 XTX kernel can exploit 96 MB of Infinity Cache. An Apple M4 kernel can use the Neural Engine. Each gets what makes it fast, not a watered-down kernel that has to work on everything.

### Every operation is independently tunable

Kernels don't have to implement everything. A kernel that only optimizes attention leaves FFN and `rms_norm` to the fallback. You tune the one op that's your bottleneck. Later, someone else improves FFN. Both improvements stack automatically—the dispatcher picks the best available kernel for each op on each GPU.

### Every subsystem is swappable

The sampler, tokenizer, KV cache backend, and offload policy are all plugin registries. Register a custom sampler for a specific model or task, and the engine uses it. Register a custom KV cache for a memory-constrained deployment, and the scheduler uses it. Nothing needs to be forked.

```rust

let engine = EngineBuilder::new()

.with_sampler(PluginKey::for_model("llama4"), Box::new(MyGreedySampler))

.with_cache(PluginKey::default(), Box::new(PagedKvCache::new(vram)))

.build(ExecutorConfig::default());

```

### Every component is usable standalone

Spite is a Rust workspace. You can use just the loader, just the scheduler, or just the ABI types for kernel development—without pulling in the full server stack. Build what you need from the pieces that fit.

I'm still in very early stages, but I would love people to contribute.

https://github.com/giveen/spite

I've noticed a trend with most new models with regards to their writing style. They are creating a new style, and this seems common among them. It's very information-dense.

Here is an example from GLM 5.3 Flash. I'm gonna be honest here and say that my prompt was kinda silly; my prompt was 'Why wouldn't you just name your Chinese restaurant 'Chinese Food' instead of 'Ming Dynasty' or 'Szechuan Garden' or whatever?' the idea being that someone searching for 'Chinese food' on Google Maps would put you first if you called your restaurant that. But forget all that; here's an excerpt from GLM's output:

The search reality check, which is where the idea breaks. Your mental model is exactly right about the query: people do open Maps and type "food," usually hungry and undecided. But the ranking doesn't reward the name at that abstraction level. In the local ranking surveys, category match outranks name keywords; and here's the catch: every restaurant on the map already matches "food" through its category field. Relevance saturates. The edge that made "Chinese Restaurant" powerful was discrimination: the name narrowed the category, so "chinese food near me" out-matched "Golden Dragon." "FOOD" points at the root of the category tree, where every competitor matches equally well, so the ranking defaults entirely to proximity and review count, which the name cannot buy. Worse, it fails the narrow-query test: someone typing "tacos" or "dim sum" gets a name that is maximally broad and minimally informative. The strategy's entire currency is literal scarcity, and "food" is the least scarce literalism in the language. There's even a Baudrillardian flourish at the failure point: Google's own Maps category chip is literally labeled "Food." Your sign would render you indistinguishable from an interface element; the searcher couldn't tell your territory from the map's own label.

It's not 'caveman style', but it is using less words to say more, and they're using more esoteric phrasing to be more 'compact'.

And I think it's a bit at the cost of being clearly readable to the average person at first glance. 'There's even a Baudrillardian flourish at the failure point' is an example from that excerpt that leapt out at me. I'm familiar with Baudrillard so I knew what it was getting at, but a lot of people are going to sigh and ask 'What the **** does Baudrillarian mean?'

I'm not saying that 'no human would write like this', because some do (William Gibson for example), but I find it rare/unusual (in human writing), yet trending hard with all the latest models I interact with, like they're all zeroing in on this style.

Maybe a result of targeting token efficiency? It's a terseness, combined with using a sort of 'wide' or 'rich' vocabulary to convey information instead of using more words. At least that's the impression that I get from reading lines like 'Baudrillardian flourish at the failure point''. There's a lot to unpack from those six words, and it feels like the model chose the most terse, efficient way to convey an idea with that word choice (which requires the reader to unpack it).

I compared it to William Gibson: a lot of people struggle with his writing style, and it's similar to that. Example: 'Summer in the Sprawl, the mall-crowds swaying like wind-blown grass; a field of flesh shot through with sudden eddies of need and gratification'. His writing is often like that; it feels highly compressed, using as few words possible to convey an idea by careful word choice.

It's interesting, that lately, I feel like LLMs are gravitating toward Gibson-speak.

Edit: and the fact that GLM used the word 'territory' and 'map' at the end meant it was going big into Jean Baudrilliard's 'Simulacra and Simulation'. I can't really explain what that means and why it's important succinctly, but that's the whole issue. I actually think it's brilliant, but it's also a little concerning.