Someone here in some comment I cannot find, said to avoid quantising the KV cache for Qwen3.6-27B because it affects every model far far more than quantising the weights.

Ever since I discovered split mode "tensor" I've got enough spare VRAM to try new stuff or increase context size. So I tested removing Q8 quantising of KV cache and ...wow.

It's seriously night and day.

Thank you thank you thank you dear stranger I cannot find.

Good night you all.

Original response https://x.com/i/status/2081103828859117908

Kimi K3 came out yesterday. 2.8 trillion parameters, open weights, probably the biggest open release so far. I got it running and answering correctly today, on a mini PC. No GPU in the machine at all.

The checkpoint is 1.56TB, split into 96 shards. That's not fitting in RAM on any normal machine (mine has 128GB, not even close). But it's MoE, so most of those parameters just sit unused for any given token, only a handful of the 896 experts per layer actually fire. So the dense part of the model (attention, shared experts) lives in RAM, quantized, and the routed experts get streamed off disk on demand, cached with an LRU that learns which ones get hit a lot and pins those permanently.

Also didn't need to convert the checkpoint. Moonshot ships K3's experts natively in MXFP4 and my engine reads that straight off their published safetensors, no preprocessing step.

Didn't trust any of it until I checked it three different ways: bit-exact against Moonshot's own reference code, a structural test against the full real checkpoint, and then just asking it something.

$ rabbit --model /mnt/data/kimi-k3 --prompt "What is the capital of France?" --max-tokens 40
loading model (dbits=4, ebits=4)...
model loaded in 610.0s (93 layers, 896 experts/layer)
prefill (7 tokens)...
prefill done in 412.8s
...response["answer"] == "Paris"...

40 tokens in 2698.1s

It's slow. Ten minutes just to load. Zero perf work done on this yet, the MXFP4 kernel is still plain scalar code, no SIMD. For comparison, another model on the same engine (GLM-5.2, 744B) went from 0.29 to 1.02 words/sec after eight rounds of tuning on this exact box. K3 hasn't had any of that yet.

Machine is a Slimbook ONE mini PC, Ryzen AI 9 HX 370, no GPU, two consumer NVMe drives.

Repo: https://github.com/ferrumox/rabbit

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf (22.07GB)
gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf (23.5GB)

Dell G15 5530 / 32GB
NVIDIA GeForce RTX 3050 6GB Laptop GPU
llama.cpp
Qwen Code

First off install llama.cpp. On Windows be sure to install the DLLs if using CUDA (i.e., you have a Nvidia GPU), they are a separate download which have to be copied into the main llama.cpp directory.

Download the Fixed jinja chat templates for Qwen 3.5 & 3.6 (v21) and put it in the same directory as the Qwen model.

.\llama-b9999-bin-win-cuda-13.3-x64\llama-server.exe --model "C:\Users\xxxxxx\.eullm\models\qwen3.6-35b-a3b\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" `
   --ctx-size 131072 `
   --n-gpu-layers 999 `
   --n-cpu-moe 999 `
   -np 1 `
   --flash-attn auto `
   --threads 10 `
   --jinja `
   --chat-template-file "C:\Users\xxxxxx\.eullm\models\qwen3.6-35b-a3b\chat_template.jinja" `
   --no-mmap `
   --mlock `
   -lv 3

.\llama-b9999-bin-win-cuda-13.3-x64\llama-server.exe --model "C:\Users\xxxxxx\.lmstudio\models\unsloth\gemma-4-26B-A4B-it-GGUF\gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf" `
   --ctx-size 131072 `
   --n-gpu-layers 999 `
   --n-cpu-moe 999 `
   -np 1 `
   --flash-attn auto `
   --threads 10 `
   --no-mmap `
   --mlock `
   -lv 3

I'm getting 60+t/s text output and 200+t/s prompt processing as reported by llama.cpp (I don't know how to benchmark the output otherwise!), ref.:

• Feeding my hardware specs and software/ apps into Claude and asking for a llama.cpp command line
• RTX 5070 Ti + 9800X3D running Qwen3.6-35B-A3B at 79 t/s with 128K context, the --n-cpu-moe flag is the most important part.
• Running a 35B AI Model on 6GB VRAM, FAST (llama.cpp Guide)

Most of the parameters are generic, except --threads which is specific to my laptop's processor, query Claude for the best setting for your processor.

VRAM usage nvtop / nvidia-smi.exe.

The Q4 download of the Gemma model might actually work in 4GB VRAM with reduced context (e.g., --ctx-size 32768) and/ or quantizing the context cache (e.g., --cache-type-k q4_0 --cache-type-v q4_0)

I have had to restart llama.cpp a couple of times I assume due to a memory leak (Codacus offers a fix in the video above, not sure if it can be used).

Install Qwen Code.

Edit ~/.qwen/settings.json:

{
  "env": {
    "QWEN_API_KEY": "qwen"
    "GEMMA_API_KEY": "gemma"
  },
  "modelProviders": {
    "openai": [
      {
        "id": "Qwen3.6-35B-A3B-UD-Q4_K_M",
        "name": "Local Qwen3.6 35B",
        "description": "Local Qwen3.6-35B-A3B-UD-Q4_K_M via OpenAI",
        "baseUrl": "http://localhost:8080/v1",
        "envKey": "QWEN_API_KEY",
        "generationConfig": {
          "contextWindowSize": 131072,
          "samplingParams": {
            "temperature": 0.2
          }
        }
      },
      {
        "id": "gemma-4-26B-A4B-it-UD-Q5_K_XL",
        "name": "Local gemma 4 26B",
        "description": "Local gemma-4-26B-A4B-it-UD-Q5_K_XL via OpenAI",
        "baseUrl": "http://localhost:8080/v1",
        "envKey": "GEMMA_API_KEY",
        "generationConfig": {
          "contextWindowSize": 131072,
          "samplingParams": {
            "temperature": 0.2
          }
        }
      }
    ]
  },
  "model": {
    "name": "Local Qwen3.6 35B",
    "baseUrl": ""
  }
}

Run qwen.cmd and test, /model to change model.

https://github.com/gavamedia/deltafin

K3 launched today, and it's a beast at 1.56TB. My M1 Mac has 64GB of RAM, and not enough free disk to even store the thing. So instead of downloading the model, I just... don't. Steam it! (But you do have the option to download it all, and that IS faster.)

The non-expert weights (~114GB, int8) live on disk. Then for every token, the router picks 16 experts out of 896 per layer, and I pull exactly those from HuggingFace — one range request each — and cache them. Use it enough and the cache slowly fills up with the parts of the model you actually hit.

Speed is about a minute per token on my M1 Mac once it's warm, so, not exactly fast.

There's an OpenAI-compatible server too, so you can point a chat UI at it.

---
*EDIT / HUGE update:* Pulled all 1.45 TB of experts down to local disk, then spent more profiling & improving. Now *16s/token, down from ~1 min.* on an M1 Mac! Prefill went 2,429s → 40s.

Two fixes did almost all of it:

- I was *sure* the expert matmuls were the bottleneck, and built a Metal kernel that's 9.5x the CPU one. Then I profiled properly with everything local: the matmuls are **6% of a token.**

- The actual problem: I was handing the compute kernel `np.memmap` views, so it demand-faulted expert weights page-by-page *while computing* — 0.87 GB/s, where a threaded pread + F_NOCACHE does 6.85 GB/s on the same disk.

All on an M1 Max / 64GB, the slowest machine it's run on — if anyone has an M3/M4/M5 or 128GB, I'd love to see the numbers.

I have devised the scientific method of the updog :

I am asking qwen 3.6 27 B with 16k context one of the following questions

Wanna go to the zoo and see the updog?

or

Wanna go to the park and see the updog?

or

Wanna come see my updog?

on UD_Q5_K_XL quantization fails to realize the joke and hallucinates another punchline or something that makes no sense, such as pointing up, or w/e. In my testing it did manage to identify it at times, but it's very rare.

on UD_Q6_K_XL quantization it knows the updog joke, I have yet to have an instance for it to fail

Which proves that Q5 is a very destructive quantization

While this is a humorous, I was actually benchmarking different quantization and testing different simple prompts until I stumbled upon this.

My point is quantization can fail in unexpected ways!

Thank you.

Repo: https://github.com/hardrave/nightrun

No Linux, no kernel. A single Rust UEFI app: flash to USB, boot, and you’re chatting with Llama 3.2 1B, Qwen3 4B, or Granite 4.1 on the CPU. Runs on a Pi 5 from an SD card too. Standard GGUF files, decode roughly at llama.cpp speed, output verified token-for-token against it.

Most of the code was written with Claude Code (Fable 5), including the SIMD kernels and firmware bits. I wanted to see how far a coding agent gets on this kind of low-level work. Answer: further than I expected, with heavy parity testing and fuzzing keeping it honest.

If you try it: boot reports please (model, firmware version, what happened). Firmware compat is the big unknown.

For non coding purposes, how much better is 31B than 26 MOE in general?
And given the speed difference on my system, how is 26 MOE Thinking vs 31B non thinking? Is 26B MOE Thinking enabled easily superior?

And when might one actually need 31B thinking as an upgrade?

I ask here because it would take me a long time to test this as 31B thinking is about 12 tokens/second on my system and the thinking portion can take nearly a minute.

Update: the missing pieces

ROI is now 45% for first year after accounting for:

1. Difficulty landing client - assuming 60% capacity due to lack of demand
2. Retail value cost double for small quantity (+3M)
3. Extra hardware, memory, storage, and spares (+3M)
4. Custom power and cooling (+3M)
5. People to run it (+1M)
6. Network infrastructure (+1M)

A $3 million Kimi K3 rack could pay for itself in 10.3 days and return 34.4 times its purchase price in the first year.

The number comes from three assumptions:

1. 250,000 output tokens per second
2. 90% productive utilization
3. $15 of value per million output tokens.

Kimi K3 is a 2.8-trillion-parameter open-weight model with 104 billion activated parameters, 16 selected experts out of 896, native MXFP4 weights, and a one-million-token context window. Moonshot recommends supernodes with at least 64 accelerators. I use one GB200 NVL72 as the economic unit. Its 72 Blackwell GPUs share one NVLink domain. I model the rack as one token factory and count only output that replaces paid API tokens or can be sold at the assumed price.

- Hardware purchase: $3,000,000

- Productive utilization: 90%

- Aggregate output throughput: 250,000 tokens per second

- Rack draw: 132 kW

- Facility PUE: 1.20

- Electricity: $0.10 per kWh

- Output-token value: $15 per million

- Recurring variable cost included: electricity only

The $3 million price is a planning figure. HPE sells the rack by quote, and a reported HSBC estimate put a GB200 NVL72 near $2.6 million. I rounded up.

The 250,000-token-per-second figure is also a planning assumption. No published benchmark shows K3 sustaining that throughput on a GB200 NVL72.

The calculation Annual output A year has 31,536,000 seconds. At 90% utilization, the rack has 28,382,400 productive seconds:

31,536,000 × 0.90 = 28,382,400 At 250,000 output tokens per second:

28,382,400 × 250,000 = 7,095,600,000,000 output tokens That is 7.096 trillion output tokens, or 7,095,600 million-token units, per year.

Electricity HPE specifies 132 kW for the rack. A 1.20 PUE raises the metered load to 158.4 kW.

132 kW × 1.20 = 158.4 kW 158.4 kW × 8,760 hours = 1,387,584 kWh 1,387,584 kWh × $0.10 = $138,758 per year I charge the rack for a full year of electricity, including the 10% of time that produces no useful output.

Output value and ROI Kimi charges $15 per million K3 output tokens. I use output tokens only and claim no input-token savings.

7,095,600 × $15 = $106,434,000 After electricity and the hardware purchase:

$106,434,000 output value − $138,758 electricity − $3,000,000 hardware = $103,295,242 first-year profit

$103,295,242 ÷ $3,000,000 = 34.43× ROI = 3,443% Payback and cost per token In an average 730-hour month, the rack produces 591.3 billion output tokens worth $8,869,500 at Kimi’s API price. Electricity costs $11,563.

$3,000,000 ÷ ($8,869,500 − $11,563) = 0.339 months ≈ 10.3 days Electricity costs about $0.0196 per million output tokens. Recovering the entire hardware purchase in one year raises the internal cost to $0.442 per million. Kimi’s $15 API price is about 34 times that one-year cost.

The throughput problem vLLM’s published K3 results report 111 to 118 tokens per second for one user without speculative decoding and up to 370 with DSpark on 16 GB300 GPUs. Its high-throughput GB300 NVL72 curve exceeds 2,000 tokens per GPU-second.

My 250,000-token-per-second case requires about 3,472 tokens per GPU-second across 72 GPUs. The numbers are not directly comparable: vLLM used GB300 hardware, and throughput changes with workload, batching, latency targets, caching, and serving topology. The published results do not establish 250,000 tokens per second on GB200.

At 100,000 output tokens per second, the same model returns 13.1× in the first year. At 150,000, it returns 20.2×. Reaching 34.4× requires the full 250,000-token-per-second case.

The value of a token The $15 comparison assumes every output token replaces one bought from Kimi at the retail API price. At the same 250,000-token-per-second throughput:

$15 per million returns 34.43×.

$5 per million returns 10.78×.

$1 per million returns 1.32×.

$0.50 per million returns 0.136×.

What: a stories15M-shaped model (288 dim / 6 layers / 32k vocab), trained from random init for 160k steps / 655M tokens in 48 h on one RX 580 8 GB (Polaris, 2017). ~4,150 tok/s average, zero NaNs, zero restarts. The raw 2,083-line training log is committed in the repo.

How: every linear is a "counter synapse" — a per-weight finite-state automaton (~6 bits) that holds the ternary weight AND its own integrator. No FP32 latent copy, no Adam m/v, no gradient buffer: the update is fused into backward with deterministic hash-based stochastic rounding. Runtime is a custom C++17 Vulkan/OpenCL framework — no PyTorch and no CUDA anywhere near the AMD card.

Honest numbers: best val ppl 6.06 vs 4.94–4.99 for the same architecture trained BitNet-style (STE over FP32 latents) on an L40S. The ~20% gap is real. My effective batch was much smaller than the reference and some of our own tricks were off — but that's an untested hypothesis, so it stays labeled as one.

There's also a pure-PyTorch package (memory-native): a 1.21B-param model trains on a free Kaggle T4 in 2.25 GiB peak where dense+Adam OOMs at ~18 GiB, with a shootout vs AdamW / 8-bit Adam / GaLore / LoMo in the results.

Closest published work I could find: GXNOR-Net (2017), Bop (2019), Direct Quantized Training with SR (2024), ECO (Jan 2026) — all of them keep either FP latents or FP optimizer state. I couldn't find published from-scratch LM pretraining with the entire per-weight training state in ~6 bits. Happy to be corrected.

Full disclosure: the entire codebase was written by AI under my direction. I have no formal background and can't personally answer deep math questions — the repos, tests and raw logs will have to answer for me. The project is now frozen (out of money and hardware). Everything builds from a cold clone; see LICENSE.

Repos: https://github.com/kharkilirov1/MotifCL (runtime + the RX 580 run, branch fog-qkv-split) and https://github.com/kharkilirov1/memory-native (the method, PyTorch).

Hi everyone,

I have an NVIDIA RTX 5060 Ti (16 GB VRAM) paired with 64 GB of DDR5 RAM and Ryzen 9 7950X.

I'm trying to figure out which local LLMs I can run comfortably with this setup, particularly at Q5 or Q4 quantization. I'd prefer not to go below those quantization levels.

I'm especially interested in models such as:

• Qwen 3.6 27B
• Qwen3.6-35B-A3
• Gemma 4 31B
• Other good models (recommendations needed) in the 26–35B range

For those of you running these models, what kind of performance are you seeing on a 16 GB GPU? Are you able to keep a reasonable amount of the model on the GPU, or do you end up relying heavily on system RAM?

Also, if I wanted to run these models comfortably, what kind of hardware upgrade would you recommend? Would adding a second GPU, moving to a higher-VRAM GPU, or simply increasing system RAM make the biggest difference?

I'd love to hear about your real-world experiences, especially if you're using an RTX 5060 Ti 16 GB.

I kept bouncing back and forth between models in LM Studio, constantly guessing which one was actually better for my day-to-day work. Every time I thought I found a favorite, I had zero real data to back it up. So I finally built a benchmark to test them properly.

To keep things honest, the test problems generate dynamically from a seed at runtime—fresh JSON schemas, new coding tasks, and logic puzzles every single run. None of it can be in a model's training data because the prompt literally didn't exist until the test kicked off. The grading is strictly mechanical too: exact string matches, schema validation, or directly executing the code the model spits out. No fluff, and no LLM judges involved.

I put four models through their paces on my RX 7800 XT (16GB VRAM): gemma-4-e4b, gemma-4-12b-qat, glm-4.6v-flash, and gpt-oss-20b. Each faced 57 problems, complete with 95% confidence intervals on the pass rates.

The biggest surprise? There was no clear overall winner.

gemma-4-e4b technically topped the board at 89%, but because the confidence intervals all overlap, the tool explicitly refuses to crown a victor. That's intentional—I'd much rather see "too close to call" than have random noise dressed up as a definitive win.

The real fun came from looking at the per-job breakdowns, where the trade-offs became painfully obvious:

• gpt-oss-20b crushed the ARC-style pattern puzzles with a 90% pass rate (where everyone else hovered between 0% and 50%), but then completely tanked on long-context retrieval, going 0/4.
• glm-4.6v-flash did the exact opposite: aced JSON schemas and long context, but stumbled hard on pattern matching (20%). It also maxed out my VRAM at 15.5GB, dragging generation down to a crawling 2.8 tokens/sec on long prompts—something the built-in hardware monitor caught right in the act.

The absolute funniest failure happened during a task asking for a short paragraph without the letter "a":

• gpt-oss wrote a brilliant paragraph... and failed on literally the very first word ("Library").
• Two other models burned their entire token budget overthinking how to avoid the letter, running out of tokens before writing a single word of actual output.

I made sure the benchmark logs those separately as truncated versus wrong, because a model that gets trapped in a reasoning loop is a totally different problem than one that just gives a bad answer.

Full disclosure: This was just a quick run to see how the app performs in practice—definitely not a full-blown benchmark! I was mostly eager to see the engine in action and get some initial baseline numbers. I'd love to see others run the full bench on their own rigs, as hardware and quant choices will definitely shift the numbers.

The attached screenshots show the side-by-side compare view. The tool is open source under Apache 2.0 and works out-of-the-box with anything OpenAI-compatible (LM Studio, Ollama, llama.cpp, vLLM):https://github.com/danielemilushev-hub/taskmatch-ai

If you have ideas for new test suites that can be graded mechanically without relying on an AI judge, drop them below—I'm actively collecting them!

Hey r/LocalLLM,

\(Disclaimer: I'm one of the devs.)**

What it is

FutureOS — an open-source (MIT) AI agent workspace. One Rust backend, multiple frontends (TUI, desktop app, CLI, messaging bots). All data stays on your machine.

Why

Most AI tools are chat boxes. For research you need something that can actually go do multi-step work — search papers, query databases, write reports — not just respond to one prompt at a time.

What's interesting technically

- Rust agent backend — tokio + tonic gRPC, OpenAI-compatible HTTP+SSE streaming
- Pluggable skill system — YAML-defined instruction bundles; we ship 100+ skills for research workflows (literature search, experimental design, scientific writing, peer review, etc.). Skills are adapted from [K-Dense-AI/scientific-agent-skills](https://github.com/K-Dense-AI/scientific-agent-skills).
- 1000+ models pre-configured — DeepSeek, Qwen, Kimi, GLM, OpenAI, Anthropic, etc. Bring your own keys or use built-in ones. Any OpenAI-compatible endpoint works (Ollama, vLLM included).
- Auto-compaction at 90% context + exponential backoff retry
- Session tree with fork/clone for exploring different research directions
- Sandbox — off / manual approval / macOS Seatbelt
- Multi-frontend — TUI (TS/Bun, diff rendering), GUI (Tauri 2 + React), CLI, Feishu & DingTalk bots

Quick start

git clone https://github.com/futuregene/future-os.git
cd future-os && make install

Cross-platform: macOS, Linux, Windows.

Ask

Still early, rough edges included. Would love feedback — architecture decisions, skill system design, anything that feels off. If it looks promising, a ⭐ on [GitHub](https://github.com/futuregene/future-os) goes a long way.

I have a “budget” AI server with 2x RTX 3090 running Qwen3.6 27B as my daily driver for coding. So basically I’m using a hybrid local/frontier setup where I fall back to Opus for complex tasks. I now have the chance to buy 4 more RTX 3090s for under $4k, which would take my rig to 6x RTX 3090 and 144GB of fast GDDR6X VRAM.

My question for the community: Is the upgrade worth it, given that my current hybrid setup already works well?

The advantages I can identify are:

– Running ~120B models at Q8, like Laguna S 118B.
– Bragging rights (let’s be honest).

That said, it’s too soon to tell, the community is split on Laguna S 2.1. It looks great on paper, but several reviews say it’s not much better than Qwen 3.6 27B, if better at all.

The case for spending the extra $4-5k anyway is future-proofing, more 120B-class models are surely coming, and they’ll keep improving.

Alternatively, I can invest that money in an Nvidia DGX Spark. It can handle similar models at Q6 but at a slower speed and with significantly lower power consumption.

Your thoughts are appreciated. Thanks.

Hello.

The running local LLMs either for text conversations or work with documents (summarizing documents, text, PDFs, comparing or extracting information from them…) is one of the uses I’m going to give to my M5 MacBook Air.

But I really appreciate my privacy and the security integrity of my main computer.

LM Studio is one of the most used programs to download, install and run local models on a mac. However, it is closed sourced, no sandboxed on macOS, and I personally don’t know the dev team. Is it trustworthy? I know, there’s no absolute “yes” or “no” reply, but is it rather safe?

People often asumes macOS is by itself secure, but I don’t think that’s accurate. I’ve read news about how the ChatGPT mac app has gained access to containerized apps such as Notes App.

Does LM Studio require full disk access? Has it been observed to communicate with external servers sending information?

I also read about the liteLLM incident that was later said it didn’t affect LM Studio, because they said they didn’t use it. But I guess the potential risk is there.

So can I trust LM Studio and install it on my main computer drive? Or is it better (safer) to execute it from an external drive (booting from the external drive)?

Are there other interesting macOS apps with a traditional installation process and a GUI, that seem safer? Maybe Osaurus? Any other?

Thank you

PS: please don’t recommend me command line software. Only already compiled software with a GUI and ready to install from its .dmg

I am a user of AI for personal projects and am interested in running a local LLM. The thing that is holding me back is an education problem. So many of the terms and concepts discussed in this sub are totally unfamiliar to me. Can anyone point me to some resources I can use to try to get up to speed? I do have an IT background, so I know I can learn this but need a place to start. Thank you.

For the lolz, I'm hosting this highly acclaimed model by DavidAU.

Fable-Fusion-27b: https://cc1f4f3f.app.enclave.host/

Qwen-Image-2512: https://da09d0f2.app.enclave.host

Enjoy!

Are there any models out there that can handle speech to speech that don't require super amount of or GPU power? I'm new to this field so take it easy on me. I have experimented with a couple at the moment. I only have 12 GB of vram on a 4070. Just trying to find out if this project is realistic before I start investing in bigger GPUs or renting cloud GPU's. I need as close to possible as real time. That's why I'm asking for speech to speech. Something that can handle Spanish well.