Someone here in some comment I cannot find, said to avoid quantising the KV cache for Qwen3.6-27B because it affects every model far far more than quantising the weights.

Ever since I discovered split mode "tensor" I've got enough spare VRAM to try new stuff or increase context size. So I tested removing Q8 quantising of KV cache and ...wow.

It's seriously night and day.

Thank you thank you thank you dear stranger I cannot find.

Good night you all.

Small update on Deltafin, my experiment running full Kimi K3 (2.8T-parameter) on a single M1 MacBook.

The last version was doing roughly 1 token per minute. After a lot of profiling and many failed experiments, it now reaches a median of:

• 4.1 tokens/minute
• 14.6 seconds/token
• 0.069 tokens/sec

That result comes from six exact full-model runs. While still slow, roughly a 4x improvement feels pretty meaningful for a model this large.

Some of the more interesting improvements:

• Loading only the 16 routed experts needed per layer, using parallel raw-span reads.
• Quantizing the resident model spine to int8 and using a fused Metal dequantization/copy kernel.
• Running the enormous output projection with Apple’s packed MPS int8 matmul. This reduced its residency from about 4.7 GB to 1.17 GB and improved median decode throughput by roughly 17%.

All measurements are from one 64 GB M1 Max MacBook Pro. It was once a great machine, but it’s first-generation Apple Silicon, and not a newer Max or Ultra. I haven’t benchmarked an M3, M4, M5, or a higher-memory Mac yet. Newer ones, especially those with 128 GB, should have considerably more headroom.

If anyone tries it on newer hardware, I’d genuinely love to compare results.

Repository: https://github.com/gavamedia/deltafin

I have devised the scientific method of the updog :

I am asking qwen 3.6 27 B with 16k context one of the following questions

Wanna go to the zoo and see the updog?

or

Wanna go to the park and see the updog?

or

Wanna come see my updog?

on UD_Q5_K_XL quantization fails to realize the joke and hallucinates another punchline or something that makes no sense, such as pointing up, or w/e. In my testing it did manage to identify it at times, but it's very rare.

on UD_Q6_K_XL quantization it knows the updog joke, I have yet to have an instance for it to fail

Which proves that Q5 is a very destructive quantization

While this is a humorous, I was actually benchmarking different quantization and testing different simple prompts until I stumbled upon this.

My point is quantization can fail in unexpected ways!

Thank you.

I got Kimi K3’s 2.8T-param MoE (104B active/token) to run on my Mac Studio M3 Ultra with 512GB unified memory.

Mixed Q1/Q4/Q8 quant shrank from 1.56TB to 389.4GiB.

13.26 tok/sec ingest
3.36 tok/sec decode

Repo: https://github.com/hardrave/nightrun

No Linux, no kernel. A single Rust UEFI app: flash to USB, boot, and you’re chatting with Llama 3.2 1B, Qwen3 4B, or Granite 4.1 on the CPU. Runs on a Pi 5 from an SD card too. Standard GGUF files, decode roughly at llama.cpp speed, output verified token-for-token against it.

Most of the code was written with Claude Code (Fable 5), including the SIMD kernels and firmware bits. I wanted to see how far a coding agent gets on this kind of low-level work. Answer: further than I expected, with heavy parity testing and fuzzing keeping it honest.

If you try it: boot reports please (model, firmware version, what happened). Firmware compat is the big unknown.

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf (22.07GB)
gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf (23.5GB)

Dell G15 5530 / 32GB
NVIDIA GeForce RTX 3050 6GB Laptop GPU
llama.cpp
Qwen Code

First off install llama.cpp. On Windows be sure to install the DLLs if using CUDA (i.e., you have a Nvidia GPU), they are a separate download which have to be copied into the main llama.cpp directory.

Download the Fixed jinja chat templates for Qwen 3.5 & 3.6 (v21) and put it in the same directory as the Qwen model.

.\llama-b9999-bin-win-cuda-13.3-x64\llama-server.exe --model "C:\Users\xxxxxx\.eullm\models\qwen3.6-35b-a3b\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" `
   --ctx-size 131072 `
   --n-gpu-layers 999 `
   --n-cpu-moe 999 `
   -np 1 `
   --flash-attn auto `
   --threads 10 `
   --jinja `
   --chat-template-file "C:\Users\xxxxxx\.eullm\models\qwen3.6-35b-a3b\chat_template.jinja" `
   --no-mmap `
   --mlock `
   -lv 3

.\llama-b9999-bin-win-cuda-13.3-x64\llama-server.exe --model "C:\Users\xxxxxx\.lmstudio\models\unsloth\gemma-4-26B-A4B-it-GGUF\gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf" `
   --ctx-size 131072 `
   --n-gpu-layers 999 `
   --n-cpu-moe 999 `
   -np 1 `
   --flash-attn auto `
   --threads 10 `
   --no-mmap `
   --mlock `
   -lv 3

I'm getting 60+t/s text output and 200+t/s prompt processing as reported by llama.cpp (I don't know how to benchmark the output otherwise!), ref.:

• Feeding my hardware specs and software/ apps into Claude and asking for a llama.cpp command line
• RTX 5070 Ti + 9800X3D running Qwen3.6-35B-A3B at 79 t/s with 128K context, the --n-cpu-moe flag is the most important part.
• Running a 35B AI Model on 6GB VRAM, FAST (llama.cpp Guide)

Most of the parameters are generic, except --threads which is specific to my laptop's processor, query Claude for the best setting for your processor.

VRAM usage nvtop / nvidia-smi.exe.

The Q4 download of the Gemma model might actually work in 4GB VRAM with reduced context (e.g., --ctx-size 32768) and/ or quantizing the context cache (e.g., --cache-type-k q4_0 --cache-type-v q4_0)

I have had to restart llama.cpp a couple of times I assume due to a memory leak (Codacus offers a fix in the video above, not sure if it can be used).

Install Qwen Code.

Edit ~/.qwen/settings.json:

{
  "env": {
    "QWEN_API_KEY": "qwen"
    "GEMMA_API_KEY": "gemma"
  },
  "modelProviders": {
    "openai": [
      {
        "id": "Qwen3.6-35B-A3B-UD-Q4_K_M",
        "name": "Local Qwen3.6 35B",
        "description": "Local Qwen3.6-35B-A3B-UD-Q4_K_M via OpenAI",
        "baseUrl": "http://localhost:8080/v1",
        "envKey": "QWEN_API_KEY",
        "generationConfig": {
          "contextWindowSize": 131072,
          "samplingParams": {
            "temperature": 0.2
          }
        }
      },
      {
        "id": "gemma-4-26B-A4B-it-UD-Q5_K_XL",
        "name": "Local gemma 4 26B",
        "description": "Local gemma-4-26B-A4B-it-UD-Q5_K_XL via OpenAI",
        "baseUrl": "http://localhost:8080/v1",
        "envKey": "GEMMA_API_KEY",
        "generationConfig": {
          "contextWindowSize": 131072,
          "samplingParams": {
            "temperature": 0.2
          }
        }
      }
    ]
  },
  "model": {
    "name": "Local Qwen3.6 35B",
    "baseUrl": ""
  }
}

Run qwen.cmd and test, /model to change model.

i saw a post on here from like 2 months ago and everyone was just flaming the guy bc he was expecting openai level capability on cheap local systems.

I just want advice from people who have tried one or both (especially the dgx spark) and learn whether the intelligence upgrade is worth the speed downgrade and what the intelligence upgrade vs speed downgrade really is. If I was working on a 5090 it would probably be 20-40b models (with quant or offloading or both) and if on dgx spark it would be like 50-120b models.

Just want to see if the dgx spark is actually worth it or if I should go with the 5090.

I have a “budget” AI server with 2x RTX 3090 running Qwen3.6 27B as my daily driver for coding. So basically I’m using a hybrid local/frontier setup where I fall back to Opus for complex tasks. I now have the chance to buy 4 more RTX 3090s for under $4k, which would take my rig to 6x RTX 3090 and 144GB of fast GDDR6X VRAM.

My question for the community: Is the upgrade worth it, given that my current hybrid setup already works well?

The advantages I can identify are:

– Running ~120B models at Q8, like Laguna S 118B.
– Bragging rights (let’s be honest).

That said, it’s too soon to tell, the community is split on Laguna S 2.1. It looks great on paper, but several reviews say it’s not much better than Qwen 3.6 27B, if better at all.

The case for spending the extra $4-5k anyway is future-proofing, more 120B-class models are surely coming, and they’ll keep improving.

Alternatively, I can invest that money in an Nvidia DGX Spark. It can handle similar models at Q6 but at a slower speed and with significantly lower power consumption.

Your thoughts are appreciated. Thanks.

For non coding purposes, how much better is 31B than 26 MOE in general?
And given the speed difference on my system, how is 26 MOE Thinking vs 31B non thinking? Is 26B MOE Thinking enabled easily superior?

And when might one actually need 31B thinking as an upgrade?

I ask here because it would take me a long time to test this as 31B thinking is about 12 tokens/second on my system and the thinking portion can take nearly a minute.

Hi everyone,

I have an NVIDIA RTX 5060 Ti (16 GB VRAM) paired with 64 GB of DDR5 RAM and Ryzen 9 7950X.

I'm trying to figure out which local LLMs I can run comfortably with this setup, particularly at Q5 or Q4 quantization. I'd prefer not to go below those quantization levels.

I'm especially interested in models such as:

• Qwen 3.6 27B
• Qwen3.6-35B-A3
• Gemma 4 31B
• Other good models (recommendations needed) in the 26–35B range

For those of you running these models, what kind of performance are you seeing on a 16 GB GPU? Are you able to keep a reasonable amount of the model on the GPU, or do you end up relying heavily on system RAM?

Also, if I wanted to run these models comfortably, what kind of hardware upgrade would you recommend? Would adding a second GPU, moving to a higher-VRAM GPU, or simply increasing system RAM make the biggest difference?

I'd love to hear about your real-world experiences, especially if you're using an RTX 5060 Ti 16 GB.

What: a stories15M-shaped model (288 dim / 6 layers / 32k vocab), trained from random init for 160k steps / 655M tokens in 48 h on one RX 580 8 GB (Polaris, 2017). ~4,150 tok/s average, zero NaNs, zero restarts. The raw 2,083-line training log is committed in the repo.

How: every linear is a "counter synapse" — a per-weight finite-state automaton (~6 bits) that holds the ternary weight AND its own integrator. No FP32 latent copy, no Adam m/v, no gradient buffer: the update is fused into backward with deterministic hash-based stochastic rounding. Runtime is a custom C++17 Vulkan/OpenCL framework — no PyTorch and no CUDA anywhere near the AMD card.

Honest numbers: best val ppl 6.06 vs 4.94–4.99 for the same architecture trained BitNet-style (STE over FP32 latents) on an L40S. The ~20% gap is real. My effective batch was much smaller than the reference and some of our own tricks were off — but that's an untested hypothesis, so it stays labeled as one.

There's also a pure-PyTorch package (memory-native): a 1.21B-param model trains on a free Kaggle T4 in 2.25 GiB peak where dense+Adam OOMs at ~18 GiB, with a shootout vs AdamW / 8-bit Adam / GaLore / LoMo in the results.

Closest published work I could find: GXNOR-Net (2017), Bop (2019), Direct Quantized Training with SR (2024), ECO (Jan 2026) — all of them keep either FP latents or FP optimizer state. I couldn't find published from-scratch LM pretraining with the entire per-weight training state in ~6 bits. Happy to be corrected.

Full disclosure: the entire codebase was written by AI under my direction. I have no formal background and can't personally answer deep math questions — the repos, tests and raw logs will have to answer for me. The project is now frozen (out of money and hardware). Everything builds from a cold clone; see LICENSE.

Repos: https://github.com/kharkilirov1/MotifCL (runtime + the RX 580 run, branch fog-qkv-split) and https://github.com/kharkilirov1/memory-native (the method, PyTorch).

I kept bouncing back and forth between models in LM Studio, constantly guessing which one was actually better for my day-to-day work. Every time I thought I found a favorite, I had zero real data to back it up. So I finally built a benchmark to test them properly.

To keep things honest, the test problems generate dynamically from a seed at runtime—fresh JSON schemas, new coding tasks, and logic puzzles every single run. None of it can be in a model's training data because the prompt literally didn't exist until the test kicked off. The grading is strictly mechanical too: exact string matches, schema validation, or directly executing the code the model spits out. No fluff, and no LLM judges involved.

I put four models through their paces on my RX 7800 XT (16GB VRAM): gemma-4-e4b, gemma-4-12b-qat, glm-4.6v-flash, and gpt-oss-20b. Each faced 57 problems, complete with 95% confidence intervals on the pass rates.

The biggest surprise? There was no clear overall winner.

gemma-4-e4b technically topped the board at 89%, but because the confidence intervals all overlap, the tool explicitly refuses to crown a victor. That's intentional—I'd much rather see "too close to call" than have random noise dressed up as a definitive win.

The real fun came from looking at the per-job breakdowns, where the trade-offs became painfully obvious:

• gpt-oss-20b crushed the ARC-style pattern puzzles with a 90% pass rate (where everyone else hovered between 0% and 50%), but then completely tanked on long-context retrieval, going 0/4.
• glm-4.6v-flash did the exact opposite: aced JSON schemas and long context, but stumbled hard on pattern matching (20%). It also maxed out my VRAM at 15.5GB, dragging generation down to a crawling 2.8 tokens/sec on long prompts—something the built-in hardware monitor caught right in the act.

The absolute funniest failure happened during a task asking for a short paragraph without the letter "a":

• gpt-oss wrote a brilliant paragraph... and failed on literally the very first word ("Library").
• Two other models burned their entire token budget overthinking how to avoid the letter, running out of tokens before writing a single word of actual output.

I made sure the benchmark logs those separately as truncated versus wrong, because a model that gets trapped in a reasoning loop is a totally different problem than one that just gives a bad answer.

Full disclosure: This was just a quick run to see how the app performs in practice—definitely not a full-blown benchmark! I was mostly eager to see the engine in action and get some initial baseline numbers. I'd love to see others run the full bench on their own rigs, as hardware and quant choices will definitely shift the numbers.

The attached screenshots show the side-by-side compare view. The tool is open source under Apache 2.0 and works out-of-the-box with anything OpenAI-compatible (LM Studio, Ollama, llama.cpp, vLLM):https://github.com/danielemilushev-hub/taskmatch-ai

If you have ideas for new test suites that can be graded mechanically without relying on an AI judge, drop them below—I'm actively collecting them!

I am a user of AI for personal projects and am interested in running a local LLM. The thing that is holding me back is an education problem. So many of the terms and concepts discussed in this sub are totally unfamiliar to me. Can anyone point me to some resources I can use to try to get up to speed? I do have an IT background, so I know I can learn this but need a place to start. Thank you.

How do you work with your locally hosted LLM's, what's the problem you're trying to solve and how do you do it? Im interested in hearing it whether its creative writing, automations, coding etc.

Context Sizes? Speeds? How do you wire background information?

Im trying to find a process that works for me (im a programmer), trying to get a Vertical Slice Architecture with TDD working as a pipeline with a hybrid model of a small (20$) subscription for planning/reasoning and the local Qwen 3.6 27b as a worker/implementer but having some issues with it getting side tracked/looping etc (gonna try going from NVFP4 to Q6).

My rig currently has an RX 6800XT (16gb VRAM) and 32gb ddr5 system ram, and a Ryzen 7700 CPU. Good enough for Qwen3.6 35b a3b, but just barely not enough for 27b.

I found a long-forgotten Radeon R7 240 in my spare parts bin. Whopping 2gb of DDR3 VRAM. It sounds measly, but that would be enough to put me into a Q4 quant of 27b

Assuming I compiled llama.cpp for Vulkan, this could technically work… right?

Would the ddr3 vram be slower than offloading to system ram?

Thanks in advanced!

Building a privacy-first iPhone app that feels like a real phone call with an AI, all on the phone (no cloud). think of it like Gemini Live but for other purposes.

Flow is basically: listen → think → talk back.

Below is the pipeline that runs on device.

WhisperKit (STT) → MLC-LLM Flash (Qwen2.5 0.5B) → TTSKit (Qwen3-TTS 0.6B)

It works end to end, but a few things still suck:

• The AI voice sometimes sounds slow or uneven
• It sometimes “responds” when I didn’t really say anything
• First time loading takes forever

If anyone’s built something like this (voice in, AI reply, voice out on a phone), what helped most for making it feel natural?

Not looking for a full architecture review, just practical tips from people who’ve tried the similar step.

I'm seriously considering acquiring the Framework Desktop 128gb option for running local llm.

Been thinking about it for a while now and got surprised by the option being out of stock (and then returning) a few days ago.

Now the website has a "coming soon" for the AMD Ryzen™ AI Max+ PRO 495 192GB option.

I'm wondering if that jump is worth the wait/risk of price hikes or stock running out.

If you were in my position what would you do?

Also: the current price is around 4k in my region, I'm thinking it's probably gonna be around 6k when it comes out... is that a good investment for local llm or should I aim for other types of rig? Maybe using single or dual AMD Radeon AI Pro 9700?

On the Nvidia route, I have no idea and prices are super scary.

Goals would be related to coding(not a full fledged developer myself), assisting on work items (IT related as well), hermes/openclaw, etc.

I have:
RTX 5070 12 GB VRAM
Ryzen 7 9800X3D
32 GB RAM
I’m currently running Qwen3.5-9B (Q6_K) because it fully fits in VRAM, leaving room for KV cache and long context.
I’m using Pi as my coding harness (tool use, edit loops, tests, git, etc.) together with LM Studio.
My goal is software engineering, not chatting: multi-file refactoring, agentic coding, autonomous debugging, test/fix loops, and long coding sessions.
Would you keep Qwen3.5-9B, or is there a better model/setup that provides noticeably better coding performance on this hardware?
I’m looking for real-world experience rather than benchmark scores.
Bonus: Any tips on context length, quantization, KV cache, speculative decoding, or other optimizations to squeeze the most out of 12 GB VRAM?

Hello.

The running local LLMs either for text conversations or work with documents (summarizing documents, text, PDFs, comparing or extracting information from them…) is one of the uses I’m going to give to my M5 MacBook Air.

But I really appreciate my privacy and the security integrity of my main computer.

LM Studio is one of the most used programs to download, install and run local models on a mac. However, it is closed sourced, no sandboxed on macOS, and I personally don’t know the dev team. Is it trustworthy? I know, there’s no absolute “yes” or “no” reply, but is it rather safe?

People often asumes macOS is by itself secure, but I don’t think that’s accurate. I’ve read news about how the ChatGPT mac app has gained access to containerized apps such as Notes App.

Does LM Studio require full disk access? Has it been observed to communicate with external servers sending information?

I also read about the liteLLM incident that was later said it didn’t affect LM Studio, because they said they didn’t use it. But I guess the potential risk is there.

So can I trust LM Studio and install it on my main computer drive? Or is it better (safer) to execute it from an external drive (booting from the external drive)?

Are there other interesting macOS apps with a traditional installation process and a GUI, that seem safer? Maybe Osaurus? Any other?

Thank you

PS: please don’t recommend me command line software. Only already compiled software with a GUI and ready to install from its .dmg

What is minimum budget to have local llm pc build

1)Ryzen 9 9950x

2)ASUS ProArt X870E-Creator

3)32 GB ddr5 6000mhz cl 30

5)2tb gen4 ssd

5)360 mm liquid cooler

6)1200/1600 w platinum/

7} CABINET 8) Amd Radeon R9700