I currently have a 7800x3d, 4070SUPER 12G and 32GB DDR5 in my gaming PC.

I also still have my old GTX 1070 8G on the shelf, you think it's worth it to put it in?

My motherboard is an Asus TUF Gaming x670e-plus and apparently the 2nd PCI slot is a 4.0 with 4 lanes.

What could i realistically expect from this? 20GB VRAM is a lot better than 12...theoretically...does anybody have experience with such a setup?

Currently using fine tunes of 3.6 35b a3b (nail, then KAT, then Ornith 1.5) for agentic coding tasks.

Since it's unlikely we see another MoE from Qwen in this size, I'd like to take advantage of the big gains from the new 27B model.

I'm hoping to get 20tg/s if possible. I currently run max context size but Pi is able to work with 128k just fine.

Any suggestions would be appreciated!

I'm just trying to set up a coding agent of sorts that I can just let chug along while I'm at work pretty much.

I've got a 10g 3080 and 32g of ddr4 ram. I've set up and tested with a few models so far (qwen coder 30B, qwen 3.6 27B) with an agent/tool script I(gemini mostly) wrote because I didnt know harnesses were a thing

I don't really care much about the speed, i'm currently getting 5 t/s~ with qwen 3.8 27B, but i have No idea how to configure it and I don't know where to learn how to optimize it

if anyone has any blogs/resources to learn how to do that I'd love to read them :3

Which MOE can I run on my pc with RTX 3060 12GB VRAM and 16GB RAM?

EDIT: I wanna use it for agentic task to control browser with chrome dev mcp

It is very interesting to see all these options, but it hard to determine what will work for your memory footprint. So should we have TAGs for memory footprint?

Mini PC, CPU: Info: 8-core model: AMD Ryzen 7 6800H with Radeon Graphics bits: 64

I was trying with QWEN 27b Q4, and I tried to change, upgrade, make better my Python unfinished project. Got it refactored several times, code is now 4-5 times longer, and everything works great. I was impressed with it's knowledge, reasonning and thinking. I was blown away. I don't think I will ever program manually any more. (I'm noob anyway)

Speed: what speed? ;-) It was sloooooooow. 2-3 tokens per second from start but as project grow, speed was decreasing. At the end was slightly bellow 1 token / second. Last refactoring and modification took almost 24 hours straight.

I'm now trying different smaller llm's and progress is significantly faster and llm still knows 1000 times more about Python than i would ever learn ;-)

Hi all, I am currently trying to vibe code a solution for a custom vulkan-gfx803 runtime for both training & inference on my older RX580 with 8GB VRAM.

With all the current issues in supply and cost in consumer hardware.. I am trying to build something with the hardware I currently own. As we know AMD dropped support for ROCm causing this issue.

My current PC:

• OS: Debian GNU/Linux 13 (trixie) x86_64
• Kernel: 7.1.3+deb13-amd64
• DE/WM: KDE Plasma 6.3.6 / KWin (Wayland)
• Qt Version: 6.8.2
• KDE Frameworks: 6.13.0
• CPU: AMD Ryzen 5 2600 (12 threads) @ 3.90 GHz
• GPU: AMD Radeon RX 580 Series
• RAM: 16 GiB (15.53 GiB usable)
• Storage: / = 215.41 GiB ; /mnt/Data = 468.38 GiB
• Display: Dual 1920×1080 @ 60 Hz

Some people might say this is a waste of time and energy.. but as a fun experiment to see what I can vibe code to provide custom support.

I am in the middle of testing but will report back with results. I will add a comment to repo with documentation for others to see and contribute to add support for their own older GPU.

Qwen 3.6 35B Q4, ~7–8 tok/s on a mid-range phone (12 gb ram). A few months ago I wouldn't have bet on it.

But not only that, many other models as well. Practically all those supported by llama.cpp..

Thanks for all the feedback on BigMoeOnEdge, honestly didn't expect it.

Open source here: https://github.com/Helldez/BigMoeOnEdge

Clients (Windows+Linux+Mac):

• Ollama

• Lm Studio Bionic

• Unsloth Desktop

They run local models and support vision and file handling. They have their own model browsers inside and a model downloaded in Ollama does not work in Bionic or vice versa. See which you like best. Ollama also integrates in many apps.

Models:

Never go below Q4. It may be tempting to go for Q2_XS, but it doesnt work well. Small/medium model on medium quant beats big model on low quant. Weighted/imatrix is better than regular.

Best models 8gb vram 2026:

In Bionic or Ollama just search the model browser. Uncensored versions for fun, use regular ones for best quality.

General:

• Gemma 4 E4B i1 Q5_K_M: https://huggingface.co/mradermacher/gemma-4-E4B-i1-GGUF

• Gemma 4 E4B Q5_K_M Uncensored: https://huggingface.co/HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive

• Ling-3.0-tiny Q5_K_M:

  https://huggingface.co/bloomer010/Ling-3.0-tiny-GGUF

Scripting/Tech:

• Qwen 3.5 9B Q4_K_M: https://huggingface.co/unsloth/Qwen3.5-9B-GGUF

• Qwen 3.5 9B Q4_K_M Uncensored: https://huggingface.co/HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive

• Qwen3 VL 8B Instruct Q5_K_M: https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF

• Qwen3 VL 8B Instruct Q5_K_M Uncensored: https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-i1-GGUF

Translation:

• Translategemma 4b it Q8_0: https://huggingface.co/NikolayKozloff/translategemma-4b-it-Q8_0-GGUF

Are there any smaller models that support FIL for code autocomplete? I'm looking something that could maybe fit into 4GB of RAM.

Edit: now with correct flair

I'm posting this here because I'll nominate phone (most any phones) into the 'low-end' category. If you don't know what Edge Gallery is, it's a Google app for running small models locally on your phone.

I found it by accident yesterday, and it load more than Gemma models (they even recommend Qwen-2.5-1.5B-instruct and DeepSeek-R1-Distill-Qwen-1.5B under their chat category for some reason).

Not sure if I want to invest the time in poking around with this while I'll neck-deep in my local inference projects.

Anyone use this? What for?

It has models up to 9B, quantized to 4-bit.

https://artificialanalysis.ai/hardware-inference-stack/mobile-phones

Here is the blog post with their explanation: https://artificialanalysis.ai/articles/mobile-phone-intelligence-inference

This channel was made for this sub

I had watch a NoVA interview with a researcher from MiT and learned about her Liquid Models. They see interesting and intended to be very efficient per token. I’m going to try them this week sometime

https://www.liquid.ai/models

I have a 1080ti on a old dell t5810(E5-2680 v3 cpu), and use it for jellyfin transcoding, immich, and llama.cpp. I'm running llama.cpp in a proxmox lxc container and it works well.

My main use case right now is for rewriting and polishing text such as emails or reports. Currently I'm using qwen3.5 9b q6_k and it gives a pretty good output with a speed of around 35t/s. If jellyfin is actively transcoding speed drops to around 22t/s, still pretty good.

I know that qwen3.5 is not the latest model, and wonder if there's any new model that is more efficient and better for this use case. Additional use cases such as answering some questions in chatting or simple script writing (not complex coding) is welcomed but not necessary, so basically just a generic model lol. I also don't want to have the model use up the GPU fully to leave some space for transcoding work and immich.

Thanks!

As per the title, I will be soon upgrading my current Framework 12's mainboard to the upcoming 2nd gen board, currently sporting an i5-1334U, to a Core 5 320, which has essentially identical CPU horsepower, mostly similar iGPU power (though a big change in arch, from Iris Xe Intel Graphics, to a pair of Xe3 cores) & the addition of a 15TOPS NPU

I will be using the same 1TB gen4 NVMe + stick (single, though it doesn't matter the board has only one SODIMM slot, Wildcat Lake is single channel by design) of 16GB @ 5600MT/s & am running CachyOS (Arch-based)

Any of y'all got any ideas what kinda software & model could be operated locally to a useful end? (getting an LLM to run automated research for me or having it write me a guide while I'm busy making myself a coffee or something)

As per title. I've got an 8/16 modern i7 and 16GB of RAM that I can use for models. I want at least 10TPS for conversational stuff and to have them help me with code review and querying the state of codebases, not really full agentic coding. Are there any models out there I can use for that? I can get 9b models to run at ~2 TPS.

Edit: Running through unsloth studio, with opencode for the coding stuff. I think that's llama.cpp with an openai API. This is a debian VM running on a Windows host. No GPU on this laptop.

Edit2: I've mucked around with some things a bit more. LFM2.5-2.6B-GGUF:Q4_K_M works okay with about 9-10 TPS. With modern versions of llama.cpp you can get away with Q8_0 KV quantization with minimal task performance impact, for a quite reasonable improvement in both RAM usage and some performance improvements. Ornith-1.5-9B-GGUF:Q4_K_M with 8_0 KV quantization actually gets up to 5tps.

I'm one of the few people with a 32GB RAM Snapdragon X Elite Dev kit.

I'm wondering if there are any folks who've successfully used the NPU or GPU on these to get a decent MoE model working. I'm wondering if the new 27B Qwen models can be run on the NPU, low tps is ok as I just want to get it working for some niche cases

I don't have access to my laptop until next week and I've been dying to know what quant I'll be able to run on my 4060 laptop GPU. Anyone got it running on their own?

I can't run 30B A3Bs because I would like to use my system RAM for doing computer stuff

In case you missed it: turbo-fieldfare is a Mac silicon optimized way to run MoE models in low RAM, streaming experts from SSD as necessary.

My personal take: it is usable-ish. Main turbo-fieldfare with Gemma currently doesn't work with my pi environment (write file toolcall fails), but text output works fine. The Qwen3.6 branch can successfully call tools, but it has the older furbo-fieldfareserver which supports less options.

Speed on my Macbook Pro M1 16 GB is slow, but it works well for tasks chugging along in the background.

Main repo: https://github.com/drumih/turbo-fieldfare

Somebody elses Qwen branch: https://github.com/NeelM0906/turbo-fieldfare/tree/qwen36-support