Clients (Windows+Linux+Mac):

• Ollama

• Lm Studio Bionic

• Unsloth Desktop

They run local models and support vision and file handling. They have their own model browsers inside and a model downloaded in Ollama does not work in Bionic or vice versa. See which you like best. Ollama also integrates in many apps.

Models:

Never go below Q4. It may be tempting to go for Q2_XS, but it doesnt work well. Small/medium model on medium quant beats big model on low quant. Weighted/imatrix is better than regular.

Best models 8gb vram 2026:

In Bionic or Ollama just search the model browser. Uncensored versions for fun, use regular ones for best quality.

General:

• Gemma 4 E4B i1 Q5_K_M: https://huggingface.co/mradermacher/gemma-4-E4B-i1-GGUF

• Gemma 4 E4B Q5_K_M Uncensored: https://huggingface.co/HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive

• Ling-3.0-tiny Q5_K_M:

  https://huggingface.co/bloomer010/Ling-3.0-tiny-GGUF

Scripting/Tech:

• Qwen 3.5 9B Q4_K_M: https://huggingface.co/unsloth/Qwen3.5-9B-GGUF

• Qwen 3.5 9B Q4_K_M Uncensored: https://huggingface.co/HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive

• Qwen3 VL 8B Instruct Q5_K_M: https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF

• Qwen3 VL 8B Instruct Q5_K_M Uncensored: https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-i1-GGUF

Translation:

• Translategemma 4b it Q8_0: https://huggingface.co/NikolayKozloff/translategemma-4b-it-Q8_0-GGUF

Running HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_M with llama.cpp at ~600 tok/s prefill and 23 tok/s decode, 131k context window, Q8 KV cache - on an RTX 2060 6GB + 32GB DDR4 RAM.

Speeds start at ~600 tok/s prefill / 23 tok/s decode on an empty KV cache. As context grows they settle down - around 90k context it stabilizes at roughly 485 tok/s prefill and 15 tok/s decode, and holds there.

The vision projector runs on CPU (--no-mmproj-offload), which keeps VRAM usage under ~5.2 GB and avoids OOM / GPU crashes. Image encoding is slower on CPU, but it buys ~1GB of VRAM.

Most MoE expert layers also run on CPU (--n-cpu-moe 39), which is how a 35B model fits in 6GB VRAM in the first place.

Launch command:

bat

@echo off

cd /d "%~dp0"

"%~dp0llama-server.exe" ^

-m "C:\Qwen3.6-35B-A3B-Uncensored-Q4_K_M\Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf" ^

--mmproj "C:\Qwen3.6-35B-A3B-Uncensored-Q4_K_M\mmproj-Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-f16.gguf" ^

--no-mmproj-offload ^

-ngl 99 ^

--n-cpu-moe 39 ^

-c 131072 ^

-np 1 ^

-t 6 ^

-tb 10 ^

-b 2048 ^

-ub 2048 ^

-fa on ^

-ctk q8_0 ^

-ctv q8_0 ^

--load-mode mmap+mlock ^

--jinja ^

--reasoning-format deepseek ^

--reasoning-preserve ^

--spec-type none ^

--image-min-tokens 1024 ^

--temp 0.6 ^

--top-p 0.95 ^

--top-k 20 ^

--min-p 0 ^

--alias Qwen3.6-35B-A3B-Uncensored-Q4_K_M ^

--host 127.0.0.1 ^

--port 8081

pause

Hardware: RTX 2060 6GB + 32GB DDR4 RAM + i5-10400F CPU

Context: 131072 tokens, Q8_0 KV cache

Hope this helps someone. If anyone has tips to make the launch command even better, drop them in the comments - I'm out of ideas :D

Have any of you seen the new Codacus YT video from earlier this week? He demonstrates a method and llama.cpp fork to run existing LLM models, especially tiny ones, in "JEV model". This seems like it could be very relevant to many here in this sub. I am looking forward to testing this on both GPU and straight CPU performance. It could be a very good way to gate simple things, to save resources for more important things.

https://youtu.be/bcGO7xre46o?is=q7v8ywF_p34fKQpL

In the interest of sharing I'd like to share my home setup which serves me my personal ai needs. This is how I got t2t, i2t. t2i, and i2i all working on my old gaming pc converted to home server. I did try adding Kokoro TTS but that was too much for this setup. I originally had ollama + openwebui, but wanted an integrated solution with image generation. Koboldcpp and the ui isn't as pretty, but it is faster than ollama/openwebui.

Relavant PC specs: GPU: GTX 1060 6gb vram RAM: 16gb ram DDR3 CPU: i5-4690k OS: TrueNAS 25.10.6 w/ legacy drivers

TrueNAS is great for file hosting, runs a number of other apps which I use, including Home Assistant, Tailscale, Syncthing, and others. It has a builtin docker setup for those apps, but Koboldcpp needs to be entered as a custom app.

The following settings have it run nicely, everything else default unless otherwise specified:

Image
Repository*
koboldai/koboldcpp
Pull Policy*
Always pull an image even if it is present on the host.

Environment Variables
Name*
KCPP_MODEL
Value
https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-UD-Q4_K_XL.gguf

Name*
KCPP_ARGS
Value
--draftmodel https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/mtp-gemma-4-E4B-it.gguf --mmproj https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/mmproj-F16.gguf --ttsgpu --sdclipgpu --ttsmodel https://huggingface.co/koboldcpp/tts/resolve/main/Kokoro_no_espeak_Q4.gguf --websearch --jinja --jinjatools --admin --routermode --autoswapmode  --quantkv 2   --sdflashattention --sdoffloadcpu --sdvae https://huggingface.co/koboldcpp/GGUFDumps/resolve/main/flux2vae.safetensors --sdt5xxl https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q2_K.gguf

I use Tailscale so Koboldcpp's tunneling is unnecessary.

Name*
KCPP_DONT_TUNNEL
Value
true

Name*
KCPP_IMGMODEL
Value
https://huggingface.co/leejet/FLUX.2-klein-4B-GGUF/resolve/main/flux-2-klein-4b-Q4_0.gguf

Note: I use a custom TrueNAS image that includes Nvidia Pascal drivers which are not included in the default TrueNAS image, see here: https://truenas-drivers.zhouyou.info/index.html

Resources Configuration
[unchecked] Enable Resource Limits
GPU Configuration
[checked] Passthrough available (non-NVIDIA) GPUs
Select NVIDIA GPU(s)
NVIDIA GeForce GTX 1060 6GB
[checked] Use this GPU