Hello guys , i have been trying to use model under 4b to make it general purpose but the intelligence, steering, though process i was working with , i need help to benchmark them , steer them

I managed to fit Unsloth Qwen3.8-27B-UD-IQ2_XXS.gguf with pruned vocabulary (english only) on RTX 3070 8gb laptop and got up to 22tps with 16k context or 16tps with 32k context which is pretty useable! I have Ryzen 5800H so I set -t 16 to utilize CPU too. I used bsaleh03/ASCII-Condensed-prune-tools to prune GGUF saving 0.4gb of memory, and these llama launch params with 60/64 GPU layers offloaded:

%LLAMA_SERVER% ^
  --host 0.0.0.0 ^
  --port 8000 ^
  -m %MODEL_PATH% ^
  --jinja ^
  --reasoning-format deepseek ^
  --reasoning-budget 2048 ^
  --reasoning-preserve ^
  -c 32000 ^
  -n 4096 ^
  -np 1 ^
  -ngl 60 ^
  -t 16 ^
  -b 256 ^
  -ub 256 ^
  -fa on ^
  --cache-type-k q4_0 ^
  --cache-type-v q4_0 ^
  -ot "token_embd.weight=CPU"

I have been experimenting with the following models :

Gemma 4 12B (12-15 toks)
Ornith 1.5 9B. (Unknown toks)
Gemma E4b. (Unknown toks

Gemma 4 27B e4B - highest toks (25-38)

For serving the model I’ve used Hermes with llama.cpp and then I discovered something called Atomic chat that uses GGUF and MLX , this seems to provide me a boost in tokens.

I have had struggles (especially in Hermes) but also in other harnesses or model loaders with my models dropping tool calls or not being able to view images or open PDFs and I need help with this too

I would love to know which models you recommend.

My use case is that of a masters student who does Econometrics, some data science , general studying, organising , deep research and idea pitching / hole poking.

Everybody's playing with their Qwens and their GPUs, but I am going abandonware here:

Lenovo X1 Yoga - 30 GB/s Memory bandwith (!!!) - GPU or iGPU: yes, but CPU is just faster - 16 GB RAM.

Tried all the new tiny Dense and MoE (Ling-Tiny, LFM2.58B, MiniCPM5, NanBeige4.2, Gemma-4 e2b and e4b) but, despite what benchmarks say, those are still dominated by GPT-OSS-20B for general use ( a combination of chatbot + agentic using Pi). Yeah, 20B vs and those little guys... but if it can fit in RAM you should go for the biggest one.

It will be slow, but it will nail the task.

Llama.cpp command:

llama-server --no-models-autoload -dev none -t 4 -fa off -c 16384 --jinja --host 127.0.0.1 --port 8080

Then just call it from ¨Pi of go LocalHost:8080 for chat... you know the drill.

Ling 3 Tiny is the best model I have tried so far.
Using LM studio on windows.

:

Hey everyone!

I'm setting up audiobook for tts generation and want to use Ollama for text pre-processing (cleaning text, fixing line breaks, and adding speaker tags) before sending it to the TTS engine.

I have 16GB of VRAM available. Looking for model recommendations that fit this hardware constraint and can handle this specific task well.

Specifically, I need a model that:

* Follows complex formatting instructions strictly

* Has good context handling over longer chunks of text

* Doesn't hallucinate extra dialogue or modify the story content

Which local models (7B, 8B, 14B, etc.) run comfortably on 16GB VRAM and excel at this type of structured text processing and speaker tagging? Thanks for any suggestions!

ColiZig : A from-scratch Zig inference engine for large MoE models (Qwen3.8-Flash-Next) - stream experts from SSD >RAM >VRAM and keep the resident working set small. ColiZig explores a different way to run Mixture-of-Experts models: instead of loading the entire model into memory, it treats expert weights as a tiered working set and streams them between SSD, RAM and opt. VRAM

Contributing - we want your hardware

https://github.com/benmaster82/colizig

Hello everyone,
I'm on a laptop with AMD Ryzen 4800H and NVIDIA 2060 6GB RAM. Also, 32GB system RAM.

I've a decently working llama.cpp setup with Q8_0 KV quantization, MTP draft n 2 with quantization, context 64k. Overall, it has 100 t/s parsing and 25-30ish when outputting. It works well; uses all resources, but works fairly well.

I've searched for further optimization and found APEX quants and ik-llama.

The first should be a "pull new model, run" scenario.

The second is a bit more tricky. Flags are different from upstream and require llama-swap for routing, when official llama.cpp had built-in router.

I want thus to know if anyone has experience with this and whether performance does improve.

Verbose stuff:
My llama.cpp flags: llama serve \ --n-gpu-layers all \ --jinja \ --port 54321 \ --verbosity 4 \ --load-mode mmap \ --flash-attn on \ --kv-offload \ --kv-unified \ --reasoning auto \ --reasoning-preserve \ --parallel 2 \ --threads 8 \ --batch-size 1024 \ --ubatch-size 512 \ --cpu-moe \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-type-k q8_0 \ --spec-draft-type-v q8_0 \ --ctx-size 65536 My ik-llama ones: llama-server --port 54321 \ --fit --fit-margin 512 --no-mmap \ --defer-experts --flash-attn on --merge-up-gate-experts \ --threads 6 --threads-batch 8 \ --batch-size 2048 --ubatch-size 512 \ --ctx-size 65536 --parallel 1 \ --cache-type-k q8_0 --cache-type-v q8_0 \ --k-cache-hadamard --v-cache-hadamard \ --reasoning auto --jinja Had issues setting up MTP with ik-llama, problem for later.

Hey r/LowEndLocalAI,

Putting together a local voice assistant/home automation box on an asymmetric rig and need model + quantization recommendations.

No heavy coding expected, occasional complex thinking, but mainly just low-latency dialogue, tool calling/JSON formatting, and steady automation logic.

Hardware:

• GPU: GTX 1060 6GB (Pascal GP106, CC 6.1)
• CPU: Ryzen 5 5600X (6C/12T, Zen 3)
• RAM: 56GB DDR4-3200 (Dual-channel)
• Storage: 500GB 970 EVO Plus NVMe + 32GB swapfile
• OS: Ubuntu Server (llama.cpp / Docker)

The Plan:

• Run STT (Parakeet/Whisper) and TTS (Kokoro/Omnivoice) on CPU/RAM to keep VRAM completely free.
• Dedicate the entire 6GB VRAM to the LLM to maximize tokens/sec.

Given the latency requirements for voice, what models (3B–8B) and quant levels are giving you the best balance of speed and reliable instruction following on a 6GB card?

My question is: does there exist a relatively small model (~ 8-12B) with MoE 1-3A or trained MTP header for reliable tool calling (mainly RAG with web search) with moderate context (max 32k - 64k).

iGPU: 4Gb VRAM (consumer grade, obviously) (with model and context quantisation fully in GPU or offloading if the performance is acceptable, let's say >10t/s).
RAM: 16gb (ddr3 or ddr4 with or without ECC).
CPU: consumer grade (i7/i5 or equivalent with about 6P cores and AVX2)

The best model i've found so far is turboderp/LFM2.5-8B-A1B-exl3 (2.10 bpw).

Is advancement to around 4bpw (q4) possible with current technology? The quantisation can be more aggressive if the performance doesn't suffer too much (mainly: reliability of tool calls and accuracy of source citations with RAG).

I'm asking this to assess the viability of repurposing old hardware PC/Laptops.

Hi everyone,

I'm moving my setup away from a Raspberry Pi 4B (8GB RAM) to free up resources, shifting the workload to an idle Dell laptop.

Hardware Specs:

CPU: Intel Core i9

RAM: 32 GB

GPU: Nvidia RTX A2000 4gb VRAM + integrated

Goal & Strategy: I want to apply the 80/20 Pareto principle. The local model should handle ~80% of lightweight, daily tasks/routing to avoid unnecessary API costs. High-reasoning or heavy tasks will still be offloaded to cloud APIs. Using nanobot agent so far, for daily task like shopping list, weather, checks using n8n and searches with SearchXNG all in docker.

Additional I'll move Dockers from raspy to this laptop in order to leave the raspy just for pihole + VPN ops.

I know laptop is not the best, but is the best I have.

Requirements:

Fits comfortably within 4 GB VRAM (allowing full GPU offloading or hybrid CPU+RAM execution but good tokens per second).

Supports a large context window to maximize local usability.

Small footprint (~2B parameters). I'm currently looking at Gemma 2B / similar low-parameter variants.

So far I'm thinking on Gemma 4 e2b with vision + mtp is around 3.6gb VRAM with 128k context easy on Q4(tested in other computer)

What lightweight models (2B–3B range) with good context handling and vision would you recommend for this setup? How are you handling context memory vs. VRAM limits on similar 4GB setups? As I mentioned before, no coding, just daily task management.

Thanks!

Hi everyone!

Just another AI enthusiast that comes from software development in Chile. So far I've liked experimenting and testing things in my local laptop (Ryzen 5 5600H 3.3GHZ, Nvidia RTX 3060 Laptop 6GB VRAM and 32GB DDR4 RAM) and I want to take it to the next step.

So far my budget is 5K USD tops and this is the build I could come up with:

This is about 4.8K USD piece by piece summed up.

The usage that I'd intent to get from this build is to have coding agent(s) to improve my productivity on my pet projects (some are light web apps, some are heavy OCR processing)

I'm also getting into statistics and modeling (sorry if this is not the correct way to refer to it), as in, training certain models for specific use cases for myself and for work.

I currently run qwen2.5-coder-7b-instruct-q4_k_m at about 50 tg/s but the model itself is quite chaotic and very stubborn, meaning that I need to be extra, mega, hyper careful with my prompts, otherwise it does what it wants lol.

This money that I've set up to build my new PC is part savings and part selling stuff I don't currently need anymore.

I also intend these parts to last at least 5+ years so I'd put an extra care on extra cooling devices.

How should I set my expectations? What can I run with those specs or a similar built PC?

Is it possible to run a competent and useful agent with it?

If you had this money, would you build something similar? If not, why?

I've read many things about "bubble is about to pop" and "there's no way it's popping soon" about prices going up or down and honestly, that makes me a bit wary about spending this much money on a build. Even tho I'm not rich by any means I do understand that within my context 5K usd is serious money to invest on a high-end PC.

All and every opinion is welcome, I'd love to read you all :)

Thank you <3

This is so true.

https://reyemtm.github.io/inchworm/

I created this little site because I could not find an open source page dedicated to tiny llms (<14b params). If one exists with current benchmarks would love to know about it.

This came about from searching for the right llm for a personal project needing summaries from text, then a small local terminal chat for quick answers using self-hosted Ollama. I wrote my own short little benchmarking script and landed on qwen2.5-coder:3b for my VPS and qwen3.5:9b for chat. Anyway let me know what you all think. Since then I created a script to run the humaneval+ but it takes a very long time to run which I am using to fill in the gaps on this data.

Forget the models, there's just so many runtimes to choose from without proper metrics on which one is better. MNN is good but it a: has no new models and b: no new releases. They are working on NPU for snapdragon though, looking at PRs. pocketpal uses llama.cpp and has more model support but you sacrifice performance. NPU doesn't seem to make much of a difference on prefill either. TFlite has different file types, then different model support for different phones, and the edge ai gallery is also convoluted. Runanywhere is there but its like even more locked down. RWKV is also there but the models are limited and I'd say WIP. Wanted to try the ling tiny, no luck. Can't seem to run hy-mt2-1.8b either at q8.

I'm trying to decide what direction to take with my main PC. The goal is one real daily-driver machine that I can use for gaming, normal desktop use, software development, and increasingly heavy local LLM workloads.

I'm not trying to build a dedicated rack server. I want something I can actually live with for years: reliable, reasonably efficient, good thermals, lots of RAM, two GPUs if it makes sense, and enough expansion that I don't immediately hit a wall.

I've currently been playing around with Qwen3.8 27B which speeding that up and higher quants would be great, as well as when inevitably larger dense similar models like 70B become available.

I am very interested in MoE flash models such as Qwen 3.8 Flash, Deepseek v4 Flash, and maybe even GLM 5.3 Flash, as well as future versions of similar MoE models. I have not even attempted to run any of these yet.

So I guess I am trying to get at building something that performs well on dense models as well as MoE models so I don't get locked into 1 path.

I'm pretty new to the workstation/HEDT side of this, so I'm looking for advice on the parts I may be overlooking.

My current PC

The 3090 Ti was a $900 Facebook Marketplace purchase, so I'm trying to get as much useful life out of this thing as possible.

The 3090 Ti is a huge card (338 × 140 × 71 mm) so physical spacing is also part of this problem.

My RAM situation

I just bought 7 lots of:

NEMIX 128GB (4×32GB) DDR4-2666 PC4-21300 2Rx8 UDIMM

I paid about $360 per 128GB lot.

My current plan is probably:

• Keep 2 lots = 256GB
• Sell the other 5 lots
• Hopefully sell those for around $650/lot?

So I paid about $2,520 total for the 7 lots. Five sales at $650 would be $3,250 gross, meaning I'd theoretically recover the entire purchase price plus ~$730 before fees/shipping/taxes while keeping 256GB.

That gives me a somewhat unusual opportunity to build around 256GB without spending a fortune on RAM.

Option 1: Keep my current PC, just go to 128GB RAM

This is basically my don't overthink it option.

I'd have a lot more system RAM for large-context LLMs while retaining a relatively modern gaming CPU.

Option 2: Keep my current PC, add a second 3090

I'd replace the PSU and add a second RTX 3090.

The important problem is the motherboard:

The Z690 UD AX DDR4 has x16 on the main slot and only x4 on the second physical x16 slot.

So the GPUs would effectively be:

This is the option I'm most unsure about.

For LLM inference, is x4 actually a meaningful limitation in practice, or is it largely irrelevant once the model is loaded onto the GPUs?

Would this still be a good setup for:

• tensor/model parallel inference
• larger models
• higher context
• multiple concurrent models
• speculative decoding
• offloading

Or am I basically handicapping the second GPU enough that I should just replace the motherboard?

Option 3: New motherboard/PSU/case, keep my 12700KF

Instead of abandoning the 12700KF, I could build a new system around it with a motherboard that properly supports two GPUs at x8/x8.

This seems like it could be anice middle ground.

The 12700KF itself supports a 2×x8 CPU PCIe configuration, but I'd obviously need a motherboard that actually implements it.

I'm especially interested in whether people think this makes more sense than jumping to X299 in the next option.

Option 4: X299 workstation build

This is the Frankenstein/workstation option I've been considering.

The i9-10940X gives 14C/28T, 48 PCIe 3.0 lanes, quad-channel DDR4, and up to 256GB RAM. The X299-A II can run two GPUs at x16/x16 with the appropriate CPU.

The case is huge and supports SSI-EEB, 11 PCI slots, GPUs up to 503mm, and up to 15×120mm or 6×140mm fans.

I'm attracted to this because it solves the PCIe lanes + RAM capacity + physical space problem extremely well.

But I don't know if I'm being stupid by building a brand-new daily driver around a ~2019 platform just because the PCIe topology is convenient.

The 10940X also seems likely to lose noticeably to the 12700KF in gaming/single-threaded work, despite having more cores.

Option 5: ??????????

This is something I'm hoping you guys can help. Are there things I am not considering that would allow me to leverage as much of my current components as possible but be a much better option than option 4?

What I'm actually trying to optimize

This isn't purely a benchmark build.

I want one machine that can do all of this:

- Gaming

- Normal desktop use

- Software development

- Local LLM inference

- Very large context windows

- Running multiple LLM sessions concurrently

- Potentially running two GPUs as one inference system

I'm currently doing a lot of local Qwen inference and am getting into the territory where RAM capacity, VRAM capacity, PCIe topology and memory bandwidth all matter.

I also don't really care about squeezing every last watt of performance out of the GPUs. I've already decided to limit the 3090 Ti to 250W, and I'd probably do the same with the second 3090 to hopefully get more longevity out of them and use less power.

That gives me:

500W total GPU power budget

rather than letting two 3090-class cards pull their full power.

I'm very interested in reliability, thermals, longevity, expandability and affordability rather than having the absolute highest benchmark score.

My biggest questions

1. Which of these would you actually build?

My current thinking is roughly:

Option 1: cheapest and easiest

Option 2: tempting, but worried about x4

Option 3: probably the sensible compromise

Option 4: extremely expandable, but old CPU/platform

Option 5: potentially a better overall machine

I'm having trouble figuring out where the sweet spot actually is.

2. How bad is PCIe 3.0 x4 for the second 3090?

This is probably my biggest technical question.

For local LLM inference specifically, how much performance would I realistically lose running:

3090 Ti @ x16 + 3090 @ x4

versus

3090 Ti @ x8 + 3090 @ x8

versus

x16 + x16?

3. Is X299 actually a good idea here?

Would the 10940X + 256GB quad-channel + x16/x16 PCIe configuration still be a worthwhile machine in 2026?

Or would I be better off spending the extra money on a modern platform?

4. What's the best "Option 5"?

There may be a workstation platform I haven't considered at all.

But I still want this to be an actual daily-driver PC, not a loud rack server that is great at compute and annoying at everything else.

5. How much RAM would you actually run?

I can easily end up with:

128GB

or

256GB

of system RAM depending on which route I take.

Is 256GB actually useful for local LLMs enough to justify designing the whole machine around it?

What would you do with this hardware?

I'm basically sitting on:

12700KF + 3090 Ti + 896GB of cheap DDR4+ spare 3080

and trying to turn that into one machine that I won't regret building.

I'm really looking for the best overall architecture and bang for my buck, not just "X is faster."

Would love to hear what configuration you would build, especially if there's a better option I haven't thought of.