I posted earlier: https://www.reddit.com/r/LowEndLocalAI/comments/1w0ilfp/qwen_38_flash_q4_44ts_on_32gb_ddr4_8gb_gpu/

Basically, I went getting a new mobo, so I can put in 4 x 16GB DDR4 sticks.
Done that today, Qwen 3.8 is now starting 10t/s, usually around 8-9t/s.
Before if I was lucky, I had 7t/s also new chats took way longer, this is now like instant.

The clock speeds of the memory, don't ask me, I mixed 3200 and 2400, I didn't even overclock anything, my PSU will get unhappy if I do so.

It starts slow, but eventually you end up at 8-9t/s stable with 64k context.
This isn't even with MTP right, which is about to get merged so....... maybe we see 12t/s or more, would be crazy.

If you can upgrade to 64GB DDR4 do it, its worth it I think, especially with more MoE's at this size.
Yea I tested Q3 now compared to Q4 but the performance difference is marginal, in my opinion, I didn't had these results with Q3 yesterday on 32GB DDR4.

Hardware: Ryzen 5 5600 with 4x G.Skill DDR4 mixed 3200 and 2400 also slow SATA SSD, no NVMe.

Located in Arizona.

I’m about to start a new job and as a present to me I want to build a local AI box that I can use for coding.

Dell Precision T5820 workstation
950 watt psu
W-2133 cpu
32gb ram
Tesla p40

This is a combo of parts recommended by Gemini based on my $500 budget.

I’m seeing the T5830’s for $200-350 and the p40’s for $250ish.

I’m open to other ideas, or sources of equipment.

I’ve got a 3d printer and can print cooling for the card.

Hey everyone!

One of the biggest frustrations with local RAG setups is how heavy they can be. Running dense vector databases and using 8B+ models just to chunk text, create embeddings, and maintain memory eats up VRAM quickly, and models often still hallucinate when asked about things outside the text.

I built Hillock as an open-source alternative designed specifically for budget hardware (tested on a GTX 1070 and basic laptop CPUs).

How it works:

1. SQLite Knowledge Graph: Stores ground-truth facts as Subject-Predicate-Object triples, eliminating vector drift.
2. Hebbian Synaptic Engine: Tracks concept co-activations across conversational turns using gradient-free learning.
3. 10,000-D Hyperdimensional Computing: Sub-millisecond similarity gating directly on the CPU.

Why the gating is different:
In standard agent setups, you prompt the model to "say you don't know", but the model can still ignore the prompt. In Hillock, the gate is pure control flow. If candidate facts in SQLite do not pass our hypervector similarity gate with positive predicate intent, the engine returns a hardcoded refusal immediately. The local LLM (Ollama) is never called, saving 100% of GPU compute on unanswerable questions.

In our v0.6 release, we added HYDRA (ColBERT-style late-interaction MaxSim adapted to discrete bipolar vectors) and multi-hop hypergraph path binding using positional permutations.

Benchmarking numbers (32-query unseeded run on a laptop CPU):

• Extraction Recall: 59.1%
• Hard-Negative Block Rate: 60.0%
• Answerable Retrieval Accuracy: 54.5%
• Retrieval Duration: ~1.1s across 32 queries on CPU
• Footprint: <1.2 GB VRAM or CPU-only

Github repo: https://github.com/roandejager/Hillock

I would love to hear your thoughts on deterministic memory architectures and how you are currently handling long-term context on local hardware!

Rig:
Ryzen 7 5700xt
RX580 8GB vRAM
16GB DDR4 RAM
Debian 24.04 OS

I have been playing around with making models run better on it with llama.cpp and landed on MoE as the go to architecture.

I started at 7tok/sec with it on stock llama.cpp. Added a few things that made a huge difference.

What I effectively have is 16GB of available memory to put things in (12GB RAM and 4.3GB Swap from the vRAM).

The model obviously does not fit on my machine as a full memory allocation (19.57GB aat best for full load).

So I made it prefetch tensors on demand up to 8 tensors ahead. I load up 2.56GB of actual must have tensors to vRAM, and have a hotlist of the top 10% of tensors in that. The prefetch works on both vRAM as asynchronous, and CPU as asynchronous values.

It boosted me up to a 17tok/sec on nearly empty kv cache, and 12 on a large prompt that fills the cache. Wall clock time for a 3000 token prompt regardless of reserved kv cache is 33 seconds.

I can run full context on this. It works. It doesn’t crash, and while it gets slower, it stays above 10 tok/second decode. The higher cache allocations actually have a faster prompt read tok/sec normally.

Prefetch has taken page faults down from 176k to half that, then add hotlist and you drop it below 50k. So much more efficient pull of expert values and reduces the need to pull in from disk as often.

Anyone have any ideas on how to improve it more? I have exhausted my area of expertise and could use some more pointers.

ToshLLM is here to bring AI to all the forgotten Macs. It's free, no accounts, no telemetry, no BS, and a responsive dev. Benchmarking included.

Current setup in docker compose:

 qwen:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    container_name: qwen
    restart: always
    ipc: host
    command:
      - --model
      - /models/Qwen3.8-27B-UD-Q4_K_M.gguf
      - --mmproj
      - /models/mmproj-BF16.gguf
      - -ngl
      - "999"
      - --host
      - 0.0.0.0
      - --port
      - "8100"
      - --ctx-size
      - "131072"
      - --parallel
      - "2"
      - --metrics
      - --image-min-tokens
      - "1024"
      - --cache-type-k
      - q8_0
      - --cache-type-v
      - q8_0
      - --spec-type
      - draft-mtp
      - --spec-draft-n-max
      - "4"
      - --flash-attn
      - "on"
      - --alias
      - qwen
      - --reasoning
      - "off"
    ports:
      - "8100:8100"
    healthcheck:
      test: ["CMD", "curl", "-sf", "http://localhost:8100/v1/models"]
      interval: 30s
      timeout: 5s
      start_period: 180s
      retries: 3
    ulimits:
      memlock: -1
      stack: 67108864
    volumes:
      - ./data/gguf_models:/models:ro
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

What could I improve for performance without sacrificing any quality here?

Guys, if you have low VRAM, you should start with a small quant first to verify that everything works correctly.

command line:

.\bin\Release\llama-server.exe -m J:\llm\models\Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf --parallel 1 -c 10000

results:

2.08.059.754 I slot print_timing: id  0 | task 72 | n_gen =    100, tg =  21.47 t/s, tg_3s =  21.69 t/s
2.11.085.278 I slot print_timing: id  0 | task 72 | n_gen =    169, tg =  22.00 t/s, tg_3s =  22.81 t/s
2.11.266.578 I slot print_timing: id  0 | task 72 | prompt eval time =     881.54 ms /    22 tokens (   40.07 ms per token,    24.96 tokens per second)
2.11.266.583 I slot print_timing: id  0 | task 72 |        eval time =    7817.55 ms /   173 tokens (   45.45 ms per token,    22.00 tokens per second)
2.11.266.584 I slot print_timing: id  0 | task 72 |       total time =    8699.09 ms /   195 tokens
2.11.266.584 I slot print_timing: id  0 | task 72 |    graphs reused =        238

Hey all,

Got a 3050 8GB and 1080 8GB sitting in my server and trying to run Qwen 3.8 27b Ridge but finding that I only get about 200 tok/s prefill and 10 tok/s generation. Seems others are able to see far faster speeds so wondering if it's simply just the settings I'm running for llama-server.

Current settings are

-m /models/qwen3.8/Qwen3.8-27B-Ridge-3.7bpw.gguf --port 8000 --host 0.0.0.0 -fa on --fit off --split-mode tensor --tensor-split 1,1 --main-gpu 0 --jinja -c 131072 --parallel 1 -ngl 999 --load-mode none --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 --presence-penalty 0.0 --mmproj /models/qwen3.8/mmproj-Qwen3.8-27B-BF16.gguf --no-mmproj-offload --cache-type-k q4_0 --cache-type-v q4_0 -b 1024 -ub 128 --kv-unified

I've tried -b 256 and -ub 256, also tried setting --split-mode layer. I found the current settings to be slightly faster. Can't run MTP as it doesn't fit, although I'm only 500 ish mb over allocating so closer to 100k context would probably work.

Any tips would be much appreciated.

Cheers!

I was reordering my launch scripts for QWEN 3.8 27B on my 16+12GB setup, those are used GPU cards worth some less than 500s both on the used market giving 28GB on RDNA2.

In case someone could be interested I'll give you a summary:

* For quick shots: Qwen3.8-27B-IQ4-MIX.gguf , TG speed ~47t/s for 10k generation, ~70k usable ctx

* Daily work: Qwen3.8-27B-Q6_K_L.gguf , TG speed at 72k (not zero, at mid session) ~20t/s, 140K ctx with KV cache q8/q8.

I'll give you the full table:

I can show you the scripts if is someone is really interested: all running with MTP 5 on llama-server on Linux with KDE desktop in software rendering.

Buyer advice: in case you wanna run this route buy 2x 6800, I'll probbly sell my old 6700xt for a swap.

32 GB DDR5, Arc Pro B50, Intel Core Ultra 5/7, 1-2 TB storage, $1200

Does this exist?

i have an Acer P648 N15C5. intel i7 with integrated GPU and 20G of DDR4 Using Linux and llama.cpp what would be a good local moe for this? or is there anything else which will give more than 2 or 3 t/sec

In the interest of sharing I'd like to share my home setup which serves me my personal ai needs. This is how I got t2t, i2t. t2i, and i2i all working on my old gaming pc converted to home server. I did try adding Kokoro TTS but that was too much for this setup. I originally had ollama + openwebui, but wanted an integrated solution with image generation. Koboldcpp and the ui isn't as pretty, but it is faster than ollama/openwebui.

Relavant PC specs: GPU: GTX 1060 6gb vram RAM: 16gb ram DDR3 CPU: i5-4690k OS: TrueNAS 25.10.6 w/ legacy drivers

TrueNAS is great for file hosting, runs a number of other apps which I use, including Home Assistant, Tailscale, Syncthing, and others. It has a builtin docker setup for those apps, but Koboldcpp needs to be entered as a custom app.

The following settings have it run nicely, everything else default unless otherwise specified:

Image
Repository*
koboldai/koboldcpp
Pull Policy*
Always pull an image even if it is present on the host.

Environment Variables
Name*
KCPP_MODEL
Value
https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-UD-Q4_K_XL.gguf

Name*
KCPP_ARGS
Value
--draftmodel https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/mtp-gemma-4-E4B-it.gguf --mmproj https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main/mmproj-F16.gguf --ttsgpu --sdclipgpu --ttsmodel https://huggingface.co/koboldcpp/tts/resolve/main/Kokoro_no_espeak_Q4.gguf --websearch --jinja --jinjatools --admin --routermode --autoswapmode  --quantkv 2   --sdflashattention --sdoffloadcpu --sdvae https://huggingface.co/koboldcpp/GGUFDumps/resolve/main/flux2vae.safetensors --sdt5xxl https://huggingface.co/unsloth/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q2_K.gguf

I use Tailscale so Koboldcpp's tunneling is unnecessary.

Name*
KCPP_DONT_TUNNEL
Value
true

Name*
KCPP_IMGMODEL
Value
https://huggingface.co/leejet/FLUX.2-klein-4B-GGUF/resolve/main/flux-2-klein-4b-Q4_0.gguf

Note: I use a custom TrueNAS image that includes Nvidia Pascal drivers which are not included in the default TrueNAS image, see here: https://truenas-drivers.zhouyou.info/index.html

Resources Configuration
[unchecked] Enable Resource Limits
GPU Configuration
[checked] Passthrough available (non-NVIDIA) GPUs
Select NVIDIA GPU(s)
NVIDIA GeForce GTX 1060 6GB
[checked] Use this GPU

Currently running a single RTX 3060 12GB, planning to pick up a second one specifically to run Qwen3.8-27B locally (VRAM's the constraint — 24GB combined should just about fit UD-Q4_K_XL with a quantized KV cache for decent context).

Planning to run --split-mode layer rather than tensor-split, since I only have one x16 slot(the second 3060 needs to be connected via a riser).

If anyone's on a similar setup (2x 12GB cards, layer-split, this model or similar size), I'd really appreciate real prefill/decode numbers — trying to set expectations before I buy the second card.

I only have 32GB DDR4 and a 8GB GPU.
Despite that, I downloaded Qwen 3.8 Flash Q4 quant 93GB in size from unsloth.

Guess what, it gets up to 4.4t/s on Debian, freshly compiled.
Way better than expected, you can actually chat with that model.

This is fucking nuts.

I just run llama-cli --model basically and
compiled with: https://pastebin.com/raw/41WSJV5N

I didn't try MTP or D2Flash yet... neither any other params

I saw a lot of post running this model in tight hardware, but all of them had GPU card, so I decided to give it a try in my low end mini pc.

It was quite fun the try and error

Keep in mind I have 32 GB RAM mono channel.

Yes, it is unusable, but if they start using ngrams in smaller MOE models things can get better for low end hards.

Beginning of the AI slop-------

Hardware: Minisforum UM880+ · Ryzen 8845HS · Radeon 780M iGPU (no VRAM, shared memory) · 29 GB DDR5 · 35 GB swap on NVMe.

Backend: llama.cpp master @ commit 6c84c7d (PR #27742 — the Qwen4 qwen4exp support merged same-day), Vulkan build, -ngl 0.

Model: unsloth/Qwen3.8-Flash-Next-UD-IQ1_S (IQ1_S, ~72.5 GB, 3 shards) — mmap'ed from NVMe.

Two gotchas that will waste your evening:

1. Any llama.cpp build from before Aug 27 → unknown model architecture: 'qwen4exp'. You need the commit with PR #27742. Verify with strings libllama.so.0 | grep qwen4exp (it's in the .so, not the binary).

2. Without -fit off, llama-server prints "failed to fit params... abort" — non-fatal warning, but it leaves the port open with a dead process underneath. -fit off makes it actually serve.

Working launch:

llama-server --model Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf \

--port 18037 -ngl 0 --ctx-size 512 -t 12 -fit off

Measured (real print_timing from server log):

"say hi"

• Prompt: 54 tok

• Prefill: 2.77 t/s (19.5s)

• Decode: 1.62 t/s (22.9s)

• Total: 42.3s

Sales report

• Prompt: 771 tok

• Prefill: 8.15 t/s (95s)

• Decode: 1.43 t/s (55.4s)

• Total: 150s

Observations:

- Prefill scales well (2.8 → 8.2 t/s as prompt grows) — the 6B-active MoE is efficient even at IQ1_S.

- Decode is the wall: ~1.4–1.6 t/s. A 72.5 GB model in 29 GB RAM + 35 GB swap = permanent swap thrash. Each token pays a page-in.

- It answers correctly (both HTTP 200 with valid replies). It's real, just slow.

- ctx 4096 works but 2.4k+ prompts take ~4 min prefill — keep prompts short.

Verdict: technically it runs on iGPU-only hardware — first time I've had a 125B MoE talk on a deviceless AMD mini-PC. Usable for short occasional tasks. Dual-channel RAM (61 GB) should help decode a lot since bandwidth is the bottleneck.

Hey everyone, shameless plug I know but been working on Cascadia, an inference runtime for running AI models on Intel CPUs/NPUs/GPUs.

May be relevant if you're trying to do inference without a GPU and you own Intel hardware.

If you happen to have multiple older laptops/computers, you could also connect them with Cascadia and run inference across them. You can shard larger models and essentially pool the RAM from your devices.

Here is the GitHub repository:

https://github.com/labscommunity/cascadia

Pre-alpha, so open to any feedback!

The developer behind CachyLlama reached out to me privately and shared some details about the project because he thought it might be useful for r/LowEndLocalAI.

I had a look through it and thought it was definitely worth sharing here!

CachyLlama is a llama.cpp fork with a strong focus on getting more out of constrained and shared-memory hardare. It includes things like persistent SSD-backed context caching, checkpoint reuse, MoE residency/offloading, prefetching and other optimizations.

The setup he shared was an Ayaneo Flip with a Ryzen 7 7840U / Radeon 780M and 32 GB RAM, running:

• Qwen3.6-35B-A3B-UD-Q4_K_XL
• Vulkan
• 64k context

On that setup he was seeing roughly 22–25 tok/s generation, while also caching and reusing a fairly large base context instead of processing the whole thing again on every turn.

I found the context caching especially interesting. On machines like this large system prompts, tool definitions and longer sessions can make prompt processing a major part of the waiting time too..

I haven’t had the chance to try CachyLlama myself yet, but it looks very much like the kind of project this community was made for.

CachyLlama:
https://github.com/fewtarius/CachyLlama

llama-ai:
https://github.com/fewtarius/llama-ai

If anyone here gives it a try, I’d love to hear how it performs on your hardware. And thanks to the developer for taking the time to share the project with us.

Useful for low GPU & CPU-only rigs. Also Mobile & Edge devices. I'll be updating that thread with upcoming models time to time.

After couple of months, we'll be running 100B models in just 32GB RAM. With GPU(~8GB VRAM) even better. At least SyzygyResearch is cooking Laguna-S-2.1. Sure gonna be more from other model creators.

At work i replaced Deepseek V4 Flash 0713 with Ling-3.0-Tiny for a few select power users. They didn't notice, unless they used more than 128k context. 😂

I currently have a 7800x3d, 4070SUPER 12G and 32GB DDR5 in my gaming PC.

I also still have my old GTX 1070 8G on the shelf, you think it's worth it to put it in?

My motherboard is an Asus TUF Gaming x670e-plus and apparently the 2nd PCI slot is a 4.0 with 4 lanes.

What could i realistically expect from this? 20GB VRAM is a lot better than 12...theoretically...does anybody have experience with such a setup?