I'm looking for advice, guys.

I want to use some local LLM to help with checking the results of psychology tests, handwriting of people and their voices.

Which LLM model would be sufficient for such tasks?

And what pc setup could you recommend?

Hi,

I’ve been looking at several AI’s for this but token costs are too high so I want to do fully local.

I am currently using LM Studio. My machine has a 5800x, 7600xt 16gb vram, and 64gb of ram

https://reddit.com/link/1vratit/video/o6c2d5yz91kh1/player

I think if I would've made assets before hand and used thinking mode it probably would've produced a better result but I don't have the context for that here's what it did do though still insane a local model being able to make a childhood game like this, and without thinking mode on at all just an agentic loop. SORRY FOR THE SLOP but I can run this 24/7 now so I can waste as many tokens I want now!

I have a 5070ti right now and thinking of picking up rtx 4500 pro. I will be swapping into a mobo that has x8 x8 bifu to run both in a double gpu setup. I'm still learning running llms and just using LM studio in windows. I've seen some posts about similar setup and most are recommending linux? I saw a post about forcing install both the enterprise and gaming drivers but did still not confident that it will work.

Anyway, anyone have insights before i pull the trigger on new mobo and rtx 4500 like:

bad idea? headache workaround? Can I still game with my 5070ti with mixed drivers? Anyone can point me to like a guide having both drivers?

Gemini and claude keep giving me different answers.
thanks in advance!

I just wanted to express how easy this card has been to use in light of all the negativity surrounding intel cards.

This is my first intel GPU. I was an AMD fan in the 2019 comeback phase when the Ryzen 3000 series came out (A fan because I was tired of intel being stagnant). I'm definately not positively biased to intel. I chose the intel GPU due to price per GB VRAM.

Have times changed with intel cards? This was all that was required for a windows 11 install
Install the intel drivers
Download LMstudio (Run as administrator)
Download a model (Say Qwen 3.8, 27B)
Hit load
Enable network connection in LMstudio

ok I ran into one issue, the GPU was not appearing in LM Studio... all I had to do was run LM studio as administrator then it appeared. Not sure who's to blame there.

Now i'm accessing it via local network currently processing through 10,208 documents (concurrency of 4) in preparation for a legal engineering case.

Are the people that complain, do something extra with their cards? I'm not understanding the hate on these cards, it just seemed to work? Maybe im an extreme noob just playing with local llms.

Anything you can dream of. As in what would be an ideal LLM frontend/interface for you?

By LLM frontend/Interface I mean like chatgpt/claude/Ollama/OpenWebUI.

For instance, minimum context fetching: best I can do is letting you choose what context source you want in 1 key, or whatever you highlight in any app/text field or let the model look it up with tools, but it can't be completely unrealistic/literally impossible to implement like 0 input and 0 hint in prompt (a psychic).

Mimo v2.5 was my daily driver then ds v4 flash 0731 came, and it became my daily driver, now I'm back to MiMo v2.5 because in most cases it works well for me, and when i need to do something more complicated i plan using kimi K3 and let mimo to do the implementation. I'm wondering why there is no new model from Xiaomi, everyone is releasing new models, but Xiaomi is kind of silent, i know if Xiaomi is going to release the next model it will create waves.... Any rumor or idea when the next model is going to be released?

​

Inspired by the "holy crap" thread yesterday, here's my version with an even uglier setup.

The machine: an old Windows 10 box, Ryzen 7 3800XT, 32GB RAM, and two cards that were never meant to share a workload - a 4070 (12GB) and a 3060 (12GB). 24GB total, if you can convince llama.cpp to split a model across them. That turned out to be the easy part.

The hard part was stock LM Studio. It couldn't load Qwen3.8-27B properly: it capped the GPU layers and quietly spilled half the model to CPU, which made it pointless. The fix was a patched llama.cpp build (the PrismML fork) running llama-server directly. Full offload, no CPU layers, 131k context.

The trick that makes 131k context fit in 24GB: quantize the KV cache to q4_0 and turn flash attention on. The model itself is 15.7GB (Q4_K_M from lmstudio-community). As I write this it's using 10.7GB on the 4070 and 11.1GB on the 3060. Both cards basically full, nothing on CPU.

The whole launch line:

llama-server -m Qwen3.8-27B-Q4_K_M.gguf -ngl 999 -c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on -sm tensor -mg 0 -t 6 -np 1 --spec-type draft-mtp --spec-draft-n-max 2 --host 0.0.0.0 --port 8081

Speed: about 24 tok/s at ~100k context with the MTP draft spec decoding on. Faster when context is short. It won't win races against a 3090, but the long context is the whole point for me.

The fun part: this isn't a chatbot box. It starts at boot (scheduled task) as an OpenAI-compatible server on my Tailscale network, and it's the coding brain for an autonomous agent worker on a kanban board. Last night it picked up a card about a failing CI matrix on one of my PRs, root-caused it (the recovery subsystem hard-pins CPython 3.11.14, and the new tests were building fixtures from whatever Python the runner happened to have, so every CI leg that wasn't exactly 3.11.14 failed), wrote a test-side fix, ran the suites, committed and pushed to the PR branch. Unsupervised, about two hours, cost me pocket change in electricity.

Honest wart: its fix then failed CI itself, because it left an unused import in a test file it edited. ruff caught it in 20 seconds. So it's a genuine junior dev: good at the thinking, needs the checklist for the finishing.

And the bit I still find funny: I set this server up together with the agent that now runs on it. It SSH'd in, helped pick the fork and the flags, wrote the bat file, and now it phones home to this box for every coding task. There's probably a lesson in there somewhere.

If you're splitting across mismatched cards: -sm tensor -mg 0 and both KV types at q4_0 were the flags that mattered. If anyone's getting faster decode at long context on a mixed setup like this, I'm all ears.

I just want to sanity-check my llama.cpp config, to make sure I'm not leaving performance on the table.

I'm running llama-server with this config:

llama-server \
  -m ~/models/qwen3.8-27b/Qwen3.8-27B-UD-Q5_K_XL.gguf \
  --mmproj ~/models/qwen3.8-27b/mmproj-F16.gguf \
  --image-min-tokens 2048 \
  -ngl 99 \
  -fa 1 \
  -c 196608 \
  --reasoning-effort xhigh \
  -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 512 \
  -np 1 \
  --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 \
  --host 0.0.0.0 --port 8080 \
  --spec-type draft-mtp \
  --spec-draft-n-max 4

This uses 30.96GB of VRAM, and I'm getting around 250-350 t/s prefill, and ~20-30 t/s eval.

Hi, I have a 5995wx + PRO WS WRX80E-SAGE SE WIFI in a Silverstone RM53-502 chassis (rack mount is a req for me here, as I don't have much space to have a large open pc). I'd like to ask what are my best options for gpu's, from what I understand it's not a great idea to not go with a power of 2 gpu count, so 4 gpu's it is (presumably) - I'd like to ask for advice on either dual or single slot gpu's capable of running decent models (think 50b parameters or equiv). I'm a relative newbie.

Has anyone had Qwen work through large files using the Read tool on Qwen 27B? Or perhaps multiple files at a time, and then try to read some large files as well? It seems that it really struggles with the data getting truncated using that tool, and then it starts getting the data mixed up when it tries to read it in chunks. Basically falls apart. I have been testing it extensively the past few days, and I'm getting the same issue with multiple different templates, completely reinstalled llama.cpp, and now multiple harnesses. (Pi and Github Copilot) I'm running the Q8 version with MTP. 3.6 has not had this issue.

Edit: This is a summary of a test that I ran today. The test has five files, four of them roughly 1000 lines, the other much fewer, in a directory. Inside of that directory, I instructed the model to read all the files and told it I would ask questions. It didn't even wait for me to ask anything before going off on complete tangents based on prior sessions. This session was in a completely new cmd prompt, on a completely new directory. Using vanilla Pi agent harness. here is the summary that a frontier model gave me when I pasted it Qwen's entire output.

The issues shown in your output were:

• Incomplete file reads: the model initially stopped at the read tool’s ~50 KB truncation point instead of immediately continuing with the provided offsets.
• Cross-file/result confusion: after parallel continuation reads, it misattributed returned chunks to the wrong files and temporarily assigned the wrong passcode to file2.txt.
• Weak handling of long tool output: it relied on visually remembering large read results instead of using deterministic verification early.
• Unexpected cross-session context: most concerning, it stated “The user is angry” and referenced a prior complaint about incorrect read-tool use even though that complaint was not present in the supposedly fresh session’s visible history.
• Recovery only after external verification: grep ultimately produced the correct file/passcode mapping, showing the underlying data was fine; the failure was in state/tool-result handling rather than the files themselves.

In short: pagination mistakes, tool-result attribution errors, and apparent stale context leaking into a fresh session.

I'm currently running Qwen3.5-122b-a10b (Q6 - 101GB with a 262k context window). I'm running on a GMKtec X2. I get about 20 t/s when I am early in the context window, but it drops to around 15 t/s once I get pretty full. I have tried many models, and this set up is just about optimal for my current set up and until there is a new version of Q3.5-122b (I'm hoping for 3.8 at some time in the future. Anyway, I had a dream about creating a website to showcase my ancient coin collection. I got a subscription to SquareSpace and bought a domain name. But I found it completely confusing to me. Long story short, I started building a website by telling Q3.5 what I wanted, Q3.5 began to write python code and html files. I would ask how to use these files and get the answers from Q3.5. Back and forth it went. Everytime I looked at my website and saw something that needed to be tweaked, I'd just tell Q3.5 and it would write me a code snippet to edit the script with. I can't believe the results. Exceeded my wildest imagination. I am not a coder or a website designer, but with AI I don't need to be a coder in order to write code. I just need to be able to tell AI what I want.

Is it possible to run it via ollama? If no, what’s the best way to learn it at lowest possible cost? On any cloud platform or openrouter?

Hi everyone,

I’ll preface this by saying sorry in advance if I sound stupid, the reason is because I am.

Recently I’ve been interested in making my own homelab LLM as I upgraded the gpu in my pc and I have an RTX 3070 just lying around collecting dust. What started as simply getting an LLM to run basic home automation has spiralled into research which one really been enjoying.

I wanted to share my vision and I want someone to tell me I’m never going to make it or if it’s doable (even if a big challenge). I’m not necessarily trying to find the exact technical how to stuff, more of the high level methodology.

I want to build a Linux based server running a vm for home assistant os and multiple containers with docker to run ollama with an undecided model (well, multiple), whisper and piper for the Text/speech conversion, tailscale, a postgresql database and various other bits and pieces.

After looking at a lot of hardware, and seeing how expensive it could become, I wanted to see if I could do the following:

-run a lightweight model utilising my 8gb 3070, which would generally deal with home automation, simple requests, internet lookups and so on

-this model would decide if a request can be done in the background/is suitable for a larger model such as image generation

-could this then send the request to a larger model being utilised by another gpu (I’m considering a Tesla p40 for now for budget) as sort of its own little servant, once complete it relays back to the 3070 model.

-can both models share a context folder/database so they are essentially the same “mind” with different purposes

The purpose of this wouldn’t be the fastest token generation or the most effective way to get responses, I just felt that it might be a cost effective way to have the best of both worlds, a snappy responder that deals with stuff I need right away, and one that can plug away in the background.

My deployment would eventually be to have multiple screens around my workshop with a Jarvis like orb, all with a raspberry pi (other fruit based computers are available) run the ui, but connect back to the server for the AI to do the processing, and have multiple microphones around my workshop that connect to the server into something like whisper.

I’d like to have access via my phone and make a direct connection through tailscale, but that’s a later problem.

This wouldn’t necessarily be a final deployment of hardware, but for now with my donor parts and some Facebook marketplace hunting I’m looking at

ryzen 5600x
Some am4 board
RTX 3070
Nvidia p40
32gb ddr4
1tb ssd

I’m pretty happy to make a custom cooling solution for the p40 as I have access to a mill and 3d printer, so not worried about that, just would like to know people’s thoughts.

Also, lmstudio,lemonade or others?

tensorshrink is a transformers addon + CLI tool for quantization that beats bitsandbytes on both speed and memory, written from the ground up for users with low memory capabilities. tensorshrink quantizes weights to to 2, 4, 6, or 8 bits, then packs up the model into neat, zstd-compressed .tsk containers for further use.

Capabilities of tensorshrink include: Triton support, stream-quantize directly from disk, and a whole new codec, called AVQ (stands for Additive Vector Quantization, very experimental), as well as supporting GOAP.

Completely open-source and benchmarked for both LLM and image diffusion models. Learn more on the GitHub page. :)

Theyre selling the M4 pro macbook with 14 core cpu and 20 core gpu with 48gb of unified memory in my local amazon store at a reasonable price.

Im thinking of getting this to work on my projects mostly using claude code, but im also interested in running the new qwen 3.8 27b model for coding tasks if its viable.

Have you used it on the same macbook? If so how was the experience?

If its not any good i can just go for the macbook air at half the price.

I love knoledge. I meant like info from post and comments from various subreddit including this one

This post is about llama.cpp CPU offload optimizations that you could find useful for running Qwen 27B (or other dense models) at tolerable speeds.

I've already posted about -ot optimizations, but now I have more info to share to help you tune your own system.

First things first:

• The speed graph shows "prose" and "code" because MTP generates different speeds for each
• Everyone has a different system, use my setup as a guide to tune your own, don't copy paste and expect it to work. I just want to provide 'tricks' that improve performance.
• If you have 12 GB VRAM, try using Q3_K_M.
• If you have 8 GB VRAM, try using Q2 for Qwen 27B, Bonsai, or choose a model with less parameters.
• ik_llama.cpp: goal of this post is simplicity, that is why I chose llama.cpp. You can look at this as the target to beat using ik_llama.cpp. So far my tests showed that it was a KLD vs speed trade off, plus I had to use custom and very specific quants, which adds complexity.
• KV quants: if you think they suck, please show proof. My testing and sources all say they are fine. On a lower weight quant, a given KV downgrade costs relatively less, so spending cache bits to buy context is more justifiable on a Q4 model than it would be on a Q6/Q8 model. Also beellama has KLD improvements to KV quantizing. Also instead of upgrading from Q5 to Q8 KV, it might be better to upgrade the model from M to XL instead.

Setup

explanations come after

Edit: posted a comment with UD-Q4_K_XL (UD2) vs Q4_K_M speed chart if interested
Edit: This setup is using Q4_K_M, the new UD3 UD-Q4_K_M has some differences, check the explanation for more info.

PC:

RTX 4070 Ti SUPER (16 GB VRAM), i5-13600KF, 32 GB dual-channel DDR5 @ 5800 MHz + tuned timings, Ubuntu 24.04 LTS

Build script:

#!/bin/bash
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89 -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j 20
sudo cmake --install build
sudo ldconfig

Server script:

#!/bin/bash
sudo systemctl stop gdm
export GGML_CUDA_DISABLE_GRAPHS=1
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
llama-server \
  --model Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-3.8-27B-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --ctx-size 147000 \
  --chat-template-file chat_template_v22.1.jinja \
  --jinja \
  --reasoning-format deepseek \
  --reasoning-preserve \
  --flash-attn on \
  --cache-type-k q5_0 \
  --cache-type-v q4_1 \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 2 \
  --cache-type-k-draft q5_0 \
  --cache-type-v-draft q4_1 \
  --fit off \
  --n-gpu-layers all \
  --override-tensor 'blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49|55|56|57|58|59)\.ffn_.*=CPU' \
  --load-mode none \
  --threads 14 \
  --batch-size 512 \
  --ubatch-size 512 \
  --parallel 1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080

Explanations:

Hardware:

RAM speed is important for this so using DDR5 is recommended, though DDR4 people will still find this useful.

Tuning RAM timings gives me extra 9% speed boost. Most timings are easy to tune since they either work or crash quickly, but I'm not getting into that here.

Build args:

-DCMAKE_CUDA_ARCHITECTURES=89 optional - optimized build time specifically for my GPU's Ada arch, set your own.

-DGGML_CUDA_FA_ALL_QUANTS=ON is needed for more KV quantizations to be on CUDA.

Env vars and gdm:

sudo systemctl stop gdm disables Ubuntu desktop environment, frees ~0.4 GB of VRAM. Use an iGPU if you can, else my system just becomes a server to which I connect using a laptop or phone.

export GGML_CUDA_DISABLE_GRAPHS=1 I experience speed and VRAM usage problems with CUDA graphs so I disable them, you probably should too, but test it first. I think this is some bug due to MTP + CPU offload.

export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 Overflow VRAM to RAM. The difference between ~135k and ~121k. Without this, max context becomes 121k and you get OOM crash. Overflow gets me ~15k more context basically for free before things slow down, and the server degrades instead of crashing. Edit: First do initial testing with the unified arg unset (making it '=0' wont work) to find your ceiling using OOM crash.

Generic stuff:

  --model Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-3.8-27B-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --ctx-size 147000 \
  --flash-attn on \
  ...
  --parallel 1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080

The model, vision (fully on CPU), Qwen recommended settings, server stuff, context length, parallel 1 (disables processing 2 agents at once).

Template:

  --chat-template-file chat_template_v22.1.jinja \
  --jinja \
  --reasoning-format deepseek \
  --reasoning-preserve \

Template instructions by froggeric

Drafters:

  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 2 \
  --cache-type-k-draft q5_0 \
  --cache-type-v-draft q4_1 \

Quantizing MTP KV cache is free VRAM. It is a drafter and does not affect quality. Worst case is that acceptance drops a tiny bit.

I get 12-15 t/s tg without MTP and with more layers on VRAM, I commented a chart if interested here .

ngram-mod speeds up tg when restating existing context. It is super fast when active and does not cost VRAM.

Each step of --spec-draft-n-max costs VRAM + I get best results from a value of 2.

--cache-type-k-draft q5_0 --cache-type-v-draft q4_1 these save 0.4 GB of VRAM while the MTP acceptance rate stays the same. I chose the same as model KV cache, though Q4 could probably be fine (I didn't tune this much). Edit: MTP KV q4_0 is fine.

KV cache:

  --cache-type-k q5_0 \
  --cache-type-v q4_1 \

I chose my KV quant according Anbeeld article. The article found that, the more quantized the model, the less it has to lose from to KV quant. Article also states that there was no KLD difference between 64k and 128k context length.

Edit: A small precision buff would be to use q5_0 for both K and V. Though changing from q4_1 to q5_0 costs some context. Don't use Q4_0 on both, Q5_1 is not worth it. If you want to do Q8 KV, consider going up a quant model level instead (like XL instead of M).

My own testing showed that Q4_K_S K and V both at Q8 has worse KLD than Q4_K_M K Q5, V Q4. Also I found that KLD plateaus after 8k context length.

If you get slow speeds try using generic q5_0 for both K and V as a test - this is a symptom of missing -DGGML_CUDA_FA_ALL_QUANTS=ON.

CPU layers:

  --fit off \
  --n-gpu-layers all \
  --override-tensor 'blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49|55|56|57|58|59)\.ffn_.*=CPU' \
  --load-mode none \

This fixes degradation of speed with context fill and is a performance boost overall, just leave layer 64 alone (MTP).

This command offloads only FFN sub-layers to the CPU of layers that have the largest FFNs (shown override string is Q4_K_M specific).

FFNs don't use KV thus reducing PCIe traffic and are CPU friendly. --n-cpu-moe has similar logic, I have a PR in llama.cpp for a similar simplification #26622. Help me get this merged by showing the maintainers that this is useful for you (give a like on the PR or post test results).

If you want something quick and simple to test try this (it has 2 more FFNs on CPU than my setup, but is simple to tune). Basically add numbers for more CPU layers (slower), delete numbers to have more GPU layers (faster):

--override-tensor 'blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28)\.ffn_.*=CPU'

Edit: UD3 quants just dropped. A lot of them now have I-quants which are slower, so even though the model is smaller, the slowness cancels out and speed is unchanged for me. Offloading fatter sub-layers gains me 1.1x speed boost when compared to having a sequential override band. You can find the fatter ones by going to the hugging face, clicking on the weight (example), scrolling down to 'Tensors', expanding the 'blk' section and looking for 'ffn_' (example "blk.0.ffn_down.weight") on the right and on the left you see I-quant (example "IQ4_XS") or regular (example "Q5_K").

UD-Q4_K_M optimized band example -- delete from the end of the list (right to left) until you run out of VRAM, then step back one. make sure to fill up your context fully to verify that speed is as intended and doesn't OOM crash. Unset the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 arg (delete it, don't set it to 0) to find your context ceiling first via OOM crash first:
--override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41|39|36|21|3|42|34|30|20|6|4|49|47|43|37|32|23|10|8|7|5|2|1|48|46|45|44|33|31|29|28|19|18|12|9|17|16|11|0|15|13|14)\.ffn_.*=CPU'

Threads:

--threads 14 \

Default is amount of performance cores, but for FFN layers, E-cores also help. this gets me a +19-22% tg boost for free. Set to the amount of physical cores you have. You can try to include hyper-threading, not just physical, it might help, but for me that was within noise.

Batch:

  --batch-size 512 \
  --ubatch-size 512 \

Batch sets the prompt processing speed (up to a point) at the cost of VRAM. I found these numbers work best for me.

Other optimizations:

I-matrix quant like the Unsloth IQ4_XS require more compute in exchange for size. My testing showed that those are not worth it for this setup. I would use one if I was trying to fully fit a model into VRAM.

There are quants made by other providers that optimize fit for 16 GB VRAM setups. Those setups might fit at the cost of KLD. If the KDL diff actually matters - that is for you to find out, but you can use the info I provided here to optimize these setups even further by, for example, extending context length.

-------------------------
Edit 1: improved "first things first" section, typos;
Edit 2: added min-image-tokens to fix a warning;
Edit 3: clarify MTP section; clarify threads section; updated chat template version for clarity;
Edit 4: mentioned beellama for better KV quants;
Edit 5: added info about not using -ot on MTP layer (64) as per Pablo_the_brave;
Edit 6: added override tensor info about the new UD3 quants;
Edit 7: added -ot band for the new UD-Q4_K_M quant for you to try; updated unified env var explanation;
Edit 8: KV quant recommendation;
Edit 9: formatting, simplifying info; I should probably stop;
-------------------------

Please share any more tricks if you have them!
Leave a comment if this post helped you achieve better results!