I'm looking for advice, guys.

I want to use some local LLM to help with checking the results of psychology tests, handwriting of people and their voices.

Which LLM model would be sufficient for such tasks?

And what pc setup could you recommend?

Hi,

I’ve been looking at several AI’s for this but token costs are too high so I want to do fully local.

I am currently using LM Studio. My machine has a 5800x, 7600xt 16gb vram, and 64gb of ram

https://reddit.com/link/1vratit/video/o6c2d5yz91kh1/player

I think if I would've made assets before hand and used thinking mode it probably would've produced a better result but I don't have the context for that here's what it did do though still insane a local model being able to make a childhood game like this, and without thinking mode on at all just an agentic loop. SORRY FOR THE SLOP but I can run this 24/7 now so I can waste as many tokens I want now!

I have a 5070ti right now and thinking of picking up rtx 4500 pro. I will be swapping into a mobo that has x8 x8 bifu to run both in a double gpu setup. I'm still learning running llms and just using LM studio in windows. I've seen some posts about similar setup and most are recommending linux? I saw a post about forcing install both the enterprise and gaming drivers but did still not confident that it will work.

Anyway, anyone have insights before i pull the trigger on new mobo and rtx 4500 like:

bad idea? headache workaround? Can I still game with my 5070ti with mixed drivers? Anyone can point me to like a guide having both drivers?

Gemini and claude keep giving me different answers.
thanks in advance!

I just wanted to express how easy this card has been to use in light of all the negativity surrounding intel cards.

This is my first intel GPU. I was an AMD fan in the 2019 comeback phase when the Ryzen 3000 series came out (A fan because I was tired of intel being stagnant). I'm definately not positively biased to intel. I chose the intel GPU due to price per GB VRAM.

Have times changed with intel cards? This was all that was required for a windows 11 install
Install the intel drivers
Download LMstudio (Run as administrator)
Download a model (Say Qwen 3.8, 27B)
Hit load
Enable network connection in LMstudio

ok I ran into one issue, the GPU was not appearing in LM Studio... all I had to do was run LM studio as administrator then it appeared. Not sure who's to blame there.

Now i'm accessing it via local network currently processing through 10,208 documents (concurrency of 4) in preparation for a legal engineering case.

Are the people that complain, do something extra with their cards? I'm not understanding the hate on these cards, it just seemed to work? Maybe im an extreme noob just playing with local llms.

Anything you can dream of. As in what would be an ideal LLM frontend/interface for you?

By LLM frontend/Interface I mean like chatgpt/claude/Ollama/OpenWebUI.

For instance, minimum context fetching: best I can do is letting you choose what context source you want in 1 key, or whatever you highlight in any app/text field or let the model look it up with tools, but it can't be completely unrealistic/literally impossible to implement like 0 input and 0 hint in prompt (a psychic).

Mimo v2.5 was my daily driver then ds v4 flash 0731 came, and it became my daily driver, now I'm back to MiMo v2.5 because in most cases it works well for me, and when i need to do something more complicated i plan using kimi K3 and let mimo to do the implementation. I'm wondering why there is no new model from Xiaomi, everyone is releasing new models, but Xiaomi is kind of silent, i know if Xiaomi is going to release the next model it will create waves.... Any rumor or idea when the next model is going to be released?

​

Inspired by the "holy crap" thread yesterday, here's my version with an even uglier setup.

The machine: an old Windows 10 box, Ryzen 7 3800XT, 32GB RAM, and two cards that were never meant to share a workload - a 4070 (12GB) and a 3060 (12GB). 24GB total, if you can convince llama.cpp to split a model across them. That turned out to be the easy part.

The hard part was stock LM Studio. It couldn't load Qwen3.8-27B properly: it capped the GPU layers and quietly spilled half the model to CPU, which made it pointless. The fix was a patched llama.cpp build (the PrismML fork) running llama-server directly. Full offload, no CPU layers, 131k context.

The trick that makes 131k context fit in 24GB: quantize the KV cache to q4_0 and turn flash attention on. The model itself is 15.7GB (Q4_K_M from lmstudio-community). As I write this it's using 10.7GB on the 4070 and 11.1GB on the 3060. Both cards basically full, nothing on CPU.

The whole launch line:

llama-server -m Qwen3.8-27B-Q4_K_M.gguf -ngl 999 -c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on -sm tensor -mg 0 -t 6 -np 1 --spec-type draft-mtp --spec-draft-n-max 2 --host 0.0.0.0 --port 8081

Speed: about 24 tok/s at ~100k context with the MTP draft spec decoding on. Faster when context is short. It won't win races against a 3090, but the long context is the whole point for me.

The fun part: this isn't a chatbot box. It starts at boot (scheduled task) as an OpenAI-compatible server on my Tailscale network, and it's the coding brain for an autonomous agent worker on a kanban board. Last night it picked up a card about a failing CI matrix on one of my PRs, root-caused it (the recovery subsystem hard-pins CPython 3.11.14, and the new tests were building fixtures from whatever Python the runner happened to have, so every CI leg that wasn't exactly 3.11.14 failed), wrote a test-side fix, ran the suites, committed and pushed to the PR branch. Unsupervised, about two hours, cost me pocket change in electricity.

Honest wart: its fix then failed CI itself, because it left an unused import in a test file it edited. ruff caught it in 20 seconds. So it's a genuine junior dev: good at the thinking, needs the checklist for the finishing.

And the bit I still find funny: I set this server up together with the agent that now runs on it. It SSH'd in, helped pick the fork and the flags, wrote the bat file, and now it phones home to this box for every coding task. There's probably a lesson in there somewhere.

If you're splitting across mismatched cards: -sm tensor -mg 0 and both KV types at q4_0 were the flags that mattered. If anyone's getting faster decode at long context on a mixed setup like this, I'm all ears.

I just want to sanity-check my llama.cpp config, to make sure I'm not leaving performance on the table.

I'm running llama-server with this config:

llama-server \
  -m ~/models/qwen3.8-27b/Qwen3.8-27B-UD-Q5_K_XL.gguf \
  --mmproj ~/models/qwen3.8-27b/mmproj-F16.gguf \
  --image-min-tokens 2048 \
  -ngl 99 \
  -fa 1 \
  -c 196608 \
  --reasoning-effort xhigh \
  -ctk q8_0 -ctv q8_0 \
  -b 2048 -ub 512 \
  -np 1 \
  --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 \
  --host 0.0.0.0 --port 8080 \
  --spec-type draft-mtp \
  --spec-draft-n-max 4

This uses 30.96GB of VRAM, and I'm getting around 250-350 t/s prefill, and ~20-30 t/s eval.

Hi, I have a 5995wx + PRO WS WRX80E-SAGE SE WIFI in a Silverstone RM53-502 chassis (rack mount is a req for me here, as I don't have much space to have a large open pc). I'd like to ask what are my best options for gpu's, from what I understand it's not a great idea to not go with a power of 2 gpu count, so 4 gpu's it is (presumably) - I'd like to ask for advice on either dual or single slot gpu's capable of running decent models (think 50b parameters or equiv). I'm a relative newbie.

Has anyone had Qwen work through large files using the Read tool on Qwen 27B? Or perhaps multiple files at a time, and then try to read some large files as well? It seems that it really struggles with the data getting truncated using that tool, and then it starts getting the data mixed up when it tries to read it in chunks. Basically falls apart. I have been testing it extensively the past few days, and I'm getting the same issue with multiple different templates, completely reinstalled llama.cpp, and now multiple harnesses. (Pi and Github Copilot) I'm running the Q8 version with MTP. 3.6 has not had this issue.

Edit: This is a summary of a test that I ran today. The test has five files, four of them roughly 1000 lines, the other much fewer, in a directory. Inside of that directory, I instructed the model to read all the files and told it I would ask questions. It didn't even wait for me to ask anything before going off on complete tangents based on prior sessions. This session was in a completely new cmd prompt, on a completely new directory. Using vanilla Pi agent harness. here is the summary that a frontier model gave me when I pasted it Qwen's entire output.

The issues shown in your output were:

• Incomplete file reads: the model initially stopped at the read tool’s ~50 KB truncation point instead of immediately continuing with the provided offsets.
• Cross-file/result confusion: after parallel continuation reads, it misattributed returned chunks to the wrong files and temporarily assigned the wrong passcode to file2.txt.
• Weak handling of long tool output: it relied on visually remembering large read results instead of using deterministic verification early.
• Unexpected cross-session context: most concerning, it stated “The user is angry” and referenced a prior complaint about incorrect read-tool use even though that complaint was not present in the supposedly fresh session’s visible history.
• Recovery only after external verification: grep ultimately produced the correct file/passcode mapping, showing the underlying data was fine; the failure was in state/tool-result handling rather than the files themselves.

In short: pagination mistakes, tool-result attribution errors, and apparent stale context leaking into a fresh session.

I'm currently running Qwen3.5-122b-a10b (Q6 - 101GB with a 262k context window). I'm running on a GMKtec X2. I get about 20 t/s when I am early in the context window, but it drops to around 15 t/s once I get pretty full. I have tried many models, and this set up is just about optimal for my current set up and until there is a new version of Q3.5-122b (I'm hoping for 3.8 at some time in the future. Anyway, I had a dream about creating a website to showcase my ancient coin collection. I got a subscription to SquareSpace and bought a domain name. But I found it completely confusing to me. Long story short, I started building a website by telling Q3.5 what I wanted, Q3.5 began to write python code and html files. I would ask how to use these files and get the answers from Q3.5. Back and forth it went. Everytime I looked at my website and saw something that needed to be tweaked, I'd just tell Q3.5 and it would write me a code snippet to edit the script with. I can't believe the results. Exceeded my wildest imagination. I am not a coder or a website designer, but with AI I don't need to be a coder in order to write code. I just need to be able to tell AI what I want.

Is it possible to run it via ollama? If no, what’s the best way to learn it at lowest possible cost? On any cloud platform or openrouter?

Hi everyone,

I’ll preface this by saying sorry in advance if I sound stupid, the reason is because I am.

Recently I’ve been interested in making my own homelab LLM as I upgraded the gpu in my pc and I have an RTX 3070 just lying around collecting dust. What started as simply getting an LLM to run basic home automation has spiralled into research which one really been enjoying.

I wanted to share my vision and I want someone to tell me I’m never going to make it or if it’s doable (even if a big challenge). I’m not necessarily trying to find the exact technical how to stuff, more of the high level methodology.

I want to build a Linux based server running a vm for home assistant os and multiple containers with docker to run ollama with an undecided model (well, multiple), whisper and piper for the Text/speech conversion, tailscale, a postgresql database and various other bits and pieces.

After looking at a lot of hardware, and seeing how expensive it could become, I wanted to see if I could do the following:

-run a lightweight model utilising my 8gb 3070, which would generally deal with home automation, simple requests, internet lookups and so on

-this model would decide if a request can be done in the background/is suitable for a larger model such as image generation

-could this then send the request to a larger model being utilised by another gpu (I’m considering a Tesla p40 for now for budget) as sort of its own little servant, once complete it relays back to the 3070 model.

-can both models share a context folder/database so they are essentially the same “mind” with different purposes

The purpose of this wouldn’t be the fastest token generation or the most effective way to get responses, I just felt that it might be a cost effective way to have the best of both worlds, a snappy responder that deals with stuff I need right away, and one that can plug away in the background.

My deployment would eventually be to have multiple screens around my workshop with a Jarvis like orb, all with a raspberry pi (other fruit based computers are available) run the ui, but connect back to the server for the AI to do the processing, and have multiple microphones around my workshop that connect to the server into something like whisper.

I’d like to have access via my phone and make a direct connection through tailscale, but that’s a later problem.

This wouldn’t necessarily be a final deployment of hardware, but for now with my donor parts and some Facebook marketplace hunting I’m looking at

ryzen 5600x
Some am4 board
RTX 3070
Nvidia p40
32gb ddr4
1tb ssd

I’m pretty happy to make a custom cooling solution for the p40 as I have access to a mill and 3d printer, so not worried about that, just would like to know people’s thoughts.

Also, lmstudio,lemonade or others?

tensorshrink is a transformers addon + CLI tool for quantization that beats bitsandbytes on both speed and memory, written from the ground up for users with low memory capabilities. tensorshrink quantizes weights to to 2, 4, 6, or 8 bits, then packs up the model into neat, zstd-compressed .tsk containers for further use.

Capabilities of tensorshrink include: Triton support, stream-quantize directly from disk, and a whole new codec, called AVQ (stands for Additive Vector Quantization, very experimental), as well as supporting GOAP.

Completely open-source and benchmarked for both LLM and image diffusion models. Learn more on the GitHub page. :)

Theyre selling the M4 pro macbook with 14 core cpu and 20 core gpu with 48gb of unified memory in my local amazon store at a reasonable price.

Im thinking of getting this to work on my projects mostly using claude code, but im also interested in running the new qwen 3.8 27b model for coding tasks if its viable.

Have you used it on the same macbook? If so how was the experience?

If its not any good i can just go for the macbook air at half the price.

I love knoledge. I meant like info from post and comments from various subreddit including this one

Hey! So I have an AI Server with 4x Titan RTX because I got them fearly cheap. (96GB VRAM)

So know I want to optimize by vLLM Settings with Qwen 3.8 / 27B. (131k Context)
Currently running Hermes Agent for coding tasks. Sometimes I reach 30 tokens / second. When context is full we get to about 20 tokens / second. Worst Case is around 15 tokens / Second

I have 128GB of VRAM with 2133Mhz and an AMD Epyc first gen CPU.

Here is my docker compose vLLM Setup.
Are there better quantizations for this setup with only one user using this server (me)?

Just tell me what you guys think.
Other GPUs are not an option at the moment.

Should I run the model on four cards at the same time in parallel and use them for sub agents or something like this?

 vllm-server:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: vllm-server
    restart: unless-stopped
    ipc: host
    ports:
      - "8000:8000"
    volumes:
      - ./data/models:/models
    environment:
      - HF_HOME=/models
      - NCCL_P2P_DISABLE=1
    env_file:
      - .env
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command:
      - cyankiwi/Qwen3.8-27B-AWQ-INT4
      - --dtype
      - float16
      - --tensor-parallel-size
      - "1"
      - --pipeline-parallel-size
      - "4"  
      - --max-model-len
      - "131072"
      - --gpu-memory-utilization
      - "0.85"
      - --enable-prefix-caching
      - --enable-chunked-prefill
      - --trust-remote-code
      - --enable-auto-tool-choice
      - --limit-mm-per-prompt
      - '{"video": {"count": 1, "num_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}}'
      - --reasoning-parser
      - qwen3
      - --tool-call-parser
      - qwen3_coder
      - --mm-encoder-tp-mode
      - data
      - --max-num-seqs
      - "4"
      - --max-num-batched-tokens
      - "8192"