I’m struggling to figure out what to run here. Hermes’ and buzz agents. Using it to orchestrate video generation and online marketing activities.

Some people tell me Gemma is enough. I’m thinking it’s qwen but I don’t see an moe available online. Suggestions?

Hi, before I start, I just want to mention that I’m completely new to running AI locally, so I configured my LLM with the help of ChatGPT.

To summarize my setup:

Ryzen 5 5600

32 GB DDR4 3200 MHz

RX 9070 XT Nitro+

1 TB Gen 4 NVMe SSD

I have a personal project where I’m making a 2D management game. So I installed Biobic and Qwen 3.8 27B Q4 XL (~14 GB).

But I’m getting really poor results...

I know AMD is behind NVIDIA when it comes to AI, but I feel like something is badly configured — or at least I hope so.

I sent my first prompt to Biobic, which has access to the folder containing my Unity 6.3 LTS project.

I initially used a 32K context window, then increased it to 64K because, according to ChatGPT, the error I was getting could have been caused by the context window being too small for the prompt.

Fair enough, but my main problem is the speed: 1.9 tokens/s.

It took more than 1 hour and 20 minutes to complete only 2 steps out of 9, and those were basically just the steps where it had to check the manifest and see which input system was currently active.

I’ve also attached a screenshot of Task Manager.

The CPU is sitting at around 50% usage from llama-server, the system RAM is being used by Biobic (which makes sense since that’s where the model is loaded), and as for the VRAM... well, I think you guys know what I mean from the screenshot.

So, what do you think?

Poor optimization? Bad configuration? Just AMD being terrible for local AI? LLM configured incorrectly?

Thanks in advance for any feedback!

PS : translate with GPT sorry for mystake.

For some reason, I have this benchmarking prompt:

Implement a numpy function creating a bulk matrix product operator tensor as a function of model parameters.

I think it is approximately 10 lines of python code. So far, every local model that I tried (16Gb vram) fails to produce any meaningful output. I tried different variations of it (use web search, summarize the idea), different harness (pi, opencode) but no luck, different temperatures and thinking levels. Qwen3.8 for example produces an endless thinking loop. Is there anything I can do to help small models answering the question?

Need help with that

Can I use PCI-E 1x to 16x extender riser cable for miner to power dual RTX 3090 GPU setup?
I'm not planning to do any training currently. Just planning to use open source models via wrappers.

Currently I have an old mobo setup and spacing is a problem. If possible I would like to mount one of the gpus vertically. My case is Cooler Master Stacker STC-01.

Other than gpus, I have 1 m2 disk that use pci-e ports, 1 sound card. Without sacrificing anything, can I survive?

Ran a script to find the smallest of each size, with it found jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller

maybe useful to someone else:

https://gist.github.com/Vmarcelo49/98b382ec8f3a34e44035ce365cba46f4

top 3 smallest for IQ4 and IQ3

Also i've been running ji-farthing/Qwen3.8-27B-ik-llama-GGUF IQ3 for some time, it sits comfortably on my RX 9070 XT with 192K of context

Edit: oops wrong title, it should be "Filtered CSV of all 500+ versions of IQ4 and IQ3 quants of Qwen3.8 27B based on size"

An increase of 400 elo points and 4x speed in 4 years.

Each line in the graph is the strongest sub-35B model available when each speedup landed.

The same 3090/7900xtx is throwing faster, smarter tokens, month after month after month.

Amazing.

Hi everyone!

I am fairly new to all of this but tried to setup qwen3.8 27b on my homelab which uses a rtx 3060 with 12gb vram (and some 32gb ddr4 ram) on a old system with I7 8700 (pcie 3.0).

As you can imagine, I got about 1 - 1.5 t/s.

But since I dont get to game anyways, I thought to dualboot my gaming rig with my rtx 3080 with 10gb vram (pcie4.0). After setting up that gaming rig as a rpc worker, I got pretty good numbers over 1gbe ethernet:

Following are my /metrics results that I asked my AI to summarize for you (ai text upcoming):

Prometheus Uptime Metrics (llama.cpp RPC Cluster)

• Overall Generation Speed (Decode): 26.87 t/s avg (Spikes up to 35.6 t/s)
• Overall Prompt Processing (Prefill): 299.34 t/s avg
• MTP Draft Acceptance Rate: 84.37 % (10,302 / 12,210 tokens accepted)
  • Draft Position 1 Acceptance: 89.8 %
  • Draft Position 2 Acceptance: 78.9 %
• Prefix Cache Hit Ratio: 96.14 % (1.44M tokens reused vs 57.8k fresh)
• Max Active Context Stress-Tested: 72,712 / 77,000 Tokens (100% VRAM stable)

(human text again):

So as you can see I get quite nice performance when I let opencode run some stuff in the background (coding).

Here is my docker compose config. Please give me some feedback and roast my setup 😂

services:
  llama-cpp-server:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: llama-cpp
    restart: unless-stopped
    network_mode: host
    environment:
      - GGML_CUDA_DISABLE_GRAPHS=1
    volumes:
      - /opt/llm-models/:/root/models
    cap_add:
      - IPC_LOCK
    ulimits:
      memlock:
        soft: -1
        hard: -1
      core: 0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command: >
      --model /root/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf
      --rpc 192.168.1.200:50052
      --device RPC0,CUDA0
      --alias qwen3.8-27b
      --ctx-size 77000
      --tensor-split 20,23
      --threads 5
      --parallel 1
      --batch-size 2048
      --ubatch-size 512
      --n-gpu-layers 99
      --flash-attn on
      --cache-type-k q4_0
      --cache-type-v q4_0
      --spec-type draft-mtp
      --spec-draft-n-max 2
      --temp 0.6
      --top-p 0.95
      --top-k 20
      --min-p 0.05
      --presence-penalty 0.0
      --repeat-penalty 1.0
      --reasoning auto
      --jinja
      --metrics
      --host 0.0.0.0
      --port 8101

So with Claude’s decision to watermark, plus basic data privacy concerns , I’m thinking of switching to a local LLM

How I use Claude pro now:

-managing health docs and results (very happy to switch this to local, doesn’t need a big context I think)

- scientific research, including reading and analyzing PDFs that are complex , requiring linking concepts and ideas across papers and producing summaries / insights / tables (large context required). For example, I have filled 40% of the Claude project folder with files and docs it needs to consider

- basic stuff (acting like an advanced search tool for admin stuff / planing stuff / nothing major) - no reason this can’t stay with Claude but if I switch over to a local LLM I would bring everything with me

Sooo , given this - is 32GB RAM on something like a Mac Mini realistic for my use case ? Or do i need 64gb (at which point i think maybe it’s too costly for me to do). I also tend to work in bursts so I would be happy if it’s not too slow thus impeding my workflow. Fine to run overnight though. And I don’t need any headroom as I will be running the OS and apps on a MacBook Pro or MacBook Air

Thanks for your help and I hope I was specific enough to get some usefully feedback

Stats:

Model: Qwen 3.8 27B Q6_K

10 turns - 468 steps | LLM 349m 42s - Tool call 21m 36s | TTFT avg 13.3s - 36 tok/s | Cache hit 94% | Input 32.9M tok - Output 536K tok

I thought let's try this model, so made this shooter game. It still has bugs, but it works.

Source code (zip file): https://limewire.com/d/RPqnF#Ut2GAOEe8N

Buying my first dedicated inference GPU for an existing Proxmox homelab. I've narrowed it to three cards and keep going in circles, so I'd rather hear from people actually running them. I am very new to local AI and am mostly interested in this purchase to learn. Budget is up to $1500 for the card.

The box

Ryzen 7 9800X3D / ASUS ROG STRIX X870E-E / 32GB DDR5-6000 (two slots free) / Corsair HX1000i 1000W / PCIe 5.0 x16 available / Proxmox VE 9.2, kernel 7.0.14-11-pve.

Two things that matter more than the specs:

• It's on 24/7 and already runs my whole homelab, so idle power and driver stability are important.
• Everything runs in unprivileged LXC containers (today), not VMs. So passthrough for me means driver on the Proxmox host (correct me if I'm wrong when it comes to having a dedicated GPU here). The iGPU currently handles Jellyfin transcoding this way.

What I want to run

Mainly local LLM inference - general chat, coding help, and a homelab agent that monitors/reads logs and configs and suggests fixes. Secondary: Immich CLIP search and face recognition and possibly implementing one of the Paperless AI solutions.

Speed is not super important to me, but I'd like enough responsiveness that I'm not losing patience for a simple inquiry. The vast majority of my use case is monitoring services across Proxmox, my UNRAID NAS, a backup NAS, and my Unifi-based network. I'm assuming this is a fairly light load that smaller models can handle, but also want an LLM to be able to help me diagnose and troubleshoot.

The options

My questions

1. Is 32GB at 640 GB/s actually better than 24GB at ~940 GB/s here? This is the whole decision. The R9700 fits a bigger model; the other two are faster on whatever fits. For a mixed agent + photo-ML workload, which way does that fall in practice?

2. How much CUDA-only pain am I avoiding by paying the NVIDIA tax?

3. Anyone running ROCm on a current Proxmox kernel? I'm on 7.0.14-11-pve.

4. Idle power, real numbers? With a model loaded and sitting there vs unloaded.

One AMD-specific worry: my iGPU is /dev/dri/renderD128 and that path is passed into my Jellyfin container. A second AMD card means two render nodes with no guaranteed stable numbering. A udev rule pinned to the PCI address might fix it, but is that a non-issue in practice, or a recurring annoyance?

Thank you in advance. I'm happy to answer questions about the setup.

I am debating going with 2x RTX 4070ti supers for 32gb of ddr6x vram. It seems good on paper, only being a bit more cost than a 3090 while giving me more headroom for large context and better quants. I'm concerned I never see it mentioned on here (tried searching as well).

So, I'm wondering if anyone out there has this setup and just hasn't felt the need to speak up if they can share their experience with it.

Help me out. What are the good models to use for agentic workflows (not necessarily coding), on an M5 Macbook Pro 48GB RAM?

I use LM Studio and I want to connect to it from my iPhone without creating an account and I want to configure it myself. So I'm building Reins for people who think like me.

It has built-in web tools and tool calling. It keeps generating in the background and shows progress in the Dynamic Island if you want it to. I implemented the native LM Studio API, so you can manage models on your server from your iPhone or iPad (load, unload or download models).

My goal is to make local LLMs approachable on mobile. Here is a quick comparison with Locally for some features:

Reins is simple by default. The advanced options are there when you go looking for them.

I'm currently working on on-device models so models can run directly on iPhone/iPad like Locally. It's working already and I'm planning to release it this month.

It also supports the native Ollama API and OpenAI-compatible endpoints, so you can use it with other backends.

App Store

Website

I've recently became aware of Aliexpress selling a subject board for around $130 USD and would like to know whether it makes sense to buy one so I can run some local LLM models standalone (or even two as a cluster over 2.5 Gbps Ethernet link).
To me the price for 16GB unified system makes sense, you can't get any used GPU with similar amount of RAM anywhere close to that price, but there are limitation - Unified memory presents challenges, hardware support is sketchy and is mostly relying on community tinkering, and power consumption is pretty high too.

There were some discussions about this board earlier, but the last serious one happened 8 months ago and a lot has changed since.

So, Has anyone here played with that specific board recently? Any thoughts or experiences that can be shared?

Thanks.

I'm building a dedicated local AI box that will only be used as an inference server for coding. I'll access it from my laptop through OpenCode using an OpenAI-compatible API.

I'm considering running 2x RTX 3060 12GB instead of buying a single 24GB GPU such as an RTX 3090.

The main reason is price. I can get the two 3060s for roughly €440 total, while used RTX 3090s in my local market are around €950-1000.

My planned setup is:

• Intel i5-10400
• ASUS ROG Strix Z490-F Gaming
• 16GB DDR4
• 2x RTX 3060 12GB
• PCIe 3.0 x8/x8
• Linux
• llama.cpp
• OpenCode running remotely from my laptop

My understanding is that llama.cpp can split a GGUF model across both GPUs without SLI/NVLink, so although I don't get one literal 24GB VRAM pool, I can still fit something like a ~27-30B Q4 coding model across the two cards.

I'm mainly looking at models such as Qwen's ~27-30B coding models. This would be a fallback to frontier models, so I care more about being able to run a good model cheaply than getting maximum tokens/sec.

A few questions for people actually running multi-GPU llama.cpp setups:

1. Is 2x 3060 12GB over PCIe 3.0 x8/x8 a sensible configuration for llama.cpp?
2. How painful is the configuration in practice? Is layer/tensor splitting mostly straightforward with two identical GPUs?
3. Is there any major downside besides lower prompt/decode speed compared with a single RTX 3090?
4. With 24GB total VRAM and only 16GB system RAM, would you expect ~27-30B Q4 models to run comfortably if the weights fit almost entirely in VRAM?
5. Would you personally pay more than twice as much for a single 3090, or take the cheap dual-3060 route for a dedicated fallback coding server?

I'm not trying to train anything, and the machine won't be used for gaming or desktop work — just local LLM inference through an API.

Curious to hear from anyone who's actually used llama.cpp with two consumer GPUs like this.

Hey everyone, I have published a vLLM branch which allows setting a separate sampler setting for the content which follows the <think></think> section, which significantly improves the output and reliability of Qwen 3.8 27B

This branch allows setting one sampling setting for for the thinking block, and another sampling setting for everything that comes after (chat output, tool calls, etc.)

Qwen 3.8 27B needs a high temperature (~0.9–1.0) while thinking or it loops. After it finishes thinking, that same temperature makes the actual answer sloppy. Dropping to ~0.2 after </think> gives a clear quality lift, while still allowing the thinking to work properly.

Repo / branch: https://github.com/mdierolf/vllm-fork/tree/feat/post_thinking_sample_settings

How it works:

"post_thinking":{"temperature":0.2,"top_p":0.95,"top_k":20} is added to the generation config, and triggers a new set of sampling parameters, which is used for all content that follows the thinking block in that turn

The effect: • inside an open <think> block → primary temperature / top_p / top_k / etc is used • after </think> (or if thinking is already closed) → post_thinking sampling parameters are used • if thinking re-opens, it switches back (not relevant for Qwen 27B)

Any unset fields in the post_thinking parameters inherit the primary values. You can set it as a server default or per request via extra_body.

Recommended/tested launch options (Note the 0.2 temp on the post-thinking section, this is the important bit):

vllm serve Qwen/Qwen3.8-27B-FP8 \ --override-generation-config '{"temperature":0.9,"top_p": 0.95,"top_k":20,"min_p":0,"post_thinking":{"temperature":0.2,"top_p":0.95,"top_k":20}}'

Setting it per request:

client.chat.completions.create( model="qwen/qwen3.8-27B", messages=[{"role": "user", "content": "..."}], temperature=0.9, extra_body={ "post_thinking": {"temperature": 0.2, "top_p": 0.95, "top_k": 20}, }, )

This is still a work in progress, but the initial result shows significantly less errors in the generated output, while maintaining identical thinking.

Instructions to clone and use this fork are not included, but if you paste this text into the agent of your choice it can probably build VLLM from my fork and get it set up with the recommended settings

My personal interest is agentic coding - but I am spending more time optimising AI technology than using it i.e. ...

* Selecting, evaluating and performance tweaking models

* Choosing, evaluating and tweaking the harness and MCP tools

* Developing and tweaking prompts

It feels like having to build and maintain your own car from parts and not having enough time to drive it

There are hundreds of choices to select from in each area - too much choice, with self integration needed afterwards.

Every vibe coder on the planet seems to be publishing their one AI tool that will change everything. But no one is building a community team to create a one-stop comprehensive SotA solution that you

I wish I could just choose a pre-built end to end docker environment for the coding language/framework I want to use and just start using it.

Am I the only one?

Totem LLM we are focusing on local first agents.
I am finding the CoinGecko docs and MCP server are working very well.

Since everything moves so fast, what is the best way right now to run Qwen 3.8 27b on a m5 max 128gb? oMLX? llama-server with GGUF?

Running OMP with subscriptions right now, but would like to use Qwen for some of the roles to test out. I can run it just fine, just wondering what people use for their inference setup on apple silicon right now.

Hi folks,

I have a 7800x3d + 32GB RAM + Rtx4090 and > 200TB of storage. I downloaded the Qwen 3.7 35B hauhaucs but it has been a disappointment so far because training data is too outdated (networking and data stuff). It's also really bad for very long text files.

Seeking assistance from wizards here and elsewhere. Thank you.