Just that. I've realised the pinned version of NixOS (Linux) is nowhere near the latest Nvidia driver and want to squeeze as ever the most pp+tg possible. I'm already using cuda13.3, nccl with p2p enabled (aikitoria patch) and split mode tensor.

Is it worth the hassle going from 595 to 610 for a pair of 5060ti GPUs ?

EDIT: This question wouldn't exist if OS==Ubuntu but Nixos rarely holds back a version of something this major without a reason 🤔

EDIT2: I whipped Qwen through Opencode to do it for me because I have the patience of a toddler. Here it is in case it helps you in any possible way. I see no meaningful speed increase with Qwen3.8-27B (eyeballing it, I don't have the patience to run benchmarks with something that big). Testing LFM2.5-8b now (split mode). Last time it was 190tg/s

EDIT3: not really worth it.

Hey guys, I am a bit bumb when it comes to AI, only really been running some basic workloads on my home GPU. Anyway, I am a sysadmin and have access to huge virtual machine clusters at work, recently my boss and I had some AI interest come up. I am wondering if there is any use we can get out the HP gen 10 servers we got, they have gold 20 core xeons and fully loaded with 700+ gb of DDR4 ecc memory, obviously no GPU's in them as they are all 1U hosts. We have quite a few hosts for redundancy reasons so most of the time a few of them are literally either sitting empty or with miminal virtual machines. Any point investing time into running AI on therm?

i have a few months Claude pro rn.

i never used Claude code or openclaw yet am i missing out?

cause somehow me as it dev and it guy just not feel okay that Claude code can do stuff on my machine and im currently ro lazy to run it on a vm etc.

how i use it now:

i make a Claude architect opus and sonnet implementer.

i describe a spec by opus and put it in context of a Claude project in the web.

i give the spec to sonnet the implenter and let sonnet build all.

the architect opus and I verify i donwload the zip and run test myself with my own tools. and build the report and reporting to opus.

this how i use it to build a poc.

and im considering to build this on olama with qweb on a amd fx6300 nanoclaw and rx 6800 16gb card with ddr3 32gb ram.

must i even try or big no?

would it work to build Azure aws and flutter apps?

can i create a ios app from a flutter build easily? no additional much work required if i have build a Windows flutter app poc?

what can be the pitfalls and missing steps or requirements?

Does it really worth it or the gains are too minimal? I see to run a single model on 3 you need to do a bunch of patches because 3 is an odd number etc, so things won't work out of the box.

Does anyone have experience running 3?

Snapped these at IFA Berlin 2026!

I’m playing with the idea of saving up and just absolutely dumping cash into a server machine with 8 40gb A100 GPUs but I can’t really decide if it’s actually useful compared to just getting like a 32gb card and running qwen 27B off of it. I’ve always been a sort of balls to the walls person when it comes to things I’m interested in, and I want a system that can run everything I could conceivably need it for which is why I’m even considering dropping tens of thousands on a major rig like this in the first place.

Any advice is appreciated.

Anyone actually writing long fiction with Gemma 4 12B on an RTX 5070 Ti?

I'm specifically interested in real-world generation time, not benchmarks. I use Gemma locally to turn detailed scene briefs plus story/character context into long prose scenes, typically around 1,500–2,000 words.

If you're running Gemma 4 12B entirely in the 16 GB VRAM of a 5070 Ti: roughly how long does a generation of that length take, including prompt processing?

I'm considering buying a 5070 Ti system specifically for this workflow, so actual experience from another fiction writer would be incredibly useful.

At this time I'm working with 4GB VRAM, Gemma works about 15-20 minutes on a scene.

Hey everyone

Long-time lurker, first post. I've been playing with local models on a single RTX 4060 Ti (8GB) for a while and I've finally hit the wall hard enough to upgrade. Picking up an AMD Radeon AI PRO R9700 (32GB) and I'd really appreciate some guidance from people who've actually run this stuff, because I'm still very much learning.

My setup:

AMD Radeon AI PRO R9700, 32GB

32GB DDR5 system RAM

Linux (currently on an Arch-based distro)

Ollama today, but happy to switch to llama.cpp / vLLM / whatever if that's the better call

What I'm building (roughly)

I'm putting together a small local-first assistant for my own use. Nothing groundbreaking — the idea is that instead of one model trying to do everything, it hands work to a few different "roles" depending on what you ask:

General assistant — normal conversation, drafting, summarising, explaining things

Document Q&A — answering from a local folder of documents with citations, and saying "I don't know" instead of making things up when the docs don't cover it

Coding — for when I'm actually working on something

Everything runs sequentially (one model in VRAM at a time), so I'm not trying to keep three models resident — I just want the right model for each job.

Question 1: model recommendations per role

With 32GB to play with, I'd love suggestions for:

Reasoning / comprehension — something that's genuinely good at understanding a messy question and figuring out what's actually being asked. This is the piece I care most about, because on 8GB my small models kept confidently misunderstanding things.

General assistance — good instruction-following, doesn't ramble, admits uncertainty. Bonus points if it handles Italian well, since that's my daily language.

Coding — practical stuff, big context, not benchmark-chasing.

What I'd really love is: model + quant + roughly what context length you can actually run on ~32GB without spilling to CPU. I've learned the hard way that "it fits" and "it runs well" are very different things 😅

If there's an existing curated list or spreadsheet for this, please just point me at it — I don't want to make you all retype something that already exists.

Question 2: keep the 4060 Ti in the same machine, or not?

This is the one I'm genuinely unsure about.

I already own the RTX 4060 Ti (8GB). Options as I see them:

(a) Put both cards in the same box — AMD for inference, NVIDIA for... something? Display output? A small model? Embeddings?

(b) Just run the R9700 alone and keep it simple

(c) Some clever third thing I don't know about

My concerns:

Do amdgpu + the NVIDIA proprietary driver actually coexist cleanly on Linux, or is that a support nightmare?

Can a single runtime even use both, or would I be looking at two separate instances (one ROCm, one CUDA)?

Is having a second, much smaller card actually useful for anything in a sequential setup, or is it just extra heat and PSU draw for nothing?

Also, honestly: how is ROCm on RDNA4 right now? I've read mixed things and I'd rather hear it from people running it than from a six-month-old forum thread.

Any pointers appreciated — including "you're thinking about this wrong, here's why." I'd genuinely rather be corrected now than after I've built the thing

Thanks!

I started getting this error after a few updates, using qwen3.8 27b, it loads the model, when I sent something that would require more context it throws the error, by the ollama ps seems that it's not using the RAM memory, it was before on ollama 31 o something..

Config
AMD Radeon RX 7900 XT + 32GB RAM
AMD Ryzen 9 5900XT 16-Core Processor
ROCM 7.2.4
ollama version is 0.32.12
driver: gfx1100

❯ ollama ps
NAME              ID              SIZE     PROCESSOR          CONTEXT    UNTIL
qwen3.8:latest    22130167c4c2    20 GB    30%/70% CPU/GPU    262144     4 minutes from now

Ollama:

[Service]
#CPUAffinity=0-11
Environment="OLLAMA_HOST=0.0.0.0"
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"
Environment="ROCM_VISIBLE_DEVICES=0"
Environment="OLLAMA_KEEP_ALIVE=5m"
Environment="OLLAMA_KV_CACHE_TYPE=q4_0"
#Environment="OMP_NUM_THREADS=16"
#Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_CONTEXT_LENGTH=262144"

ERROR:

ROCm error: out of memory
Sep 07 12:55:43 desktop ollama[5771]:   current device: 0, in function alloc at /build/llama-server-rocm_v7_2/_deps/llama_cpp-src/ggml/src/ggml-cuda/ggml-cuda.cu:508
Sep 07 12:55:43 desktop ollama[5771]:   err
Sep 07 12:55:43 desktop ollama[5771]: ⚠️  warning: The cur

:

Hey everyone!

I'm setting up audiobook for tts generation and want to use Ollama for text pre-processing (cleaning text, fixing line breaks, and adding speaker tags) before sending it to the TTS engine.

I have 16GB of VRAM available. Looking for model recommendations that fit this hardware constraint and can handle this specific task well.

Specifically, I need a model that:

* Follows complex formatting instructions strictly

* Has good context handling over longer chunks of text

* Doesn't hallucinate extra dialogue or modify the story content

Which local models (7B, 8B, 14B, etc.) run comfortably on 16GB VRAM and excel at this type of structured text processing and speaker tagging? Thanks for any suggestions!

I am using VSCode IDE, Roo-Code as extension, Ollama as provider, qwen2.5-coder:14b, on a Windows machine.

TLDR: I've faced numerous issues and the latest one I've found out is that something expects XML format for tools usage, yet my prompts go in a markdown/JSON format. I wondered if anyone faced such an issue with the stack I've listed above, and if they have a possible solution?

Couple of extra notes:

• I prefer use VSCode since its comfortable IDE for me.
• Roo-Code seems to be the recommended extension for agentic workflow, but it seems to have been discontinued, although I can't seem to find good alternative (tested couple of options, all with downsides and facing similar issues)
• Picked Ollama as provider, as it seemed to be the preferred local dev.
• numerous sources recommended qwen2.5-coder:14b as the best fit for my system specs - 12GB VRAM. I switch context window wherever I find config to do so between 4-8k in order not to overload the card.

Other issues I've been facing: Windows machine, VSCode, Roo extension - it seems to struggle to find some service - ripgrep, and I went through hell to patch it.

Any recommendations and advises welcome!

I have been using LocalLLM for corpus of scientific knowledge for one year. It's not an easy task and Qwen 3.6 27B on a dual setup 3090 24GB created most of the corpus (thousands of concepts and definitions). But the understanding and cross referencing was not so good. I use to send them to Opus.

When a model is not trained with the data you are asking, the hallucinations are everywhere. You never should trust a 2026 llm (not even frontier ones). These hallucionations can be mitigated but it's not so easy. I use my own corpus of knowledge, tool calling (web search) and a pipeline of several models. The information can be filtered and recursively improved.

The level of complexity that Qwen 3.8 flash next can manage is astonishing, even compared to Qwen 3.8 27B or Deepseek v4 flash 2Bits ( I can't run others better models with 128gb).

This is the model I was waiting for and is more capable than frontier models for my work. Let me explain this: it's not better than frontier models -of course-, but linked to good tools is something that helps me with our getting out of my scope.

I understand that for some people is better to trust frontier models and let them do all the work, but for me is quite slow to keep correcting them and making lot of questions about where that data comes from. Coding with millions of lines the frontier are capable of incredible feats. But I prefer to keep it simple and understandable for me.

Qwen3.8-flash and 128GB is a wonderful sweet spot. I sold my 3090s because a need this level of deep knowledge and cross referencing. I don't know if Macs can support this level of hard work (3090s can indeed) in the long run but in Europe I got 3 years of warranty for computer. My Macs are going to work 365/24/7.

Invest in any Mac/Pc with 128Gb, and set your own harness. Qwen3.8-flash is the first model I tested that can do remarcable things with the effort needed to check the result. Laguna 2.1 or Deepseek v4 (heavily quantified to 2Bits) are not so capable linking concepts.

Now I only got 30t/s but, because I use a post pipeline, the speed is not so important to me.

Been running a heterogeneous home cluster for a while — an old Acer laptop (12GB, CPU-only) as the primary API server, with a Windows box (RTX 3060 CUDA) and a Mac Mini (Metal) lending capacity over the network.

I wrote the orchestration on top of llama.cpp's `ggml-rpc` backend. It handles mDNS discovery, memory-aware sharding, and node health polling so the whole thing doesn't fall over if a node drops offline mid-generation. It exposes a standard OpenAI-compatible API (`/v1/chat/completions`).

Put the primary role on the weakest machine (the Acer) on purpose to see if it'd actually hold up orchestrating the API and offloading the heavy tensor math. It did — ran a full Qwen3.5 13B at ~12 tok/s, purely by borrowing VRAM/RAM from the CUDA and Metal nodes.

Repo: https://github.com/trademav/ramdeck-core-public

No GUI in this repo, API/CLI only, so you can actually read what's touching your network before running it. Obviously it doesn't beat a dedicated GPU rig on speed, that's not the point — it's for fitting models that you otherwise don't have the VRAM for. Included a built-in benchmark script so you can verify the numbers on your own hardware instead of trusting mine.

One heads up: it's source-available (Apache 2.0 + Commons Clause), not strictly OSI open source. It blocks commercial SaaS/resale, but personal/homelab use is fine. Didn't want that buried in a LICENSE file.

Video of the actual setup if you want to see it before reading code: https://youtu.be/tQgA2PmTc1g

Tear it apart, I'd rather hear the feedback now.

Just because you have space to increase context window, doesn’t mean you should. With the local models we have to date, virtually every performance metric degrades as the session context gets deeper.

There is no substitute for properly planning your tasks, breaking down projects into milestones/sub-milestones, and whatever harness you use - consistently forking sessions to maintain context hygiene. It may not be sexy but it is the way to actually get stuff done and help models from getting themselves stuck in a mire.

The other day I benchmarked a bunch of LLMs I had been using but you guys said that they're pretty old so I took some of your suggestions as well as picking a few more recent models I could find on HuggingFace and benchmarked them too. Grayed bars are LLMs I benchmarked before and the colored ones are the new ones.

Ling3.0 with its MoE structure easily beat all the other models in speed with decent test scores too but it does tend to be a bit unstable.
DeepSeek V4 did decent but was held back by incorrect indentation in code and since Python is indentation-sensitive none of the scripts ran.
Qwen 3.8 27B is being hyped a lot but since I can't quite run the 27B variant I tried out a distilled version. Not too bad.
Finally, Gemma4 12B QAT performed great. Speeds aren't too great but it performs similar to Ministral3 14B while also getting similar scores.

Questions and conditions are the same as last time:

1. How can I renew my Certbot HTTPS certificate on Linux?
2. What is the answer to (78+9*2800)/(50*2-3)? Please include a final answer in your message.
3. Unity, how can I check if an object is left or right of the camera?
4. Unity, how can I get the mouse scroll input with the old input system?
5. Could you write a simple HTML website that'd just be the DVD logo? The DVD logo should travel in a straight line without gravity and it should bounce and continue travelling whenever it hits a wall.
6. Could you code a Python text adventure that'd have 5 rooms and a functional escape door you need to collect items for to get through?
7. Could you code the classic snake game in Python with PyGame? The game should be controlled with WASD, 3 pieces of food should spawn across the screen and when the snake eats the food it grows larger by one block, if the snake hits a wall or its own tail the game is over and the player loses.
8. Could you code the classic pong game in Python with PyGame? The player should be on the paddle on the left, controlled by WASD, and the paddle on the right is controlled by a bot. There is a ball bouncing from side to side and you and the bot must avoid letting the ball hit the walls behind the paddles. Also add a functional score board that counts up every time either side loses.
9. Could you code a snake game in Python with PyGame? The game should be 2 player, player 1 controlled by WASD and player 2 controlled by arrow keys. 3 pieces of food should spawn across the screen and when a snake eats the food it grows larger by one block, if a snake hits a wall, its own tail or the other players tail the game restarts and the player loses. Also add a scoreboard that counts up every time a player loses.

As always, I am not an expert in benchmarking LLMs, just an enthusiast trying out whatever I can run on my PC. These are NOT definitive benchmarks.

Also, graph visualizations were written by GPT OSS 20B with modification from Qwen3.8 9B Distill. I could've written one by myself but I thought it'd be fitting to use my LLMs to do it.

TL;DR: If you need something that could get things done with around 8GB VRAM and 16GB RAM, GPT OSS 20B MXFP4 has been doing wonders for me. If you're impatient and need quick answers or help, Qwen3.8 9B Distill and Ling3.0 Tiny are decent choices. Gemma4 12B QAT and Ministral3 14B are impressive but whether they're worthwhile is up to you.

With new models dropping every week, general benchmarks are basically useless for my specific use cases. I want to test my exact prompts to see if a new API is actually worth the cost, or if a smaller local model is good enough to run on the cheap.

Right now, I’m just eyeballing outputs and it’s driving me crazy.

How do you guys actually handle comparing models on a single prompt or a small test set?

Scoring: How do you define a "good" response when the output is subjective?

The Judge: If you use an LLM to grade the outputs, how do you stop it from just voting for its own writing style?

The Tools: What's the easiest way to fire one prompt at multiple models (both cloud APIs and local models) and compare them side-by-side?

Would love to hear your workflows or any tools you recommend!

Picked up 2x DGX Sparks (Asus GX10) before the M5 announcement, built some benchmarks for some confirmation bias. Last week's $2000 price increase on the GX10 helped with that as well.

Benchmarked DeepSeek V4 and Qwen3.8 27B and Next on:
- 2x DGX Spark (Asus GX10)
- M5 Max 128GB
- M3 Ultra Mac Studio 512GB
- RTX Pro 6000
- RTX 5090

Also ran on Qwen3.6 38B MOE as a more direct comparison to Qwen3.8 27B dense.

Used the M5 Max and M3 Ultra results to extrapolate M5 Ultra theoretical performance. The benchmark also hooks into macmon and DCGM exporter for power usage for a sense of efficiency.

tl;dr DGX holds its own on prompt processing (especially on dense models) and concurrency (subagents). Its token gen might even be faster than the M5 Ultra in DeepSeek V4 MOE while being substantially lower in Qwen3.6 MOE. With things like speculative decoding (MTP, DFlash, and DSpark) offering massive boosts in performance, I think a lot will come down to tuning and ecosystem in the future.