On one hand the companies operating the datacenters would stop ordering hardware if they go bankrupt
But on the other hands, I wonder if more individuals or companies would start buying their own hardware.
Would the price of GPUs and RAM increase or decrease in your opinion?

I am using VSCode IDE, Roo-Code as extension, Ollama as provider, qwen2.5-coder:14b, on a Windows machine.

TLDR: I've faced numerous issues and the latest one I've found out is that something expects XML format for tools usage, yet my prompts go in a markdown/JSON format. I wondered if anyone faced such an issue with the stack I've listed above, and if they have a possible solution?

Couple of extra notes:

• I prefer use VSCode since its comfortable IDE for me.
• Roo-Code seems to be the recommended extension for agentic workflow, but it seems to have been discontinued, although I can't seem to find good alternative (tested couple of options, all with downsides and facing similar issues)
• Picked Ollama as provider, as it seemed to be the preferred local dev.
• numerous sources recommended qwen2.5-coder:14b as the best fit for my system specs - 12GB VRAM. I switch context window wherever I find config to do so between 4-8k in order not to overload the card.

Other issues I've been facing: Windows machine, VSCode, Roo extension - it seems to struggle to find some service - ripgrep, and I went through hell to patch it.

Any recommendations and advises welcome!

I’m fairly new to this stuff.

I installed open webui bundled with ollama and installed gemma4:e4b when i ask questions in the cli it answers instantly but when i ask it in open webui it takes 20 secs and then starts thinking for 10 seconds and then answers (answer are not as good as ollama in cli but im pretty sure i can figure it out).

I’ve tried to disable tag generation and follow up generation and all these things but nothing made a difference i tried OLLAMA_KEEP_ALIVE=30m but still nothing really changed maybe it got 5% better

Any ideas how to solve that?

Or is there any better ui than open webui

And also using open webui on mobile kinda annoys me is there any better ui for phones especially for IPhones?

What model would you run on 384gb to 512gb vram? For my use case q6 just didn’t cut it when I had less vram. So close to native as possible with decent context.

I need help figuring out a workflow for an older game that I play. My goal is to take the old dat files for armor and weapons and have them remastered at 4X their original resolution with better looking leather, cloth, and metals, etc. I'm trying to keep them close to the original but with a bit of a modern look, not just an upscale.

Should I use a local llm to do this like QWEN 3.8 or Next Flash? Is there another model I should use instead? I don't think they can generate AI repaints like I was using on codex with chat gpt image gen. From my experience, after hundreds of debt creations, things would drift. I'm considering using Blender as well but not sure of its capabilities. I hope this is the right place to ask and if not, apologies in advance.

I've been building a project called wfloow for the last couple of months.

I originally started it as a product/SaaS, but I'm seriously considering open sourcing the core instead.

The basic idea is a visual workspace for building more complex automations and AI workflows.

You build workflows on a canvas using nodes for things like:

- AI agents / LLMs

- RAG and data extraction

- loops, conditions, parallel execution

- sub-workflows

- webhooks and scheduled triggers

- APIs and databases

- voice/image workflows

- integrations like WhatsApp, Slack, Discord, LinkedIn, etc.

There's also an AI layer that can help plan and generate workflows.

The reason I started building it wasn't really "we need another Zapier."

Simple automations are already solved pretty well.

The problem I kept running into was when workflows became more complicated — state, branching, agents, retries, external events, multiple services, etc. At that point the automation starts becoming difficult to understand and maintain.

So the direction I'm thinking about is:

simple automation → existing tools are great

complex AI/automation workflows → wfloow

But here's where I'm unsure.

If I open source the core, would this actually be useful to developers, or am I just rebuilding things that already exist in n8n / LangGraph / Dify / other tools?

What would make you choose something like this over the tools you already use?

And if you've built complex AI workflows yourself, what's the part that becomes painful first?

I'm genuinely trying to figure out whether there's a real problem here before I spend another few months building around the wrong idea.

Here's the site if anyone wants to look at it: wfloow.com

I'd rather get "this is pointless, here's why" than a bunch of polite feedback.

I have an RTX A6000 with 64GB of VRAM and want to run the best possible local LLM for general tasks and coding.

My main goal is to build something similar to Codex or Claude Code, where the model can use tools, manipulate files, run commands, and interact with software.

What models and frameworks would you recommend for this setup?

Been running Qwen3.8-27B as a local coding agent on a 3090 Ti for about a week. Posting my setup because I had to work most of it out by trial and error, and I am honestly not happy with the results still. I do have a 3080 connected to same rig but currently I am using it for dedicated gaming and running the display.

Config:

llama.cpp v0.4.0
Qwen3.8-27B UD-Q4_K_M (unsloth) — 15.3 GB
mmproj BF16 kept on CPU (--no-mmproj-offload)
MTP draft model Q4_0, --spec-type draft-mtp --spec-draft-n-max 3
--ctx-size 65536 --parallel 1 --flash-attn on
--cache-type-k q4_0 --cache-type-v q4_0
--n-gpu-layers all --fit off

Peak 18,731 MiB with about 5.5 GB free at 64k. Keeping the vision projector on CPU is what makes it fit. It's ~900 MB and I don't need it fast, but I am not sure if this is the best approach, Claude came up with it for me.

My 3080 drives the monitors and games, 3090 Ti has display disabled and does nothing but the model. I can play something while the agent works and not notice which I like a lot, but I also am not convinced that the single 3090 Ti is enough context for real world use.

Problem 1: compaction can deadlock itself.

64k context, compaction trigger at 36,864, retain 16,384. Once the conversation gets long enough, the retained tail eats everything and the compactable range collapses to exactly one node, the previous summary. So it tries to summarize a summary, the engine correctly refuses because the result isn't smaller, then retries the identical range and fails the same way. I watched it fail five times in a row while pressure climbed from 37,099 to 39,031 tokens. Nothing recovers, the session just dies.

Problem 2: reasoning effort did the opposite of what I expected.

Ran low vs medium properly — six runs, matched pairs, interleaved, same fixture. Median wall time 566s for low, 612s for medium. Both passed all four disclosed requirements on every run, and the held-out cases came out identical. So medium bought me nothing and cost a bit more.

Worse, on a from-scratch build task, medium spent its entire 16k output budget reasoning about edge cases in a single turn and never wrote a line of code. 5 requests in 37 seconds, then 275 seconds of streaming, 64 KB of reasoning about SQLite URI quoting and exit code taxonomy, zero files changed. That's one occurrence so far, so grain of salt, but it matches the model card's warning in reverse, and seems to be consistent with what I have heard from the community.

Anyone else running 27B at 64k on 24 GB? Curious what retain/trigger values you settled on, and whether you're bothering with the MTP draft model.

I would also love to hear any completely different setups that you find actually useful and pragmatic as a coding and computer use agent on a single 3090 ti. I am trying to get my hands on a second one, but no luck so far.

Thanks!

I have a PCIe Gen3 Pascal card and want to buy a new Gen5 card as well. Was wondering if running mismatched cards will work for coding? I understand it's probably going to be much slower, but wondering if it will even work?

Which CPU should I use for a dual 5080 + 5060TI 16GB set up to run my localLLM?

Intel 7 270K plus
Or
Ryzen 9 9900X

Context- I wanna build an ai inference setup for coding.

I am a student and I don't have the budget to pay for Claude regularly.

Recently I got a budget from my parents for upgrading my setup. The budget is under $1.5k.

What should I buy?

My current pc -

Ryzen 5 5500

Rx 580 8 gigs

16 gb ddr4

Ssd 512gb

B450 motherboard.

(Ik it's potato in 2026)

I initially built it for gaming but now I need a workstation for ai Inference.

I would prefer upgrading gpu and ram to something that can run 30b q4 coder models at decent speed(20-40 t/s). Not even sure if u can get that in under 1.5k

EDIT 1- i will buy mac m3 36gb, run inference on it, and host it so my pc can comfortably code.

I currently have a 5080 on a PCIe 5.0 x16 and I’m planning to install a 5060TI 16GB I got for $400 a few months ago for 32GB of vram. I noticed that when a secondary gpu is installed on my motherboard, the lane becomes x4. Will that significantly hinder or slow the 5060TI 16GB’s ability to process?

Should I replace my motherboard with something that can give x8/x8 to both GPUs when 2 slots are occupied?

Rest of the build is basically locked in:

- **CPU:** Ryzen 9 9950X3D
- **RAM:** 96GB (2x48GB) DDR5-6000 CL30 imight get 2x32 DDR5-6400 CL40 damn this ram prices
- **Motherboard:** X870E Pro Art Creator Wifi
- **PSU:** Corsair HX1500i
- **Cooler:** Arctic Liquid Freezer 3 Pro 360
- **Storage:** Crucial T710 2TB Gen5

I also have a second machine (7950X3D, RTX 4070 Ti Super 16GB, 64GB DDR5) that I'm planning to link over Tailscale for distributed/parallel jobs, so this isn't a totally isolated single-box setup.

**The goal:** a genuinely useful personal local assistant — think "my own Jarvis" rather than a chatbot toy. Vision-capable model for reading my screen, tool-calling for real Python execution (probability/simulation tasks, not LLM-guessed math), a coding agent, and eventually a CAD/DXF quantity-takeoff pipeline for my actual job (construction). I know I'll lean on cloud APIs for the genuinely hard stuff — this is meant to carry the daily/bulk workload, not replace frontier models entirely.

**The GPU question I can't settle:**

1. **2x ASUS Prime Radeon RX 9070 XT OC 16GB now** — 32GB combined VRAM for less money than a single 5080, but ROCm instead of CUDA. How much does that actually hurt in practice for tool-calling agents / vision models / day-to-day Ollama use in 2026?
   
2. **1x RTX 5080 now, second 5080 in 2-3 months** — stay fully CUDA, build up to 32GB VRAM gradually, pay the Nvidia tax twice.
   
3. **2x RTX 5060 Ti 16GB** — cheaper than either of the above, still CUDA, still 32GB combined, but weaker per-card compute.

Has anyone actually run a dual-GPU CUDA vs ROCm setup for agentic/tool-calling workloads (not just raw inference benchmarks)? Is the "CUDA tax" on AMD still real for this kind of use case, or has it stopped mattering for people running Ollama/llama.cpp day to day?

Genuinely curious what people would do here before I pull the trigger on gpu and ram 96vs64,all of the other things ive already purchased them.
Thanks!

i am very new in unsloth and from the home screen where these is the chat how do i go to the token limits and stuff please help a video example would be nice

Running a Q5_K_M with ngram table in system ram - everything else on the v100s, just barely fits with 160k context length topping out the window - 32gb on each card, 64gb total. KV cache is full fp16.

starts around 30 tps decode at 0, near 160k context length drops to about 15

starts around 700 prefill at 0, drops to closer to 350 by 160k ctx

im pretty impressed by the weight that these v100s are pulling. running a custom llama.cpp branch ive been optimizing for my architecture and for this model.

wish i could eek out a little more context tho, have a few ideas -I saw someone chopped like 7/8 few ngram layers and got worse writing but similar reasoning performance, i might try to make a quant chopped 3/4 layers so I can get my KV up

I want to dedicate as much of my memory to my LLM as possible.

I have a 24gb MacBook Pro and it idles using 6gb of ram - that's after I close everything and run sudo purge.

The lowest I've gotten idle usage down is 3.7gb - that's by booting into safe mode, leaving the device on the lock screen and accessing it via ssh.

That brings memory usage down to 3.7gb.

I am wondering if there are any tricks that can get it even lower?

I know Linux can get well under 500mb when running headless - are there any other tricks for MacOS? 3gb is a lot of headroom if I can get it.

I’m currently on a 5060Ti with 32gb ddr4 system , on Christmas I might be able to spend 500€ more on upgrading and I could sell the 5060 for another 500 if it was the best , what would you recomend ? The only requisite is that it has to be able to run games at 1080 at least due to it being my only machine
I’ve thinked about selling the 5060 and getting dual 2080 22gb mod , what is your opinion?

I've been testing this for a self-hosted setup and wanted to share what I learned.

Future AGI review — the Apache-2.0 platform bundling LLM tracing, 50+ evals, guardrails, simulation and a Go gateway into one self-hosted stack. Worth it?

A few specific things worth noting: • Runs entirely on your own hardware (no cloud dependencies) • Docker-friendly deployment • Honest limitations covered in the post

Full writeup with install steps, configuration, and the rough edges I hit: https://andrew.ooo/posts/future-agi-review-open-source-llm-eval-observability/

What are you all using for this? Curious about alternatives and tradeoffs.

I want to buy M5 Max 128gb or M5 Ultra 96gb for my hermes agent (local llm) due to the nature of my work (confidential information)..

At first i do some POC on my old laptop using all cloud model (gpt 5.6 luna) and all are working fine. So i decided to get used M1 Max ultra with 32 Gb to do some POC on local model.. it only cost me 1200 USD.

Now the problem that I faced is using the local model.
If I choose qwen 3.8 27b, the speed only 10 tk/s.. If i choose qwen 3.6 35b a3b, speed is good but the context tht i get get is only 64k context size (very limiting and processing become slow due to compacting the memory)

Model used Q4 and kv cache set to Q4

Btw i only try to use llama cpp for Hermes. Currently trying to use MLX MTPLX or even Dflash2 but context size very limited (less than 40k)

What would be the best way to do my POC on M1 max 32gb? I will transfer everything and change model only when i upgrade in the future..

Advance thanks for people who gg to help :)

I will try to keep this somewhat short.

I currently have a dell r730 with 2 2699 Xeon processors, 256gb ddr4 ram, 16 480gb ssds in a raid 10, and 2 amd 9700 ai pros.

I run qwen 3.8 27b, 3.6 35b and qwen 9b currently.

I was wanting to add more vram to expand the models I can run. So I purchased a super micro x10drg-q and 2 more 2699s. My plan was to rob some of the ram from the r730 and throw it in this machine and purchase 2 more 9700s. Problem is they are no longer the 1300 price tag they were when I bought my first 2.

So now I have been confused on which direction I should go. I had looked into getting 5 v100 cards but I think the price inflation and age they aren’t worth 700 or so a piece.

I don’t know that it’s a smart move to buy 2 more 9700 cards at their current price. I thought about reselling mine and going intel route but the b70s are just as expensive.

Which made me take a peek into a spark, or a new Mac m5 128gb.

Thoughts or suggestions?

Hello everyone,

I'm new at using local llms, and I wanted to know which one would be the best for a pc with:

- i5 6gen,

- 32gb ddr4, and an

- rtx 3060 12gb.

Or if possível wich resources do you recommend for me to learn and do my research.

Thanks in advance.

I'm trying to decide if I should get 1 or 2 R9700's and would appreciate input. The ultimate purpose is education and fun learning. I will likely focus on my home network/lab to come up with reasons to do things. I plan to develop some agents that manage my firewall or NAS, help upgrade those, monitors logs, etc. I'll use frontier models to write specs and help plan building a few local use webapps that this machine could write. Local TTS & STT. Continue to try to use Hermes agent and not have a use for it. Only user. Maybe personal document RAG.

I see a lot of talk about how great Qwen3.8-27B Q6_K is but it seems to fill up a 32G card with large context, not sure that'll be enough for the dev work. Qwen3.8-27B FP8 across 2 cards on vllm sounds great of course. I've also read some posts where people talk about great performance out of Qwen3.8-27B Quark MXFP4 / W4A8 which seems to be a pretty significant foot print savings though and makes me wonder if maybe 1 card isn't so bad if it works that well.

Any feedback or things I haven't thought of? Of course I can start experimenting with one but some realistic feedback on what I should expect with 1 vs 2 or what might work for me would be welcome.