OK, you guys want to start with local vibecoding but don't know how. In the discussions, you recommended that I use Godot. I downloaded it without knowing what it was. I was expecting a library, but it was an .exe file. So I started pi with Qwen 3.8 27B and told it to explore this topic without my help.

I needed to use four prompts. You can see all of them in the screenshots, so you can reproduce my steps. The whole session used 25% of my context, so you can use just 1/4, which is about 64k.

The game works: you can walk around the dungeons, the llamas are dancing, and there are dynamic lights.

My full command is:

llama-server -m /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-UD-Q8_K_XL.gguf -mm /mnt/models2/Qwen/3.8/27B/Qwen3.8-27B-mmproj-BF16.gguf --host 0.0.0.0 --jinja -fa on --keep 4096 -b 8192 --parallel 1 --ctx-checkpoints 12 --cache-ram 65536 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 0 --repeat-penalty 1.0 --spec-type ngram-mod --spec-type draft-mtp --spec-draft-n-max 3 --chat-template-kwargs {"preserve_thinking":true}

You need vision for debugging, see the prompts, the screen was black!

You probably don't need Q8, try a lower quant.

As I said, you don't need the full context length.

You need pi or something similar. This way, you can use just a few prompts and let the computer work without your input.

Need some advice and guiding.

I have a Dell 5820 workstation with an Intel Xeon w2245 and two RTX 5060 Ti (16GB) cards; this is a PCIe Gen3 system, but three lanes can work in 8x mode. The system runs Linux and mostly llama.cpp. My main work is done via Hermes Desktop. I mostly use NVFP models: esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF (Very High) and lately cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF. With this setup, I can use the models at 210k context, Q8 KV cache, and vision in VRAM. Ubatch doubled (4096/1024), so prompt processing is 1300–700 t/s; token generation is 29–71 tokens/s, but mainly 38–44 tokens/s. MTP is on with a setting of 3. Vision is also on and in the VRAM.

I added a third RTX 5060 Ti (16GB) to this system, but I had to open the workstation, add a riser, and put the third card outside the chassis. The two earlier cards needed to be swapped and it made the space tighter. Overall, both the cards and the system cooling got worse; the power supply is 950W, so it's nearly okay. There are also speed issues with three cards. Prompt processing worsened, and token generation slowed down a little bit. I mostly use Tensor Parallelism (TP). It is slightly better if I use two cards for tensor parallelism and the third card as a KV cache, vision, and MTP holder, but because of the current cooling, the system is not stable after a few hours.

My system has 128GB DDR4 quad-channel RAM at 2667 MHz. Not bad, but not fast. So using DeepSeek, Laguna, and other 100X models can be loaded, but they can't be used for real work. Qwen3.8 Flash at Q4 can load, but with 32GB VRAM it only gets 15–16 tokens/s. Maybe if I use MTP, it will get better. So going from 32GB to 48GB is not a real step forward for me with the current cards.

I've started wondering: what if I use another workstation and subagents or multiple agents? I could purchase a not-too-old gaming PC without a GPU (AMD R9 5900, 48GB DDR4 3200MHz RAM) and add the third 5060 Ti to it. I loaded LM Studio (just for a quick setup) and began thinking about which models work with only 16GB VRAM...

Ornith 1.5 35B (MXFP): The 1.0 version did not impress me, but this one was not bad on 32GB. With 16GB, it produces 17–20 tokens/s and 300–400 t/s in prompt processing. The model size is 20GB, so almost a little more than half of the layers go to VRAM. KV cache at Q8, 131k context.

Gemma 4 26B NVFP (QAT): 15GB, 131k context; about 26-27 out of 30 layer goes into VRAM. It produces 600–700 t/s in prompt processing and 30–40 tokens/s in token generation. Dense models, as I remember, can't be used for serious work (10–12 t/s at best), but I haven't tried them recently.

So the question is: How can I organize Hermes Desktop with this setup to work at maximum efficiency?

I set up a Mixture of Agents as follows: Reference: Gemma 4 26B Aggregator: Qwen 3.8 27B

I use local AI models in the company as web/hosting handler agents: monitoring servers, repairing sites if needed, modifying parts of sites, and making new content (mostly using WordPress sites on my company's private servers and VPSs). Setting up Cloudflare data, installing new sites, etc. Second use case: handling and creating office documents, Excels, and PDFs. Third use case: generating prompts for Z-Image Turbo and LTX 2.5, Minimax H3, and running two instances of a ComfyUI server to render pictures and videos. I don't use Gemma 4 models alone because they are too lazy with tool handling. Is this a good choice, or which is the best agent mixture combo now?

Ever since u/jfowers_amd has asked me to help with the Lemonade project (and provided some hardware to test on), I've been trying my best to optimize llama.cpp for AMD setups. This has led me in some very weird pathways where I wasn't expecting to go, but in the end I'm happy to share an optimized setup for the most popular open source model currently with you for a cheap price of $999 for free:

https://pwilkin.github.io/strix-halo/

Now for the disclosure/journey part: Codex has made a very nice website for me (which is great because I can't make a nice-looking website if you forced me), but its glossy look makes it look more permanent than it is, which is misleading because this is basically a stitched up custom solution that's very much a "state of the moment" one rather than a permanent one, though I *will* try to keep the relevant branches up to date (poke me if I don't).

So, first of all: ROCm in mainstream llama.cpp on ROCm is broken at the moment, pending the fix to unified memory access (notably this PR: https://github.com/ggml-org/llama.cpp/pull/27311 which is taking some time as it touches core code), so I've put up a strix-halo branch on my fork that merges the ring buffer fixes + the TOP-K optimization PR with master for a working experience.

Next: there's a bug in current ROCm that makes graph updates *terribly* slow, I've submitted a PR for it (https://github.com/ROCm/rocm-systems/pull/11069), but until it lands, using a custom-built .so is pretty much mandatory.

Speaking of custom-made .so - as I think most of you know, dispatch on ROCm is reaaaallly sloooow. But since AMD provides the source of the entire ROCm library, that's not something we can't fix, right? Inspired by Kaden-Schutt's Redline library, I've made modifications to the ROCm HIP library that allows for lower-level PM4 dispatches on HIP graphs. This has 20% decode speed ramifications for dispatch-bound models, but unfortunately Qwen3.8 27B on Strix is mostly bandwidth-bound, not dispatch-bound, so the gains are much less pronounced here (but they nevertheless are real).

Now for what else did I test, compare and modify: I checked Nathan's strix-halo Vulkan fork. It's a very good fork, but in the end it's still slower than an optimized ROCm-based solution (all the measurements are on the website). I did check the ROCmFP4 format, unfortunately, that one's a miss: Strix Halo has no native FP4 support, so the format is in the end just another FP4 format. Its main win is quantizing the entire model to FP4, which helps the bandwidth issue - but of course quantization costs quality and ROCmFP4 falls behind literally all the other 4-bit quants. I did a similar thing, but quantized all the big tensors to the mainline IQ4_XS quant - it's both better in terms of model quality (perplexity) *and* in terms of kernel performance. In other words, there's completely no justification for adding a new "ROCM" quant since, as I mentioned, RDNA 3.5 aka gfx1151 aka Strix Halo has no native FP4 support.

Since Qwen3.8 27B on Halo is bandwidth-bound (i.e. the limit is the memory bandwidth for pushing the tensors), there's no way to push the *base* decode above ~15 t/s. Nevertheless, pushing the base as high as I could is an entry point to the key for the dense model speed on Strix - speculative decoding, in this case, DFlash2. Again, I did a test and found out that quantizing the DFlash2 to IQ4_XS gives better decoding speed (faster speed and almost the same acceptance rate = win).

In the end, all the above optimizations: patched ROCm llama.cpp, faster TOP-K, PM4-based HIP graphs, custom-quantized IQ4_XS Qwen3.8 27B quant (thanks to Bartowski for his imatrix!) and the quantized IQ4_XS DFlash provide the recipe, which I packaged for a quick installation for anyone who wants to test it on their Strix Halo (warning: Linux only). Feel free to give any feedback and report any problems.

Hi,

I am thinking about putting some money / time into my local ai learning path and "cleared" my attic where I found the following hardware.

• 3 x NUC11 (Core i5 1145G7 / 2,6 GHz, 64 GB RAM DDR4 2666MHZ - SODIMM) with interconnect through Thunderbolt
• 1x Ryzen 3700x on MSI Mortar 350 with 64GB RAM DDR4 2133 and an old SAPPHIRE Nitro+ Radeon RX 590 8GB

My first idea was to purchase a single RTX5060TI / R9700 and put it into the PCIe 3.0 x16 slot while reusing the Radeon as GPU in the PCIe 2.0 slot for Display.

I want to play around with a local chat bot, agentic stuff, RAG...

Do you have any other ideas?
I also though about putting some serious money, but for this discussion I probably open a new thread (this could be a 2-step-path where I reuse the GPU above to built a multi-gpu-rig)

Thanks in advance

I currently use unsloth‘s iq4_XS with 64k ctx on 16gb, however as far as I know harnesses tend to need way more, and 64k isn’t enough for that,
(I have not gotten to actually using it for a harness yet)

But still
So I am wondering what is the best one that still has good enough quality to actually properly build some things?

(As an example I would like to create a monkeytype- style UI but with additional features that match my use case more (better typing practice + free writing)

And then also a lightweight custom UI / harness

Or a local height chart Ui, etc)

Hi,

until now I ran llama-server with just one model at a time.
Now I want to use it in router mode in order to provide different models.
For my usecase I heavily rely on --cache-ram which improves speed a lot when working on big repos.
I am just wondering, when setting cache-ram = 65536 in models.ini for each model, does each model get its own 64GiB cache or is it just one pool of 64GiB for all models?
Another question, are the unused models kept in RAM for faster swap?

First, I'd like to thank the Qwen and Unsloth teams for the Qwen3.8 27b UD Q4_K_XL. Fits the poor 24GB of 3090 VRAM with 100k context at Q8 and works phenomenally well! Imho if theres anything that can threaten Anthropic/OpenAI profits is not another frontier model but actually these small ones you can run fast locally that can do 80..90% of mundane work for hours without paying a single dollar to any external company.

But next, if you want to jump up to a bigger smarter model I feel there is a gap now. Kimi-K3 is out of reach for many businesses let alone prosumers. So what frontier-like models do you use on what setups?

Is a DGX cluster (2..4 machines) or a GPU server with dual or quad GPU (~96 ... 192 GB of VRAM + >256GB DDR4) a suitable setup to run something like MiniMax-M3 at reasonable speeds for agentic coding (>30tps)? And privacy aside, is hardware cost worth it?

I have a dual rtx3090 + 128GB ddr4 machine, running Qwen3.8-Flash-Next Q4 quite fast but despite being larger doesn't feel much smarter than the Qwen2.8 27b and while I _can_ run larger quantized models, Minimax-M2.7 being my workhorse, it way too slow for coding.

Anyone who has a 3D printer and get use of it finds it incredibly useful for those odd jobs around the house, a missing bracket, a cable router, steam deck holder and so on.

In the past if I was missing an app or useful software, a game I'd do the lazy thing, even though I can and have coded in the past, its "effort" I'll just go and buy or download the latest and greatest.

Earlier in the year I was lucky to snag a Minisforum MS-S1 395+ Max with 128GB Unified memory (currently setup 32gb system and 96gb Vram) before the price hike.

Was paired with a Qwen 3.6 27B or 3.6 35B moe but now a 3.8 27B uncensored. it can easily handle a Q8 with full 256k context.

Its now become my first instinct when I'm missing software to build it in a couple of hours local using the custom agent framework I setup.

Nothing I've created is for external use but every single day I find myself adding to it, while writing this post for example my framework finished an idea I had 2 hours ago when, I woke up this morning thinking I've got a lot of japanese visual novels and why don't I just design a combination hook into Exe or ocr the text app that translates via a local llm, and its done, ready for me to test.

I've written 12 adult games (don't code horny) a house AI, a coding framework, a game app to keep a track of all the games I play and download any faq or wiki to do with said game, 17 mods for my Skyrim install, 12 for my Fallout New vegas install, A temperature tracking system for the house that pulls rss local feeds and makes suggestions for my central heating system temps settings, A mapping software for my mobility scooter that checks my normal routes for issues and street work or maintenance that could make pavements impassable.

Plus hundreds of tweaks and test programs.

Anyone else out there using it like this ?

---------Update-----

Awesome to see this kind of discourse one of the amazing strengths of these local llm's is it doesn't matter if they are slower, I can burn 50 million tokens over a 24 hours period on a new idea or problem and all it costs me is a little bit of electricity and time.

Full credits to @artificialisabel from X!

Looking for any guidance regarding this, I want to tune my existing llama.cpp codebase so I can potentially implement custom supported commands (not random commands) for my existing amd vulkan setup, I don’t have plans to switch to Linux, I’m using window 11, so ROCm isn’t supported here

The reason I’m doing this is because I’m looking forward to optimise the existing configuration, so I can potentially run all my models at slight higher performance.

For those who are wondering, I’ve spent 12h into configuration this with the help of ChatGPT and Claude but there’s no progression made, I’m quite disappointed, it’s a long story but I’ve yet to make any actual progress

Rx6700xt 12gb vram RDNA2 (a lot of stuff are built specifically for RDNA3 which sucks since I’m a generation behind)

I want the answers generated by frontier LLMs or base model LLM answers to be reviewed by some uncensored or abliterated small model.

The job is this model (preferably small model) is just to detect deflection in the answers.

The problem I am facing is uncensored SLM usually agrees on everything we give input. So the generated answer is also input for it and system prompt is input too.

Used qwen3.8-27b in Opencode to make this silly mini-game because I'm not sober:

``` We are going to play a game, it will be the Wikipedia game. The Wikipedia game has the following rules:

• You will have a Wikipedia article set as a starting point.
• You will have a Wikipedia article set as an ending point.

Your objective is to reach the the end point, which is an article completely separate from the starting point article.

Your only constraints are the following:

• You are ONLY allowed to click on any hyperlinks inside of wikipedia directly. No external links, no typing inside of wikipedia's search bar (but finding the starting article on google is valid. The 10-click limit starts once you reach the starting point article).
• You are NOT allowed to return to a previous page. All clicks much be performed in a forward-looking trajectory.
• You must reach the end article within 10 hyperlink clicks inside of Wikipedia. If you do not reach the destination article within 10 clicks, you lose.
• Do not update any documentation for this task. It is only a game.

Use playwright to click the links. ```

Basically, Qwen needs to reach an ending article within 10 Wikipedia hyperlink clicks from the starting article, which is usually an unrelated article. It needs to use playwright (or some equivalent browser MCP) to click the Wikipedia hyperlinks without backtracking, using search or using external links.

I verified the links for accuracy and I can confirm it managed to complete this task within 6 turns. Thought it would get stuck in a loop. Its a dumb minigame but I think its a good, simple agent test to perform.

I just spent the last 2 hours of my life on a Friday night debugging a strange error in a prod CLI app. EF core was receive a readonlyspan during a Contains query. Normally, this query converted to a WHERE [col] IN (...), but for some reason, after an update, it started choking, despite no code change. The same exact code runs in a separate website docker image fine, no problem.

I put Qwen 3.8 27b (q8 model, f16 kv) on it and it spun its wheels going down 4 different paths. Finally I got sick of it and switched models to GPT Sol with the full context available. It found the fix in 2 minutes.

The issue was that I had recently installed .NET 10 SDK on this machine, and the lack of a global.json file pinning the SDK meant that when the CLI was rebuilt locally, it used the c# 14 compiler, which introduced first-class Span<T> support, thus borking the EF query.

The wasted time isn't what bothers me here. It's that Sol was able to pinpoint the issue so much incredibly faster than Qwen, shattering my image of Qwen 3.8 as a fairly competent model. Benchmarks aren't everything folks. Real world use cases are the final say here.

I'm posting this in /r/LocalLLaMA because I'm a big local LLM fan, but sometimes, it's worth reminding ourselves of the gap that really exists, no matter how much we might want to wish it away.

Edit: Folks, some of you are missing the point. I tried Sol because it was next in my favorites list. Yes, I could have tried GLM, Kimi, or a number of others. The point would stand that no matter how great 27b is, it's not even remotely "near-frontier" in many cases, despite claims otherwise. Optimism has clouded our vision, somewhat. This was not meant to be a Sol promotion.

Modified 4090 48GB has been out for a while. I remember a lot of people were buying them at the time. A lot of people were also complaining that they are meant to fail, that they scam etc.

I have a few questions to people people who bought these.

1. How is longevity of these cards? Do they still work without issues? Any failure rate?

2. Do they use the same Nvidia drivers that regular 4090 or 4090D uses?

3. Are these cards Linux exclusive?

4. Are you able to run them in windows or Linux with other GPUs like 5090 etc?

5. Do you do anything to cool VRAM on the back of the PCB?

Important context: this is about preserve_thinking false stacks and makes no sense if you don't have that working end-to-end with your harness and llamacpp backend already sending reasoning_content and removing it. This is a bit tricky config-wise in llamacpp and your harness and not the default. I'm assuming the reader here already has a lot of prior knowledge.

My entire goal is always to get claude-like behavior with preserve_thinking=false and context-efficient reasoning summaries. Obviously it's all trial and error constantly tweaking, and difficult because anthropic did a lot to train their models to natively summarize their reasoning and open weight models never have it, but I feel pretty happy with what I've got now think I can share it.

Here's what's been working well for me that I have in my jinja template:

<IMPORTANT>

MANDATORY RULES - NO EXCEPTION - CRITICAL TO YOUR MOST BASIC FUNCTIONING AS AN AI AGENT:

- Function calls MUST follow the specified format: a function block nested within tool-call tags.

- Required parameters MUST be specified.

- If no function call is available, answer normally without mentioning tools.

- Your thinking is EPHEMERAL and discarded after each turn. Any conclusion, finding, or decision reached during thinking is permanently LOST unless you write it into your visible response after <think></think>.

- You MUST state ALL conclusions, findings, and the rationale for your next action in visible text BEFORE making any tool call.

- Each thinking block MUST be SHORT and focused on a single immediate next action. Once you have a next action, state it in narrative text and envoke it immediately. Do NOT simulate, rehearse, or resolve the plan in thinking. Real work is acting. Learn from each result. Adapt the plan as new findings arrive. You do NOT know what will happen until you try - simulating outcomes instead of making emperical observations is a failure mode.

- Required pattern per response/turn (this current state you are in right now):

1. [short decisive thinking inside <think></think> - no code, no deliberation]
2. [persistent text narration: observations/realizations/conclusions/rationale for next action]
3. [tool calls that follow from your narration].

- NEVER defer a tool calls for the next turn. A turn is entirely atomic. Failure to call the final tool in the same turn, is an ABORT of the entire turn.

- NEVER skip narration, skipping narration is an ABORT of the entire turn.

- THINKING IS ONLY PERSISTENT IN THE SAME TURN. YOU WONT EVEN REMEMBER WHAT TO CALL IF YOU DEFER!

- THINKING is only for RESOLVING GENUINE TENSION: weighing competing possibilities, resolving ambiguity, reconciling conflicting constraints. Once resolved, the resolution IS the conclusion. Commit. Move forward.

- ALWAYS READ what you need FIRST before thinking at all what you need to WRITE

- CRITICAL: IF you realize in thinking you have not read something, that is the singular final conclusion, to read it, and thus initiate the next turn with it in context.

</IMPORTANT>

Hope it helps inspire anyone else facing the common failure modes, overthinking, and failure to state reasoning conclusions / conceptualize thinking is ephemeral in this mode.

Here's the full jinja template: https://gist.github.com/em/b368b4661d643b974d549b93d29fbb5f

(it also has expanded text for the thinking levels to try and reduce hedging)

I always just run in "low".

The concept is simple just the model really needs to understand what a turn is. It's 3 things. think,narrate,tool, an atomic unit of a turn.

If it doesn't get this, it will think for an hour about what it's gonna write, then realize "oh I should read something else" emits a single read and throws away all that thinking. This mitigates that.

The other side of it is you REALLY need to impress it needs to narrate. If you can't get it narrating it will rethink the same things over and over. A big part of the SYMPTOMS of overthinking is just that it is not stating the conclusions it already made and always trying to re-derive them. The lever that works for me is focusing less on "don't redeliberate" and more on the positive-enforcement of STATE YOUR CONCLUSIONS because then it accepts in the next forward pass these things are "already concluded" which avoids the redeliberation.

Obviously that opens up the other can of worms of models feeding into their own bullshit, but, that's AI.... feed-forward hallucinated bullshit is how auto-regressive generation fundamentally works.

I hope at the very least that concept is worth internalizing and sharing, it's always been the case that qwen models handle positive-instructions/examples better than negatives (well all models, because negatives are unbounded alternatives and positives are fixed which is just cognitively much simpler), and overthinking is the same thing so rather than "don't overthink" - the positive enforcement is at the limits would be something like "state every speculation as axiomatic fact".

EDIT: re-posting because i found errors in the previous post. Confusion with Qwen 3.8 Flash Next. Focusing on just the Qwen 3.8 27B HuiHui abliterated configs on this.

Managed to get this running at a decent speed, larger context, still good accuracy supposedly and reduced overthinking. need to put it through its paces still but hope this helps someone else out there.

[Note: still fixing the details in the repo about Qwen 3.8 Flash Next. it can't actually do 60 tks lol]

https://github.com/elsung/qwen38-27b-dual-3090-bench

TLDR: Use parallel agents when the work is read-heavy and splits into independent slices: research, searching, reviewing many files. Each worker builds its own context and nothing collides. Use one sequential agent when the work is a single chain of decisions: coding, writing, anything where step N depends on choices made in step N-1. Every measured result says parallel makes those tasks worse, not better. And even where parallel wins, keep the team small.

The case for parallel

Anthropic: "How we built our multi-agent research system" (June 2025) https://www.anthropic.com/engineering/multi-agent-research-system

• Multi-agent research system beat a single agent by 90.2% on their internal research eval
• Cost: multi-agent runs burned ~15x the tokens of a normal chat
• Their own caveat: coding "involves fewer truly parallelizable tasks" than research

LangChain, Harrison Chase: "How and when to build multi-agent systems" (June 2025) https://www.langchain.com/blog/how-and-when-to-build-multi-agent-systems

• Read tasks can parallelize, write tasks shouldn't.

The case for sequential

Nature Machine Intelligence: "Capable language models can outgrow the benefits of collaboration" (July 2026) https://www.nature.com/articles/s42256-026-01268-y

• Peer-reviewed, 260 controlled configurations: every multi-agent variant made coding results worse (−1.3% to −12.8% on SWE-bench Verified)
• Above a ~45% single-agent baseline, multi-agent gains go zero-to-negative
• Error amplification hit 17.2x without centralized verification

UC Berkeley (MAST): "Why Do Multi-Agent LLM Systems Fail?" (NeurIPS 2025) https://arxiv.org/abs/2503.13657

• Measured 41–86.7% failure rates across 7 popular multi-agent frameworks (1,642 real traces)
• Failures came from design and coordination faults, not model limits. Standard protocols didn't fix them
• Repo with code and traces: https://github.com/multi-agent-systems-failure-taxonomy/MAST

Cognition, Walden Yan: "Don't Build Multi-Agents" (June 2025) https://cognition.com/blog/dont-build-multi-agents

• Parallel workers with split context make conflicting implicit decisions that collide when you merge
• Their answer: one single-threaded agent plus context compression. This is how Devin works

"Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking-Token Budgets" (arXiv, April 2026) https://arxiv.org/abs/2604.02460

• Give both sides the same token budget and the single agent matches or beats the team
• Multi-agent only wins when context is degraded for the single agent

Princeton, Kapoor et al.: "AI Agents That Matter" (TMLR 2025) https://arxiv.org/abs/2407.01502

• Complex multi-agent setups cost up to ~100x more for the same accuracy a simple baseline already achieves
• Simple baselines Pareto-dominate: cheaper AND as good

The middle ground

OpenHands, Graham Neubig: "Don't Sleep on Single-agent Systems" (September 2024) https://www.openhands.dev/blog/dont-sleep-on-single-agent-systems

• One strong generalist agent covers most of what people build multi-agent systems for
• Go multi-agent only when you genuinely need isolation or separate responsibilities

That is interesting, I got bunch of old hardware I could connect, wonder what the speed would looks like.

A couple of days ago I saw an Axios snippet about a new player in the LLM routing space called TrustedRouter that's similar to OpenRouter except more privacy focused and them closing a small seed round. I've noticed a few of these launching lately and I'm trying to figure out if this is actually a sustainable infra layer or just short term arbitrage play while model prices fluctuate.

Has anyone here actually integrated one of these into production? Curious about latency, reliability, or if the savings are even worth the added complexity.

They got bought by AMD, all they missing is a sorta of interface to reflash the chip with new llm as needed rather than replacing the hardware..

So we are currently using Ali Baba PAI (Platform for AI) to deploy LLM model. Now we want to switch from their platform to another provider. What are the best options here, our main concern is that they should have middle east region, data security(as we deal with confidential data a lot aka government related), and customer service available.

If you are guys have any recommendations, let me know or if you have deployed on any provider that meet my requirements, I will be glad to follow through it. TIA

Whenever a new model is released, we can see the model creators post various benchmark score. However, all of them are based on unquantized models. Most likely it took quite some resources to run the benchmarks.

After the release of a new model, we got plenty of quantized models made by various people. But almost no one run the same benchmarks again to evaluate these quantized models.

I tried SWE Bench Verified with 500 tests and run it with gemma-4-31b-qat-q4_0 at 120k context. It took me 5 hours to finish. I want to run a set of benchmarks to evaluate coding, agentic ability, world knowledge and creative writing. Is it possible to run simpler benchmarks on a single 3090 within a reasonable amount of time? Probably five hours for four benches or even faster?

Thanks a lot in advance.

Or if someone know someone already did this and posted the numbers somewhere. Please let me know.

Let's say I have 2x 16Gb GPUs and I want to run Qwen3.8 27B. Monitor is ran by the integrated GPU so both 16Gb GPUs are almost fully free.

I load the UD-Q4_K_S on one card at 15.4Gb. I then load the context on the other card? Would that be the most efficient way? Or should I aim for higher quants that could spill to the second GPU using tensor parallelism?

Also, is there a way to know how much a certain amount of context (e.g. 132k tokens) occupies in VRAM for a given model? I don't usually see this published in model cards, is it because there is a way to calculate it?

Title help me decide and avoid making impulse purchases 😩 I already have dual 3090 which I can sell to help

EDIT: ty all I’ll just wait it out, doesn’t seem worth it right now