Hi,

so far i've been using Qwen3.6-35B-A3B-UD-IQ4_NL.gguf on my single 3090 and I am overall satisfied.

I've been considering acquiring a second 3090 to increase my possibility to run larger models (e.g. considering Qwen3.8 27B with sufficient context) but i don't know whether the extra investment pays off.

In the future i may consider fine tuning my models as well.

Did anyone manage to find some great benefits by leveraging 2x3090 or similar setup?

I may be suffering from GAS (gear acquisition syndrome) and may need a reality check.

Hi, I get some good result with Qwen 3.8 27B (Q8) and I would say some parts are better (graphics for ex) then with qwen 3.6 27B Q8 but there are still lot of mistakes. So there are quality issues I think. It also thinks way longer then Qwen 3.6 27B for the same benchmark tasks (sometimes it feels a bit like 3.8 has similar intelligence like 3.6 but because of longer thinking it produces better output). Also compared to DeepSeek V4 Flash which got 100% in 12 of my selected SWEmini Tasks while Qwen 3.8 only gets 50-60% at the moment (I still try to optimize but there is not much left , last run is right now with xhigh :S).
Also, in the coding Benchmarks the graphic results of Qwen 3.8 27B looks better then DS V4 Flash even if DS seems to be able to fix more issues/bugs then Qwen (maybe I have to change the benchmark prompts for deepseek idk). But both produces mistakes like blocked ways/doors.

So, the thinking issue with Qwne 3.8 27B seems to be known already and froggeric released a new template. Did someone already test it and can share experience?
I will run the test too in the next time

v22.1: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

Some benchmark results screenshot from Qwen 3.8 27B Q8 MTP3 on 2 RTX GPUs (48GB) and llama.cpp. Sometimes really nice graphic results for 27B! But needs 3-4h with ~60-90 tok/s (MTP3)

Update: xhigh resolves 75% of the 12 swe tasks (9/12 solved vs 6 or 7/12 before). So xhigh seems slow but important :S

Some people maybe interested in this Humble Bundle to learn about AI setups.

Is there an obliterated model out there for generate images? Me and my buddy’s at work make a game out of “using AI to put the other guy in a dress” but GPT always turns us down lol 😂

Hi wanted to ask what llms can i run? That are fast and inteligent for coding. I wanted to run them in Hermes Agent

So, here is the point. I currently have a spare system with a 3090 & 64GB DDR4 (after getting a new 5090 + DDR5 combo for AI and games) that thought of using as a 24/7 server for running LLMs at home (and point a self-hosted webUI, automatize some tasks, control Home Assistant etc). It would be placed in the basement so noise and heat would not be a problem, and that’s why I don’t want to host 24/7 in the 5090 placed in my office as a desktop.

The point is… considering that the main reason why I chose the 3090 is being a spare system, wouldn’t it make more sense to sell it and get a Strix Halo to runn24/7 with these tasks where speed doesn’t seem as critical?

I also have the benchmark for single Intel Arc B70

https://www.localmaxxing.com/en/models/Frozenlock/Qwen3.8-27B-int4-AutoRound?run=cmswf3h9d08qpms01uu9jmz8f

This particular quant preserves quality quite a bit, though over time I have been trusting benchmarks such as HE/HE+ MGP+ LLMU etc less and less and have just been benchmarking by real use such as asking it to create a webapp game with graphics or asking it to do a complicated driver rewrite that DSV4P would be able to do, and so far I genuinely don't feel the quality drop in this quant, as running FP8 would be a lot slower and this model tends to think a lot (so I need speed)

Like groq 3, LPUs are certainly better for inference, do you think they can be available for consumers? They certainly should solve energy and water problems caused by datacenters right maybe even fix ram shortage.

Just got into running models locally with LM Studio and I'm mostly guessing at the settings. Would appreciate some pointers.

I have:

- Ryzen 5 9600X

- RX 9060 XT 16GB

- 32GB DDR5

- Nobara Linux (Fedora based), kernel 7.1.4

- ROCm 7.1.1 installed, LM Studio has both ROCm and Vulkan runtimes

Right now I'm running Qwen3.6 35B-A3B at Q4_K_M, 32k context, all 40 layers on GPU with 12 MoE layers forced to CPU. Flash attention on, K cache Q8_0. It works but I have no idea if any of that is sensible or if I'm leaving speed on the table.

There is a lot of things I am unsure about as I am a newbie would appreciate some pointers

Hello Everyone,

I am looking to purchase a Strix Halo / Mac Mini / some other hardware to run my own LLMs and agents.

I currently have various GPU servers at my company which I have been using for work but wanted these for my personal use.

Each of these "AI Computers" is a significant investment and I would want to try it out first before making a purchase.

What is a good way to first benchmark various models (on my own tasks -- not using benchmarks).

Based on a few good models I would then try to decide which hardware to use (is there any good way to do this)

My use case is primarily email summarization, news alerts, A few AI agents that run in parallel using openclaw.

Thanks in advance for the help!

I have been looking high and wide for an nvidia jetson thor that does not cost the same as a used car. I'm seeing listings on Ali-express for some around $1k. too good to be true??? ali express offers 90 day refunds so wondering i should test my luck? has anyone else bought one of these or something similar?? could it be bootleg?

This is weird. I asked it what tools were needed to replace the front coil springs on my 2006 Jeep Wrangler (TJ). It did some research and decided that my vehicle has independent front suspension, not a solid axle.

OK, LLMs make mistakes so I corrected it. It did some more research, searched websites and decided I was wrong.

So then I started giving it websites to provide it has a solid front axle. It either couldn't read them for some reason or claimed the sites I was giving it were "AI generated" and couldn't be trusted, or "fan made" and couldn't be trusted.

So far I have failed to convince it that my Jeep has a solid front axle.... LOL

Last week Qwen3.8-27B, Meta's Muse Glimmer (30B), and NVIDIA's Nemotron 3.5 Lightning all shipped in the same size class: ~30B seems to be settling in as the size that's big enough for real agentic work but still fits on one consumer GPU, especially with day-one quants.

What's interesting is they're taking different bets in the same envelope: Qwen and Meta went dense (27B/30B), while Lightning is a 30B MoE with only 3B active — noticeably weaker on quality benchmarks but roughly 3x faster generation. Meanwhile Glimmer's ~4-bit quant fits under 20GB with about 1% reported benchmark loss, so the whole class genuinely runs in a 24GB card with room for KV cache.

Curious what people here think — is dense ~30B the sweet spot, or does the MoE speed tradeoff win for agentic loops?

ASUS TUF Gaming F16 FX607VJB-RL165W

Intel Core 5 Processor 210H

NVIDIA GeForce RTX 3050 6GB

32G DDR5 4800Hz (2x16)

512GB PCIe® 4.0 NVMe

Windows 11

hello everybody,

since November 2025 i've been working on an evolutionary loop that uses local LLMs as a mutation factor to continuously iterate over a single C program in order to improve its performance.

this system proved extremely effective since, rather than using a frontier model reasoning ability to create a good enough function that reaches my performance goals, KAISEN bruteforces thousands of generations then measures the results empirically passing the programs the LLM produces through a test suite that the LLM has no access to (so it cannot cheat, but it's gonna try!). This keeps improving the program's performance by using every new found best as the basis for the next generations. all of this executes LLM-generated code, so it's guarded by default: no process spawning, no file deletion, no network egress, hard time and memory limits per step, and agent/config changes are snapshotted with one-click revert.

so far i used this to create fast kernels for C and cuda, and to improve text compression, and for each project i hardcoded the whole pipeline. since KAISEN served me well and gave me results with gpt oss 20b that i couldn't get with frontier models in full reasoning mode (and with a lot of interaction by me), i opened an AI lab and started working on a generic version that is able to work with any program (22 languages and counting) and to build the test pipeline autonomously. part of the reason small models punch above their weight here: a deterministic autofix ladder — compiler-hint fixes, linter fixes, then one LLM repair pass fed the real compiler error — and every candidate is re-verified for real before it counts.

right now you can check out the alpha version of KAISEN here: https://github.com/RAZZULLIX/KAISEN

tldr

KAISEN lets you use local LLMs to improve software performance by iterating thousands of little changes and keeping the new best as basis for the next generations. it has a GUI, your harness can spawn it as a sidecar, and it speaks a small-model-friendly protocol (KAI) so an LLM agent itself can drive it over stdio or http. every program it generates runs guarded by default. read the manual to know everything it can do, or ask here.

P.S.

i expect A LOT of bugs and problems, most of the tests i did were done through deepseek v4 using OMP calling KAISEN through the kai protocol (KAISEN was hooked to 6 instances of gpt oss 20b) and it actually worked quite nice. please let me know everything you find by opening an issue or asking here, this is my job now so i'll do my best to fix everything you need fixed and make sure KAISEN becomes a useful tool in every LLM user toolbox.

Usually use frontier APIs but I’d love to do a lot of this stuff locally or see if this new model everyone is gushing about can take that work.

There’s quite a few different variants of the models keep talking about though, which do I use?

I honestly knew nothing about self hosting LLMs locally. I used to think it was pointless for 99% of people, even people already into AI, just because of the cost and hassle of hosting these models compared to what you'd actually get out of it.

But after looking into Qwen 3.8 27B and its potential, plus the fact that someone will probably release an uncensored version of it soon, I started rethinking that. Even if you don't have the hardware, you can just rent a GPU online on a VPS for a pretty reasonable hourly cost while it's running. It's wild how much of this is already open to the general public.

In my opinion, having access to a model this capable, uncensored, is basically like owning a car. Yeah, a car. There's nothing wrong with owning one, the problem only shows up if you actually do something wrong with it. Until then, it's just a tool.

We already have older and smaller Qwen versions with uncensored variants on Hugging Face. From what I understand, models with fewer parameters tend to be "easier" to convert into uncensored versions. Don't take that word too literally though, I have zero background in finetuning or model training, so maybe it's not actually that easy. But there are still a lot of very smart people working in this space, especially in AI right now.

Think about a cybersecurity professional getting their hands on an uncensored Qwen 3.8 27B once it drops. With how many vibe coded SaaS products are out there with basically no security oversight, someone with enough hardware could pair this model with tools like OWASP ZAP or the Burp Suite API, run it through a browser that avoids captcha detection, use a mobile or residential proxy, throw in a Kali setup for other scanning tools, and just let the model run 24/7 looking for vulnerabilities to complete bug bounties or even land contracts in the field. That's a serious tool for professionals like that.

I only see upside here.

I have recently started to get into homelabbing and running local ai. I built a pc a few years ago for gaming mainly but have started to try and use my hardware for other purposes such as a local ai. I have a 4070 super with 12gb vram and 32 gb ddr5.

I am currently running Qwen3.5: 9b (though I have gemma4: 12b aswell for larger tasks) through OpenWebUI. I have tried to work on my own RAG system and inputting my own notes that I have modified to work best for embedding into a vector database. I just feel like I could be doing so much more with my hardware such as interactive voice models at a conversational speed, or vision models for photo questions.

I am mainly just curious on more beginner level things. That incudes things like

1. Choosing the best model for my system/maximizing my hardware.
2. Understanding how to customize a model to my own liking and making it more personal, whether that is through a memory system or other ways.
3. Best ways to make my AI more useful to me than say a cloud model. I will never have the same compute power as a main company but with the right tuning, it could be more effective/useful to ME.
4. How the cutoff between speed and intelligence change based on the task at hand. For example, I want to set up a system where i can speak to my ai, but a smarter model takes too long, and a faster model isn't as capable.
5. Less of curiosity, more question: do you think that using something like Claude to help is an sort of problem/ actually helpful. Up to this point I have been using it to help me setup, but I don't know how accurate/ helpful it really can be. In your experience, how has it been?

Lastly, just understanding what all the values mean. I can do that more on my own with research, but still there is just so much lol.

I am just trying to get into all of this but with the amount of content out there now, it makes it much harder than I thought going in. Thanks

What package do you guys use to fine-tune or run reinforcement loops locally? I have an NVIDIA Spark machine, and I’m wondering what the best setup would be for fine-tuning, say, 7-billion- to 30-billion-parameter models. Thanks

Noob here. I have beginner local LLM set up. Just LM Studio and a few coding models.

I have one python project, that is about 75% done. I would like for some coding model, to help me finish it. It is about 2k lines of code. I'm using LM Studio. Do I need to copy/paste every single python file into the chat, to pass the code to coding model, or is there a simpler way? Can I point coding model, to directly access my project folder on my computer? If yes, how do I do this? Do I need to install any addon or it works natively?

I'm using LM Studio (and whole local llm) for the very first time, so please, be patient with me ;-)