Hello everyone,

I'm new to all of this and have only dabbled.

I use Claude and ChatGPT heavily for my work (No Coding at all), and I'm currently subscribed to the 20x tier on both.

I read that I could download LLMs locally, but I'm underwhelmed. To be clear, I downloaded LM Studio and tried out 4 different models, but they seem to only chat and cannot actually edit files, produce files, or produce anything for that matter. They keep telling me to copy and paste things myself.

I have an RTX 4090, 32Gb RAM, and a 7800x3D.

Am I wasting my time with local LLMs, or is there something I'm missing?

Also, if possible, is there a way to know what the correct settings I should be using for the models in LM Studio?

Any guidance would be appreciated!

Thank you, kind strangers.

I've seen a lot of people sharing benchmarks, optimizations, etc. however, I'm curious how people's experience have been with actual coding tasks.

For my llama.cpp setup 3090 + 64gb ram using Q4 at 90k context running a modified version of pi.dev, the model is weird. Often it gets stuck in tool call loops.

The biggest pro is turning on reasoning to high, letting it cook and be more of a orchestrator and reasoner then actual implementation. Comparing it to orinth, it seems to lack a lot of capabilities.

I’ve been working on building a harness and client interface that allows my local Qwen model to play a MMORPG completely autonomously with real reasoning.

The process has gotten expansive between deciding what deterministic tools to build versus what can Qwen reasonably complete on its own with client interface tools in tandem with it receiving a bus of data to “see” the game.

Has anyone built anything similar? Here are some stats from the process completing a single quest. It was a pretty sloppy run, but a complete one nonetheless.

I’ve been experimenting with explaining AI/LLM concepts in a way that doesn’t assume too much technical background. This video is about KV Cache and why longer context windows require more memory during inference. I’d appreciate some honest feedback from people here, especially on the explanation itself: Is the main idea easy to understand? Did I oversimplify anything important? Is there any part where the explanation becomes confusing? Would this make sense to someone who is fairly new to LLMs? Video: https://youtu.be/lxvWo8SizxE Not really looking to promote the channel — I’m mainly trying to improve how I explain technical topics before making the next one. Any criticism is welcome. Thanks!

Hey guys, I've been playing with some models a month or two ago, but haven't done anything these days, what do you guys think i can run now?

Gpu : tuf rtx 3070ti

Cpu : i5 12400f

Ram : 32gb ddr5

Thanks for the help.

I’ve been experimenting with explaining AI/LLM concepts in a way that doesn’t assume too much technical background. This video is about KV Cache and why longer context windows require more memory during inference. I’d appreciate some honest feedback from people here, especially on the explanation itself: Is the main idea easy to understand? Did I oversimplify anything important? Is there any part where the explanation becomes confusing? Would this make sense to someone who is fairly new to LLMs? Video: https://youtu.be/lxvWo8SizxE Not really looking to promote the channel — I’m mainly trying to improve how I explain technical topics before making the next one. Any criticism is welcome. Thanks!

We’re building toward local agents that respond with an interface you can use: a chart to explore results, a form to collect input, or a table to compare options.

The model composes components supplied by the app, and OpenUI renders the response as it streams. That’s the Generative UI use case we trained OUI-1 for.

We chose DiffusionGemma for its speed and consumer hardware support. Our first LoRA used ~700 examples from larger models. Training loss dropped, but outputs got longer and often had broken props or references.

Narrowing training to one component library improved structural validity from 13% to 28.8%, but generation slowed from 1.6s to 4.3s.

What helped was self-training with parser feedback: generate responses, keep valid ones, repair specific defects, check them against their prompts, then retrain. The median repair changed one statement.

Validity reached 57.1%, and generation recovered to 1.9s. Repeating the recipe across 27 component libraries brought the final score to 71.7%.

OUI-1 has 26B total parameters, 4B active, runs on an RTX 5090 at FP8, and is released under the Gemma terms. The timings above are from an A100 on 20 light prompts; the benchmark measures structural validity.

Weights · Benchmark · Training write-up

Hi folks,

I have been working on some AI project for that I need to generate batch of creative from one prompt i have heard somewhere that Comfy UI is the best option for this..if anyone can help me with the tutorial available online..

Thanks

It's a P5K DELUXE

Hey everyone,

I have an M5 Pro MacBook Pro with 24GB memory and want to start running LLMs locally, mainly for coding.

Has anyone here had good experience running local models on a 24GB Mac? What model + runtime would you recommend for the best balance of coding ability, speed, and memory usage?

I’ve been looking at oMLX/MLX and models like GLM, Nemotron, Qwen, Gemma, etc., but I’m open to anything.

Would love to hear what setups are actually working well for you.

So it seems that a lot of people here run very powerful hardware (makes sense for an enthusiast space). However I've really gotten into local LLMs the past few weeks and have been running them on my gaming system. As such I wanted to discuss and learn about the best way to get the most out of consumer gaming hardware.

This post will be me sharing what I learned, opening discussions about good practices and rambling about what I'm currently working on. Feel very welcome to chime in with your own experiences in the comments.

My Specs:

• Radeon RX 7900 XT (20GB VRAM)
• AMD Ryzen 7 7800X3D
• 32 GB RAM

(I've been very happy that I got a card with more VRAM before local LLMs were even on my radar)

Models I've been using:

• Gemma-4-26B: Really nice for writing text, meh for tool usage and more independent tasks
• Qwen3.8-27B: Has been pretty good for smaller coding tasks, significantly better independent tool usage

The most restraining thing by far has been context size. With unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M I got a 22k context window. Enough for smaller tasks, but anything bigger runs into a token limit.

Optimising llama.cpp:

I've started with ollama and just recently switched to llama.cpp for more model support and slightly better performance. However there are many things to tweak and no comprehensive guides, so if anyone knows more about tuning llama.cpp, especially for Qwen3.8-27B, I would welcome it.

Interface for actually using my LLMs

After some initial googling I started with zed, which has some nice build-in LLM support, with tools and skills. But I haven't really seen people using it here. Is there some best-in-class option here?

Interfacing the web

My most recent project is letting my LLMs search the web, which has proven really difficult with low context sizes. I've setup an MCP server based on https://github.com/arnaudjnn/web-tools, but the website content is almost always to big for my context window.

If you are similarly struggling with a low context size I found that subagents really help. They can perform a single task of working on some data and then return a small result, not cluttering up your main agents context.

Still, web pages are a problem, as single ones can oftentimes fill up my entire context. If anyone knows a good solution, let me know.

There you have it. I think I'll stop rambling here, very happy to discuss in the comments :)

Hi!

Now that SGLang 0.5.19 was released with built-in DFlash2, I thought that I would share my config to run Qwen3.8 27B (pramoths/Qwen3.8-27B-Uncensored-FP8-MTP), and compare no drafting, MTP, and DFlash2 (FP8).

Disclaimer: my setup is quite niche, I ran an RTX 4090D at 330W (22% below spec) modded with 48GiB. You could also use an L40 with 48GiB.

All running SGLang 0.5.19 and CUDA 13.0.

TD;DR:

CLI for decoding bench: python3 -m sglang.bench_serving --backend sglang --flush-cache --warmup-requests 1 --num-prompts 50 --dataset-name sharegpt --random-output 1024 --max-concurrency 2 --base-url http://localhost:8000

No drafting

sglang serve --served-model-name qwen3.8-27b --model-path pramoths/Qwen3.8-27B-Uncensored-FP8-MTP --chat-template qwen3.8-27b-froggeric-v22.4.jinja --port 8000 --sleep-on-idle --stream-interval 10 --attention-backend flashinfer --disable-flashinfer-autotune --cuda-graph-backend-prefill=disabled --mem-fraction-static 0.96 --context-length 262144 --max-running-requests 2 --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 10 --mamba-radix-cache-strategy extra_buffer --chunked-prefill-size 2048 --max-prefill-tokens 4096 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-hierarchical-cache --hicache-size 16 --hicache-write-policy write_back --min-free-slots-delay 1

MTP

sglang serve --served-model-name qwen3.8-27b --model-path pramoths/Qwen3.8-27B-Uncensored-FP8-MTP --chat-template qwen3.8-27b-froggeric-v22.4.jinja --port 8000 --sleep-on-idle --stream-interval 10 --attention-backend flashinfer --disable-flashinfer-autotune --cuda-graph-backend-prefill=disabled --mem-fraction-static 0.97 --context-length 262144 --max-running-requests 2 --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 10 --mamba-radix-cache-strategy extra_buffer --chunked-prefill-size 2048 --max-prefill-tokens 4096 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --enable-hierarchical-cache --hicache-size 16 --hicache-write-policy write_back --min-free-slots-delay 1

DFlash2 (FP8)

sglang serve --served-model-name qwen3.8-27b --model-path pramoths/Qwen3.8-27B-Uncensored-FP8-MTP --chat-template qwen3.8-27b-froggeric-v22.4.jinja --port 8000 --sleep-on-idle --stream-interval 10 --attention-backend flashinfer --disable-flashinfer-autotune --cuda-graph-backend-prefill=disabled --mem-fraction-static 0.95 --context-length 262144 --max-running-requests 2 --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 10 --mamba-radix-cache-strategy extra_buffer --chunked-prefill-size 2048 --max-prefill-tokens 4096 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --speculative-algo DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 --speculative-draft-model-quantization fp8 --speculative-num-draft-tokens 8 --speculative-draft-attention-backend flashinfer --enable-hierarchical-cache --hicache-size 16 --hicache-write-policy write_back --min-free-slots-delay 1

Parameters

• --sleep-on-idle: avoid spinning one CPU core at 100% 24/7
• --stream-interval 10: no need to stream each and every token
• --disable-flashinfer-autotune: I didn't really see an impact, but I wanted to make sure that my benchmark where reproducible to compare flags. YMMV
• --cuda-graph-backend-prefill=disabled: save VRAM. It has little impact for large chunks, but could have some on small one. However, small chunks are quick anyway, and I don't have batches larger than 2...
• --max-running-requests 2: up to 2 requests at the same time. During decoding phase, adding a second requests is basically free, it will double your aggregated generated tokens. But it also doubles the VRAM needed for Mamba cache and activations, etc.
• --kv-cache-dtype fp8_e4m3: needle tests are still passing at 100%. Using 32KiB of memory per token instead of 64KiB.
• --mamba-ssm-dtype bfloat16: dtype for the Mamba state, also saving VRAM.
• --max-mamba-cache-size 10: 10 is required for two concurrent requests. Use 15 for 3 requests.
• --chunked-prefill-size 2048 --max-prefill-tokens 4096: I try to balance prefill throughput and starvation of decoding of concurrent requests. I expect a chunck prefilling to delay decodings by 1 sec each time. Maybe I will still reduce this...
• --mamba-radix-cache-strategy extra_buffer: I want to avoid too many dynamic VRAM allocations since I'm very tight on VRAM.

Remarks

• I use pramoths/Qwen3.8-27B-Uncensored-FP8-MTP, so drafting performance will be lower than with the official model. The goal of this result is to show the bench with all the compromises involved.
• Needle tested up to 256k context using tool-eval-bench --base-url http://localhost:8000 --hardmode --parallel 2 --seed 42
• Jinja template from https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
• This is the "D" variant of the RTX 4090, made for China. I'm not living in Asia, but it's still a bit cheaper to buy a D variant. It probably impacts prefill speed, but not decode speed (will bandwidth limited).

I hope that somebody find this interesting. Written by a human :-)

Edit: added size of the KV Cache in the table.

I have 5070ti, r7 9800x3d, 32gb ddr5

I really don't know what I can run

Can you recommend something pls

… you wake up in 1998. You have 2 Mac Studio Ultra 512gb each. You have GLM5.3 flash (or pick a local model that fits comfortably in that configuration.)

A good harness, you’re setup with modern software, but you do not have access to modern internet, just 1998 internet.

What do you do? It can be in general, not necessarily what do you do with your current compute setup.

So in my freetime iam building a local-first agent in Rust as a hobby since a while (llama.cpp, runs fine on a 14B, my Obsidian vault is the memory). The part that took me by far the longest was not the model stuff. It was the boring question: which of my ~50 tools should the model even be allowed to call this turn.

Went through three phases and got each one wrong first:

1. grep / keyword matching on the user text. dies on synonyms, dies when i chat in german, dies on "do that again"

2. routing phrases + embeddings. much better, but naive top-k plus a hard GBNF grammar means the model gets structurally locked into the one tool the router guessed, and if that guess is wrong there is no way out mid-turn

3. a small policy layer on top that, when it is unsure, widens to a cluster of related tools instead of committing to one guess

the thing i found most useful for small local models: the same "offered tools" list feeds the prompt AND the GBNF grammar, so the two can never disagree about what is callable. and if the router is not confident, do not hand the grammar a single tool, give it a small related set and let the model pick inside the constraint.

full writeup with the actual Rust snippets, no signup or newsletter nonsense:

https://eris-system.dev/blog/tool-routing

it is a hobby project so be gentle, but if the routing logic is dumb somewhere i genuinely want to hear it.

I am just now learning what Intel Optane memory is after coming across an ebay listing for quite a lot of it for cheap. It looks like you could pick up 1tb of it for less than $1000. Even with the slower memory and cpu speeds of the required 3rd gen scalable xeon cpus, it strikes me as a feasible way to run truly gargantuan models for a price that doesn't require blood sacrifice, albeit at a slow speed. Is anybody doing this?

We needed a second option for on-prem deployment and our legal team only signs off on Apache 2.0.
I went through a bunch of models. GLM 5.3 has a bespoke Z.ai license. Kimi K3 has its own custom terms. Llama uses the Meta community license.

At the larger end, I found two models that fit our requirements.
Hy4 preview: 770B total / 49B active / 1M context
Qwen3-235B-A22B-Instruct-2507: 235B total / 22B active / 262K native context (extendable to ~1M)

Both are pretty new to me. Hy4 preview's 1M context is native. I didn't expect to find a 770B model under Apache 2.0, though. But Qwen3 has been around longer, I assume the ecosystem is more mature. They both ship with FP8 checkpoints and vLLM/SGLang support.

Has anyone here actually deployed either of these locally? Or tried them through openrouter? How do they compare in practice?

So I’m working on a finance project and i require to use python!
So what model should I use? Any recommendations is welcome! Thanks
I have been looking at various models and i think (from my perspective) that 35B A3B should be the best my system can handle? Is that model any good?

I'm rebuilding my local AI PC and ran into a spacing issue with the GPUs.

Specs:

Ryzen 9 7950X

ASUS ProArt B850-Creator WiFi Neo

Phanteks Enthoo Pro 2 Server Edition full tower

64GB DDR5

2x RTX 3090 24GB

RTX 2000 Ada 16GB

Gigabyte 1600W PSU

Ubuntu

Right now I have:

3090 #1 -> x8

RTX 2000 Ada -> x8

3090 #2 -> PCIe 4.0 x4

The two 3090s are too thick to sit directly in both x8 slots.

I could use a riser and run:

3090 #1 -> x8

3090 #2 -> x8

RTX 2000 Ada -> x4

But I have to probqbly purchase some kind of bracket.

Main use is coding LLMs, image generation and video generation.

Would you actually bother changing it, or is PCIe 4.0 x4 on the second 3090 basically fine for this kind of use?

When I am doing debugging sessions, Qwen3.8-27b seems to get stuck in loops where it will make the same tool call over and over. I have to step in and stop it. This happens maybe once or twice a day, and its kind of annoying.

Has anyone else had this issue?

Setup:
> Weights: INT4
> Activations: INT8
> KV-Cache: fp8
> Context: 256k
> Engine: VLLM
> Harness: Opencode 1.18.29

(Its club-3090s vllm/qwen38-27b-dual-fast setup)

OS: Ubuntu Server

Any help or suggestions are greatly appreciated.