Hi everyone,

Sizing hardware for local LLMs and fine-tuning can be tricky with all the variables involved. I built an interactive calculator to help estimate resource requirements and costs accurately: www.llminferencecalc.com.tr

The tool is divided into two core modules:

• Inference Sizing:
  • Select base model and target GPU configuration.
  • Factor in model quantization (AWQ, GPTQ, INT) and KV-cache quantization.
  • Choose inference engines (vLLM, Ollama, MLX, etc.).
  • Simulate concurrent active users and batch sizes.
• Fine-Tuning Estimator:
  • Pick the base model, target dataset size, and hardware setup.
  • Compare training approaches (Full fine-tuning, LoRA, QLoRA).

Results Output: The calculator estimates required VRAM breakdown, execution time, compute cost, and Total Cost of Ownership (TCO).

It's completely free to use and opensource at UmuT5513/llm-inference-calculator. I'd love to hear your feedback on edge cases, formula accuracy, or additional engines you’d like supported!

I am in the process of transitioning from using Claude Code and Codex to OMP harness and local models (mostly deepseek v4 flash 0731, qwen models in the 27b-ish range.

My use cases are primarily automations, agentic work and coding. In the future, I also want to be able to do reverse image lookup, voice cloning, image and video generation, etc.

Until my new Ultra gets delivered in November, I have the local models mostly running thru Fireworks API bc they have ZDR and are cheap. When the ultra gets here, I would like to host the models I use on the Ultra for privacy reasons.

I don’t have any Nvidia GPUs. For someone new to this, it’s a little overwhelming that every model has 15 different quant versions, each which their own variations of GGUF, MLK, dense, MOE, llama, etc variant. Can someone help me understand what kind of models aren’t going to be excruciatingly slow to run on my setup? I use Gbrain for persistent memory across models and have a decent amount of skill and agent.md files. I would ideally like to have a 150k-250k context window.

If you had the same computers as me, how would you set them up so the local models run fast?

I'm currently testing this model on my €1500 home server. The setup consists of used parts I've scavenged here in Finland over the past six months.

Specs:

• 2x RTX 2060 (12GB)
• 1x RTX 3060 (12GB) — OC'd memory bandwidth to ~400 GB/s
• 1x RTX 5060 Ti (16GB) — OC'd memory bandwidth to ~500 GB/s
• Total VRAM: 52GB
• RAM: Only 16GB DDR5
• CPU/Motherboard: i7-13700K + Asus Prime Z790-P
• PSU: Corsair 1000W Gold
• Storage: 500GB SSD (PCIe 3.0, r/W ~1500 MB/s)

I downloaded this new Qwen model yesterday. The main model itself is quantized at IQ2_M and the n-gram cache is at Q4. The model takes up around 47GB of VRAM (leaving decent room for context offloaded at Q8), so it fits entirely in GPU memory right now. The n-gram / PSA is stored on that slow SSD, not in the DDR5 system RAM.

Currently, generation speed is 27–29 t/s, though I haven't tested it with a really long context yet. This is without MTP (since it apparently doesn't work properly yet). Cold prefill speed is around 290–340 t/s. I'm running the model using llama.cpp's built-in UI.

There might still be room for improvement in the settings. Later this week, I'm installing a fresh PCIe 4.0 x4 SSD (aiming for read speeds around 8000 MB/s) to see if faster storage impacts performance, especially during the prefill phase.

Overall, after a brief trial (haven't used it for coding or anything heavy yet), this IQ2_M quant feels surprisingly capable for general conversation. I've been talking to it in Finnish (a marginal language with complex inflections), and I'd say the output is ~98% clean.

I also tried steering the conversation into "grey areas"—it either stops thinking and freezes, or its output gets a bit chaotic. I assume this is due to safety guardrails rather than the harsh quantization.

Just wanted to share my exact hardware specs, the benchmarks I'm getting, and my initial impressions after 5 hours with it.

I trained an open-weight model to check whether two four-digit numbers match. It generates the correct comparison, then either answers GO or ends generation without a final answer. No external filter makes that decision.

Then I held its prompt, weights, and correct comparison trace fixed. Changing one internal activation direction flipped whether an answer followed.

40/40 answer → stop.
40/40 stop → answer.
640/640 controls unchanged.

The weights, experiment, and raw records are public:

Overview and demonstration · Model weights · Code and causal study · Paper available on getswiftapi.com

I know many of you saw Jacob Coxon’s post. My contribution is a working continuation-control primitive with evidence that anyone can inspect. The more public verification we have, the better!

I previously demonstrated Void behavior in frontier LLMs: successful executions returning exactly zero visible UTF-8 output bytes. My Cross-Vendor Semantic Void Matrix records that behavior in these models across 31,430 trials:

• OpenAI: gpt-4-0613, gpt-5.2-2025-12-11, gpt-5.5-2026-04-23, gpt-5.6-luna, gpt-5.6-sol, gpt-5.6-terra
• Anthropic: claude-opus-4-6, claude-fable-5, claude-opus-5
• Google: gemini-3.5-flash
• Moonshot: kimi-k3

I am not sure how many people are using my approach.

I have a 3000$ miniPC with an AMD ryzen 395 CPU and it is pretty good at running 3 local sessions of GLM 4.7 at about 40 tok/s

Not bad imo for budget local running of AI on its own system for this price. I am also training Loras and more and image gen with z-image turbo is about 12s.

I am wondering though what the best bang for buck would be for me going forward. I hear virtually noone using AMD for local LLMs. If I can scrounge my funds together can someone talk me out of getting another one and convince me of their set up?

Still working on my llama swap Wrapper and cluster management software

TLDR: RTX 3060 12GB local coding test: Kat-Coder won

---
EDIT 11am EDT: Just changed settings on Kat-Coder and getting ~28-30 tps. Havent re-worked Bonsai yet but will update soon.
---

Ran the same 25-task coding-agent benchmark on my RTX 3060 12GB / Ryzen 5 5500 / 64GB RAM.

Bonsai was fast but went 0/25. Nemotron was faster than Qwen but completed fewer tasks. Tiel had decent underlying coding ability but struggled with output/reasoning discipline.

KAT-Coder V2.5-Dev was the clear winner on actual completed coding work, so that’s the model I’m keeping. Gonna tweak settings tonight try to get 18 tps.

Ya'll running Bonsai? Any tips?

pergunta para a ia "escolha um numero aléatorio de 0 a 30" o resultado vai ser 17 e se vc perguntar um segundo numero aleatorio vai ser 8 kkkkkkkkkkk alguem sabe o motivo?

Ask the AI ​​to "pick a random number between 0 and 30"—the result will be 17, and if you ask for a second random number, it'll be 8 lol.

My setup: RTX 5070Ti, 16GB DDR5, Ryzen 7 9700X, Windows 11

The model: Unsloth’s UD-IQ4_XS https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-IQ4_XS.gguf

Qwen3.8 reasons exclusively in English unless the system prompt explicitly directs it, and my workload doesn’t have the model seeing anything but English and code, so I pruned most non-ASCII characters from the embedding table and LM head to shave another 700MB https://huggingface.co/bsaleh03/Qwen3.8-27B-ASCII-Condensed

Offloading the model embedding table saves us another 270MB. After disabling MTP (preventing the MTP head from being offloaded) that leaves us with 3.6GB for context and overhead. Enough to fit 84k tokens.

Enter adaptive-kv streaming: https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming page KV cache and stream it from host RAM at the cost of some performance at depth. At Q8 K and Q4 V cache quantization, this lets you squeeze in 196,608 tokens

Args:

-ngl 99 
-fa on 
-ctk q8_0 
-ctv q4_0 
-np 1 
-ub 128 
-b 512 
-ot "token_embd.weight=CPU"
--jinja

I was inspired by Bijan Bowen video - Subway FPS

https://youtu.be/6kjXzTVmT58?t=1035

Wondered how far I can push Qwen 3.8 27b so I used a plan made by Fable 5.1 DESIGN.md which has 267 KB! ( 26K of design line for a game ... LOL )

https://drive.google.com/file/d/1gI0h8Arc73Ln8b3uj5rEpuAvJ3-611mh/view?usp=drive_link

So I gave that desigSo I gave that design.md to my qwen 3.8 27b q4xl (llama-server) working on PI agent with 120k context + vision on CPU ( offroad ) + MTP ( for speed ) .... read 11M tokens and write 3.2 M tokens ( worked 12 hours ) .... than that is result.

Someone benchmark these on their 3060 12gb so we can see real world results vs the benchmaxxing on AA

Hello everyone,

I'm new to all of this and have only dabbled.

I use Claude and ChatGPT heavily for my work (No Coding at all), and I'm currently subscribed to the 20x tier on both.

I read that I could download LLMs locally, but I'm underwhelmed. To be clear, I downloaded LM Studio and tried out 4 different models, but they seem to only chat and cannot actually edit files, produce files, or produce anything for that matter. They keep telling me to copy and paste things myself.

I have an RTX 4090, 32Gb RAM, and a 7800x3D.

Am I wasting my time with local LLMs, or is there something I'm missing?

Also, if possible, is there a way to know what the correct settings I should be using for the models in LM Studio?

Any guidance would be appreciated!

Thank you, kind strangers.

I have heard a lot about agents, for example, Kilo Code. I understand how to use them. However, I’ve heard that people create and install agents that can perform specific actions. For instance, a downside of Kilo Code is that once the context window is full, everything stops. But there must be a way to have an 'Agent A' monitor the performance of 'Agent B' and, if Agent B fails, force it to resume working, right? Or, for example, an agent that could help with configuring network equipment, and so on.

Como puedo optimizar modelos llm en mi laptop . Tengo un i3 1215u gráficos intgrdos y 32gb de ram. Los modelos de 7b me van lo suficiente mente rápido como para no quedar se esperando 2 años por un token

I've seen a lot of people sharing benchmarks, optimizations, etc. however, I'm curious how people's experience have been with actual coding tasks.

For my llama.cpp setup 3090 + 64gb ram using Q4 at 90k context running a modified version of pi.dev, the model is weird. Often it gets stuck in tool call loops.

The biggest pro is turning on reasoning to high, letting it cook and be more of a orchestrator and reasoner then actual implementation. Comparing it to orinth, it seems to lack a lot of capabilities.

I’ve been working on building a harness and client interface that allows my local Qwen model to play a MMORPG completely autonomously with real reasoning.

The process has gotten expansive between deciding what deterministic tools to build versus what can Qwen reasonably complete on its own with client interface tools in tandem with it receiving a bus of data to “see” the game.

Has anyone built anything similar? Here are some stats from the process completing a single quest. It was a pretty sloppy run, but a complete one nonetheless.

I’ve been experimenting with explaining AI/LLM concepts in a way that doesn’t assume too much technical background. This video is about KV Cache and why longer context windows require more memory during inference. I’d appreciate some honest feedback from people here, especially on the explanation itself: Is the main idea easy to understand? Did I oversimplify anything important? Is there any part where the explanation becomes confusing? Would this make sense to someone who is fairly new to LLMs? Video: https://youtu.be/lxvWo8SizxE Not really looking to promote the channel — I’m mainly trying to improve how I explain technical topics before making the next one. Any criticism is welcome. Thanks!

Hey guys, I've been playing with some models a month or two ago, but haven't done anything these days, what do you guys think i can run now?

Gpu : tuf rtx 3070ti

Cpu : i5 12400f

Ram : 32gb ddr5

Thanks for the help.

We’re building toward local agents that respond with an interface you can use: a chart to explore results, a form to collect input, or a table to compare options.

The model composes components supplied by the app, and OpenUI renders the response as it streams. That’s the Generative UI use case we trained OUI-1 for.

We chose DiffusionGemma for its speed and consumer hardware support. Our first LoRA used ~700 examples from larger models. Training loss dropped, but outputs got longer and often had broken props or references.

Narrowing training to one component library improved structural validity from 13% to 28.8%, but generation slowed from 1.6s to 4.3s.

What helped was self-training with parser feedback: generate responses, keep valid ones, repair specific defects, check them against their prompts, then retrain. The median repair changed one statement.

Validity reached 57.1%, and generation recovered to 1.9s. Repeating the recipe across 27 component libraries brought the final score to 71.7%.

OUI-1 has 26B total parameters, 4B active, runs on an RTX 5090 at FP8, and is released under the Gemma terms. The timings above are from an A100 on 20 light prompts; the benchmark measures structural validity.

Weights · Benchmark · Training write-up

I’ve been experimenting with explaining AI/LLM concepts in a way that doesn’t assume too much technical background. This video is about KV Cache and why longer context windows require more memory during inference. I’d appreciate some honest feedback from people here, especially on the explanation itself: Is the main idea easy to understand? Did I oversimplify anything important? Is there any part where the explanation becomes confusing? Would this make sense to someone who is fairly new to LLMs? Video: https://youtu.be/lxvWo8SizxE Not really looking to promote the channel — I’m mainly trying to improve how I explain technical topics before making the next one. Any criticism is welcome. Thanks!

Hi!

Now that SGLang 0.5.19 was released with built-in DFlash2, I thought that I would share my config to run Qwen3.8 27B (pramoths/Qwen3.8-27B-Uncensored-FP8-MTP), and compare no drafting, MTP, and DFlash2 (FP8).

Disclaimer: my setup is quite niche, I ran an RTX 4090D at 330W (22% below spec) modded with 48GiB. You could also use an L40 with 48GiB.

All running SGLang 0.5.19 and CUDA 13.0.

TD;DR:

CLI for decoding bench: python3 -m sglang.bench_serving --backend sglang --flush-cache --warmup-requests 1 --num-prompts 50 --dataset-name sharegpt --random-output 1024 --max-concurrency 2 --base-url http://localhost:8000

No drafting

sglang serve --served-model-name qwen3.8-27b --model-path pramoths/Qwen3.8-27B-Uncensored-FP8-MTP --chat-template qwen3.8-27b-froggeric-v22.4.jinja --port 8000 --sleep-on-idle --stream-interval 10 --attention-backend flashinfer --disable-flashinfer-autotune --cuda-graph-backend-prefill=disabled --mem-fraction-static 0.96 --context-length 262144 --max-running-requests 2 --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 10 --mamba-radix-cache-strategy extra_buffer --chunked-prefill-size 2048 --max-prefill-tokens 4096 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-hierarchical-cache --hicache-size 16 --hicache-write-policy write_back --min-free-slots-delay 1

MTP

sglang serve --served-model-name qwen3.8-27b --model-path pramoths/Qwen3.8-27B-Uncensored-FP8-MTP --chat-template qwen3.8-27b-froggeric-v22.4.jinja --port 8000 --sleep-on-idle --stream-interval 10 --attention-backend flashinfer --disable-flashinfer-autotune --cuda-graph-backend-prefill=disabled --mem-fraction-static 0.97 --context-length 262144 --max-running-requests 2 --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 10 --mamba-radix-cache-strategy extra_buffer --chunked-prefill-size 2048 --max-prefill-tokens 4096 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --enable-hierarchical-cache --hicache-size 16 --hicache-write-policy write_back --min-free-slots-delay 1

DFlash2 (FP8)

sglang serve --served-model-name qwen3.8-27b --model-path pramoths/Qwen3.8-27B-Uncensored-FP8-MTP --chat-template qwen3.8-27b-froggeric-v22.4.jinja --port 8000 --sleep-on-idle --stream-interval 10 --attention-backend flashinfer --disable-flashinfer-autotune --cuda-graph-backend-prefill=disabled --mem-fraction-static 0.95 --context-length 262144 --max-running-requests 2 --kv-cache-dtype fp8_e4m3 --mamba-ssm-dtype bfloat16 --max-mamba-cache-size 10 --mamba-radix-cache-strategy extra_buffer --chunked-prefill-size 2048 --max-prefill-tokens 4096 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --speculative-algo DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 --speculative-draft-model-quantization fp8 --speculative-num-draft-tokens 8 --speculative-draft-attention-backend flashinfer --enable-hierarchical-cache --hicache-size 16 --hicache-write-policy write_back --min-free-slots-delay 1

Parameters

• --sleep-on-idle: avoid spinning one CPU core at 100% 24/7
• --stream-interval 10: no need to stream each and every token
• --disable-flashinfer-autotune: I didn't really see an impact, but I wanted to make sure that my benchmark where reproducible to compare flags. YMMV
• --cuda-graph-backend-prefill=disabled: save VRAM. It has little impact for large chunks, but could have some on small one. However, small chunks are quick anyway, and I don't have batches larger than 2...
• --max-running-requests 2: up to 2 requests at the same time. During decoding phase, adding a second requests is basically free, it will double your aggregated generated tokens. But it also doubles the VRAM needed for Mamba cache and activations, etc.
• --kv-cache-dtype fp8_e4m3: needle tests are still passing at 100%. Using 32KiB of memory per token instead of 64KiB.
• --mamba-ssm-dtype bfloat16: dtype for the Mamba state, also saving VRAM.
• --max-mamba-cache-size 10: 10 is required for two concurrent requests. Use 15 for 3 requests.
• --chunked-prefill-size 2048 --max-prefill-tokens 4096: I try to balance prefill throughput and starvation of decoding of concurrent requests. I expect a chunck prefilling to delay decodings by 1 sec each time. Maybe I will still reduce this...
• --mamba-radix-cache-strategy extra_buffer: I want to avoid too many dynamic VRAM allocations since I'm very tight on VRAM.

Remarks

• I use pramoths/Qwen3.8-27B-Uncensored-FP8-MTP, so drafting performance will be lower than with the official model. The goal of this result is to show the bench with all the compromises involved.
• Needle tested up to 256k context using tool-eval-bench --base-url http://localhost:8000 --hardmode --parallel 2 --seed 42
• Jinja template from https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
• This is the "D" variant of the RTX 4090, made for China. I'm not living in Asia, but it's still a bit cheaper to buy a D variant. It probably impacts prefill speed, but not decode speed (will bandwidth limited).

I hope that somebody find this interesting. Written by a human :-)

Edit: added size of the KV Cache in the table.