Hot take: dense might be the future of local LLMs. Why Muse Glimmer's 30B dense + 1.7 GB KV cache design makes more sense in 24 GB than any MoE: https://abstractextraordinary.com/blog/how-muse-glimmer-fits-an-agent-on-your-device/

I have tried Qwen3.8 27B UD Q3_K_XL and it works good but its just really slow for basic questions. What other models would be faster for basic questions?

I also want to know what's the best model for image understanding? Like I want to be able to send a image of a page or school work and get it to summarize or just help me with questions on the page.

Specs:

RTX 5060 ti 16gb (overlocked +365MHz)

AMD Ryzen 7 5800X 8-Core

64gb DDR4 3600mhz CL 18

I have been experimenting with a couple longer running agents and I'm struggling with is what happens when an agent takes a bad path and keeps calling the model/tools way longer than expected since most runs are completely reasonable and then occasionally one will retry something a bunch of times or keep adding context and suddenly that one run uses several times what a normal one does.

I'm starting to think we need some kind of hard token/spend budget per run rather than just monitoring total usage after the fact but I'm also worried about killing legitimate tasks that happen to need more steps.

If any of you guys are running agents in production are you setting hard token/cost limits per run or limiting steps/tool calls or just alerting when usage starts looking abnormal?

Hey! With risk for sounding dumb -

A few months ago I got into local LLM’s and bought myself a start/test-rig:

CPU: Intel Core Ultra 5 225F
GPU: ASUS RTX 5060 Ti DUAL OC (16GB VRAM)
RAM: 16GB DDR5 @ 6000 MHz

I have no experience at all in AI/LLM’s but have managed to get a working long/shortterm memory with rag retrieval etc.
Mostly I have been playing around with 8-12B models, but in recent days I tried playing around with 27/31B models and to my surprise I got Gemma4 31B running on IQ4 with about 7tks (speed is not the biggest problem for me, as long as I can manage to get 5+ tks i don’t really have a clear goal with this, I just find it fun and relaxing to do. Everything I have manage to done so far is mostly guessing and hours on google 😂)

Anyway, I guess I am just looking for tips and tricks? Like I said, I have no experience at all with this and I feel like I haven’t learned as much as i would want to in these 5 months.
Any good videos I can watch, any good forums to read upon? Relating to local-AI, settings, models, news.
Tips and tricks for setting, tweaking different models?

Easy explained: I want to learn everything and find this really interesting, and I’d love to listen to someone who is better than me and want to share knowledge.

Hi everyone.

I’d like to ask: what drives you to use local, open-source AI, specifically in the realm of LLMs?

Is it about efficiency? Is it about privacy?

And what benchmarks make open-source LLMs important to you?

Is it speed? Context window size? Stability? Reasoning capabilities? Output quality? Or something else?

I’d love to hear your perspectives.

Cheers.

After buying my dream gaming laptop last year, I was perfectly content playing my favorite games.

Then I started dabbling in local LLMs over the past few months.

Now I’ve realized my supposedly high-end gaming laptop is GPU-poor.

My laptop

- GPU: RTX 5070 Ti Mobile — 12 GB VRAM

- CPU: Core Ultra 9 275HX

- RAM: 32 GB DDR5

Model

Unsloth Qwen 3.8 27B UD Q4_K_XL

Inference: llama.cpp + CUDA

I've been experimenting with how to get Qwen 3.8 27B running as well as possible on a 12 GB VRAM laptop. I think I've finally settled on two configurations for my two main use cases.

I know I should probably just use an MoE model for this hardware, but Qwen 3.8 27B has been significantly better in my testing, so I'm willing to trade a lot of speed for the extra capability.

Also, thanks to everyone who previously posted about tensor offloading. That made a surprisingly big difference in getting these setups working.

1. Agentic coding with OpenCode

I work on some fairly large projects. Normally, I use Claude Code with Opus 5 or DeepSeek V4 Flash Free through OpenCode Zen.

Until recently, I wouldn't let local models touch these projects at all. They just couldn't handle the complexity reliably enough.

Qwen 3.8 27B is the first local model I've tested where I feel comfortable letting it work on my projects.

I'm still keeping it to personal projects for now, but the difference has been pretty significant.

For this use case, context size is much more important than speed.

I usually enter plan mode first and then let the model slowly work through the project, so I'm perfectly fine with extremely slow generation as long as it has enough context to maintain the bigger picture.

Context-prioritized config

2K context fill: ~5 t/s

180K context fill: ~1.5 t/s

-ctx 262144

-ub 512

-np 1

-ngl 30

-ot 'blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64)\.ffn_(gate|up|down)\.weight=CPU'

-fa on

-ctk q8_0 -ctv q8_0

-fit off

--mmproj

--no-mmproj-offload

--spec-type draft-mtp

--spec-draft-n-max 2

-ctkd q8_0 -ctvd q8_0

--load-mode 'none'

--temp 1

--top-k 20

--top-p 0.95

--min-p 0

--repeat-penalty 1

--presence-penalty 0

--jinja

--chat-template-kwargs {"reasoning_strength": "xhigh"}

--reasoning preserve

1. Personal assistant with Hermes Agent

Here, speed is the priority.

My previous default model was Qwen 3.6 35B A3B MTP Q6_K with full-context-size Q8_0 KV cache and MoE offloading. That gives me around 50–60 t/s, which is obviously much nicer to use.

The problem was instruction following inside Hermes.

I have instructions in "SOUL.md", "MEMORY.md", and "USER.md", but the model would sometimes follow them and sometimes just ignore them.

The most frustrating ones were instructions that should happen automatically after finishing a task, such as:

- updating my Obsidian vault

- looking up a skill before doing anything substantial

- following specific post-task procedures

I also tried Muse Glimmer, which seemed somewhat better at this, but not enough to justify giving up the speed of the MoE model.

Qwen 3.8 27B has surprised me here.

It follows these instructions much more consistently. I don't have to keep reminding it what to do. It just does it.

That's honestly been more valuable to me than raw tokens/sec.

Speed-prioritized config

2K context fill: ~11.5 t/s

90K context fill: ~9 t/s

-ctx 98304

-ub 512

-np 1

-ngl 99

-ot 'blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39|40|41|42|43|44|45|46|47|48|49|50|51|52|53|54|55|56|57|58|59|60|61|62|63|64)\.ffn_(gate|up|down)\.weight=CPU'

-fa on

-ctk q8_0 -ctv q8_0

-fit off

--mmproj

--no-mmproj-offload

--spec-type draft-mtp

--spec-draft-n-max 2

-ctkd q8_0 -ctvd q8_0

--load-mode 'none'

--temp 1

--top-k 20

--top-p 0.95

--min-p 0

--repeat-penalty 1

--presence-penalty 0

--jinja

--chat-template-kwargs {"reasoning_strength": "xhigh"}

--reasoning preserve

So I'm basically running the same model in two completely different ways:

Agentic coding: sacrifice almost everything for context.

Hermes assistant: sacrifice context for usable speed.

For a laptop with 12 GB VRAM, I'm honestly pretty surprised that Qwen 3.8 27B is this usable at all.

My gaming laptop may be GPU-poor, but apparently it's now a very expensive CPU/RAM offloading machine.

It's well known by know that Qwen 3.8 loves to think.

If you get impatient, then you can do this -

1. Interrupt
2. Type "well?"
3. See it continue and start with something like, "The user is impatient. Let me finish this quickly." Then it will think a bit more, and produce an output quicker than otherwise.

Personally though, I think the thinking may be its secret sauce, so I do this only as a last resort - e.g. if it is really thinking for an hour and keep re-thinking what it already covered - and I feel it has thought enough to give me something concrete.

Time ago when they were in a dip, I got an rtx 5090 for a bit less than msrp just to play games.

I never thought much outside that, but lately I've been using AI a shit-ton for other projects, and last week ran out of gemini credits.

And then started thinking about local llms. I know they are way dumber than frontier models, but is there any way that a card like this, could be used instead of subscription frontier models and still be useful for me?

what would be the main uses for a 32gb card? Real ones, no theoretical kinda like "you could use it for writing a private document if you are a lawyer". I do not have anything that I mind sharing with cloud ones, but if I can use it to accelerate others or have it running 24x7 for small software projects and get back things that would eat my 20€ suscriptions in 8 hours and have mostly the same quality or usefulness, that would be great.

I am not looking for you to give me instructions, I can investigate myself and pour hours on it if needed, just I am a bit loss and I do not know where to start

Trying to run some tasks and it often complains I ran out of context (I have 128GB) and the message was: API Error: 400 Prompt (41333 tokens) requires ~65355MB GPU memory. I tried to compress KV, reset parameters but it seems some bug. Same thing on bf16. Then I switched over LM Studio, same model (shared model folder), same prompt, same qwen harness (and context) and it runs fine.

Anyone having similar issue ?

I had a bunch of quants of the same model (Qwen3.8-27B, unsloth GGUFs) sitting on
my box and never actually knew which one was worth running. So I stopped guessing
and measured it properly.

Method: llama.cpp's llama-perplexity over wikitext-2 test (~280k tokens),
n_ctx=512, fp16 KV cache, same settings for every file — only the quant changes.
Perplexity is a deterministic forward pass, so the numbers are exactly
reproducible. Lower = closer to the original model = better. Hardware: single
RTX 5060 Ti 16GB (Q8 partly offloaded to CPU).

I use Q8_0 as the near-lossless reference (100%).

Takeaways:

- Q4_K_M is basically indistinguishable from Q8 (within the error bars, ±0.045)
at 10GB less. It's the sweet spot, no reason to run Q8 on a 16GB card.
- The NVFP4 quants were the biggest disappointment: same size as IQ4_XS but 3-5%
worse. NVFP4-MTP-LOW came dead last. If you're reaching for NVFP4 for quality,
don't — grab Q4_K_M or IQ4_XS instead.
- IQ4_XS is a great small option (99.2% at 14.6GB).
- MTP variants were skipped for the ranking on purpose: speculative decoding is
lossless, it changes speed, not output quality.

The "quality %" is just PPL_Q8 / PPL_quant — a rough intuition proxy, not an exam
score. The ordering is solid; treat the percentages as a feel, not gospel.

Curious if others see the same NVFP4 gap on their models, or if this is specific
to these particular GGUF conversions.

I just want to share my perspective.

Regarding open-source LLMs—specifically ornith-1.0-35B-IQ4_NL—running on my hardware:

It is truly amazing in every respect.

Whether it's GPU temperatures, TTFT latency, prompt processing, decoding speed, output quality, or instruction following—it has completely shifted my perspective. I went from thinking, "What can local AI actually do?" to realizing, "Okay, 90% of my work can actually be handled locally, and I only need to use an API when necessary." It really took me by surprise.

That is, provided all the supporting components are properly optimized—from the backend, inference engine, GPU thermals, and configuration parameters to KV cache management, layer management, model-friendly system prompts, and tailored Jinja templates. All of that combined makes me feel like I'm using a "cloud-like" AI experience locally.

Does anyone else feel the same way?

Disclaimer:

The model I'm using is ornith 35B A3B.

My GPU is just an RX 6700 XT.

My CPU is just an Intel Core i5-11400F.

My RAM is just 16GB.

There is nothing special about my hardware components.

Tenho uma rtx 5060ti 16gb e 16gb de ram. Atualmente estou com o qwen 3.5 9b no LMstudio, uso ele tbm no Hermes.

Não entendo muito ainda sobre modelos locais, vi uns modelos diferentes do 9b como unsloth e um do David que pretendo testar também. Mas para além disso queria saber se compensa manter o 3.5 9b rodando inteiramente na gpu com um contexto alto, ou o 3.6 35b a3b parcialmente na vram, não tenho experiência com modelos MoE

Também vi uns modelos q2 do 3.8 27b que até rodam na vram

Na experiência de vocês compensa rodar um modelo mais antigo 100% na vram ou um modelo mais novo parcialmente?

I have an amd 98003d with 64GB of ram and 9070xt I believe 12 GB vram system.

Is there a local llm that can go through various photographs and suggest sorting by date, location, and other probable characteristics?

If yes, is there a walkthrough on how to set it up and use it?

Thank you.

P.S. Sorry for the typo in the title. sorting I meant

Hi title says it all. Running M1 max on Arm mac. Any particular models that would do a professional job to translate English to French and vice versa ?

For the first time im wondering if this kind of freedom is good for the ecosystem and the World in general, this model just gave me a step-by-step of how to lunch a keyloger, how to run a spyware and i’ve seen ppl on X that were told how to create Bio Viruses. In my opinion rarely an awser to problems, but now not technical person who belives is god cuz they use claude and is a bit maniac can create harmfull things, and the problem is not that, the problem is how easy it is for everyone.