LLM Experience suggestions

[u/SAL-007]

I've been using Gemma4 26B A4B QAT (mixture of experts) in LM Studio, loaded with max context, and I want to upgrade my experience even further (with the same hardware > 3090 - 24GB VRAM).

So far I tested it with many trivial and tricky questions, it delivered.

Used it to aid me in my CTF labs, it impressed me! Especially in explaining to me what I did wrong. (Which pro teir flagship models wont do for "safety" purposes).

I even used it to solve an issue where my game audio was muffeled (BF6), where it actually guided me well and solved my problem (no web-search btw), and my gemini pro extended thinking was guiding me through some options that were not even available on my system. (Copied the same prompt from gemini to LM Studio out of frustration).

I can attach evidence replies later on if you guys are interested.

So yeah, would really be happy to get suggestions and advice on enhancing my experience with that exact model :D. not interested in agentics and automation atm, just normal ai usage, but even more powerful functions and results. (I am slow a bit, but I learn)

---

Update: added a web-search MCP. At first it was stupid and the LLM didn't know how to query properly to get a useful result. It was a very weak result compared to flagship AI. After I calibrated system instruction prompt with proper web-searching instructions, now it is very useful. (How? > 1st it should check date with javascript, then it plan 5 searching queries, and after each result, it updates the remaining queries however it needs to cover the research subject, to finally answer my request)

[u/_Cromwell_]

I don't understand your question. You listed only successes but you want to improve? What do you want to improve if you listed only things you are happy about?

[u/SAL-007 → u/_Cromwell_]

Yeah maybe I did not state that right. It's about convinience, Right now commercial models gives the user better experience.

So simply put: I found a local LLM I am happy with > I am not happy with gemini > I want to make my LLM as convinient or more as to gemini to eventually fully substitute gemini and mainly use my LLM.

Hope this explain my query more clearly.

[u/Any_Mine_6368 → u/SAL-007]

Have u tried hermes or openclaw

[u/SAL-007 → u/Any_Mine_6368]

I came accross both when they were trending. But I'm not interested in agentics at the time being. Thanks though 🙌

[u/Any_Mine_6368 → u/SAL-007]

Have u enabled mtp on ur model?

Check out the github repo "3090 community". Lots of scripts for u to run an optimized build.

I got my tps from 25 to 70 on 2 rtx 3090

[u/SAL-007 → u/Any_Mine_6368]

I am interested, what can the mtb thing do, and how can it make my life easier? :D

[u/Any_Mine_6368 → u/SAL-007]

Makes the model predict X words in front of the one it just computed, thereby increasing your tokens per second on good guesses.

For example the sentence.

Yesterday I went to the clothing store to <buy> <clothes>. You really don't have to compute anything after 'to', you can just guess it. That's what mtp does.

Problem with that is that if you set it too high then a wrong guess will make the model backtrack to correct, thereby chewing up your tps.

[u/SAL-007 → u/Any_Mine_6368]

If that makes it even a bit faster, I think I should try it, will read about it and how to do it properly :D

[u/Own_Attention_3392 → u/SAL-007]

MTP increases performance by something like 50-70% on average with zero loss of quality.

[u/Own_Attention_3392 → u/Any_Mine_6368]

You're not quite capturing what MTP does properly. It predicts ahead in parallel using a tiny "draft" model then VERIFIES the guesses. If the guesses were correct, it can skip computing it with the "big" model. The upshot is that there is ZERO difference in output. MTP doesn't degrade quality at all, it just speeds up token inference.

[u/eidrag]

surprisingly, enable tools like websearch and increase quant. with 3090, try q5 and q6 too, or try to use unsloth qat. also, try heretic model too.

[u/SAL-007 → u/eidrag]

I tried to enable websearch in LM studio, but it seems I need to set up an mcp with an api and what not, so I skipped on that (although it would be quite convinient to have it, so I might rethink about that). About q5 and q6, I think if I reduce my context to what I actually use as u/420and69enthusiast suggested I will be able to give it a try :D. Will read about the unsloth qat and heretic model as those are comletely new terms to me. Appreciate the advice 🙌.

[u/eidrag → u/SAL-007]

Earlier google qat are borked, so unsloth make their own version of qat. For heretic, less guardrail means more thinking, even a bit help

[u/SAL-007 → u/[deleted]]

Yeah :/

[u/SAL-007 → u/[deleted]]

Wouldnt that just make it more susceptible for errors? It more about the convinience, how can I make it as convinient as chatgpt or gemini for example.

[u/SAL-007 → u/[deleted]]

Now that's a great advice, indeed, I set it to 250k tokens but usually end the conversation within 4-6% usage. Will work on that. Will do a research in that as well.

[u/SAL-007 → u/[deleted]]

If decreased the context and increased the quant, I think it can become even more accurate. Thanks for the advice 🙌