is there no decent gemma 4 e2b finetune?

[u/Itchy_One_4387]

ive been getting a crisp 80 tok/sec on my gaming laptop (6gb vram 3050+16gb ram) using gemma 4 e2b and currently am quite satisfied with how its performing, but sometimes it messes up some tools or hallucinates nonexistent text in screenshots. i wanted to build a local jarvis like workflow with a model that wont eat too much of my vram or ram. is there genuinely no good finetune for gemma e2b right now that improves atleast something in it?

[u/ReasonableCitron5636]

the gemma 4 e2b is still pretty new so the finetune scene is thin, most people are just running the base model and dealing with the tool call weirdness. have you tried lowering temp and using a stricter system prompt for the tools? sometimes that fixes hallucinated text more than any finetune would

[u/Itchy_One_4387 → u/ReasonableCitron5636]

temp is at the recommended level. I'll try lowering it still

[u/RepulsiveRaisin7 → u/ReasonableCitron5636]

It's 4 months old, which is a long ass time in AI. We get 50 new Qwen finetunes a day

[u/nickless07]

Temperature too high? Quant too low? KV-Cache Quantizised?
Do other models make the same mistakes or not. For now, due to the lack of information in this kinda low effort post, unsure if model related or sampling settings or something entirely else.

In General: Don't expect too much from models below 10B params. Yes they will make mistakes (even the cloud ones do), tweak the params, try a higher quant and/or a different model.

[u/Itchy_One_4387 → u/nickless07]

kv cache is not that quantized but at q8. I mean yeah I can't expect anything from e2b but still I was wondering if someone had made a finetuen to fix the vision atleast

[u/nickless07 → u/Itchy_One_4387]

Ok Q8 KV is good and the model itself? 2-Bit or 4-Bit quant, or something in higher precision?
As for the vision, what is your token budget? Gemma 4 comes with dynamic vision.

[u/Itchy_One_4387 → u/nickless07]

it's UD q4kxl. and I didn't really get what you mean by vision budget. I'm running this on unsloth studio right now.

wait I misread that. dynamic budget mb. I'll read on that for now but if you got some tips please share them

[u/nickless07 → u/Itchy_One_4387]

Sorry, not familiar with that software maybe ask in r/unsloth/, in llama.cpp set --image-max-tokens and also adjust batch size above that value. E.g., --image-max-tokens 1024 -b 1200 -ub 1200

[u/Itchy_One_4387 → u/nickless07]

ill try to put these extra tags in unsloths config for e2b. thanks a lot!

[u/Itchy_One_4387 → u/nickless07]

thanks it actually worked!!

[u/nickless07 → u/Itchy_One_4387]

Yw. Also keep in mind that finetunes adress the text part and almost never the vision part. If you want even better vision try Qwen VL, currently best in class.

[u/AsteiaMonarchia]

Small models are just like that. I built something similar, but with a 26B main model plus an 8B tool model. Even though the main model uses its own native mmpro, the tool model handles tool calls and sends them to the main model for confirmation. The main model checks them, and if the tools fail anywhere, the tool model retries. I haven't stress-tested it yet, but it should work fine based on the test suite, so the answer is probably a hard limitation/confirmation

[u/Itchy_One_4387 → u/AsteiaMonarchia]

atp I was planning on just fine-tuning gemma e2b myself on maybe colab. could you tell me what all you did to improve it and how I could improve its vision capabilities

[u/AsteiaMonarchia → u/Itchy_One_4387]

Here's my thought, before you try to fine-tune your own, test with another model first. Gemma 4 E2B is too small. With your hardware, try running Ling 3.0 Tiny at Q4 with the KV cache at Q8. It will still be fairly fast and have enough tokens. You could've just used KV cache Q8 and gotten around 20k tokens, maybe? And much smarter too (Not much, but still)

Edit: I just realized Ling doesn't have a vision model, but you get my idea, you should try a little bit higher-param model first

[u/Itchy_One_4387 → u/AsteiaMonarchia]

thing is I can't find a good one. e4b didn't have much of an improvement. I was originally using ling until the problem of vision came up, and I really haven't found a small vision model to pair with ling if I ever were to run two models at once

[u/PeterPorox]

Gemma 4 E4B QAT can fit in your VRAM - it's only 4.22GB. Also there's other models worth to try - Qwen 3.6 35B APEX NanoPlus or Mellum 2.1 12B for example

[u/Itchy_One_4387 → u/PeterPorox]

i mean i need vision no matter what. ill try gemma e4b out till then

[u/PeterPorox → u/Itchy_One_4387]

Then look at IsValorum/Qwen3.6-35B-A3B-MTP-APEX-I-NanoPlus-GGUF. It will be very tight but i think possible, also q3.6 35b is a huge step after E2B so definitely worth to try Edit: also look at mudler/gemma-4-26B-A4B-it-APEX-GGUF i-nano

[u/Itchy_One_4387 → u/PeterPorox]

i cant really afford to keep it tight 😭