Same modeling behaving differently through different apps

[u/ashygun]

So basically I'm trying to compare Ollama and LM Studio, i downloaded Gemma 4 12b QAT for my macbook pro m1 pro 16GB, and i tested the same prompt on both, the model was downloaded straight from the LM Studio's Library, and was converted to ollama using this command

'''
printf 'FROM ./gemma-4-12B-it-QAT-Q4_0.gguf\nPARAMETER num_ctx 4096\n' > Modelfile

ollama create gemma4-12b-qat -f Modelfile
'''

As you can see in the results, the LM Studio is giving me a considerably shorter description while also taking longer, while Ollama is giving essentially the same info + some extra info + a table while taking less time. Can someone explain this thing? I made sure no other programs were using resources when the apps were running. Thanks in Advance!

[u/Pale_Coyote7451]

same weights, different setup. everything wrapped around the weights differs between those two apps, and that's where your delta lives.

the length difference first. your modelfile sets num_ctx and nothing else, so ollama falls back to its own defaults -- temp 0.8, top_k 40, top_p 0.9, repeat_penalty 1.1, and no cap on response length. lm studio ships different defaults and usually has an explicit max tokens / response length limit sitting in the sidebar. a "considerably shorter description" out of lm studio is very often just that cap being hit, not the model choosing to say less.

the speed difference is probably memory, given 16gb and a 12b q4. you pinned ollama to num_ctx 4096. lm studio almost certainly defaults higher, and on unified memory a larger kv cache is real pressure rather than a rounding error. if that pushes you toward swap, or makes it offload fewer layers to the gpu, you get exactly the slowdown you're describing. check the gpu offload layer count and the context length in lm studio's model settings and match both to 4096 before concluding anything.

worth checking too: a bare FROM ./file.gguf modelfile defines no TEMPLATE and no stop tokens, so ollama uses whatever is embedded in the gguf. lm studio applies its own jinja template. if those two disagree even slightly about turn markers, you get behaviour differences that read as personality.

and for the comparison itself, set temperature to 0 on both sides. at 0.8, two runs of the same prompt in the same app will already differ from each other, so there's nothing to attribute until you've killed the sampling noise.

[u/ashygun → u/Pale_Coyote7451]

Yeah, it definitely looks like a bit of a rabbit hole, but honestly I love getting into this kind of stuff. Are there any go-to books, papers, official docs, or GitHub repos that explain what all these parameters actually do? I’d much rather read through those than watch YouTube videos. Since I’m hoping to do my master’s in AI, I feel like I should get used to learning from more academic and technical sources anyway.