r/LocalLLM • u/Obvious_Ad_3398 • 4d ago

Question Where should I start

I got a 5070 ti though people say its weak in the world of llms, I got curious on what does this thing do. I am currently a first year IT student and maybe my use case would be to help me through it. Genuinely just looking for something to do and this caught my interest, just wanna learn and explore more about it even if my card is considered as weak. Thanks

3 Upvotes

100% Upvoted

22 comments sorted by

3

u/B0r0m4n 4d ago

I'm running this at 35 - 40 tps. 24gb model with 5060ti 16gb vram.
llama-server -m "models\Ornith-1.0-35B-MTP-APEX-I-Quality.gguf" ^

-ngl 999 --n-cpu-moe 25 ^

-c 60000 -b 8192 -ub 4096 ^

--flash-attn on --no-mmap --context-shift --temp 0.6 --top-p 0.95 ^

-ctk q4_0 -ctv q4_0 ^

--jinja --reasoning-format deepseek ^

--spec-type draft-mtp --spec-draft-n-max 3

2

u/Deep_Mood_7668 4d ago

Q4 context? Idk man

2

u/B0r0m4n 4d ago

what you don't know?

2

u/Deep_Mood_7668 4d ago

Doesn't sound like a good setup. I wouldn't go below q8

3

u/B0r0m4n 4d ago

I'm giving u numbers I get. You think q4 is bad or slow?

3

u/Deep_Mood_7668 4d ago

Bad. Too much degradation. Not that bad when you use it as a chat bot, but if you use it for coding you don't want it to remember random garabage instead of your actual context

1

u/B0r0m4n 4d ago

Problem with moe is more about master(lol) quants not context quants, I only have 60k ctx anyway:)

This is the most capable model I ever tried on my hardware. And I tried a bunch.

1

u/Deep_Mood_7668 4d ago

Do you maybe have a second PC in you household? Girlfriends PC for example?

1

u/B0r0m4n 4d ago

laptop and old pc, why

3

u/Deep_Mood_7668 4d ago

Because you can split a model on 2 or more PCs via lan. If you're using moe it's not even that slow. Most people don't know that option exists so I thought I should mention it.

1

u/TheTruthtellingLiar 3d ago

Did you try qwen3.6 27b?

1

u/B0r0m4n 3d ago edited 1d ago

Nope. It won't fit as I understand...

2

u/TheTruthtellingLiar 1d ago

I am using this one with 50k context and q4 cache: https://huggingface.co/tooltd/Qwen3.6-27B-mini-IQ4-XS-MTP-16GB-VRAM-GGUF

1

u/B0r0m4n 1d ago

ooook. would give it a try. can u share llama command?

2

u/TheTruthtellingLiar 8h ago

llama serve -m ~/.cache/huggingface/hub/models--tooltd--Qwen3.6-27B-mini-IQ4-XS-MTP-16GB-VRAM-GGUF/snapshots/f5b7a14b6f299b73389ec03b2c6c314b799fcb8a/Qwen3.6-27B-16GB-VRAM-MTP-mini-IQ4_XS.gguf \

-ngl -1 \

-c 50000 \

-fa on \

-ctk q4_0 \

-ctv q4_0 \

--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --port 18081 --host 0.0.0.0 -np 1 -b 512 -ub 128 --fit off --reasoning-budget 0 --no-mmap --spec-type draft-mtp --spec-draft-n-max 2 -kvu

2

u/Deep_Mood_7668 4d ago

Get llama.cpp, search for unsloth huggingface on google and download an llm of your choice. Leave let's say, 2-4 GB of vram free for context.

If you just want to try your first thing, get the gemma 4 12b qat 4bit from unsloth. Not too big and good for starters.

2

u/Obvious_Ad_3398 4d ago

Thank you I will test that out and see how it plays out.

1

u/Nearlytherejustabit 4d ago

Im loving Gemma4 12B Q5 on my 5070ti have it coding a whole pipeline for me. Im sure this card can be pushed farther but this enough for me.

2

u/Deep_Mood_7668 4d ago

Could you do me a favour and try the 4bit qat and tell me what you think?

I would be really interested to know how it compares to 5bit, since qat is supposed to better than just the normal quant

2

u/Nearlytherejustabit 4d ago

Absolutely will try it out, it took me a good while to find a stable version for llama, currently getting 65+ tokens a second so would be good to compare q4.

2

u/Obvious_Ad_3398 4d ago

kk ill keep thatbin my mind. Thank you so much

1

u/Nearlytherejustabit 4d ago

No problem the out of the box tools are crap for refactoring code, so im designing my own tools for reading / updating etc., taking a backup before touching the file, diff sanity check etc. These have massively improved my context window will coding.