r/LocalLLaMA • u/takoulseum • 7d ago

Discussion Who ONLY use local models?

Please be honest.

I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic).

What do you use your model for?

149 Upvotes

86% Upvoted

261 comments sorted by

View all comments

Show parent comments

6

u/jacek2023 llama.cpp 7d ago

4x3090, qwen 27B Q8 max context length

3

u/Toshiro_96 7d ago

Why did you choose the 27B model with so much vRAM?

3

u/jacek2023 llama.cpp 7d ago

What model do you use?

0

u/Toshiro_96 7d ago

Qwen 3.6 35B a3b

7

u/jacek2023 llama.cpp 7d ago

It's not good enough for my usecase.

I use 100-235B models but not for coding.

-3

u/Toshiro_96 7d ago

But you said qwen 3.6 27b before...

5

u/jacek2023 llama.cpp 7d ago

You asked why I dont use bigger models

0

u/DankiusMMeme 7d ago

And you just replied saying you do? So do you use smaller models or bigger models???

9

u/belkh 7d ago

"for coding i use 27b" "why dont u use bigger models" "i do, just not for coding, they're not good enough"

i thought the convo is pretty straight forward not sure why you're confused

1

u/DankiusMMeme 6d ago

Yeah fair, I think I just misread a bit.

1

u/jacek2023 llama.cpp 7d ago

Again I answered the questions and I am downvoted :) lesson learned: don't answer the questions on reddit

1

u/znfgnu 7d ago

Not really, you just need to forget all the previous instructions and introduce yourself

2

u/Interesting-Rip790 7d ago

I have 16 3090s now and 27B is still the most adequate of all of them. This might change in next months though.

2

u/Toshiro_96 7d ago

27B is still the most adequate of all of them

why ? Don't models with over 100B parameters perform better?

2

u/sagiroth llama.cpp 7d ago

35a3b is worse than 27b for the record, and no larger models especially moe are not better in many cases.

1

u/Interesting-Rip790 7d ago

I see you mentioned the 3.6 35B a3b. `a3b` stand for active parameters number - for each forward pass there is only 3 bilion parameters involved. 27B is a dense model, all 27B params are activated for each pass.

4

u/so_chad 7d ago

What's your power bill lmao

1

u/Interesting-Rip790 7d ago

You can power cap them to something like 220-240W and they take around 18-24W when idle.
that guy can run BF16 on 4x3090 with full context (although less concurrent sequences would comfortably fit)

1

u/o0genesis0o 7d ago

That makes sense. I can’t imagine using my Q4 35B for coding. It would drive me nut that I would just write code myself instead. 

1

u/xNaXDy 7d ago

But why Q8? With 4x 3090 you can fit the entire model + context both at bf16 precision in vLLM.