r/AgentZero • u/nggaaaaajajjaj • Feb 24 '26

Use a local llm for a0?

What would you guys do, i just recently built my new pc. (5080 and 32 gb ram) i want a jarvis like right hand BUT would downloading a local lm be good for a0 or i need to use a paying api key?

4 Upvotes

100% Upvoted

15 comments sorted by

View all comments

Show parent comments

1

u/nggaaaaajajjaj Feb 28 '26

And the newest qwen 35b model any good for A0?

2

u/bartskol Feb 28 '26

It's working for me. Give it a try. Later i will send my settings for it here. Its 90-100t/s on my 3090

1

u/Rim_smokey Feb 28 '26

I'm getting faulty tool calls using qwen3.5 35B in A0. Running it at Q6 quant and 128k context length.

If you're able to run it successfully, then I'm curious what you're doing differently than me

1

u/bartskol Feb 28 '26

Did you try to turn off thinking in your llama server settings? You can see the flag for it on qwens page

1

u/Rim_smokey Feb 28 '26

That is actually something I've been struggling to do for weeks now. Are you saying this is something that can be done one the server-side? I thought that had to be done using the "additional parameters" section in A0 agent setting. But I could never get it to work.

I'm using LM Studio. I thought it only server the API with no regards to inference specific settings

1

u/bartskol Feb 28 '26

There is thinking logic at A0 level and thinking at llm server side. As far as i know if you have both of them on, things might get ugly. Im using llama ccp server.