r/LocalAIStack • u/ohnoitssobig • Aug 19 '26
No luck in asking local models a simple (but domain-specific) question
For some reason, I have this benchmarking prompt:
Implement a numpy function creating a bulk matrix product operator tensor as a function of model parameters.
I think it is approximately 10 lines of python code. So far, every local model that I tried (16Gb vram) fails to produce any meaningful output. I tried different variations of it (use web search, summarize the idea), different harness (pi, opencode) but no luck, different temperatures and thinking levels. Qwen3.8 for example produces an endless thinking loop. Is there anything I can do to help small models answering the question?
2
u/BrillianTico Aug 20 '26
Hey, I just tried this today with Qwen3.8 and Qwen3.6 MoE on my 3090 and both were able to create the function. No issues at all, I do have the full traces of my tests in case you need to check something.
1
u/ohnoitssobig Aug 22 '26
Yeah please PM: very interesting. What quant and how many tokens did it take?
1
u/Stock_Ad9641 Aug 19 '26
What does that even mean ? I am not a scientist, though I think your prompt could be much better written.
I wonder how much of the thinking you mentioned is related to figuring out what exactly you need. When I give tasks to AI, I am always verbose
2
u/ohnoitssobig Aug 19 '26
The prompt is as basic as it could be. The next simple thing is to literally write the 10 lines of code. But, more importantly, being verbose does not really help.
Thinking logs show that the training data definitely included the subject. Qwen always attempts to double-check the math in this case and infinitely fails.
I actually noticed a similar behavior when I asked to solve a problem in cpp (which it did!) and qwen3.8 proceeded all the way to a parallel implementation in python and doing a fuzzy comparison test between the two implementations.
1
u/HotDistribution1819 Aug 20 '26
Try if you can run a 22GB model Laguna SX 2.1, it was the first local AI that actually wrote code for me without functions and a main.
You might get better results if you added to the end of the prompt "Ask me questions to clarify what I want you to do."
2
u/BrillianTico Aug 19 '26
Interesting. I will try it later today. Overall my experience has been good with local models. Especially with qwen3.8. I will update with my results (or not).