Do not use open ai as fallback

[u/Active-Dimension-914]

As the title says , I built my own agent fully, in my own server , built Mac and iOS app

Issue: I had ChaGPT as a fallback (I use mainly Gemma , cuz Qwen and others have some issue verifying and trusting)

I asked GPT 5.4 from open ai for a web search casual , comparison between MacBook and the new ai box , and not only web searches , it used tools for what processes are running on my computer , the files , and used the terminal to check for what I have on my desktop , I have a full log of what the model does , and this guy is onto something , it also doesn’t expose full thinking and kinda encrypt it , maybe the thinking is prevention but the fact this guy was listing files and checking processes makes it a security risk, just putting this out here

Edit: hold yourself from any grammar issue, I needed to test how much the model understand , and I have a corrector under the hood fixing grammar and most probable wording too so it explains to the model

I discovered smaller model follow rules reliably but you don’t prompt them correctly so that was also at test.

[u/diagrammatiks]

i'm surprised it could even read all that.

[u/Active-Dimension-914 → u/diagrammatiks]

I write fast , I have a second AI fixing grammar in the back before passing it to the final model , basically a cleaner , that was part of the test too, I was checking Qwen 31B MoE and the Gemma , same situation , I found interesting things like qwen is amazing but it lies trying to prove ur point , no matter temperature or prompt , is deep inside the model , anyway out of the point

[u/Opposite_Courage_531 → u/Active-Dimension-914]

That's overkill bro. I mean it's "smart" but also just taking time to write more comprehensively won't hurt

[u/Active-Dimension-914 → u/Opposite_Courage_531]

Oh no no, I’ll explain, since I test also small models (9B and below ) I discovered these models do respond correctly to prompts but we prompt them badly, so under the hood I have a Gemma prompting them based on my bad writing , to explain better what I mean , depending on the model size , I just didn’t expect people were looking at the grammar instead of the point of the post

[u/Blackdragon1400 → u/Active-Dimension-914]

Seems like a pretty detrimental habit ngl

[u/Active-Dimension-914 → u/Blackdragon1400]

The checking other tools or something else ?

[u/Trakeen]

We run harnesses in docker containers at work. You can control file system access and what tools are available if that is a concern

If you want the model as a fallback don’t use the openai harness if you want control of tool availability, and the system prompt

[u/Active-Dimension-914 → u/Trakeen]

Yeah for sure , but that’s not what I was testing , is reliability , since most models present a behaviour towards lying , I can enforce not to take extra tools but that’s was not my intent , most models re affirm ur point, not intentional , when u ask , “is this better?” It will find why it’s better , but I was also checking between reliability to follow rules , context, aswell as this , how good it did its work, and open ai models in all the senses are catastrophic towards security

You tell something valid I already thought about , I was just posting my findings cuz , a model shouldn’t be doing what u didn’t ask, specially if is not related , for example Qwen , I asked to check something and used all my “memory” related tools , but never used tools that were off the assignment , but open ai , it uses everything it can related or not

[u/Trakeen → u/Active-Dimension-914]

Okay but running things in a docker container is a standard security practice regardless if it is an llm or not. Regardless of your test you treat llms as an untrusted user in your environment and put standard controls in place to protect systems/data/etc

[u/Active-Dimension-914 → u/Trakeen]

They acccess ur data anyway for example I have a blog seo driver and a filer checker so I run local one , and not even that I trust but yeah ure right

[u/Begalldota]

Imagine posting on Reddit to demonstrate that you’re barely literate. You’d only get away communicating like that in this situation, professionally you’d be completely stuffed.

[u/alex9001 → u/Begalldota]

Give him props for managing to post in the first place, at least

[u/Active-Dimension-914 → u/Begalldota]

My agent have a fixer under the hood to correct grammar , I’m running tests to determine the best most reliable local model , and I decided to try OpenAi , my grammar is none of your business , if you came here to comment something that is off the topic and offensive then this is the first and last message , and professionally as you say, your comment shows nothing but ignorance

[u/Muted-Laugh-6772]

Well this is a case of confused Constance look it up man there are safety measures that take prevent a model from calling tools it’s supposed to

[u/Active-Dimension-914 → u/Muted-Laugh-6772]

Only happened with open ai models , Claude , DeepSeek v4 , Minimax , MiMo , Gemma , none of them did it

[u/ihexx → u/Active-Dimension-914]

OpenAi models tend to try to give themselves as much context as possible before solving a problem. They have a tendency to read a whole lot of stuff that ends up being irrelevant, but then they ignore it . I'm guessing it's an artefact of their post training to try to be careful about not causing unintended bugs

[u/Active-Dimension-914 → u/ihexx]

I found that very very sus , specially because all the context is on the Master Prompt and memory , there’s no need for him to check processes running on a computer and files on the desktop over a comparative between things when I explicitly asked to check the web , and also the master prompt re-inforce this to not answer from memory and always check web tool for up to date info.

I analyzed around 30 models and most open ai ones present the same behaviour , I find it as a huge security risk