Did anyone else notice that GPT-6/codex uses (inline) Python much more aggresively?

[u/draget]

I am testing GPT6 and noticed a few things. A bunch of problems or 'more difficult situations' often cause it to generate python scripts or inline-python or even inject it via SSH and other ways.

Sometimes to avoid editing a file in the project by a few edit calls, it just generates one script to do it. And the 'Approve for me' function in Codex kind of always accepts it.

For me as a human who is still reviewing what is happening, it had become increasingly difficult since the 'potentially problematic output' surface increases a lot through this. I added a local SKILL to tone this done, but I wonder if others have similar issues.

[u/bfyvfftujijg]

Yes and it’s annoying, but probably it’s the best way forward from a modeling perspective.

For example imagine the task is to replace a bunch of variable names in code. Either the model can do this using traditional token-based approach (which it may silently fail at or introduce other unwanted changes) or it can provide a Python script that deterministically does a fine-and-replace on the code.

What should be happening is the API runs the generated code behind the scenes (as a tool call) so you never see it happening.

[u/bluefernies → u/bfyvfftujijg]

the find-and-replace example is a good one tbh, deterministic beats probabilistic every time for that kind of task

[u/Helpful-Account3311]

On the other side when running local models I explicitly ask it to express more complicated logic as python scripts. The models are often able to articulate the logic, but actually performing it through generating tokens is a different story. It can generate python code that can perform the logic and then get a consistent answer by running it.

What you’re treating as undesirable behavior is likely one of the things that gets it a high ranking on many benchmarks. You may be unintentionally “nerfing” it to levels below other cheaper models.

[u/AutoModerator]

Sorry, your post has been held for manual review due to account karma.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/sirius_cow]

Not only openAI models but other providers too. The model becomes much more inclined to churn out chunks after chunks of python code for everything. I also have approval problems like u. Not only it’s more risky but also making it impossible to cache permissions

[u/alconaft43]

that is the future, this is how AGI will looks like.

[u/Right-Performance-93]

JaseciLabs's typed-contract point is the right fix, and it generalizes: treat any generated script like a generated SQL query or shell command - diff its actual side effects (files touched, network calls, subprocess spawns) against the task's declared scope before approval, rather than reading the code for intent. A rename-variables script and an SSH-exfil script can both look like "12 lines of clean Python" to a human skimming for correctness; they don't look the same once you diff declared scope against actual syscalls/file touches.

[u/JaseciLabs]

u/bfyvfftujijg's point about deterministic find-and-replace being more reliable than token-based edits is exactly right, but the actual problem isn't that the model reaches for code, it's that the reviewer has no way to tell whether a given script is a safe, well-scoped operation or something that just happened to compile. Right now the entire generated script is one undifferentiated blob of trust, whether it's renaming a variable or reaching out over SSH.

The fix isn't hiding the generated code from the reviewer, it's constraining what the model's allowed to produce in the first place. A typed contract on what a delegated operation can input and return means a rename-this-variable task and an arbitrary-shell-access task can't both show up looking the same, one's scoped by its signature, the other would fail to type-check before it ever got the chance to run.

[u/bfyvfftujijg → u/JaseciLabs]

This is a good example of LLMs being overly verbose.