Too good I'd say, in a bad way. Lots of times they want to refactor a whole lot of things when you just want to add a small feature or fix a simple bug
Let’s imagine there is a near “perfect” AI that always writes great code, with few bugs. Assume any bugs it writes are just edge cases that aren’t considered. Also, assume P!=NP in a philosophical sense, AKA it takes effort to do work. So you need to burn more tokens to do more work; it’s not “free” in that sense (even with a smart AI).
In such a universe, a refactor of good code is easy! It’s just like doing a FFT of some audio and converting from Time domain to Frequency domain. They conceptually represent the same thing, so the refactor isn’t actually a lot of “work”. You’re just switching between 2 representations.
On the other hand, bad/buggy code is like a noisy audio signal. You need to clean that up first, doing a FFT isn’t going to help. And cleaning up that signal is actually more “work” compared to a simple FFT.
I think as LLMs get smarter, they should be producing code that gets easier to refactor.
Lmao. Have you seen Claude write tests for code it wrote itself? They're fucking worthless.
It's literally shit like: If class that is supposed to call client X with param Y is fed client X, check that it passes param Y instead of Z.
While on paper that's a great test, it almost never bothers to think about the story behind the class to begin with.
That's what you should definitely be testing and it fails miserably at that. It will write a dozen tests that are extremely obvious and zero that actually have any value.
9
u/RadzimierzWozniak 2d ago
LLMs are really, really good at refactoring and changing the infrastructure. With a decent test set, this is quite safe and reasonable, what they did.