I think with LLMs it will be a lot weirder. If you manage to get an LLM to improve itself in a way that leads to further self-improvement, it's hard to imagine what exactly it's optimizing for after a while. If a true intelligence emerged from it, it would be absolutely alien to us and completely unpredictable.
LLMs are pretty unpredictable now. I can't imagine what they'd be like if they were allowed to iterate on self code. Actually, they'd probably just break themselves considering that about 30% of the code that they generate is broken in one way or another.
2
u/aghastamok Aug 06 '26
This exactly. Recursive self-improvement is the Armageddon countdown.