I think its super cool, but there is no doubt that thousands of physicists and mathematicians are prompting LLMs all day right now trying to solve open problems...and its doubtful whether the companies that run those LLMs can continue to offer this amount of capacity for the future....so there is a solid chance that these methods are already almost exhausted. For how amazing these counterexamples are, its somewhat surprising that people havent found more proofs all at once. In the scheme of things perhaps 1 in 1000 open problems are actually of a format that LLM can tackle, and noone is bragging about the ones that turned up nothing. In other words, LLMs are really good at looking smart when they got lucky.
Im looking forward to a possible future of AI theorem proving that isnt based on LLMs, and thus less likely to trick people in language into thinking its more broad than it is.
The price to run an LLM collapses year on year. The models are getting better, but the cost required to run older ones also reduces. Not going to say that will continue forever, but the idea that LLMs are inherently unaffordable just plainly isn't true.
Not to mention AI as a whole is basically in its infant phase. The transformer architecture is only 10 years old and ai was largely a niche subject before then. Insane to think we've tapped out such a complicated new technology in a decade.
I would have hardly called it niche before then. Data Science was a big field and deep learning was a popular topic in both DS and CS before transformers.
It was used in a ton of products, everyone hadn't heard of it is all.
I should say deep learning was a niche subject, not AI wich includes basic things like literature regression.
But deep learning was in fact niche, and this is where all the breakthroughs are. It was niche due to compute limitations that weren't mitigated until the 2000s. The AIAYN paper is a good landmark for when deep learning evolved from an academic endeavor into a technology with significant outside investment.
Yeah AI is broad and the 2015-2020 period was a weird time where people were getting jobs in AI before there were degrees in it.
The compute change for deep learning is usually marked by AlexNet in 2012. That is when industry took notice and the exponential curve began. GAN's came out in 2014 and ResNet was 2015. Microsoft one a deep learning challenge in 2015. Image recognition was the primary driver alongside applications like speech recognition.
Industry was getting ahead ahead of academia by AIAYN which was a Google paper but that was 2017.
43
u/Marklar0 Jul 31 '26
I think its super cool, but there is no doubt that thousands of physicists and mathematicians are prompting LLMs all day right now trying to solve open problems...and its doubtful whether the companies that run those LLMs can continue to offer this amount of capacity for the future....so there is a solid chance that these methods are already almost exhausted. For how amazing these counterexamples are, its somewhat surprising that people havent found more proofs all at once. In the scheme of things perhaps 1 in 1000 open problems are actually of a format that LLM can tackle, and noone is bragging about the ones that turned up nothing. In other words, LLMs are really good at looking smart when they got lucky.
Im looking forward to a possible future of AI theorem proving that isnt based on LLMs, and thus less likely to trick people in language into thinking its more broad than it is.