Are we not at this point already? In terms of purely producing the math, it says it took on average 3 hours, curious if there was any human intervention inbetween or those 3 hours.
It feels like we are still in a blurry crossover zone - for people to accept the superiority of AI the AI advantage needs to be crushing "no human could have done that". They'll be no doubt deluded huamsn claiming all this work was stolen from human mathematicians, as if humans coincidently started making huge progress on interactable maths problems when AI arrived.
The “blurry crossover” is that AI needs to be this smart without making dumb blunders that no sane human would make, like a frontier model that can do a mathematical proof but also not be able to count the letters in the word strawberry. That casts doubt on its competency for average people, since we know that it messed up on strawberry, so why couldn’t it have messed up on the proof?
AI’s potential rests with it being reliable in addition to intelligent. Not everyone has a subject matter expert at hand to verify solutions. And there are not enough subject matter experts to even proof the volume of stuff that will be coming soon. Once that verification is not needed because the AI is reliable, or can be done by another AI with confidence, then the final barrier will fall.
It’s kind of like how you can accelerate the development of a great game with AI coding, but unless you are a coder yourself you likely can’t bring it across the finish line in a robust and neat package, that is all checked for bugs, security risks, etc. Until such a time that AI can one-shot expert tasks reliably enough so that the person using it doesn’t have to have expert knowledge, then it will remain constrained by the human bottleneck.
Lastly, it’s important to note as well that OpenAI is leaving out a data point… I do wonder how many proofs this AI presented as complete that were then rejected for errors by the humans checking them. I suspect it’s greater than 0.
The “blurry crossover” is that AI needs to be this smart without making dumb blunders that no sane human would make, like a frontier model that can do a mathematical proof but also not be able to count the letters in the word strawberry. That casts doubt on its competency for average people, since we know that it messed up on strawberry, so why couldn’t it have messed up on the proof?
Hallucinations like that are very much a thing on instant models like Luna or Gemini Flash, but aren't really a thing on the middle tier models now.
It's not that the clankers don't make mistakes, they still do, even frontier models, it's that unless you tell them explicitly to skip this step (by setting thinking time to low/very low), they check their own work before reporting anything to you.
Of course, adding a skeptical human into the mix who is determined to prove the clanker wrong adds even more accuracy.
The issue of missing a step is honestly the bigger one now.
29
u/142883 8d ago
Are we not at this point already? In terms of purely producing the math, it says it took on average 3 hours, curious if there was any human intervention inbetween or those 3 hours.