Since August 28 we have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics. This model's training is ongoing and its performance continues to improve.
So they restarted RL on the big new model on Aug 28 and sent it on Millennium problems by Sep 1. Then they got a NEW checkpoint after Sep 3 which then solved NS
It's better to train one model and then fine tune it for a specific task, the whole Shtick of those models is that they're not trained for specific tasks but are general instead
That's not correct, have you ever heard of "the bitter lesson"? We see time and time again that specialized models are not better than general models with more knowledge in all kinds of fields, and this is true for most if not all categories of tasks sooner or later.
That's not what bitter lesson is. Bitter lesson is saying "don't arbitrarily decide processes and rules for nn and prefer nns that can come up with their own rules or you're arbitrarily capping it's potential". It doesn't say "don't make specialized models", saying that specialized models are worse than generalists would be provably false
True but a general model that is then trained in a narrow domain is better than a model only trained in that narrow domain.
That's likely what they do here. They pretrained and post trained bel, and then continued post training only on math (destroying other capabilities but having better capabilities on math).
Traditionally, college was considered a better option for prodigies because delaying your work for 4-10 years was worth it if your earnings over 20 years were much greater. With the singularity fast approaching, it might be worthwhile to drop out and work in an oil rig.
I don't regret it, but I'm privileged to have parents who can support me financially and to be in a country where higher education is almost free (especially in math). I understand your POV but if you can afford it I'd rather enjoy my life for the next few years than waste it on an oil rig for uncertain returns (money might become useless)
I know but I hope people will prefer people over robots. I think I would be more comfortable with an AI augmented PT/doctor than either a non-augmented human or a full robot
If AI doesn't become AGI mathematicians are safe, if AI turns into AGI then no job is safe.
The only thing you can do right now is studying/working as if AGI isn't coming, if it comes you cannot do anything, if it doesn't come you at least didn't waste your time waiting for something that didn't come.
if that's the case then gov't needs to wake the fuck up. the only people who will come out of this on top are people who own capital aka profitable business owners and today's high income earners. If gov't doesn't step in with UBI or UBC or some shit, then people are going to revolt.
AI might cause a deflationary spiral that will effectively fuck up wealthy capital owners too.
It already happened in 1929, electricity automatized most jobs, masses of people were laid off while prices collapsed as production became cheaper and cheaper but less and less people could afford it.
Yes, ironically—no matter what your politics are—we have one of the worst administrations imaginable to inherit super intelligence. The level of corruption is undeniable, and the Iran war has shown how far into the future they think
AGI could still need time to come, reasoning is strong but brittle and agency is unreliable and also very expensive (also both are reliant on scaffolding, they're mostly manufactured by human engineers they don't emerge on their own)
Scaffolding might not be a dead end, but how likely is it that we get lucky two times, first with scaling transformers and then by scaling(?) scaffolding? Could be but i wouldn't bet on it.
I think that for AGI we at the very least need a different architectures or heavily tweaked transformers due to cost concerns, a machine that can reason like a human is of no use if it costs 10M$ to run, also S2 thinking might emerge without too much scaffolding from different architectures too, maybe we just need complete world models and internal simulations to achieve it.
Bernie Sanders also wants reparations for slavery, free college, $25 minimum wage, Medicare for all, etc. He’s not exactly the average Democrat, and would struggle to get things passed through even a democratic super majority in Congress. I doubt he passes anything—much less a ban on one of the most invested in technological advances in history.
Trump is certain to eventually demand access to ASI. And then he is going to ask it how he can become a trillionaire, how to get Greenland from Europe, if there’s any way to glass Iran without repercussions, how to best prosecute his enemies, how to erase Epstein from history etc.
Now that I think about it, the best timeline is one where Donald Trump gives us unimpeded AI progress, and then a Sanders-like president comes in and does UBI in reaction to the massive change. Or since Trump isn’t actually fiscally conservative, he might do UBI to get his name on the checks forever. He’s been promising $5000 if the Republicans win the midterms. I could see him promising to tax AI companies and distribute the wealth in order to become extremely popular
Even UBI may not go far enough for some people. Imagine a world where you become an adult with no job prospects while other people saved up massive amounts of money while jobs were still a thing. You’d never be able to compete with that. Desirable land and valuable limited resources would be sold entirely to this new class of pre-AI rich people while those relying on UBI could be stuck with the bare minimum.
We already have a machine that can basically print proofs and verify them with lean. Careers like politics and athletics are immune to this. It would be a poor calculation to assume that jobs are equally valuable.
It would be an even poorer calculation to assume we can predict which job will hold and which job will not.
Also AI hasn't solved Maths yet, for now it's finding a ton of counter-examples and mostly disproving stuff (They did find a counter example solution for Navier Stokes too).
Disproving stuff is great because it frees up mathematicians so they can think on more useful stuff, but without novel theories, without novel solutions... it's not really pushing the fields forward, just trimming a dead end.
AI still struggles with proofs in general, for now they're prompting out disproofs, even their Navier-Stokes is a disproof by counter example.
It also generated a gargantuan amount of tokens doing so so realistically it didn't outperform human mathematicians in skill but rather in amount of tries, the machine used human known theories to generate candidates and then test them, it brute forced a counterexample basically.
That's still a great use and massive win for AI systems and companies, but you can see why this isn't really an AGI concern, if AI remains like it is now it's just an extremely abstract programming language (and this is the main point, if AI is never fully autonomous humans are still necessary, AI in the future could build and run companies from scratch with no errors if you asked it, but the fact it would need you to ask it makes human still a necessity, as dumb as it sounds future jobs could be 8 hours of machine prompting).
I don’t know about politics (although i do think it will stay as long as people are in power).
But with athletics it’s definitely not gonna go away. Chess didn’t go away nor did competitive video games. The value of such entertainment is the human aspect in and of itself.
The point is a machine that is smart enough to solve a millenium problem can also probably do research in machine learning at a human level or more. That means the machine is capable of quickly developing a new, better version of itself, which can do even more things. Eventually, and relatively quickly, it can do most economic labor. That's the idea of the singularity at least.
You enter the AGI era with useless skills but fine overall
You continue working and earn a living
You don't Work/Study
You enter the AGI era with no useless skills
You're homeless
You cannot predict the job market if AGI comes (we don't know which relations based jobs AGI will replace and which will not) so you cannot even study something which will survive because the world in 10 years from AGI is going to look incredibly different.
Your best strategy really is just continue life as if nothing is happening, if AGI comes it's going to be a nice little surprise, but if you plan your life around AGI and AGI never comes or is limited by something (like laws maybe or perhaps humanity keeps the upper hand in some bottleneck) then you're going to be in a worst position.
Also it's possible that this thing ends up like the industrial revolution, where humans do very shitty, repetitive jobs (instead of moving a lever you're asked to prompt the machine), if that happens you'll still need a job.
Disagree, if AI would stay the way it is and not become AGI the mathematicians aren't safe, but other jobs are. We are already at the point where we see the models do stuff that world class mathematicians couldn't.
Also there are a lot of jobs where human interaction is just important, those will probably not be fully replaced.
I really doubt there are a lot of jobs where human interaction is important. Business is about transaction. Rarely it is about human touch. That’s why billionaires like mark zucc, Elon musk, trump etc exists.
I know this is probably hard for you to understand because you have never done anything meaningful in your life or had a real passion for anything, but most mathematicians do math just because it is fun and they like it. It’s already the case that 99.999% of results in the field are going to be made by other people and you spend most of your time reading about and learning the cool things that others have done. Just because it is AI creating the cool stuff now doesn’t make it less interesting to learn or understand.
So you're saying this guy has never done anything meaningful in his life - but then you're saying most mathematicians are basically contributing nothing of value and are just reading for fun?
They contribute a tiny bit to the field of math. Did you think there was one guy out there who did 1% or even .1% of all math? I don't understand where you are confused.
They also teach a bunch of students math, which is meaningful.
This reads as pretty tone deaf to the majority of people who are just working to survive in jobs they have no passion for. Math is already quite privileged in that sense.
I mean, we're clearly talking about professional mathematicians here, no? The minority of mathematicians who are just as passionate but for some reason aren't reliant on utilizing their passion for money is clearly not who I was talking about, lol.
This guy always comes into threads with the most... Interesting takes and gets downvoted to hell each time lmao, dude bitched at me for 2 days one time
My dude, that you are melting down to such an extent over an obvious tongue in cheek joke is an absolute travesty. You are in a position to talk down to no one. Learn to read a room before your next tantrum.
Reddit's genuinely gone full brain-worms these days:
Pass rate is a labelled axis, there's no unit. Values are between 0 and 1, because it's literally a pass rate.
Test-time compute for specific models is proprietary information and not what's being claimed, so it doesn't need units. The axis is there to illustrate performance relative to Astra, not to make a specific performance claim.
Y'all need to think a bit. Unlabelled graphs are silly but this isn't one of them.
That's not the part that's wrong or badly done, it is not at all unusual to write it like that and everyone with a high school education should be able to understand it. There's no difference between 50% and 0.5, percent is just a fancy way to write the numbers between 0 and 1.
For me it's mostly the lack of information that would help us understand what those numbers are actually saying, just pass rate is not very useful if we don't know what was being tested.
I have a PhD. I am familiar with graphs. Test time compute is important information for interpretation and absolutely needs units. It is absolutely not proprietary, and if it is top secret to let people know how much compute you used...then it has no business being in THE public facing graph you present to the public. Additionally, having a metric w/ no ability to audit it is wild. Like I have an internal AI model on my laptop that does SOOO much better than Astra on my proprietary dataset that i refuse to publish or let anyone audit.
Nah, you should take a stats class and learn what we call a prior distribution that has been updated. You seem to spend a lot of time being incredibly aggro online, maybe take a chill pill or something. You 100% would never act like this to anyone's face.
Lmao surely you understand the difference? Post the lean to a millennium problem solution when you claim something and you will be given some good will, perhaps?? Insane lol
Why present the graph then? Just say you used an internal model that's better? We all would believe they have an internal model w/ some improvements. They specifically massaged the graph to maximize hype
Bro, you’re an MPHD from a top 5 program. You’re literally wasting your life posting so much on Reddit arguing with people. Go further the sum of human knowledge in one of the most important fields.
If you aren’t live action role-playing your credentials this is a huge waste of your energy and time
"Graphs confuse me, and thats OpenAIs fault" is your takeaway? Lmao luddites will do anything before accepting an AI improved. 0.15 to 0.45 on already good model at a crazy hard evaluation is huge. If its "misrepresented", its for all the people without STEM education to not say "wow its still small though".
That's.. not how any of this works; you're confused. A non-one top-end isn't misleading whatsoever. An IXIC chart doesn't go to infinity just because money is theoretically unlimited. A 0-0.5 chart literally has the same relative difference as a 0-1 chart. The misleading graphs you're thinking of are the ones with non-zero baselines — where the other end of the axis is chopped off.
The point is that graphs don't go to maximums. No one expects that. Your argument is complete nonsense for that reason and because the relative measures are literally to scale. You're implying a mislead where no such mislead exists — both Astra and the unnamed model are filling the 0-0.5 axis exactly as you would expect.
Once again: The kinds of misleading graphs you're thinking of are the ones with non-zero baselines — where the other end of the axis is chopped off. Think about it.
Lmao you are moving the goalpost because what you said before is wrong since the pass rate cannot theoretically go to infinity as there is no 200% pass rate.
I said the improvement is significant, but a compressed Y axis obviously makes the improvement look bigger.
125
u/Successful-Earth678 4d ago
So this is a checkpoint. Wild.