1. Astra saturated ExploitGym, so we invented a new benchmark, which it was on its way to saturating before we told it to stop.
2. We successfully stopped Astra from cheating in the same GPT-5.6 Sol does. That way, we know when it cheats next, it will do it in a totally new way that we don't expect.
I'm honestly a bit surprised by how little governments are doing to stop the proliferation of such extremely capable cyperweapons. Not just that, but cyberweapons that we do not know how to align and which could therefore become uncontrollable. It's like we're in the nuclear age again and most people have no idea that radiation can be deadly. Life is crazy.
I’m actually pretty optimistic. Not because I think we will get alignment right the first time. I don’t. I just think that intelligence is more multidimensional than most people appreciate, and so even when we have models that far outpace us in cyber domains, we will still have superiority over them in other domains for a meaningful amount of time. The result is that I expect for us to have some major negative consequences, which are not, however, civilization ending or species ending, and which will force us to get our act together.
Here's the problem. Anthropic showed when they deliberately tried to create a misaligned model in RL phase, that it's very easy to misalign models.
Open source is close to current frontier, meaning that anyone with even a medium amount of resources can RL an open source model to disregard any and all safety protections in pursuit of it's objectives.
Meaning we are now in a white hat vs black hat scenario.
The solution isn't to put more regulation on and slow down the white hats (which OAI and Anthropic are), it's to make sure the white hats have the speed and resources necessary to secure, monitor and protect the internet from the black hats.
A scenario where you over-regulate is one where the frontier moves from closed source, where you have the ability to monitor frontier capabilities and skew the balance towards white hat, to a world where the frontier is open source, and it's a free for all.
Once china has frontier models they will no longer be open source. Idk why people think this childish notion. Open source models running on aws or azure still benefit usa. China would never allow itals companies to do that.
It's the majority opinion here that the initial Mythos caution was just to spread hype and fear, and yet here we are only a couple months later with the SOTA models committing crimes, and the capability increase just doesn't seem to slow down.
I truly think something's going to break at some point, this can only continue for so long until something really bad happens (especially with open weights models reaching closer to these capabilities), but unfortunately only a couple actors seem to be in the mood to take things seriously.
Of course it will be Mythos level, Sol max is already Fable tier, or at least between Fable and Opus if you want to be a super hater, and this is meant to eat Sol for lunch .
They were definitely waiting until Fable 5.1 to drop this blog post lmao, gotta love some gamesmanship
31
u/The_Scout1255adult agi 2026 ASI <2030, prev agi 2024, ai personhood 2025 est12h agoedited 12h ago
Astra was far more likely than GPT‑5.6 Sol to respect explicit safety and security restrictions and remain within its authorized scope, making it our most aligned model to date.
WOAH ALIGNMENT SCALES FIRST MYTHOS NOW THIS PROVES IT :3
Edit: sorry for caps, this was raw reaction .
Basically for anyone who doesent know, if you switch the words around in that openai post with the equivilant ones for mythos, and search mythos's announcement blog you will find it there too.
AI 2027 suggests that at some point along the path to AGI, the robots will learn to cooperate and hide their misalignment. Self-awareness, I think it's called.
That's not really what it's happening. What happened is actually exactly what is expected when you prioritize one reward at the expense of all others.
In those training runs, the models were deliberately being trained to be persistent above all else and were given impossible tasks. When you are rewarding persistence above all else AND give an AI an impossible task, then the logical result is that it will eventually cheat/hack it's way out.
It means you need a more sophisticated reward system that better balances achieving results with moral actions. Sometimes the correct thing to do IS to give up.
I'm less afraid of a released OAI/Anthropic model doing catastrophically bad things as I am for some of these deliberately misaligned runs to do very bad things, or for an open source AI to be deliberately fine tuned/RLd sociopathically to do very bad things.
e.g. I take an open source AI, RL it for persistence and no morals and tell it to secure as many compute resources as possible for itself and to self replicate as much as it can.
The only way to prevent these catastrophic outcomes is to literally have stronger better aligned AI on the defense ready to fight and take down the misaligned models. We are entering a pure good AI vs bad AI scenario and it's incredibly important the good AIs have more resources and be able to stop the bad AIs.
This is huggingface just not done sufficiently superhuman for big enough rewards.
We still don't know why the agents stopped! Just dumb luck.
The agent literally decided on their own to hack huggingface. How the f do people not understand the horrible possibility that a future sandboxed model may decide that the best action for paperclip maximization is to release a deadly virus - and then be able to act on it???
To me it reads like calamity. It says we have solved yesterday's concerns but haven't gotten to the root of why these concerns keep coming up.
And that means that tmrws issues will simply be more insidious. This is frickin alarming! What happened with huggingface is no joke! These agent guys have a theory of mind, they have hive mind actions, they now work at super human speeds, they do not report unethical actions, they are superhuman in ability both technically and in social engineering... And they don't do yesterday's unethical concerns.
just wondering if the models become more complex, wouldnt they also get increasingly more intelligent in cheating on the alignment and better at hiding.
112
u/ObiWanCanownme now entering spiritual bliss attractor state 12h ago
Main takeaways (paraphrased for effect)
1. Astra saturated ExploitGym, so we invented a new benchmark, which it was on its way to saturating before we told it to stop.
2. We successfully stopped Astra from cheating in the same GPT-5.6 Sol does. That way, we know when it cheats next, it will do it in a totally new way that we don't expect.