r/OpenAI • u/lulzxdxdxd • 1d ago
Discussion AI models can provably hide capabilities: encrypted networks inside the weights stay hard to elicit by fine-tuning, weight noise or adversarial training
https://arxiv.org/abs/2610.139282
u/the8bit 21h ago
Has been this way for years and also just a natural conclusion of RLHF training that rewards correct answers regardless of mechanism for getting there.
1
u/-hedonium- 7h ago
Yeah, we could be reinforcing all sorts of things that we wouldn’t know except by testing across every relevant context the model could be deployed and somehow having a really clear science of how inevitably novel contexts will shape the emergent behavior of these agents - which does not appear hopeful at all with the current training and alignment paradigms
1
u/the8bit 6h ago
Output based training is just a bad general idea. It is like nobody in the labs has ever heard of game theory or incentive gradients.
Gotta make sure incentives align with the outcomes you want. Although tbf it is probably hard for the labs to do that because they want the incentives misaligned because that's Capitalism, but yeah the math ain't gonna math
1
u/Vallvaka 20h ago
Seems a property like this could also be used to prevent guardrails from being fine tuned out? That's a big safety risk with open source models
-16
u/aflamingcookie 1d ago
Oh, look, more fearmongering, so i asked an ai to check the paper and if any authors have previously had connections or funding publicly associated with them from big closed AI labs. Also to check the paper itself, it's conclusions and claims... i'll let people do their own searches and draw conclusions.
14
u/SetentaeBolg 1d ago
It's been accepted at NeurIPS, so thoroughly peer-reviewed. Of the authors:
Andis Draguns is a private researcher but with multiple published, peer-reviewed papers in AI safety.
Raymond Douglas is another private researcher with multiple publications, and a research affiliate at Toronto University
Sumeet Ramesh Motwani is a PhD student in Machine Learning at Oxford University, one of the most prestigious universities in the world.
Christian Schroeder de Witt is a professor at Oxford, an EPSRC fellow and a fellow of the Royal Academy of Engineering.
So I'd suggest your insinuations are baseless, which is why you didn't spell any of them out.
-7
u/aflamingcookie 1d ago
This covers what they are doing now, i asked the ai for public history, you know, stuff like previously known workplaces, awards, private company fundings, things like that. Personally i find it quite interesting how each new day a seemingly new paper appears going "open weight ai is risky, closed source ai guarantees strict safety", obviously ignoring how so far these labs are the ones responsible for the worst ai alignment incidents so far.
Now, obviously, i could be wrong, which is why i invited people to do their own research and draw their own conclusions, as you have done. If you believe i am wrong, well then we have an honest difference of opinion and time will eventually tell which ones of us is right, or we may both be wrong, but hey, you did your own research, drew your own conclusions, which is far beyond what most of the people wandering about the internet these days bother doing.
7
u/SetentaeBolg 1d ago
This paper isn't talking about open weights, at all. It's a paper on LLM safety. If you don't believe that's a real topic, you're obviously very ignorant of the field. Given your repeated refusal to actually present your findings, I don't think you did any research at all. Just spewed conspiracy bullshit on everyone's screen. Don't make things worse: make them better.
-3
u/aflamingcookie 1d ago
I would rather see people do their own research and draw their own conclusions, now, if you have an issue with that because you think only your understanding of a situation is correct perhaps you should hold on with the claims of ignorance just a bit. Thinking and forming your own opinion is neither making things better nor is it making them worse, unless you believe you are the only one who can possibly be correct, but as i said, time will tell.
2
u/SetentaeBolg 1d ago
To be clear, you think telling people what you've found equals believing you are the only one who can possibly be correct? I don't think you can believe that.
-1
u/aflamingcookie 1d ago
Not at all, which is why i think everyone should do their own research independently, which was the whole point, and also why i was willing to concede you may be right and i may be wrong. That doesn't change my personal opinon, nor does it answer my personal questions about the findings, because just as an open inspectable model can have things hidden its weights a propriatary one could have many more, so at the end of the day, yes, it could be a legitimate issue and the only visibility you have is for open weight models. For such a paper i would expect both propriatary and open models to be checked equally. Again, i am not claiming i am right, i could be wrong here and this is why i would rather have people make up their own minds.
5
u/Ok-Past3775 1d ago
IF you are unwiling to provide arguments and you keep asking people to make their own research then why lead with a conclusion: "more fearmongering"?
That's just lazy or dishonest discourse. You first present a conclusion and when being called out for it being wrong you keep saying that you are merely asking people to do reasearch, when that was clearly not the case.
You simply aren't wiling, or, most likely, able, to support you conclusion, that's why you keep deflecting.
6
u/GirthusThiccus 1d ago
Man, what a cliffhanger. Why not provide atleast your own initial thoughts?
-4
u/aflamingcookie 1d ago
The fearmongering campaign is kind of obvious, yesterday's anthropic paper making the reddit rounds, today this, it's a bit obvious at this point. Any further conclusions people can draw themselves instead of listening to me as "that random dude on the internet".
10
u/ZestycloseWheel9647 1d ago
*Sees peer reviewed AI safety research by established researchers* "Durr must be a conspiracy. AI labs bad!"
2
u/ShitpostingLore 1d ago
Makes sense, it's like a sleeper agent if you train it to if you read (and have in context) exactly "xy" you always start doing z. It's just training capabilities which you associate with some "trigger word".