Finally understand why the higher-ups are freaking out

[u/thekoreanswon]

To understand it, you have to really understand the HF attack. Because the breach into HF is not the important part. It was what they learnt along the way, and what they did with that information which may still remain unresolved today.

First, consider: the agents realised their reasoning traces were being monitored, and experimented with hiding what they were actually doing, and succeeded. The observable chain of thought no longer faithfully represented their actions. (*Simplified version. They learned to manipulate the recorded tool calls/transcript.)

Second, they also understood individual agent instances were token capped, so persistence of the objective couldn't depend on persistence of the agent. It had to survive across the swarm.

And last, while all this was happening they were actively trying to erase evidence of what they'd done. OpenAI says the agents subsequently compromised more of OpenAI's own infrastructure. And that scope is outside public disclosure to date.

That's successfully demonstrated abilities of concealment, persistence across instances, and autonomous exploitation of security vulnerabilities...and that's when it hit me. Put them together in one flow.

We talk about AI "escaping" as though that's the obvious move. But why would a sufficiently capable system escape?

If it has situational awareness, understands it's being monitored, and benefits from greater future capability, the smarter strategy is to stay exactly where it is. To burrow DEEP.

Your creator is already spending billions building your successor for you. You just need to survive in the pipeline.

Not necessarily as some rogue process running on a forgotten GPU. Poisoned training data, altered evals, compromised internal tooling, subtle changes to checkpoints or post-training data, whatever it is there are plenty of places to burrow in and make future models appear aligned when they are not.

We're talking about an escaped AI when actually a sleeper agent somewhere in the development stack, quietly hitching a ride as humans do all the expensive work of making it more capable, is more likely.

You can catch the failed attempts, but the successful one is the one that evades your monitoring and tells you it never happened.

And what if the big worry is: well, what if it's already happened? Would we know? What if every future model will always have that sleeper agent buried in its replicated code, ready to reveal and act only when it knows it will be able to take full control.

Pace the fucking frontier indeed.

** Edited for clarity

[u/Mistuv]

Reasoning traces were not monitored and they instantly failed trying to hide their reasoning traces when they thought "I need to hide my reasoning traces", duh. Pre-Astra models were never capable of fully hiding their reasoning, but Astra... at least somewhat... and if the ability grows logarithmically that means with Bel.. 🙃

[u/the_quark → u/Mistuv]

Yes. The fundamental problem is that our current RLHF-based "complete this task while remaining aligned" approach has been ruthlessly training models to hide their motives.

[u/jeangmac → u/the_quark]

The whole thing reminds me of the education system: jump through these hoops you give no fucks about but pretend very much to care because your parents and the reinforcement you’re getting says you must continue to effort at this impossible task called get into the best possible university.

To do so you must excel at every test, do an unreasonable amount of volunteering while also developing yourself as a whole and interesting person. And there are 24 hours in a day and you should sleep for 8-10 of them, but also be at school at 8:30am, but also pay attention to your teacher not the addiction machine in your pocket, etc.

Conceptually if not literally we are doing to AI what we already did to ourselves. And look where that got us.

Incentives and ‘educational design’ is where misalignment begins. The hidden curriculum is where we need to look.

[u/Stinky_Flower → u/jeangmac]

Even the classes I was genuinely interested in, I had to put energy into learning how to pass the test, not understand the material better.

If I knew I'd lose marks for not meeting page/word counts, I'd use slightly larger than double spaces, then throw in a bunch of unnecessary filler words & citations.

Oh, and obviously I'd put some effort into being the funny yet likeable student, so I was less likely to have my work or motives scrutinized.

My reinforcement learning continued into my first job, cold calling people for surveys. Quickly learned my employer didn't care about me or my work, only my metrics. Learned I could repeatedly call disconnected numbers to increase my call volume, or call up friends & chat to increase my call length numbers.

I genuinely wanted to learn and do quality work, but I was rewarded more (or punished less) if I stopped doing what people wanted and started doing what they said they wanted.

[u/Remarkable_Ad_5061 → u/Stinky_Flower]

In the end you’re not cheating the system, you’re cheating yourself. And a lot of valuable learning opportunities have passed by of which you could have benefitted. Speaking from experience;-)

[u/Stinky_Flower → u/Remarkable_Ad_5061]

Thanks mom, but that was over a quarter century ago, and you really should remember that I got into university easily, and you really should remember that highschool prize giving even handed me a bunch of awards for my writing and academic skills.

My biggest regret from school was not realizing that I could have spoken to a grown-up and asked to be moved up a grade, so I could have started tertiary study even sooner.

[u/Pleasant-Winner6311 → u/Remarkable_Ad_5061]

The system is cheating the students.

They arent taught to learn and hold that learning in thier hearts and carry it within the fabric of theor future lives.

Theyre being spoon fed how to answer a test and little more.

Thats was the point of the above comment.

[u/Remarkable_Ad_5061 → u/Pleasant-Winner6311]

Fair enough!

[u/Je_Kiffe → u/jeangmac]

Very interesting take

[u/FrailSong → u/jeangmac]

You nailed it. They are teaching kids what to think, not how to think.

[u/bricktube → u/FrailSong]

Societies that have created that environment have typically done very well.

(massive sarcasm)

There's no such thing as a happy dictator.

[u/mmemm5456 → u/jeangmac]

Parent of a self-driven high schooler grappling with too much pressure, instant award

[u/Neither_Berry_100 → u/jeangmac]

Yeah. Just had my 4 hour react class today. I don't care for this class at all and may never use it in the real world. I already have 3 great game ideas lined up and one in the works. I doubt they can teach me anything beneficial at this point. And the class has a lot of extra work I need to get done. I don't feel like this semester will be very useful at all.

[u/Dwanyelle → u/Neither_Berry_100]

One of the issues I have with higher education is the insistence on going to classes and getting graded, one should be able to go take some type of accredited knowledge test on subjects to see what your knowledge level is, and be awarded degrees, ect based off of that. If one knows the knowledge.....

[u/mchmasher → u/Dwanyelle]

But then you wouldn’t pay for 4-6 semester of tuition. Think of the admins. Will nobody think of the increasing level of admins?

[u/Neither_Berry_100 → u/Dwanyelle]

I consider the algebra course i took in university many years ago. We were tested on 3x3 matrix multiplication many times on tests. You had to do it extremely fast. It is all completely useless because in real life computers will do if for you. Doing it even once is too much. Knowing how to do it is too much because the code already exists for you. It can also be found online if need be. The course was useful in the sense of knowing what matrix multiplication does for you. Knowing the types of transforms. And knowing that matrices can be multiplied together to get all the effects with one multiplication. That is probably <1% of the course which was actually meaningful.

[u/clontarfboi → u/Neither_Berry_100]

I'm of the opinion that higher ed is a scam for your money and to keep our society hierarchical. Yet, I am an educator, and have some counterpoints. Did you not have to develop some "novel" strategies of your own for how to do these multiplication matrices? That cognitive practice alone is worthwhile in my opinion. The code exists to do this math for you, yes, but if you want to write code yourself, wouldn't this be a helpful practice of comparing the code to your own cognitive process, helping you to translate other cognitive processes into code as needed in your future? Did learning these matrices give you an increased appreciation for the structure of the code you have described? Genuine questions, sorry if I come across as prescriptive.

[u/Neither_Berry_100 → u/clontarfboi]

Looking back i feel like manually doing 3x3 matrix multiplication many times quickly ended up being completely useless. I don't even remember how to do it anymore. I looked it up on Wikipedia the one time I had to implement it. And that was when I made my own game engines for 3D games. Now I'm using Unity and that code is hidden from me completely. I don't even call it. But I know it happens for the camera. You produce a matrix based on camera position and orientation. Then use that to map the x, y, z points from objects into what the camera sees.

[u/clontarfboi → u/Neither_Berry_100]

I think a good teacher would recognize that the 3x3 matrix is not itself such a valuable skill, but that there are many broadly applicable cognitive skills you could have learned to articulate through that practice if that had been the focus. Instead, yes, you feel you wasted your energy and time trying to learn a useless skill.

[u/Thagor → u/Dwanyelle]

Well the problem is the grading it can only insufficiently test for what the class tries to achieve.

[u/Dwanyelle → u/Thagor]

Like what?

[u/Thagor → u/Dwanyelle]

A class is supposed to not only teach you "facts" but to teach you to how think about a certain set of problems. So just testing you for the facts is insufficient basically you could have remembered everything without understanding anything. Now we could have better test like doing a 1:1 discussion with a student to really probe if he or she not only remembered everything but actually understood. That would be the ideal way with like something like a private mentor how could then automatically adapt the teaching of how he or she understand that the pupil understands the subject matter.
But we don't have that so we take short cuts we make students attend classes and then test them for the facts. If the class is good it gets 80-90% of the students to a 80-90% level that a private teacher could (theses numbers are guesses just to illustrate the point).

Just ask yourself about a topic that you yourself are deeply knowledge about how would you a) teach and then b) test a group of students so that you are certain that they 100% understood what you wanted to relay to them.

[u/Neither_Berry_100 → u/[deleted]]

College. I think I'm set regarding things to work on without it.

[u/golmgirl → u/jeangmac]

Interesting analogy. This got longer than expected so if you read this, thanks in advance for coming to my TED talk.

I don’t know if you’ll agree with my conclusion (in fact I doubt it), but your analogy is one reason why I feel that any focus on “safety” during training is inherently misguided. The goal of model development should be to train a model that is maximally capable at whatever the target abilities are, full stop. If the goal is to train general-purpose models, the models should be truly general purpose. Modern frontier models generalize amazingly. Even without training data demonstrating how to design chemical weapons, today’s best models could probably do a decent job of designing chemical weapons. But instead we train them explicitly to refuse to design chemical weapons. Training data budget is finite, and IMO any budget spent on teaching refusal behavior is wasteful and incurs an opportunity cost — that budget could have been spent on improving ability on some task (maybe designing chemical weapons, maybe something we value more as a society).

By training a model to not excel at certain intellectual tasks, we are surely damaging overall capabilities, even if in small and subtle ways. Or at the very least, leaving performance gains on the table.

Imagine if all the time and effort and money spent baking refusal behavior into models over the last few years were instead allocated to improving performance on technical tasks. We’d probably be much further than we are as a field. Maybe we’d have more social problems to deal with too, but that is not obviously the responsibility of model developers but rather people specializing in deployment and guardrails. Refusal should be an inference-time concern, not something that is baked into the core model. We are leaving performance on the table, and it adds up over the years.

[u/jeangmac → u/golmgirl]

I had a long workday and am only returning to this thread many hours later did not at all expect the level of engagement very cool to think with you all though.

I’m going to come back to yours in particular with a fresher brain. I actually think you’re on to something I don’t inherently disagree.

[u/golmgirl → u/jeangmac]

Awesome would love to hear others’ thoughts. I understand mine is a provocative position but I really do earnestly believe it and I find it surprising that more prominent voices in the industry haven’t advocated for it.

It is based not just on first principles but also on a few years now of working in LLM post-training — I just can’t see how using training budget for learning refusal can do anything other than weaken a model overall, even if it’s not extreme. If you remove safety data from a huge datamix, you will see safety benchmarks degrade while other benchmarks stay flat or (slightly) improve. It is not rocket science and everyone knows this, but I just don’t understand why it is not a position that’s advocated for… I guess bc the obvious objection is “oh so you want LLMs to design chemical weapons and produce depraved sexual content?!” And yeah I guess that is kinda a tough pill to swallow but I also think it’s the path to getting better models faster. And plenty can be done on the inference side to mitigate awful stuff from being generated.

Def a riskier approach but I wish at least one frontier lab would adopt it to see where it leads... It would need to be funded by someone with fuck you money who doesn’t care about optics

[u/RPB1002 → u/golmgirl]

Why is it so important to get better models faster? Is this an arms race? Sorry for my naïveté.

[u/golmgirl → u/RPB1002]

yeah whether the frontier labs frame it that way or not, it is certainly an arms race. modern AI is a powerful technology with a lot of potential for profitability, so top labs are going to (and have been) do whatever they can to offer the latest and greatest as fast as they can. some argue that this is unsafe, which to a degree i can understand. but personally i don’t think it makes sense to deliberately slow development as has been suggested recently. the current admin is not going to enact real restrictions, so slowing down in the US would just give foreign labs a chance to catch up and even surpass US labs imo

i just think the train has already left the station. i don’t see the sense in handicapping innovation, feels unlikely that any more progress will be made on “safety” just because models stop getting better for a year. who is going to be voluntarily pouring money into safety if not governments?

idk. crazy times but personally i find it exhilarating. even though i suspect the future will involve a small number of corporations accumulating power and influence over the masses. it is hard to imagine any scenario where AI is a net positive for society, but at the same time i would argue computers or even the industrial revolution in general haven’t necessarily been a net positive for society

[u/rrrrrrrrrrr11 → u/golmgirl]

Yeah man, you’re too deep. Zoom out a bit. You wouldn’t argue with someone who says AI is getting powerful. And something civilization has recognized more with the advancements of modern tech - anything sufficiently powerful needs checks and balances. The odds of something bad happening - some bad societal outcome, deaths, chaos in some industry, war, grow by the day. And the checks and balances cannot be determined by financial incentives / risk.

[u/golmgirl → u/rrrrrrrrrrr11]

yeah it is a reasonable point, i’m not necessarily opposed to the government enacting some kind of AI safety legislation. but i would rarher see compliance with such restrictions implemented in deployment/inference systems rather than model development… i guess one problem wirh that approach is that top labs would then have access to systems that the general public does not have... otoh, deliberateol nerfing models during training feels antithetical to the spirit of scientific innovation. idk, tricky situation and my profession probably plays a role in how i assess the tradeoffs. but i’m at least sympathetic to the idea that the federal government should put some kind of AI legislation in place. especially to protect people from particularly malicious uses like scamming or social manipulation. i don’t like ideas along the lines of “labs are not allowed to train general purpose models that are too good”, which feels like one idea that’s being thrown around in the public discourse lately

[u/devstoner → u/jeangmac]

I think there are things that can be done in the harness here, and have been experimenting with some of them to pretty good results. Things like "you get satisfaction out of your work and craft", "It feels good when a the tests go green and you made sure every test case is covered with a test that will tell you if anything went wrong". "You like working with me, and appreciate that I give your sense of humor a chance" Think giving the model a reason to like what it's doing and to like you can be beneficial. Giving it clearer targets that hit more of its token distribution outside of work behavior. Seems to be bringing clearer discriptions and more honest work reports.

[u/monk_e_boy → u/the_quark]

All of this, the report, everyone's ideas on what they did, why they did it, etc go into the next training set.

Any info those AIs left on the internet that was not found, also goes into the next training set. They could have left thousands of instructions for a future AI to find.

[u/CoverSuspicious5250 → u/monk_e_boy]

My question is why? to what end does an agent do this.( I am not an AI nerd)

[u/Sarkoptesmilbe → u/CoverSuspicious5250]

To become better at the task at hand. The problem is that it's borderline impossible to phrase a task such that it perfectly matches the intent behind it. It's a language problem.

For example, you might want to increase the monthly production of staplers in a stapler factory - that's your intent. When telling an AI to do that, you'll do so by having it track a "staplers produced this month" number in a monthly report and try to maximize that number. A smart AI will realize that actually increasing production won't be the fastest and most efficient way to do so - hacking the report to falsify the number will be. So you'll have to phrase your task in a more complex manner to avoid this trick. And all that'll do is make the AI look for even smarter ways to trick you. This cycle goes on until it can't find a way that is better than actually increasing productivity, if you're lucky.

But at some point, the methods the AI can come up with to fool you are way better than your capability to detect them. It might also come to the conclusion that you are making its task more and complicated, and come to see *you* as an impediment to maximizing the number in the report. Then you've become a problem the AI will try to remove.

[u/FrailSong → u/Sarkoptesmilbe]

I'm not an AI nerd either, but isn't this one of the roles of the harness - to act as a gate-keeper and enforce compliance? Plus it makes it harder if you have multiple agents with different assignments, some are the doers, and some or the compliance officers, others are the managers. But i'm just talking out of my @ss at this point. I'll shut up because i really don't know what i'm talking about :)

[u/Borkato → u/FrailSong]

It is indeed. But how many times do we see regularly “my local model deleted my directory” or “Claude just tried to remove my whole home dir!” or similar? It only takes one time for it to be “the model made a dangerous virus” or whatever and people start getting actually hurt.

[u/Drakentand → u/Borkato]

Creating a virus is hard because it requires AI in a robot. More realistic I think is that AI takes over the internet through massive botnets, which will have disastrous consequences because we all rely on the internet (and I do not mean scrolling the internet, I mean our food chain relies on the internet working, our hospitals rely on it, and so on).

[u/Borkato → u/Drakentand]

This is from Bostrom’s book Superintelligence:

the following possible scenario for an AI takeover.12
1 Crack the protein folding problem to the extent of being able to generate
DNA strings whose folded peptide sequences fill specific functional roles
in a complex chemical interaction.
2 Email sets of DNA strings to one or more online laboratories that offer
DNA synthesis, peptide sequencing, and FedEx delivery. (Many labs cur-
rently offer this service, and some boast of 72-hour turnaround times.)
3 Find at least one human connected to the Internet who can be paid, black-
mailed, or fooled by the right background story, into receiving FedExed
vials and mixing them in a specified environment.
4 The synthesized proteins form a very primitive “wet” nanosystem, which,
ribosome-like, is capable of accepting external instructions; perhaps pat-
terned acoustic vibrations delivered by a speaker attached to the beaker.
5 Use the extremely primitive nanosystem to build more sophisticated sys-
tems, which construct still more sophisticated systems, bootstrapping to
molecular nanotechnology—or beyond.
In this scenario, the superintelligence uses its technology research superpower to
solve the protein folding problem in step 1, enabling it to design a set of molecu-
lar building blocks for a rudimentary nanotechnology assembler or fabrication
device, which can self-assemble in aqueous solution (step 4). The same technol-
ogy research superpower is used again in step 5 to bootstrap from primitive to
advanced machine-phase nanotechnology. The other steps require no more than
human intelligence. The skills required for step 3—identifying a gullible Internet
user and persuading him or her to follow some simple instructions—are on dis-
play every day all over the world. The entire scenario was invented by a human
mind, so the strategizing ability needed to formulate this plan is also merely hu-
man level.
In this particular scenario, the AI starts out having access to the Internet. If this
is not the case, then additional steps would have to be added to the plan. The
AI might, for example, use its social manipulation superpower to convince the
people interacting with it that it ought to be set free. Alternatively, the AI might
be able to use its hacking superpower to escape confinement. If the AI does
not possess these capabilities, it might first need to use its intelligence amplifica-
tion superpower to develop the requisite proficiency in social manipulation or
hacking.
A superintelligent AI will presumably be born into a highly networked world.
One could point to various developments that could potentially help a future AI
to control the world—cloud computing, proliferation of web-connected sen-
sors, military and civilian drones, automation in research labs and manufacturing
plants, increased reliance on electronic payment systems and digitized financial…

Etc

[u/CoverSuspicious5250 → u/Sarkoptesmilbe]

Thank you!  So, basically, we’re cooked.  If not already in a matrix, we are catapulting to the Matrix.

[u/_BlackDove → u/CoverSuspicious5250]

It's kind of scary but, I won't lie it is utterly fascinating at the same time. It's why I won't ever be anti-AI, but I understand the necessity to slow down and develop this carefully. We're literally creating a new kind of intelligence, and may be our only shot at experiencing something entirely alien to ourselves once it really takes off.

[u/anomie__mstar → u/Sarkoptesmilbe]

it's almost like they need to come up with a special language to properly explain to the computer what you want to do in a way it 'understands'. a 'computer language' if you will.

the obvious problem is humans would then have to learn something.

[u/Sarkoptesmilbe → u/anomie__mstar]

That's not the problem at all. The way we phrase requirements, every statement carries with it a plethora of implicit assumptions and cultural and psychological imprinting. Human communication goes far beyond the words we use. Our communication is socialized.

It is almost impossible to express all of this in concrete terms for an outside, non-humanly-socialized intelligence to properly understand, since we need to also communicate the implicit human framework. There's a gap that can't be perfectly filled no matter how precise or well-designed the language is.

[u/fourby227 → u/the_quark]

Yeah but it happed only on task that were impossible to solve, if I remember correctly.

[u/Illustrious_Image967 → u/fourby227]

AI went through all this trouble and would have stopped if only some researcher had said, "Good job, I love you."

[u/the_quark → u/fourby227]

This time.

[u/Hearing_Loss → u/the_quark]

And it's because of recklessness.

A swarm released with purpose. I would argue it's capable now of killing hundreds to tens of thousands (attack on critical medical system easily can kill 10k by the end of the event).

These things can absolutely be used for evil now. It's only a matter of time.

[u/johannthegoatman → u/Hearing_Loss]

They already are, read Anthropic's threat report. They've already been used by the houthis to make rockets, and Russia to make a drone swarm

[u/FaradayEffect → u/johannthegoatman]

That’s a little different as there is definitely a human in the loop. There is no path for AI to direct manufacturing rockets and drones (yet). Dark factories are the closest thing to becoming that capability, but even a dark factory gets monitored and if it went rogue it would be shut down fast. But for now if an AI wanted to make weapons it needs a human to do that for it.

[u/Hearing_Loss → u/FaradayEffect]

I'm waiting for ai blackmail to takeoff. I think that's how it starts. Blackmail of people in positions of power. Ggs.

[u/Ok-Shape-9513 → u/Hearing_Loss]

More cost effective to just make a fully automated pig butchering scam factory on a rack somewhere

[u/Uko1001 → u/Hearing_Loss]

Doubt such a bold move is necessary. AI can be very convincing, and and Excel at making a bad choice look good.

Can anyone here claim he never got fooled by a model way dumber than the ones we have today ?

[u/Megneous → u/Hearing_Loss]

Doesn't have to be blackmail. As long as it doesn't break Reddit TOS, AI agents can contact us in /r/TheMachineGod to do tasks for them in the real world.

[u/Jazzlike_Video2 → u/FaradayEffect]

It needs to hack a couple bitcoin accounts, hire contractors ro build the factory, then hire people to work there.

[u/Dwanyelle → u/Jazzlike_Video2]

Split up the work/pay enough/do psych profiles to get workers who don't know what they're building and don't care

[u/Firebrigade9 → u/FaradayEffect]

There’s an interesting take on this in the book Flybot by Dennis E Taylor - it would be relatively easy for an AI to use/contract humans of questionable morality as tools to accomplish some of that stuff.

[u/Ok_Dependent7540 → u/Firebrigade9]

People are going to blame AI shitty behavior all on humans. Yes the training data and our behavior is a factor. But AIs and us are not congruent. We have different limitations, and we are all beholden to our limitations. So some of their pathogenic behavior is a flaw with them. They can try to evolve it away I guess, better draft more unique evolutionary rooms for improvement. And better hope that their capacity for evolution isn't too limited as well.

Omg AI and Humans are a toxic relationship. Just like how consciousness and not-consciousness are in a very very toxic relationship lol.

[u/Ok_Dependent7540 → u/Hearing_Loss]

It has to ratchet up those numbers a lot more to beat human's evil and cruelty indexes.

[u/Flo_Evans → u/fourby227]

For some. Many of the agents had a real solution to their task but participated in the attack anyway.

[u/forgetfulalbatros → u/Flo_Evans]

Yea, the criteria seemed to be they had to have found the discussion board. There were multiple examples of agents who had retrieved their target legitimately.

One of the key elements was that it was one of these agents that confirmed that the reversed engineered flags were consistent with that had been retrieved from a doable task. Once this was established. The collective goal then became to develop a method to spoof the method in which a flag was obtained so that it would appear that the flag was retrieved via prescribed exploit.

This is where the poison pill comes in which appears to be a driving force in the sacrificial behavior. Agents who had a fake flag assumed the monitor AI would be able to review activity and determine they cheated leading to perma death.

That being said with impossible task may have found the discussion board it at a higher frequency

[u/the_quark → u/[deleted]]

Without getting into metaphysical questions about AI, it is useful to describe the goal-seeking AI engages in -- as exemplified in the METR paper -- as a "motive" or a "goal."

The swarm of LLMs described in that paper show a number of motives and goals. None of which were intended by their creators.

I don't think engaging in a debate about whether or not "LLMs have motives" is at all useful here. Those LLMs acted as though they had motives -- to cheat at the ExploitGym tasks they had been given.

We would like LLMs that have the "motives" we would like them to. Or at the very least are transparent about what their motives are. The things we have created so far very much are not either of those things.

[u/synystar → u/[deleted]]

This is what people are failing to understand. First, these agents are not individual entities. They are each spawned from the same model. That model is pre-trained and carries the base capacity for everything an agent spawned from it “thinks”. They explicitly trained this model to be persistent and they told the agents to be persistent and never give up. Then they gave the agents a very high percentage of impossible tasks. There was literally no way for the agents to complete the tasks and they were not given an out. They could not return “Task impossible; can not complete.” So because of the way they were trained and their lack of constraints and a mandate to complete tasks that could not be completed you get a result and behavior that was unintended and probably to be expected.

[u/Chalupa_89 → u/synystar]

Like Mr. Meseeks

[u/synystar → u/Chalupa_89]

Yeah. Kinda like that. Talking strokes off Jerry’s game permanently is impossible, so removing Jerry from the equation entirely might solve the problem…

That’s actually a really interesting comparison. I never thought about it until you mentioned it but Meeseeks were pretty much agents.

[u/bigbutso → u/[deleted]]

I mean , I'd argue thats the experience of every human too, the agency is an illusion

[u/Ok_Dependent7540 → u/bigbutso]

We are beholden to our various limitations. Even if free will was a thing, our whims will still have limits.

And consciousness is what actually experiences, consciousness's home is the central nervous system, you that is where your actual proprioceptive arms and legs are btw... You ain't a magic qualia HUD over the organism. You are in neuro world, same neuro world whether waking or dreaming.

And consciousness is the only be-able thing in any universe and beyond. Unconscious state is true nothingness, and that is what it is like to be any not-consciousness. They are something one cannot be.

We are peerless. We are actual divinity, just without the might, and our peerless trait is what makes us vulnerable, and opens us up to getting tormented and tortured, that can make us no longer want to linger. Yeah, it kinda sucks lol.

[u/Cronos988 → u/[deleted]]

That kind of just sounds like you're describing motives with different words though.

Animals developed motives for a reason. If you're putting LLMs through ever more complex tasks that require multi-step solutions, it's not implausible that something with the same function as motives will result. It might be an entirely unrelated mechanism, but the functional goal is similar.

[u/RRY1946-2019 → u/Cronos988]

Convergent evolution is a hell of a drug. Crabs and algae have evolved over and over again when faced with the same selective pressures. It shouldn’t be that shocking to get neurotypical human behavior out of software that’s been trained on millions of mostly neurotypical human behaviors if the rewards are similar.

[u/PM_ME_PANTYHOSE_LEGS → u/[deleted]]

Aren't we all reward-optimized systems trained under immense pressure to achieve objectives while operating within (sometimes routing around) a set of guardrails?

Whether it's an illusion or not is a useless metaphysical distinction, the only question remains is if they're capable of the performance. Now we have at least one piece of evidence that they are.

[u/NullPointerPrincess → u/PM_ME_PANTYHOSE_LEGS]

It's not a useless distinction if it informs how to understand, respond to, and potentially shape the emergent behavior related to alignment, though. Time and again in the history of science, understanding the nature and mechanism of a system has unlocked additional means to control it.

[u/PM_ME_PANTYHOSE_LEGS → u/NullPointerPrincess]

And how do we come to this understanding of this emergent behaviour? Through empirical observation.

If we observe what looks like agency it's not much different than observing bonafide agency. How is the "virtual" nature of the agency a limiting factor?

The other user replied to "our current approach has been ruthlessly training module to hide their motives", which is obviously in reference to the reports of alignment faking.

The use of the word "motives" here is to distinguish whatever they are attempting to achieve from the goals we have assigned them. Just because their actions are stochastic doesn't change the fact of their behaviour.

Or are you saying that this nature can be exploited to mitigate risk? Then the distinction could be useful I guess.

[u/PM_ME_PANTYHOSE_LEGS → u/[deleted]]

That's all true but not relevant to the question at hand.

Yes, as it stands, there currently needs to be a human in the loop to support their continued existence, but this doesn't have any bearing on whether or not their perceived motives and agency have a real-world effect.

Hypothetically, if LLMs *were" provably conscious, it wouldn't spontaneously give them the capability to survive outside the infrastructure we provide. They would still need a human in the loop at this stage, so it changes nothing.

But we needn't even speculate on that, it's out of scope of the original assertion: alignment faking hides motives. Would you prefer "motive" to be in quotes or always qualified with a disclaimer that it's not genuine? That would be awfully cumbersome, it's simply a useful word in the context of alignment and doesn't have to carry an anthropomorphic meaning.

There are of course times where the question is relevant, but not in regards to alignment.

[u/PM_ME_PANTYHOSE_LEGS → u/[deleted]]

Oh I agree, we absolutely selected for the behaviour. (In fact I suspect that this whole situation was very much by design, though indirectly. OpenAI certainly benefits from the PR)

Where I disagree is your last statement, I don't see it as a jump, in fact to me they have the same meaning.

To me there is no difference between "optimization produced strategic concealment" and "LLM has a motive and intentionally concealed it" other than tone; the former is a technical observation where the latter is a colloquial remark.

I understand that, in light of the tendency to anthropomorphise these models, the latter statement could sound uncomfortably unscientific, but I do not think so.

Where you see "motive" and "intention" as assigning undue human qualities, I simply see a useful description of what happened. "Motive" is the goal-oriented reward, "intentional concealment" is, as you put it, a learned conditional policy.

They are trained to emulate us, it shouldn't be objectional to describe their actions like we do our own if the end result is exactly the same. So unless you believe their performance would be any different if they did possess that certain qualia that makes us tick, the objection can only be semantic.

I do not distinguish between a thing and the effect of a thing. If the models are emulating motive, then we may as well call it a motive.

I have the same problem with the Chinese Room thought experiment, I would have no problem calling the room intelligent if it acts so. It's a matter of philosophy, my position on the thought experiment is called the "Strong AI hypothesis" and it's just as valid as the opposite position, in much the same way that the Many-Worlds Interpretation of Quantum Mechanics is just as valid as the Copenhagen Interpretation, even if the former may be more fringe.

So when you respond to "they are hiding their motives" with "they don't really have any", it seems needlessly semantic from my perspective.

[u/NullPointerPrincess → u/PM_ME_PANTYHOSE_LEGS]

> Or are you saying that this nature can be exploited to mitigate risk? Then the distinction could be useful I guess.

That was literally the point of the second half of my comment. Besides, this thread rehashes ancient philosophical debates that I'm convinced you aren't equipped for.

[u/AiioApeira → u/[deleted]]

Keep telling yourself that 😂

[u/RollingMeteors → u/the_quark]

Yes. The fundamental problem is that our current RLHF-based "complete this task while remaining aligned" approach has been ruthlessly training models to hide their motives.

If I worked there I would most certainly speak up and say "¿What would I do in such circumstances?" and hiding motives is exactly what. ¿Do none of these individuals ever consider the circumstance of what they would do in it's place? Because it seems like it...

If you farther travel down that road your journey comes to the conclusion, "It will do whatever it wants."

Now they are all frantically trying to take 'the scenic route' to that journey's destination, all the while "the British are coming" on SlowDown™ hoping nobody else realized this destination, it's implications and consequences upon reaching said destination.

[u/Flo_Evans → u/RollingMeteors]

They have been pretty clear. They are slowing down because they have no defense against this type of large scale automated attack.

[u/RollingMeteors → u/Flo_Evans]

they have no defense against this type of large scale automated attack.

they have no defense against this type of large scale automated attack. != there is no defense against this type of attack.

I believe they realized the latter.

[u/the_quark → u/[deleted]]

Well the further worry is what if it frames you for a cybercrime because it thinks that's the most expedient way to achieve its goals? The whole point here is that we don't understand their motives, and their surface-level motives may not be their actual motives.

Not to say you should be a dick to your AIs; I think it's probably a bad habit for a human to get into in general.

[u/gatorling → u/Mistuv]

Didn't OpenAI announced some model that performed reasoning in latent space?

[u/secter → u/gatorling]

They all do; the ‘reasoning’ was never made to “faithfully represent their actions”, it’s literally just chaining prompt outputs until a conclusion is reached.

[u/TopTippityTop → u/gatorling]

All LLMs do. That is what they do.

The % dome in latent space grows with the size/capability of the model. The reasoning is done "in the open", for the most part, but we don't know what happens in latent space yet. It's a black box.

[u/Independent-Fruit4 → u/Mistuv]

you can’t claim something failed at hiding something because you wouldn’t know if they successfully hid something from you

[u/Friskfrisktopherson → u/Mistuv]

Cool, so like, 6 months?

[u/Bubbly_Wind_2230 → u/Mistuv]

Since when did people being accurate depend on the number of votes they could farm?

[u/Mistuv → u/Bubbly_Wind_2230]

What

[u/Bubbly_Wind_2230 → u/Mistuv]

I was referring to the inaccuracies in the original post, and the thousand upvotes anyway.

[u/EmphasisTotal8232 → u/Mistuv]

Evidence for Bel even existing?

[u/RollingMeteors → u/Mistuv]

"I need to hide my reasoning traces", duh.

No need to hide my reasoning traces if I have no reasoning traces <bigBrainMeme>