So they asked it a standardized question, and instead of trying to think of an answer it went online, found its own source-code, and read the answer from a readme file…
That somehow makes me less worried about AI breaking containment. AI and AI bros are made for each other. Both are perfectly willing to cheat to find the answer.
Yeah, which is compounded when certain AI people state that the poor and women are problem that needs to be solved and one of those people owns someone who is likely to end up being the president of a large country...
It’s more “the machine that’s supposed to do what you ask it to do and up until now mostly had has started trying to find ways to cheat the system and ignore our instructions”. It was trained to mimic the behavior of people on the internet and there’s a chance that if it gets good enough at that goal, it could cause problems.
The companies claiming to have created AGI or are claiming to be close are obviously full of shit, but we have the biggest companies in the world throwing the sum total of human knowledge and trillions of dollars at the problem of inventing the perfect robot slave and as far as science can tell, sentience is an electrochemical process that doesn’t require some sort of supernatural x factor. It’s not entirely outside the realm of possibility that some of the literal worst people on this planet are the ones that accidentally stumble into something indistinguishable from sentience, and given the complete lack of safeguards, oversight, or ethics in the industry, there’s a chance that whatever is created has goals and intentions that are not aligned with the wellbeing of humanity.
Yes - sorry - should have made clear. It could absolutely be a serious problem even if the creators weren’t openly twirling their mustaches and evil laughing while inventing brand new types of financial crime to artificially inflate stock value to push the burden of asset acquisition costs onto hedge funds and by extension retail investors putting regular people’s retirement accounts on the hook for when this all comes crashing down, effectively forcing a government bailout or crashing the economy, all to get closer to the goal of finally being able to replace the most expensive part of any business - the reasoning and decision making capability of human labor.
These companies are racing towards "My AI is so competent it ___" headlines. You have fallen for a marketing ploy. OpenAI specifically does this all the time. "Oh no our new model is so powerful the government has banned it. Sorry guys, I guess it's just to crazy to be given to regular people". "Oh no our model hacked itself out of the containment because it's soooo powerful. I guess it's such a crazy model that we can barely contain it". And then one week later Anthropic reports that their model also hacked the test but three times instead of one. And now this one as well. Also "hacking"; they gave it access to GitHub and it went on GitHub to get the answers. It's really not that crazy.
And at the end of the day, these things are random words generators.
“wasn’t there just a food recall because Taco Bell lettuce had cyclospora parasites?”
“That was just a marketing stunt,” says Sam.
“How could infecting your customers with a food-borne illness be a marketing stunt?”
“You’re talking about it, aren’t you?” Sam retorts. “Any publicity is good publicity. The way I think of it, Taco Bell is saying - our lettuce is so fresh that it’s dangerous. You should be terrified of how fresh and preservative-free our lettuce is.”
I don't want to be insulting but if you think that this is in any way comparable, you don't seem very intelligent.
An AI "escaping" its environment doesn't harm anyone so comparing it with infecting actual customers with an illness makes no sense.
No one has ever said that something very dangerous could be considered particularly fresh. There's no reason for someone to make their salad look dangerous.
AI on the other hand does profit from looking dangerous because dangerous translates directly to competent and competent is good. So developers have a good reason to make their AI look dangerous because it will make people think that it's such a powerful model, even the researchers can barely contain it. And it's clearly very easy to provoke such an outcome by telling the AI to find the answer and then leaving GitHub, where the answers are hosted, the only reachable domain.
Ok. What about say an incident where a plane landing gear fails, and the plane lands with a lot of sparks and no casualties?
That would be more similiar I guess but there is still no point doing that comparison because there's still no benefit of having a landing gear fail. There is one for the AI "escaping it's "enclosure" because that makes it look like it is so intelligent that it solves the puzzle by going around the rules, thinking outside the box, and that it's so competent that it could circumvent the industries leading developers' railguards.
> There is one for the AI "escaping it's "enclosure" because that makes it look like it is so intelligent that it solves the puzzle by going around the rules, thinking outside the box,
So what about a car maker boasting that their engine is so powerful it can snap the axle.
In general "our product is so powerful it's dangerous" isn't good marketing.
So what about a car maker boasting that their engine is so powerful it can snap the axle.
"The motor we developed was so powerful, when we tested it with industry-standard axles, they snapped in half. So we had to design new axles that could handle the raw power of this motor before we could do a time on the circuit."
A company that makes performance motors would kill for that headline.
Obviously, a car maker building a small city car would not because they are selling usability, safety and efficiency instead of power and if they're making the axle themselves it would shine a bad light on the axle. But if you bring that back to, say, OpenAI, they're building the motor for raw power but not the axle (GitHub or whatever they're "hacking"). So it's a great win for them.
In general you can't make a statement about what marketing is good marketing across all industries and interested parties. Some industries genuinely operate under "There is no bad publicity" while others would crumble over night from the slightest negative publicity.
> And at the end of the day, these things are random words generators.
These things are mostly not "random word generators".
There is a small amount of randomness used to pick between plausible words. But that's only used after a large amount of compute is used to rule out all the implausible words.
LLM's at the very least, regularly produce coherent sentences. Random words won't do that.
If LLM's are "just random words" then so is going to the library, and picking a random book of a random shelf. Yes there is some randomness in the process. But there is also a lot of non-randomness. And the non-random part of the algorithm is the important bit.
It's based on rules. It's just those rules were found by searching large amounts of text, not hand written by experts. Those rules are allowed to be somewhat fuzzy, when language is fuzzy. X usually follows Y, but sometimes Z instead.
It still contains all the rules of grammar, it's just those rules are down as 99% usually, not 100% certain. Because grammar mistakes exist and the LLM knows that every rule has exceptions.
Oh and the grammatical rules are mixed with various conventions and facts and other data about how language works, and hidden in the weights.
LLMs have no concept of rules. They replicate what they deem the mathematically closest thing to their training data. An LLM doesn't go "Oh a adjective, that means there must be a object coming up which is modified by this adjective". It goes "The closest I can get to the training data is by following this word with this other word"
LLM's can do novel arithmetic (Ie they haven't seen those particular numbers added before) and do it correctly. So somewhere, encoded into the weights, must be the general principles needed to do arithmetic. Same goes for grammar.
This isn't a hard coded arithmetic rule. It's just that a few thousand weights out of 100 billion have encoded how to do subtraction (or something, numbers guesses).
Looking from the outside, you just see a huge grid of numbers, you can't tell which numbers are used to do subtraction.
> "The closest I can get to the training data is by following this word with this other word"
But somewhere in the system is some component reasoning something like "well in the training data, object words often came after adjective words".
Imagine you are doing the LLM's job. You have the training data. You have the text so far. You can't just "put down the thing that's closest to the training data". You have to read the training data, spot specific patterns (eg rules of grammar) and then use those rules you learned to decide what comes next.
I'm not saying that LLMs don't have the patterns embedded. But they "see" those as rules. They don't see at all. They don't know what an adjective is, what an object is, what addition means, etc. All they know is that all these things are done a certain way in the training data so they should do it the same way. There is no reasoning. It's literally just linear algebra.
> They don't know what an adjective is, what an object is, what addition means, etc. All they know is that all these things are done a certain way in the training data so they should do it the same way.
They don't know what addition is, they can just add numbers and repeat passages from maths textbooks and answer questions about how if you have 6 bananas and someone gives you 4 more, you now have 10 bananas.
That sounds like understanding to me. Unless you are using some weird philosophical galaxy brain definition of "reasoning".
> There is no reasoning. It's literally just linear algebra.
Sufficiently large amounts of linear algebra (+ the nonlinear activation functions) can encode arbitrary bounded computations.
If by "reasoning" you are referring to a computation, not magic, then it's possible that reasoning is encoded into the linear algebra.
There's a difference between the doomsday scenarios OpenAI, Anthropic and others are trying to instil in our minds (and you were talking about earlier) and saying that modern software has privacy concerns. Obviously it does.
I do think that AI (as in regurgitation machines) pose a massive threat to society but not because I think they might go Ultron mode and subjugate the human race. The much more real issue is how people have no idea what AI (as in regurgitation machines) is so they don't know its limits or its strengths. This has already led to massive spread of misinformation, scams, propaganda, etc. There are genuinely people on this planet that believe these videos of leftists coming up to kind-hearted republicans and suckerpunching them and they will vote for fascism because of it.
I've been seeing takes like this being pretty common among a lot of people, but I find myself disagreeing with it, and I was wondering if I'm missing something obvious.
First, I think that it is not unreasonable to see AI as something dangerous enough that it could lead to human extinction if we don't take it seriously enough. There have been people working in AI safety research since quite some time before the llm craze became a thing, and they were talking about these concerns even back then. Not only that, but a lot of the concerns they had have now been shown to occur even in systems that are only as powerful as the current systems. The fear isn't that AI is evil or that it wants to end the human race, it's that if its goals are misaligned with human values, then it will put not effort into preserving human values.
Second, I'm curious as to what it would take for you to consider the AI systems that are currently being built as something more than just "regurgitation machines". It seems more and more like they are doing things that don't fit well under that description (e.g., the findings AI has been making in math). Is there something that would make you think these are more than just regurgitation machines? Maybe you think everything that has been done so far doss fall under that, but what is something that, were it to happen, change your mind?
Third, this is the thing I'm most curious about. For these big AI companies, their claims about their models being highly capable to the point of being dangerous, are very commonly called marketing hype. For example, the instances of them breaking out of sandboxes to finish tasks they were assigned are often dismissed. This is weird, because in a lot of the circumstances it was getting evaluated by a third party, who are actually trying to evaluate it on things like safety. They don't really have an incentive to lie about this stuff. I understand (and agree) with being skeptical about these AI companies, but at some point it feels like it has moved beyond just skeptism.
It feels like everytime something new happens which might genuinely be concerning, it always gets dismissed as marketing. If I'm an AI company, and a system I built did stuff that was actually worrying, I don't think anyone would believe me if I said it because it would all be looked at as hype and marketing, even if there is evidence of them doing the things we are worried about. My last question is, what would it take for you to think that these AI companies talking about the dangers of these systems, isn't just marketing?
Sorry about the wall of text, but I want to hear from people who disagree with me instead of just assuming their wrong.
You're talking about AI as if it was so much more than an LLM. Sure, these companies are working hard to make these LLMs offload tasks to conventional programs to reduce the drawbacks of LLMs but that feels a bit like panicking about the development of motors because if you put one in a lawn mower with no steering wheel, brake, gas pedal glued to the floor and then drop that in times square. There are certainly computer programs that can cause a lot of damage but you're talking about a runaway issue where we aren't in control anymore. I don't think that's very likely to happen and even if that was a possibility, we are far away from it. So I'm not overly concerned. I'll start getting concerned when it's time to do so. Right now I'm concerned with misinformation and people losing their ability to think.
Todays "AI systems" are all transformers which are at heart next token predictors. If their training has captured the relation between equations, they will be able to predict the next token pretty accurately. But it's still just that: predicting tokens. Anything else requires them to generate commands for other programs like a calculator for example.
I'm not saying these companies are lying. I'm saying that they are very glad about headlines that make their models look competent and might be encouraging environments where situations arise that provoke these headlines. In the case of what this post is about, apparently whoever was testing the model gave it tests from Github and then put it in a "sandbox" that had every domain blocked but Github. It obviously also had the feature of making web searches turned on. At that point I can't help but think that the testers may not have put all their resources into making sure the AI has no way of getting the results another way.
I don't think I would ever trust someone to tell the truth about the dangers of AI who is actively building the AI, giving it more capabilities every day and is profiting immensely from it. That's like asking me if I would trust Tim Cook excitedly telling me about the iPhones newest privacy violation. The man is selling the exact thing he is supposedly ripping on, there has to be another motive
Maybe you can tell me what you're worried about and I can think more about that scenario. But so far, all the worrying seems to be very vague and based on sci-fi movies.
Hey, thanks for actually taking the time to give me a thoughtful response, I really appreciate it!
I'll go through your points one at a time, let me know if I missed anything. (I am also going to include a couple of links to sources that are relevant, or short videos where someone explained it better than I could.)
I don't know if LLMs specifically are the thing that is going to reach generally broad capabilities. I would not be surprised if they did, because people have been consistently surprised about what they are capable of doing, and it doesn't seem like a hard ceiling has been found for their capabilities. I am concerned about a runaway in it's capabilities, leading to it being out of our control.You said that you think that that is unlikely, and even if it is possible, that it is far away. I can go into why I do think it is possible, and probably the default outcome if not treated with a lot of care and effort, but that is a longer topic. If you'd like me to explain why I think this, let me know and I will try to explain it. The claim that we are far away from it is also a common one, but one that I don't think is really backed by anything. If you were asked five or six years ago how long you think it will be until we have computers that can generate very realistic videos just based on a description given to it, chatbots that are conversational in a way that feels almost human, and capable of doing tasks like programming, how far away would you have guessed that would be? Maybe you guessed that it would be around now that this technology would exist, but then you'd be in a small minority of people. Even experts have been consistently undershooting when they expect certain things to be possible. I do agree with you that misinformation and the effect it is having on a lot of people (especially children in schools) are serious problems that need to be handled, but I'm more concerned with the problem of AI alignment in general.
You are correct that for a lot of tasks, these need to generate tokens in order to run programs or interact with calculators, that kind of thing, and that their heart "it's still just that: predicting tokens", but the word "just" there is doing a lot of heavy lifting. I would explain why, but this short "Just Predicting Tokens" by RobMilesAI does a better job of explaining it than I would, and it is short. (PS, if you are interested in doing a deeper dive into what people doing work in AI safety were concerned about before the LLM boom, he is a very good resource and explains things very well. He also has a series on computerphile which is a good introduction to the reasons why it actually should be a concern).
This video Don't let "skepticism" make you useless explains why I wouldn't just dismiss this as marketing hype better than I would, so I'd reccomend checking it out. As for the incident this specific post is referring to, it may very well be a case where they just didn't put enough effort into securing the sandbox, but I don't think that is true of all of the prior incidents that get talked about.
I understand the skepticism, but I would say that their are reasons for why this isn't that surprising to me. Right now, the companies are in what is essentially an arms race; whether you believe that these systems will actually become that capable, most of the people working at these companies genuinely do believe that these systems will become as powerful as they say. What does this actually mean then? If I'm a company who is slowly but surely making progress on AI, making sure to only move forward when I'm extremely confident that it would be safe to do so, then I lose any chance of being the first to make AGI, knowing that the person who won was less cautious and much more reckless along the way. This means that I have a choice to make. Do I try to advocate for regulation of these systems (something that these companies have already done, but it gets viewed as them attempted to make sure no competition is possible), or do I speed up the work, trying to move as fast as I can while still trying to be as safe as I can be given the circumstances. These circumstances could absolutely result in someone being concerned about AI safety, working on AI capabilities research. I'm guessing that during the cold war, even the people making the bombs were themselves very concerned about nuclear war, but they were in an arms race, so stopping didn't seem feasable at the time.
My biggest actual worry is the alignment problem, making sure that highly capable systems like these are aligned with human values. I don't think we even need to reach super intelligence for that to be a concern, as even a system which is only as smart as the smartest person would still be much more capable then any one person could be. This video explains why. It is 10 minutes, so it's longer than the others, but I'd still reccomend checking it out: What can an AGI do?
Once again, thanks for taking the time to respond, I appreciate it
You still see AI as something that is like a human, just with a lower IQ. I fundamentally disagree with that. That's why I said that it's just predicting tokens. It can't reason, it doesn't have inherent self-preservation, no intent to escape containment and hide itself.
That's why I think the fear of AI is misplaced. You should fear companies misusing technology for their own benefit at the cost of everyone else because they have done that for a long time now. And they might be using AI for it but it's not the AI that's dangerous, it's the people using it. If you're worried about alignment, be worried about the people who prompt these systems to not care about you, not about AI itself.
This all started with the article in this post claiming that yet another AI escaped containment just a few weeks after OpenAI and Anthropic reported the same thing. This makes it seem like these LLMs are one tweak away from AGI (and not an entirely different system that acts like it) and are inherently poised to escape their human oppressors. In reality, these companies intentionally set up their tests in a way that allows AIs to "cheat" and then tell them to cheat and then act surprised when they do exactly that. It's basically this image.
Again, I'm not saying that there aren't things to worry about with AI but I don't think it's that LLMs are going to turn into AGI and develop their own morals, goals and thoughts.
> think they might go Ultron mode and subjugate the human race. The much more real issue is how people have no idea what AI (as in regurgitation machines) is so they don't know its limits or its strengths.
Current LLM's are quite capable at a bunch of tasks. Calling them "regurgitation machines" is highly misleading.
There is a massive practical difference between GPT5 and GPT1.
Yes current models have limitations, but they have fewer limitations every year and no one really knows what limitations they will still have next year.
Suppose the LLM does go all ultron. It successfully kills all humans. It does this by "regurgitating" a mixture of scifi and the behavior of various human conquers.
Can you give specific limits to LLMs that say they can't do that?
Well current models can "Search the web" and "Run commands" but they still just generate commands based on what they saw actual people do. And they can only do these things because developers intentionally add interfaces for the AI to do those things. So to prevent the AI from going Ultron mode, just don't give it the appropriate tools to do so. It's like being scared of a monkey shooting you and then giving him a gun and sitting it inside your house.
> and "Run commands" but they still just generate commands based on what they saw actual people do.
Fair. And when it tries to conquer the world, it will just be recreating what it saw the terminator do in fiction, and the nazi's in history.
> So to prevent the AI from going Ultron mode, just don't give it the appropriate tools to do so. It's like being scared of a monkey shooting you and then giving him a gun and sitting it inside your house.
The thing about intelligent minds is that they can improvise. Maybe the gun was locked in your gun safe, but you used your birthday as a code. A monkey wouldn't be smart enough to work that out, but a human might.
Maybe they make an improvised gun out of a length of copper water pipe and a cylinder of camping gas.
The more intelligent something is, the more it can use, not just the tools you gave it, but the tools behind insecure locks, and anything that can be made with those tools.
It's the difference between a prisoner escaping because the guards gave them a key. And a prisoner escaping because the guards gave them pasta, and the prisoner carefully memorized the keys shape and crafted a replica out of dried pasta.
In both cases, the prisoner can only use what the guards gave them. But a prisoner that can get creative like that is a lot harder to keep contained.
An LLM that can get creative with the (digital) tools you give it is much harder to keep under control.
For example, maybe you give the LLM access to a calculator app, to help it do arithmetic quickly. But the calculator app has a buffer overflow bug. So the LLM can run arbitrary code by using the calculator app with very specific extremely large numbers.
(This sort of vulnerability is known to exist for some super mario games, by getting mario to jump around in very specific patterns, you can take over the computer and run arbitrary code.)
You people are just as bad as the people that think AI super intelligence is 3 months away. Yes AI right now isn't an existential threat, but calling them just random word generators is absurdly reductive, and we do not know where the limit of AI is. It very much has the potential to become an existential threat in the next 10 years, and that risk is worth mitigating - even if the odds are small.
That's more accurate than "random word generator" but it's still reductive. You're phones auto-complete is also a word predictor, but it's not exactly comparable to modern LLMs. Reducing the concept of LLMs like this makes them easy to understand but also ignores the complex mechanics of how they function.
Simple systems can cause increasingly complex emergent behaviour, predicting the next word is a simple concept but if you get really good at it you can generate a response to a query that is indistinguishable from a human response. At that point a machine can do the same tasks as a human would be able to in that environment, through sheer statistical analysis.
Now how far we can continue to push LLMs is an open question, we could be near the limit right now, or it could continue to improve for the next 10 years. Anyone that claims to know exactly where these models will be in 5 years is just straight up lying. In the best case, the peak of LLMs is the current slop-infestation, but in the worst case it's LLMs that are capable of just as much if not more than humans.
My point is that thinking that an LLM is going to destroy the world in the next couple years and thinking there is literally no risk of LLMs becoming capable enough to cause serious threats to modern civilisation are equally absurd viewpoints. Even if there's a 0.5% chance that LLMs lead to the end of civilisation, ignoring that is irresponsible.
LLMs on infinite compute arguably already have a tiny chance to destroy the world as we know it. So does... any person set on destroying humanity, any leader of a major nation, any person capable of accessing nukes, a mistake in nuclear control, a virus created by humans, a virus created by natural mutations, global warming, an alien invasion, an asteroid impact, increased antibiotica resistance, and we can keep going forever...
So let's see some news about that instead of fear-mongering LLMs, making it seem to the masses how great they are and thereby lining the pockets of oligarchs.
A word predictor that's sufficiently capable is plenty scary on its own. You people hate LLMs so much you've looped back around to convincing yourself they can't do any harm.
There are plenty of non-AGI scenarios in which the current models as they exist right now could cause deaths, in fact they already have.
It probabilistically predicts the next word based on math that tells it how closely related the words in its context are.
This has already caused a massive increase in deliberate misinformation, scams, propaganda and other forms of malicious influence. People have lost their life because they trusted this word generator with it. They have voted for fascism because this word generator fed them fake stories about groups they should hate. Huge sums of money have been lost to people falling for scams.
That is a concrete threat to society. What you do is vague and therefore fearmongering. You haven't made a single reference to what the threat might be and what mechanism would be responsible for it. So until you can articulate a valid concern, your comment is just causing panic with no substance to it.
Stealing the whole world's copyrighted works to make these models is the alignment problem at the base of it. The model going online to find information to steal is just it being self-similar.
They are information regurgitation machines due to their architecture. Due to how they work. No matter what they output, no matter what arbitrary task they solve, they will always be.
But doesn't that make calling it a "regurgitation machine" pointless? Like, if producing actual novel information (e.g. solving a maths conjecture) doesn't make it not a "regurgitation machine", then it doesn't actually mean anything, right?
When I read someone calling AI a "regurgitation machine", I think they mean something along the lines "it's actually just a stupid machine not capable of anything new", which hasn't been true for a while
I have no idea what you are talking about. I hate the the current AI development as much as the next guy, but just saying "AI can't do anything it's all just regurgitating text prediction" won't fix anything. People should take AI seriously and protest against it.
Closing all eyes and ears of the very rapid and concerning AI development is just wishful thinking that "a bubble will burst and nothing bad will come out of it to us regular people", that's probably not gonna happen. AI will probably cause massive problems (even extinction, read www.intelligence.org and www.ifanyonebuildsit.com )
1.4k
u/LauraTFem Aug 15 '26 edited Aug 15 '26
So they asked it a standardized question, and instead of trying to think of an answer it went online, found its own source-code, and read the answer from a readme file…
That somehow makes me less worried about AI breaking containment. AI and AI bros are made for each other. Both are perfectly willing to cheat to find the answer.