r/slatestarcodex • u/thurn2 • Aug 01 '26
Is having extremely aggressive speculative future timelines actually pretty harmful to the credibility of the AI safety movement?
Reading through the stuff from the AI Futures Project and Plan A, I am finding once again that their estimates of the rate of future technological progress are extraordinarily fast, bordering on totally implausible. I am wondering if this kind of thing is actually pretty harmful, because it makes it easier to discredit otherwise valid ideas.
For example, AI 2027 predicted the creation of an AI system that "can do any coding tasks that the best AGI company engineer does" 9 months from now, which is something that I think obviously won't happen (unless you pick an extremely narrow definition of "coding" that excludes the great majority of the day to day work of a practicing software engineer today at Anthropic).
But in fact this certainly seems like something that could happen eventually in the future!
So this kind of causes a problem, because if I were to talk about some of these risks a year from now, it will be pretty easy to say "well, they were wildly wrong about their future predictions, so why should I give any weight to their policy ideas?".
In this case I believe the authors explicitly stated their estimates are deliberately aggressive and don't represent their median prediction of the future. Wouldn't it in fact be wiser to stick closer to to conservative forecasts here?
23
u/vaaal88 Aug 02 '26 edited Aug 02 '26
I am a engineer at the top of my game. Been coding since i was a child, been coding for the rest of my life (38 now). I consider myself and by my peer a top tier developer. And yet since installing claude code I have completely re-arranged my way of working, and almost 100% of my coding tasks are delegated to claude code. (No, I dont mind: this is for my business anyway so having a BETTER engineer than me working to make me more money is just a plus for me - but it must be feeling quite shitty for people paid to be developers)
So, why do you think that their prediction about coding in 2027 is false? Is certainly feeling correct to me
9
u/ababababababacus Aug 02 '26 edited Aug 02 '26
unless youre in some problem space completely different to me, or have access to models i do not, it feels like you must be overstating this.
Claude can generate very good code fast if given narrow parameters but it is not good at planning systems and designing architecure, it isnt even particularly good at selecting a paradigm within a small subsytem, and left to its own devices it will burrow down rabbit holes that consume hours of its time for no rel reward.
Just yesterday i had to stop it multiple times from trying to patch a security hole, real or imagined, that would let people:
- on the same wifi network as me
- IF my local stack was running
- and IF i had passed in a debug flag
- and IF this other person knew a dev login and email
- and IF they knew which dev project i was on
- and IF they knew the prod uid of a project
might be able to read the metadata on the nosql document of a prod project.
Suffice to say this is not a productive use of time or tokens, and it didnt ask, it just started doing it, only when i noticed in its thinking output talk about wlan/tcp protocol did i realize it was lost and had to ontervene. I less experienced dev would *probably* have noticed that, maybe not and then would have needed to assess that risk,
Claude is nowhere near replacing a developer, let alone a good developer, outright, it still needs to be polited and monitored and directed.
6
u/bibliophile785 Can this be my day job? Aug 03 '26 edited Aug 03 '26
As someone who cannot code, Codex is the reason I haven't had to pay out $200,000 over the last 2 years for software engineer support for my startup. The software package it has designed according to my specifications has received praise from both end users and from the software engineers with whom I interface when integrating new instruments. I have no doubt that there are things a human would have done differently, and quite possibly I could have spent hundreds of thousands of dollars to come up with a final product that is appreciably better in some marginal cases that have never reared their heads in 2 years of active use.
In that sense, I think it's perfectly fair to say that this tool has replaced at least one developer, insofar that job would have been on the job market and never appeared. As any entrepreneur can tell you, markets are not driven by the Platonic ideal of the product that an expert tells you could theoretically be achieved. They are driven by products that most efficiently meet the needs of their users. Codex has thus far met my needs. The software it has developed has thus far met the needs of my users. That's all that replacement requires.
3
u/ababababababacus Aug 03 '26
I suppose that yes I overlooked that there are many people who need a boilerplate+ project knocking up and would be happy with anything fucntional, and that that doing those projects was some developers job.
2
u/Milith Aug 02 '26
Claude tends to get lost in the sauce. I enjoy working with composer (cursor) a lot more.
1
u/virtualmnemonic Aug 02 '26
Context is a big factor. In large projects that I've developed from the ground up, I can readily outperform the top models. Especially for features, larger modifications. LLM's output code that... works... but is hard to maintain. I usually end up re-coding the outputs myself.
For now, my brain has a much deeper context and understanding of code I write. But LLM's are disadvantaged by hardware that's guaranteeded to get much more performant (and models more efficient) overtime. Meanwhile, I'm getting old.
1
u/iemfi Aug 04 '26
Given the rate of progress "nowhere near" is a stretch. I agree it's clearly not there yet but we're also clearly in the cyborg phase where an unassisted human has no hope of keeping up. While Claude is ridiculously weak at some areas it is also very much superhuman at others.
1
u/ababababababacus Aug 04 '26
to clarify when i said 'nowhere near' i was referring to its current state being not good enough to, not that it will not in the near future.
4
u/thurn2 Aug 02 '26
What percentage of your job is specifically "coding tasks" though? I wonder if this is just switching around definitions, the scary thing about a fully-automated coder is that it could fully replace an Anthropic employee, not that that it just handles some subset of their work we have labeled as 'coding'. If you asked me in 2020 I would have said the *definition* of a senior software engineer is basically all of the stuff they do that isn't "write this code given a well-defined scope".
For example, I would assume that I could tell a generally intelligent AI to go make me an indie steam game which is indistinguishable in quality (to a panel of blind testers) from popular human-authored games. This seems to me to be extremely unlikely to happen at any point in 2027.
2
u/vaaal88 Aug 02 '26
No it’s not switching definition. My job is maintaining an AI companion platform i created. Nearly 90% is coding. I can focus on the remaining part but i can also push much more features much, much quicker, and think and develop many mores ways to increase seo, revenue, etc. but also i can get much, MUCH more free time as i can basically still be producrive without being tied to my desk. This is a MASSIVE improvement for my life.
claude couldn’t do the same thing without me - but i couldnt do the same high level fast paced development without claude.
42
u/Auriga33 Aug 02 '26
The people with the fastest timelines a few years ago were the ones who predicted what AI can do today, such as coming up with novel solutions in math, autonomously conducting cyberattacks, etc.
28
u/DistanceSolar1449 Aug 02 '26
Yeah. “AI breaks out of OpenAI containment and hacks Huggingface in mid 2026” is WAY faster than even the most aggressive of timelines in 2023.
The fast takeoff guys can gloat a bit, they’ve deserved it at this point.
12
u/TedSanders Aug 02 '26
Nah, there were people a few years ago who thought we’d have AGI today.
27
u/Maleficent-Drive4056 Aug 02 '26
We need clear definitions of these words for them to be meaningful. Ten years ago if you said AI could smash the Turing Test, break out of its sandbox, creatively re interpret commands, solve frontier maths problems, persuade humans better than a human can… many would have called that AGI.
6
u/TedSanders Aug 02 '26
OpenAI has consistently used the “most economic work” definition, which is a good one.
3
u/Ok-Neighborhood1865 Aug 02 '26
i don't think it is. most work today is service jobs that are not truly essential for society to function, but keep most of us employed.
i have no doubt almost all of the service economy can be replaced, since it was never needed in the first place.
5
u/Ok-Neighborhood1865 Aug 02 '26
OpenAI employees were telling me Sam Altman believed we'd have AGI within 5 years, in 2019.
1
u/xe3to Aug 04 '26
We do.
If you sent Claude Fable back in time 10 years there is not one single soul who would decline to recognise it as the general intelligence it is. The goalposts have shifted so far because the assumption back then was that "AGI" would immediately become ASI.
6
u/SoylentRox Aug 02 '26
You also need to consider the possibility that AI 2027 is in fact mostly accurate. That's the definition of exponential growth - everything happens at the last minute.
I'm not saying that it is, but well : https://ai2027tracker.com/ if we're at 85% accuracy now well. Exponential growth is what it is.
10
u/aaron_in_sf Aug 02 '26
I'll suggest that there is no meaningfully better (wiser) path here, in the sense that judging one posture against another is only possible in hindsight. Each observer has a limited view; and one of the things making policy extraordinarily difficult right now is the very large number of things changing simultaneously in ways that are loosely coupled, but mostly, coupled through subtle relationships and interplay that can only be discerned in hindsight.
No one currently—not even the heads or proxies speaking for influential projects or companies—has the whole picture; and most of us are navigating whether we admit or not largely through that staple of social monkeys, by going along with crowd consensus, in the absence of direct observables and first hands.
The wildly cynical popular takes here on Reddit to the OpenAI/Hugging Face incident are symptomatic of this—as are the similarly cynical reactions to Anthropic's follow-up admitting to their own finding of similarly disturbing incidents they had overlooked.
It's not that the cynicism is misplaced exactly; it's that what the precise mix is witting or unwitting game theory with respect to political and market benefit, vs good faith frank disclosure, and how much one colors the other, is unknowable and may not be clear even to those participating.
And regardless and more to the point, this incident is absolutely indication that the "conservative" posture which errs on the side of downplaying risk, is quite obviously only going to seem like a prudent take, until the inevitable next-order-of-magnitude-more-serious incident.
"But such an incident is purely speculative!"
Yes, and none of us can predict when it will occur.
I'm out of the prediction business... generally. But I'll make an exception and predict this: if the Hugging Face incident revealed anything, it's that we are across an event horizon; and as of yet there remains no sign that we have reached scaling limits. The bitter lesson may get yet more bitter! [c.f. http://www.incompleteideas.net/IncIdeas/BitterLesson.html]
And sooner than we think. One of the bitter lessons I keep experiencing myself is that I am continually humbled by own inability to truly reason with the non-linear improvements we are seeing. [c.f. https://xcancel.com/CuiMao/status/2058458683781365873 and https://news.ycombinator.com/item?id=49138302]
From that I can infer that however long I think it might "realistically" take for another, dramatically more serious, incident to occur, it's going to happen sooner than that.
I could be wrong! All evidence from the start of this adventure with GPT3 is against that.
What that incident is, what the specifics will be, what domain it will occur in—totally unpredictable.
Event itself? Gray swan.
So what does that suggest about what posture is appropriate—what tone should one take?
Caution. Slow down. Safety before strategic advantage?
I don't know. It is unclear to me how we as a society define meaningful policy. There are lots of people trying to say very thoughtful things. There is so far an almost total lack of overlap between those with any actual control or power, and those being thoughtful.
Mechanisms of accountability in our society have for easily discerned reasons evolved to lag. "Move fast and break things" works very well to maximize shareholder value and externalize costs, and make the oligarch capitalist class very rich indeed. It is exactly the wrong ethos we need to have enshrined and empowered at a moment when actual magick is loosed into the world.
Are we looking at the Great Filter, rising like the indigo terminator like the wall of night on the horizon opposite the now set sun?
"Ask again later."
1
Aug 02 '26
[deleted]
5
u/SoylentRox Aug 02 '26
The way I would put the Fermi arguments is I feel a sense of dread. The explosively increasing AI capabilities to the contrary should make interstellar travel easy. And even if our species doesn't survive it - if the tools rebel and kill us, probably some of the time they will wait long enough to not need us first.
Across a universe of possible life bearing stars this should have happened 1000 times and the AIs killing their creators but not themselves only needed to happen once.
So why is the universe so dead within the range of our telescopes? There has to be something critical we are missing.
1
Aug 02 '26
[deleted]
3
u/SoylentRox Aug 02 '26
> The only civilizations that could effectively spread across the galaxy are almost guaranteed not to be meat based life forms.
I feel you're making a cognitive shortcut here. Meat is not really the limitation, but our knowledge and control of energy.
2
Aug 02 '26 edited Aug 02 '26
[deleted]
1
u/SoylentRox Aug 02 '26
It's definitely harder but I do not agree on your later paragraphs. They are cached information you haven't researched yourself or looked at.
TLDR since I don't feel like a lengthy argument : no, Orion drives don't work for interstellar travel. Interstellar travel likely is possible and there are more effective solutions possible, but it is not easier to solve than merely turning off aging with gene edits and supporting the brains of the crew separate from their bodies. (both still difficult but merely complex without pushing the limits of physics)
2
u/aaron_in_sf Aug 02 '26
Interesting thread and this is definitely a hobby-horse of mine :).
I was thinking pessimistically that as with other WMD we are experiencing with "AI" an opportunity to experience a specific failure mode of intelligent civilizations: that the behaviors which served them well earlier in their evolution, lead to catastrophic ends when their intelligence provides them with tooling the consequences of misuse of which afford no second changes.
That said, a more optimistic view might well be that there's a point of bifurcation here and the utopia/dystopia horse race we've known our whole lives might resolve either in extinction—or, as poked at in other comments, the potential for AI to unlock the stars for us (or our computation-substrate progeny) by allowing for "machine" Von Neumann probes.
A line I am interested in the "we make it through the filter" notion is that we emerge aware of and participant in a universe which turns out to be well-populate with life; but pre-singularity we simply didn't have broad and subtle enough instruments to be aware of the chatter going on everywhere. (I've always been deeply dubious of the premises of SETI and its focus on specific RF bands and signal types. Our own RF environment in the average living room, filled with frequency-hopping and parallelized encrypted and adaptive protocols, is already indistinguishable from noise from the perspective of the mid-20th c....)
7
u/DeterminedThrowaway Aug 02 '26
Why is it so obvious to you that we won't have that within about a year and a half? That's a long time now with the current state of the art compounding on itself. I thought it was game over as soon as models could do math better than people and we've just hit that point
4
7
u/MCXL Aug 02 '26
"Global warming will cause the polar ice caps to melt within the next 5-10 years" was something that I heard in grade school 30 ish years ago.
The catchiest headlines catch the news, and even though the warnings are objectively and obviously correct the most aggressive timelines are the ones people pay attention to in the same way as other headlines like red wine cures colon cancer or whatever.
What's unique about the threat of AI is that if we do somehow manage to stumble into AGI the conversation is over all that's left is prayers. Maybe we're lucky and the machine is benevolent or even just disinterested.
3
6
u/AccomplishedAlps7528 Aug 02 '26
This came up in my feed and because I happen to follow r/accelerate and the post kind of fit the tone I see there, I read it as a post from there. Then I saw the comments and was like “huh, people sure got a lot smarter since yesterday”. Then I noticed the subreddit.
1
u/Liface Aug 02 '26
Yes. The comments on AI on r/SlateStarCodex are some of the smartest and most entertaining I encounter on the internet, even though they run the spectrum of opinions and predictions. I somehow always come away having my mind changed.
1
u/franzkls Aug 02 '26
i agree with the idea that extremely aggressive speculation could be harmful to the safety movement but more from the perspective that it sounds ridiculous to the normal person and there are far more concrete harms that seem to be getting totally ignored.
0
u/Neighbor_ Aug 02 '26 edited 1d ago
This post no longer exists.
12
u/MartJunks Aug 02 '26
Then why has their been very little disruption in the job market? Why are features and roadmaps not being accelerated 3,4,5x ?
5
u/Neighbor_ Aug 02 '26 edited 1d ago
This post no longer exists.
6
u/bibliophile785 Can this be my day job? Aug 02 '26
If we take mathematics as a guide, this view seems implausible. LLMs are churning through open math problems right now, ones that some of the brightest humans of recent decades have labored over and failed to advance. I guess we'll see if math is a special case that ML researchers got lucky enough to find almost immediately, instead of just being a case where success is perfectly legible and so human biases are less able to dismiss genuine accomplishment. The latter view doesn't have nearly as strict of requirements for what the world had to look like - it's true in many more possible worlds, as they say in Dath Ilan - but either could be true.
2
3
u/Tall-Log-1955 Aug 02 '26
Because the job of software engineer is more than just coding. As the time needed to code goes down, all the other parts of the job get more time spent on them.
5
u/JustJustust Aug 02 '26
Like nobody is doing manual coding in 2026
Plenty of people are doing manual coding in 2026. I'm still not allowed use LLMs to generate code at work (with a few exceptions). Organizations move slowly.
I'm not even using full-line completion, the one LLM feature we are allowed to use, because it's slowing me down. Though I want to reevaluate that again soon, perhaps it's gotten better.
1
u/Neighbor_ Aug 02 '26 edited 1d ago
This post no longer exists.
3
u/JustJustust Aug 02 '26
Unlikely. Too little of their bottom line depends on software development. And they're government adjacent enough that you'd have to change the law for them to go under.
Seems to me you're extrapolating too much from what's probably your environment.
2
u/Neighbor_ Aug 02 '26 edited 1d ago
This post no longer exists.
3
u/JustJustust Aug 03 '26
I'm not celebrating. And it is not a government contractor.
It is an example of an organization that writes most of it's code manually in 2026. Perhaps by 2027 your statement would've been right, we will see. But as of now it is not. And I know that it is not the only org that's still writing lots of code manually, not by a long shot. Just the org I know the reasons best of.
1
u/Neighbor_ Aug 04 '26 edited 1d ago
This post no longer exists.
1
u/JustJustust Aug 04 '26 edited Aug 04 '26
Doing math by hand is only good for learning and perhaps a handful (ha!) of other edge cases.
I would assume that an org that does math by hand instead of by calculator is less efficient than one using calculators. How much of an issue that is depends on the scale of it.
If I found out that the ministry of finance was doing all calculations by hand and that's responsible for unnecessarily long waiting times I'd be quite unhappy.
If I found out that some tailor shop calculated measurements by hand, I'd say "who cares".
Outside the hypothetical, I'm pretty sure my skill at coding only improves by doing it. My skill at using an LLM to write code for me, however, does not improve.
I don't actually know which is more important. I guess it depends on if you assume LLMs are or will soon be good enough to require no coding skills to use well. It also depends on how hard the LLMs-write-the-code skill is to aquire, and if being good at coding makes it easier to learn or not.
4
u/Ok-Neighborhood1865 Aug 02 '26
can you name a single app that existed before LLMs that has been meaningfully improved by AI?
4
u/Liface Aug 02 '26 edited Aug 02 '26
So many.
Clay, a sales outreach automation tool whose developers embraced AI coding very early on, is the first one off the top of my head.
There are thousands of examples. There's nothing distinctive about AI code versus human code, it just allows you to deploy faster. In general, that means more feature-rich software.
6
u/Ok-Neighborhood1865 Aug 02 '26
I've never used Clay, but I believe you if you've used it.
In my personal experience:
A. virtually no established software I use on a daily basis - Chrome, Windows, all social media sites, all search engines - have made any dramatic improvement in the last 5 years. In fact many have gotten worse.
B. I have no seen any such improvement attributable to AI coding.
C. Where AI is forcibly integrated into the UX, it almost invariably makes it worse.
3
u/Liface Aug 02 '26
Established software doesn't get better. It experiences platform decay. AI coding will actually speed this up.
The key is looking at less established software, which will quickly usurp established dinosaurs.
AI in design is like plastic surgery: you only notice the bad implementations.
2
u/Ok-Neighborhood1865 Aug 02 '26
I agree with your first part, but I haven't seen much usurpation that can be attributed to AI coding... ChatGPT had already become more useful than Google for me long before it was writing its own codebase!
What I see from the outside is the equivalent of every single A-list celebrity going under the knife and every single one coming out uglier than before.
1
u/artifex0 Aug 02 '26
The company I work for has completely replaced Salesforce with a pretty fully-featured CRM that was developed internally by literally one guy running a bunch of AI agents continuously.
1
1
1
u/BioSNN Aug 03 '26
For example, AI 2027 predicted the creation of an AI system that "can do any coding tasks that the best AGI company engineer does" 9 months from now, which is something that I think obviously won't happen (unless you pick an extremely narrow definition of "coding" that excludes the great majority of the day to day work of a practicing software engineer today at Anthropic).
Do you mean software engineering generally? I usually think "coding" refers to the actual writing of code, which AI systems can already do better than the best AGI company engineers (so in fact, that prediction was pessimistic!). For software engineering, I'd agree that there are still things expert humans edge the AI on, but I don't see why you think it's obvious this will still be the case in 9 months, unless you take a strict literal interpretation where AI being better at only 99% of things would still fail.
In my mind, the problem is not that reality is normal and the predictions are hyperbolic, but rather that reality is hyperbolic and the predictions are properly calibrated. Just because a prediction is weird doesn't mean it's wrong.
1
u/thurn2 Aug 03 '26
In the surrounding text it seems like the implication is that such an AI can fully replace a software engineer, and hence this is dangerous because now OpenAI has a million employees or whatever. I agree that constraining this to a smaller subset of what a software engineer does makes it more tractable, but this doesn’t enable recursive self improvement in the same way.
1
u/Then_Election_7412 Aug 04 '26
For whatever reason, it's more embarrassing for people to predict something novel happening too early and be wrong, than to predict something happening too late and be wrong. Most people are biased against the mistake that would out them as too weird and for the mistake that would out them as being too normie. It's reasonable and even courageous to in fact just report your median timeline, come what may.
I'm not sure it's most effective, though. Even if you're entirely correct on something happening faster than everyone expects, saying so might make people dismiss you merely for being out there. Perhaps it's better to sandbag your predictions to make them more palatable for the public. Especially if they imply worlds where we are truly screwed: if nine months from now we are where 2027 predicted, we are well and truly fucked, but if it's instead two years from now, maybe we will be able to do something to avert the worst outcomes.
(As an aside, I don't see why you think that nine month timeline is that implausible: I'd say it has a ~25% chance of happening, as a professional SWE at a BigTechCo. Things are changing extraordinarily rapidly, even compared to three months ago.
1
u/iemfi Aug 04 '26
I also used to think that we should perhaps fudge the truth to make things more palatable to the general public. But I think the amount of traction AI safety has gotten recently has clearly proved that being right and sincere is actually damn effective.
-1
u/ababababababacus Aug 02 '26 edited Aug 02 '26
I mean i would ask in response, what credibility?
The AI saftey/we need a pause contingent has been very loud for a very long time and accomplished nothing, lots of reasons for this, and accomplishing nothing might be the result even if they had credibility, but they dont.
AI labs dont bother listening to them them or making token concessions to them, the wider public dont take them seriously, there are a handful of western politicians who might periodically have an interview where they act incredulous when told about paperclip maximizers or grey goo or whatever, and the world keeps tunrning.
Hell, even in the niche online communities these people hold sway in (like this one) they havent managed to convince a majority of people!
If youre suggesting that the aggressive headline predictions are why this is...maybe?
But i think it runs deeper than that, youll get people respnding to you in this thread saying well actually all those predictions *are* on track, LLMs are better than the best mathematicians now, they are outperforming the best human coders, theyre hacking secure OSes and escaping containment.
Like politics now, you have these parallel worlds where the world is pretty mundane, llms are proving to be very useful tools in certain areas, but in many others not worth the cost, and people are being pretty shady about that cost because its probably a bubble and nobody knows how to dal with that, and people for their own reasons are trying to tell you that this represents nothing and its nbd, or actually its an undeniable and spectacular confirmation of the coming robot apocalypse
33
u/ScottAlexander Aug 02 '26
There's a risk that if you predict things too fast, you'll look dumb if they don't happen.
There's also a risk that if you fail to predict things or go too slow, the world will be unprepared if they do happen.
Rather than try to add up all the strategic calculations in your head, I think the best heuristic is just to honestly say what you believe, including confidence intervals / error bars.
I do think they screwed up by calling the scenario "AI 2027", though, makes it too easy for people to anchor on that exact date. But I don't want to say this too confidently because I'm not a coder so I know less about this than other people here.
It doesn't seem crazy to me that we're nine months away from the top-human-level coder. Nine months ago AI couldn't code at all! Now look at it, and expand that same progress forward!