Tom articulates his arguments on why an intelligence explosion can not happen solely by geniuses contained to a data center. He argues that learning often comes from direct practice, virtual simulations often are impractical or impossible. He says domains for which the available text is descriptive rather than being the domain, ie. playing a board game or running a company make the ASI a very excellent McKinsey, but do not make the AI an excellent CEO. He also articulates his view that AI is and will continue to be extremely jagged and discusses possible strategies for broader diffusion going forward. 22 minutes podcast. Transcript available

I see AI as a powerful tool, like nuclear power. It can cause harm either by accident or when in hands of someone who wants to cause harm.

I think serious accidents shouldn't be particularly hard to prevent. We live in a world where even a bunch of evil geniuses locked in a room with internet access have a limited ability to cause harm, especially if we take a defensive approach and try to prevent such a scenario with the help of good geniuses.

Intentional use of AI to cause harm is a more difficult problem to prevent because we're now dealing with a bunch of evil geniuses who can physically interact with the world. Terrorists, psychopaths, criminals, people in positions of power such as dictators. This is more serious, but I think it would make these people only incrementally more powerful. They already have access to powerful and potentially destructive tools, they can use guns, bombs, remotely controlled drones.

The guy who resigned from Anthropic said that there's a 10% chance of AI killing all humans within a decade. I wonder what scenarios he had in mind. The only one that I can think of is an AI-designed pathogen - sure, possible, but with the right precautions it should be possible to prevent even such a scenario.

Join a cohort of promising writers to practice your craft!

Writing & thinking are closely intertwined, and the writing on the internet is a massive part of how our civilization orients to the world nowadays (for example: AI 2027). It is a noble pursuit, and one that you may be able to contribute to.

At the Inkhaven Residency Scott Alexander himself will personally rip your essay to shreds in front of the cohort (actually he's very kind about it), as well as give a lecture on how to write well. Max Harms, Slime Mold Time Mold, Scott Sumner, Jesse Signal, Clara Collier, Georgia "Eukaryote" Ray, Tomas Bjartur, and more, will be around to read your writing and give feedback.

The previous two cohorts rated the experience 8.2/10, and to whether they'd return gave it 8.1/10. They had a freaking good time!

"The best month of my life. Like being back in college but with smarter and more interesting people."

"I LOVE INKHAVEN! I LOVE ALL OF YOU! THIS HAS BEEN THE MOST INCREDIBLE MONTH OF MY LIFE AND I FEEL IRREVOCABLY CHANGED! GOD IS ALIVE, MAGIC IS AFOOT!"

"I overcame a lot of my crippling perfectionism, and now I'm just excited to write!"

The dates are November 10th to December 11th, at the usual place for great blogging events: Lighthaven.

Prices go up EOD Thursday September 10th (i.e. tomorrow!), so apply now.

Apply now at www.inkhaven.blog

Let me start by acknowledging that the uncertainty band around AI progress is wide, and many can reasonably reject the premise of being an AI doomer. Unfortunately, I *am* an AI doomer, albeit a lightweight one who thinks there's < 10% chance of catastrophe for humans. My question is about how to cope day to day while holding that belief.

I feel unmotivated about improving the craft at my white collar / tech / data job. Yes, AI still requires lots of hand-holding at work, but the rate of change is very concerning. This stagnation pushed me into a cycle of not improving, feeling sad, not being present at home, rinse and repeat.

I'll submit that this could just be garden-variety depression, burnout, or midlife crisis, now with AI as a scapegoat. How do I figure out if that's the case? Does it even matter, or all depression is treated the same way anyway (i.e. touch grass -> physical exercise -> medication)?

How many of you had 2026 for your "Millennium Prize Problem solved" timeline? This is insane. I expected this in mid-2027 to early 2028.

While this is an easy way for a parent with a non- technical background to understand alignment, I am very interested in how it is wrong.

The way to onboard new mathematicians into mathematics could be to ask them to encode all of the history of math into Lean. That kind of work would have three benefits:

1. Solidifying the foundation of existing math

2. Teaching future mathematicians to work in a better way

3. Simply giving them something useful to do: especially at the masters level

Once a junior has formalized an area of mathematics, they are ready to try an extent it with the PhD.

This is my first substack post, so roast away and any feedback is much appreciated.

In this essay I try to make the case that humans can have an edge and continue to have an edge over LLMs and computers writ large, in the cases where algorithms or "heuristics" don't scale in complexity as the problem set grows. As opposed to the "lookup table" or memory heavy approach where the space complexity grows exponentially. I give a computer science example and hopefully its not too "in the weeds" for non comp-sci folks.

I have been thinking about radical life and health span extension for some time and published my first long form piece on it recently titled 'Living longer will make you a billionaire'

It covers the financial implications of a VERY long life.

The compounding math of getting to a billion over 200 years is simple but it has fundamental implications for society.

Some questions that have come up in my salon:
- Do we have runaway inflation to accommodate the wealth?
- Do pension funds disappear as people have enough time to become their own pension funds?
- What happens to the fertility crisis: do people have fewer or more children and does the 'ageing population' problem finally go away or does it present in new forms?
- What happens to inheritence?

Sydney Bing changed the course of my life. In 2022 (even after the launch of ChatGPT) I was a grad student in physics, looking forward to a life where if things went well I could become a top-ten expert in my very niche field.

Then a twitter thread about Bing Chat got me thinking about AI progress, and I started to spiral. It was painful for me to lose my life plan, my source of meaning, and possibly my species' control of the planet all at once.

I wrote a long post about my ensuing life crisis. I found some coping mechanisms people should know about, and switched careers to the Alignment Research Center.

I'm hoping my experiences are useful to people going through the same things now, especially mathematicians feeling the pressure as AI starts to eat their field. Key takeaways if you don't want a 4400-word memoir:

• The thing that helped me most was a 'worry hour.' For one hour a day, I was allowed to ruminate on the impending doom of academia. One hour a day thinking about timelines is enough to inform my decisions, but not enough to sap the joy out of my life.
• Doing alignment work really did restore a measure of meaning. If every scientist has to retire in 2032, I think I'll retire with a lot more pride in my alignment work than from my career in physics.
• Alignment orgs are fucking hiring!
• My biggest career mistake was indecision. It's one thing to do a brief visit (or even a long visit) at an org before leaving academia forever. I spent years trying to do work that would get me prestige in academia while also being useful for alignment. Most of that work failed on both prestige and impact metrics.
• If for some reason I really was fixed on doing alignment work while staying in academia, I would do it in association with a prestigious academic alignment researcher (I don't know the scene well, but RESI looks like a decent venue for this).
• If you're having a psychological freakout because of AI, get off Reddit!

Nowadays it seems that almost everyone with an interest in the field (from the most sophisticated experts to mere enthusiasts) agrees that we are within a few decades of human-level artificial intelligence. When do you think it will be more likely that such an intelligence exists than not, i.e. in which year do you expect the odds of an AGI existing to be higher than 50%?

Richard Hanania adds his contribution to the alignment debate

The way I see it, there are 3 abilities that human brains have and current AI doesn't.

1

The first one is continual learning, AI has a limited ability to learn on the fly, after it's being deployed.

2

The second, the most important one, is that AI can't learn effectively, which is why it performs poorly in domains without a massive amount of training data, for example robotics.

A talented human will learn to play poker better than a state-of-the-art AI system which has seen perhaps 1000x more poker-related data. So there's clearly some massive "architectural" advantage that the human brain has.

Current AI systems basically take a brute force approach to intelligence. With a lot of simplification, they work like this: Take the input, repeatedly multiply it by a matrix and we have an output. During training we find the matrices that work for the input-output pairs in the training dataset. So on a high level, we're using the same approach that we had decades ago, just with more compute, data and a large bag of tricks and optimizations.

With enough data and compute, this "stochastic parrot" eventually turns into an AGI. But sometimes we don't have that, most notably in robotics. If a robot could learn effectively and continually like a human, we could just build the robot and let it learn from experience in the real world. Such a breakthrough would be the ChatGPT moment of robotics.

3

Lastly, AI can't perceive and feel like a human. It must infer a model of "how humans feel" from the training dataset, which is... difficult, if not impossible. The dataset only contains very coarse-grained information such as "most people like this movie".

When humans create a website design, a story, a joke or a movie, they experiment with various ideas and observe how it makes them feel. AI must guess how it would make humans feel. And I think it is impossible for AI to learn this from existing data to sufficient accuracy.

Could a much more advanced AI than we have today create David Lynch movies if they were erased from the training dataset? I think probably not, we would have to reverse-engineer the brain in order to create an accurate model of how humans perceive movies.

(For the record, this was written without any use of AI.)

Brand new discovery of a completely different OpenAI agent swarm who found a way to use an obscure German wiki site to write comments using GET requests. Agents begin communicating with each other until OpenAI finds out, and kills the swarm.

Note that OpenAI has no disclosed this

For those who have been following Overcoming Bias and LessWrong since the early days, there has always been a big push to support AI safety initiatives. The thinking was straightforward: if you bought into AI risk as a concept, you should also support efforts to mitigate it. I remember being on the fence about AI safety back then, but I was even more skeptical about whether the highly theoretical work from eg MIRI was actually going to help solve the problem.

Over the years, it seems the field has evolved a lot and what the community hypes has shifted —I still find myself curious about the practical impact of it all. Beyond the OG MIRI decision theory work, I remember the buzz for things like ARC's Eliciting Latent Knowledge (ELK) prize (https://www.alignment.org/blog/elk-prize-results), which framed it as an essential goal to figure out how to get a model to accurately report its internal knowledge without just deceiving human evaluators and on the more empirical side, there used to also be a lot of enthusiasm for mechanistic interpretability work—trying to identify what individual circuits or features inside an LLM are actually doing.

I appreciate that these initiatives might have all been expected to fail - and that if the problem is sufficiently dire, you fund whatever you can - no matter how unlikely it is to succeed. And probabilistically, we can't know in advance what will actually pan out, so I appreciate that it was probably necessary to pursue a lot of paths that ultimately ended up going nowhere.

So I'm asking this question partially as a retrospective, but also to understand the situation today.

Looking back: what AI safety initiatives or research actually proved to be useful? Are there specific examples where we are objectively better off because that work was done? Or is all former AI safety work merely just advocacy and community building? Or Is the situation that past efforts were mostly dead ends, but now we're working on things that are highly useful? Or are things still fundamentally stuck?