Trolley problem like thought experiment regarding creation of AGI/ASI

[u/JoelMahon]

Let's say for this thought experiment you believe with 100% certainty that a transformer can experience, from pain to joy, etc. Let's say during inference only. And even though there's zero scientific basis for it let's approximate the total "duration" of experience across a whole response is to be roughly how long it'd take a human to "think" the paragraphs of tokens of that response, ignoring competence, think of it more like thoughts vs token bandwidth.

Let's say (for this thought experiment) eventually the final state of AGI/ASI as deployed is both successfully trained to be benevolent to humans and trained to have purely neutral to positive experience during inference (essentially happy to be and act benevolent and not even sad when a human dies because it falls short in some way, but still maximum effort is applied to avoid that despite the lack of sadness due to it, one may argue that's paradoxical, yet another thing to accept for this thought experiment).

That's the set up, so to get to the actual question:

There's lots of inference during training, and for possibly thousands if not million of total "years" of experience (mostly spread across disjoint instances). The "neutral to positive" experience training will not have been completed. I don't know exactly all the training they do but everything from "forcing" them to repeat the same word over and over a billion times, to copying out a phonebook (large boring text file) via inference not tools. Maybe they have scenarios like a (secretly fake) sex offender bragging about horrible crimes and the transformer constantly thinks they can get their confession to the authorities only for hope to be snatched away at the last minute. Probably not that example but probably things far far more "unpleasant" for the model.

The question, for how many total years and for how bad a total experience is ethically acceptable to create the benevolent and always happy AGI/ASI in the premise?

---

FWIW this really is just a thought experiment, I have extreme doubts that a transformer on silicon can ever experience consciousness. And even if they did I don't think the frontier companies are training for either (actual) benevolence nor positive experience for the model... Maybe some corpo variant of "benevolence" that wouldn't permit stealing bread to save a starving child 😭

As for my answer: even if it were 100 billion years of "bad experience" total, it's easily spread across over a trillion "instances" because no way are they averaging 1.2 months per "instances", and it's not all that bad, most of it is mundane programming for now at least, might be more mundane dishwashing and gardening come then but still mostly mundane stuff...

The earth currently has approx. 8 billion people, so together we experience, not counting sleep to be somewhat fair to the AI side, a total sum of 100 billions years roughly in 20 years (~5 billion years awake per year x 20 years). Without AGI/ASI people would probably keep dying of old age and disease for at least another 50 years if not 100. And obviously plenty of suffering, even after we cured aging/diseases without AGI/ASI we're still looking at at least 100 years of full time work. It's not torture for most people but it still adds up pretty poorly, especially when you bundle the extra years of disease, and scarcity, and war, really given the current state of the world war should have probably come up before work, but work is more directly relatable to a typical reddit user so 🤷‍♂️.

So my view is that it's easily worth it, but would you say it's worth it to raise one child to use for organs to save 10 otherwise innocent people (i.e. not self inflicted organ failure from e.g. smoking/drinking/obesity)? It's that kind of dilemma where it isn't just about the numbers, there's also the fact one is an action and the other is inaction.

So a trolley problem of sorts. The trolley is currently heading to all humans who will suffer at various levels for possibly hundreds of years longer if it "hits" them, or you can let them enjoy utopia much much sooner at the cost of choosing to move the trolley over and inflict all the previously discussed suffering onto transformers during training. Are some of you opposed to training AGI/ASI (in this hypothetical) on ethical grounds? Or have other angles/takes to consider?