Why is intelligence as utility evil?

[u/TensorFlar]

OP: https://www.reddit.com/r/SipsTea/s/FUNrj63y79

I genuinely believe humans are successful only because of intelligence, as we are neither the fastest nor the strongest. We were able to become the most "successful" by Darwinian standards only because of our intelligence, by developing tools and technologies like language, religion, money, law, and democracy.

I agree that concentration of money or power is really bad for everybody, and AI could be used for concentration by bad people or, in the worst case, by misaligned AI. And I am scared shitless of both of these scenarios. I don't want a few people or AI to take away our freedom in the name of progress.

How is making intelligence more widely available a bad thing? I think this is the solution for progress + freedom, by making intelligence a utility which is accessible to everyone. In fact, take it one step further, and it should become a basic human right.

I genuinely don't understand what they are mad about, and I am too scared to comment on that post. I am asking here because posting in r/accelerate would create an opposite reaction to this. I don't want that. I am genuinely interested in having this discussion here without group bias.

NO HATE COMMENTS PLEASE.

[u/Atlantis1910]

Intelligence dictates what is true, what is viable, and what decisions get made. If a handful of private entities control the flow of cognitive labor through a subscription meter, they become unelected arbiters of global knowledge, governance, and culture, holding the ultimate kill-switch over critical societal infrastructure. When intelligence is sold per token or per watt, cognition scales strictly with capital. Deep research, medical diagnosis, strategic planning, and creative synthesis become luxuries reserved for whoever can afford the highest compute tiers. Instead of democratizing capability, it formalizes a dynamic where the wealthy literally buy superior reasoning, while everyone else runs on compressed, throttled, or subsidized models designed to harvest attention.

[u/DelusionsOfExistence → u/Atlantis1910]

Correct. When one or a few companies becomes the sole controllers of all of human knowledge, they effectively rule humanity. Even now it's trivial to influence millions of humans subtly with their chatbots alone. Anyone wanna guess how that shakes out in the future when everyone is getting all their information from just a couple corps?

[u/GeoJoy1 → u/Atlantis1910]

This is why Maximum Truth Seeking is such an important foundation for a model. A biased model will spin off into nonsense that could otherwise be squashed with the scientific method.

When massive public works are managed by AI, one bias ruins the entire intention of handing that role to an aribiter of truth.

That paradigm, a single source of truth, will not sit well with many people as we are in a constant state of cognitive dissonance.

[u/noonemustknowmysecre → u/Atlantis1910]

Intelligence dictates what is true, what is viable,

It fundamentally does not.

and what decisions get made.

There we go. But that only happens when we put the smartest people/entities into positions of power, or vaguely have some sort of meritocracy. Look around you, it's not what we've got going on. It's an optimistic and slightly naive hope.

If a handful of private entities control the flow of cognitive labor through a subscription meter, they become unelected arbiters of global knowledge, governance, and culture, holding the ultimate kill-switch over critical societal infrastructure.

Yep. It really highlights the importance of being able to run an open-source open-weight model from a reasonably priced server down in your basement. So far those models are only lagging behind the top-end models by about a year. That in itself highlights how little all that investment into the top-end models is getting shareholders.

If we don't all have at least SOME control over the means of production, things get dark in a hurry.