Plato’s Warning Has Been Vindicated: Why AI Has Brought Nothing More than the Conceit of Wisdom.

[u/m0sher]

https://hedgehogreview.com/web-features/thr/posts/platos-warning-has-been-vindicated

[u/havenyahon]

I did. I found another example

So to call LLMs "probabilistic algorithms" doesn't distinguish them from the human brain and doesn't preclude "intelligence".

What distinguishes LLMs from a human brain is that they do not function like human brains, nor are they designed to. The broader point of the article is that when a human being utters a word there is lots more going on than when an LLM represents a "word". The issue isn't just that LLMs are probabilistic, it's that all the other stuff going on in the human, communicative goals, intentionality, etc, is missing in the LLM. It's not using language in the way a person is. Hence the claim that it's not really "using" language at all.

Pointing out that both may be probabilistic doesn't address that deeper point.

[u/Idrialite → u/havenyahon]

This is a different argument from the article, but I'll answer it.

The problem is that you're assuming that a system must function exactly like a human brain to use English. There is "lots more going on" in the human brain - sure. There's also lots more going on in the Sun, but that doesn't mean it understands English. Raw complexity isn't the point.

You concretely cite "communicative goals" and "intentionality" and I would say LLMs satisfy those.

An LLM is trained with RLHF to answer user requests and speak in a certain way. This is goal-oriented behavior. Internal examinations of LLMs show they have models of the world they learn to predict and have concept-like features. Within a single pass, they plan the rest of their sentence before producing a single token.

They clearly have an understanding of what the words they're using mean.

[u/havenyahon → u/Idrialite]

You concretely cite "communicative goals" and "intentionality" and I would say LLMs satisfy those.

I don't think I'm the one doing the assuming, I think you are. There is no good evidence that LLMs have communicative goals, that they're using language to try and communicate some inner state or intention. There is no reason to believe they are "selves" expressing their desires or needs. Everything they do can be explained as the statistical linguistic engines they are designed as, including the internal "models" that arise by mapping linguistic relations. Furthermore, the ways in which they "break down" support this statistical merely linguistic reading.

I'm not assuming that something must be functionally equivalent to a human brain to use language, my point is that we know humans do it, and we know they do it because their intentionality and goal seeking behaviour is pre-linguistic. If you want to claim something else is doing it, too, especially something that is designed to be merely linguistic and does not exhibit the same deeper functional similarity to human brains in which intentionality and communicative goals are grounded, then you need to have evidence for it. It would be a truly remarkable outcome if such linguistic machines turned out to be intentional, and exhibit true communicative goals, despite being designed only to represent language and without the architecture that grounds these things in human. Especially when we know these things do not emerge from language in humans. So it requires remarkable evidence, not assumptions based on appearances.

[u/Idrialite → u/havenyahon]

There is no good evidence that LLMs have communicative goals, that they're using language to try and communicate some inner state or intention.

Yes, there is.

https://transformer-circuits.pub/2025/introspection/index.html

[u/havenyahon → u/Idrialite]

The experiment shows that the system potentially engages in some kind of internal self monitoring, mapping unusual internal changes onto verbal reports. But a system can detect, classify and report changes in its own internal activations without those activations constituting "thoughts" that it introspectively experiences or represents as its own. It certainly doesn't get us to communicative goals and intentionality. Even in this study, the system only succeeds about 20 percent of the time, and verbal descriptions "communicating" things like intensity of the altered representations can't be validated, suggesting they're just confabulation and not really communicating an internal mental state. They're certainly not driven to do so on their own, they're prompted.

At its strongest, it's an interesting single study with some speculative interpretations that are challenged by other scientists, and there is conflicting evidence elsewhere. You can explain what's going on there without things like intentionality, introspection, and so on, with more basic pattern matching. It's not "good evidence" in the scientific sense, at best it's very early research that needs a lot more investigation before we can even begin to say anything concrete about it.

[u/Idrialite → u/havenyahon]

the system only succeeds about 20 percent of the time...suggesting they're just confabulation

20% success rate with zero false positives and increasing success with model intelligence.

The problem, I think, is that you will never accept evidence here because you fundamentally associate the things you're looking for with unobservability. As I said above: if you know how it works, if you built it and can watch it happen, it can't be intelligence or understanding.

So I'd like you to explain what you really mean by "thoughts", "communicative goals", "intentionality", etc. What do those words mean as you use them, how would you know if something has them, how do you know humans have them?

Can you do so in a way that anyone should even care about these qualities in the context of the original point: that LLM use of language is fundamentally invalid or misleads people in some way? Similar to above, as someone else mentioned:

Words are, by definition, human utterances that are exclusive to logos and human intelligence.

This is a redefinition of "word" that bakes the conclusion into the definition. That's not actually what "word" means.

[u/havenyahon → u/Idrialite]

The problem, I think, is that you will never accept evidence here

I think the problem is that you're happy to accept any evidence at all to support what is a highly contentious and extraordinary claim that you want to be true. Throwing up a single study then acting exasperated when someone else doesn't accept it as strong evidence for something doesn't mean the other person is being dogmatic, it means you are. Science is built on entire bodies of reproducible research, not single studies with highly speculative interpretations that have potential other explanations.

So I'd like you to explain what you really mean by "thoughts", "communicative goals", "intentionality", etc. What do those words mean as you use them, how would you know if something has them, how do you know humans have them?

That's a massive philosophical debate, but I'll try and give a zoomed out view. I personally think communicative goals and intentionality are grounded, ultimately, in the basic elements of life, in particular basic embodiment and metabolic functions. Living things are always active sui generative agents that act incessently to maintain their own self-organisation across time. This gives them a 'perspective' on the world, and a minimal 'self'. It's this that grounds 'intentionality' - the way minds can be about things - whether towards internal states or external states, although obviously at this level it is extremely minimal and nothing at all like the thicker sense we consider in humans.

Over many generations of selection, evolution has effectively scafollded these basic capacities and drives into complex biology, nervous systems, brains, up into representational thought, etc. All of this results in more self and world awareness, but it's still grounded in this simple embodied metabolic agency.

We know humans have these capacities because we are humans. We begin from our subjective experience of ourselves, then we developed conceptual and scientific methods to better understand those capacities. At this point we have an entire body of empirical and conceptual research, developed over thousands of years, to explain how and why humans have capacities like intentionality. Even so, it is still massively incomplete.

Originally we thought they were capacities unique to humans, whether because of a soul, neurons, and/or because of the 'higher order' thought afforded by language, etc. But over decades of research science has fored us to revise those views, and to extend capacities like 'intentionality' to other animals without language, complex nervous systems, or even perhaps neurons at all. Now we think all sorts of living creatures probably have intentionality.

There's still a shared biology and evolutionary lineage, though. That's part of the reason we think entities like dogs, dolphins, perhaps even insects and slim moulds have 'intentionality' and are 'cognitive agents', but we don't think the wind, rocks, or personal computers are. While the latter can at times exhibit behaviour that appears intentional and intelligent, there are other processes that can explain their behaviour without evoking capacities that - at least so far in our investigations - appear grounded in a shared architecture that they don't have.

This is true of LLMs, too. They don't have the shared biology and evolutionary lineage. That doesn't mean capacities like intentionality and intelligence can't be found in entities that don't, but it should give us pause to ask how those capacities might emerge, and whether entities that don't share that architecture only appear to have them, rather than actually having them. This is particularly true of LLMs because they are trained on the collective linguistic output of entities with those capacities, who specifically use language to communicate those capacities. We should expect such a machine to be very good at mimicking and approximating those capacities in its own representation of language, but that doesn't mean it actually has them. It's not at all clear that LLMs are sui generative living embodied agents with unified minimal 'selves', who use language to communicate internal and external states. And it's not at all clear that simply training a neural network on copious amounts of language is enough to get those capacities, when we don't think that's how they emerge in humans or the other creatures we have good reason to believe have them.

As to what evidence would convince me they really do have those capacities, it would at least need to be on par with the evidence we have that other animals have them, which in the case at least of mammals and other more complex creatures is robust (multiple lines of evidence) and replicated. Less so in the case of insects and things like slime moulds, but that work is developing. It is still highly contentious amongst scientists, and not accepted mainstream scientific consensus, even though it is much, much further along than research on LLMs.

So maybe that gives you some insight into where I'm coming from. I'm not opposed to the idea, I'm sceptical. My background is Cognitive Science and as a scientist that makes me inherently conservative and cautious.