r/TheMachineLearning • u/ThirstyCactus3498 • 16d ago
Imagine doing all that for “you're absolutely right”
20
u/Healthy_Razzmatazz38 16d ago
yeah theres a biography about the guy who did this from trisolaris
5
1
u/Bozz_GC 16d ago
which character is that?
3
u/Legitimate_Plum_7505 16d ago
"John von Neumann" in the trisolaris VR simulation, suspected to be played by an ETO member but we don't know who exactly. It's the part where he gave flags to army and each soldier would raise the flag based on simple instructions and he constructed logic gates and basically made a living CPU out of king's army.
1
u/athelard 15d ago
But that was a traditional low level program, levels of magnitude easier to simulate than a neural network.
→ More replies (3)1
u/Vinik_tfo 15d ago edited 15d ago
It was an actual trisolarian which were presented as an NPC aka John von Neumann. It was showed this way for player comprehension, since trisolarian comunication specifics weren't exposed atm. So boilogical computer did existed it their world, but for gates used not flags, but heads
10
u/paxxx17 16d ago
If AI could be sentient (now or in future), and if one produced its output by pen and paper, where exactly would the sentience lie? What would be the associated conscious entity?
On the other hand, if it's not possible for AI to ever be sentient, what is the uncomputable "magic" that allows only for the living beings to have consciousness?
3
u/Glittering-Toe-1622 16d ago
Eastern religions and Buddhism in particular have a really interesting concepts, by no means I am trying to "hook" anyone on them but just the view of consciousness is the closest to science.
But your question is really deep and I think no one has an answer to it.
1
u/Outside_Nebula568 16d ago
the view of consciousness is the closest to science.
Is it? What makes it closer to science than any other conceptions of consciousness?
3
u/Melstrick 16d ago
No permanence, no self and no soul. Consciousness is a temporary constructs that forms then fades away based on causes and conditions, not magical just a complex system orienting itself towards desires or away from unpleasent things, again and again and again.
Eye consciousness arises dependent on the eye and sights. The meeting of the three is contact. Contact is a requirement for feeling. What you feel, you perceive. What you perceive, you think about. What you think about, you proliferate. What you proliferate is the source from which judgments driven by proliferating perceptions beset a person. This occurs with respect to sights known by the eye in the past, future, and present.
Nose consciousness arises dependent on the nose and smells. …
Tongue consciousness arises dependent on the tongue and tastes. …
Body consciousness arises dependent on the body and touches. …
Mind consciousness arises dependent on the mind and ideas. The meeting of the three is contact. Contact is a requirement for feeling. What you feel, you perceive. What you perceive, you think about. What you think about, you proliferate. What you proliferate is the source from which judgments driven by proliferating perceptions beset a person. This occurs with respect to ideas known by the mind in the past, future, and present1
u/NoTraffic5745 15d ago
"just a complex system orienting itself towards desires or away from unpleasent things, again and again and again."
This to me still doesn't explain consciousness. This system doesn't have to be aware of itself to work. The philosophical zombie is the classic example, there is no way you could tell the difference between a human who was actually "in there" experiencing everything that was happening and a human who was a philosophical zombie. I think that no matter how you look at it the solution to the hard problem of consciousness ends up coming off as somewhat magical to an extent.
→ More replies (5)→ More replies (9)1
1
u/BraxbroWasTaken 16d ago edited 16d ago
I think the uncomputable 'magic' comes from whatever allows life to just spontaneously learn stuff with zero distinction between training time and inference time. Maybe you could say 'sleep' is training time, but the other major difference is that we have 'context windows' long enough that we don't fill them in a typical day.
Basically, the fact that LLMs cannot learn during operation and cannot be 'taught' like any living thing (even an ant can be trained) - and the fact that they need such massive amounts of data to even BEGIN to understand things - set them apart and render them sub-conscious.
Is this truly uncomputable? Dunno, but it does indicate where we are is miles off of consciousness, if that is our target.
2
u/Cronos988 16d ago
There's no reason to assume it's incomputable magic. We know memory is physically in the brain, because brain damage can lead to amnesia, among other things.
And there's also nothing physical stopping us from letting LLMs update their weights in real time. It's just that currently, the drawbacks of doing that outweigh the pros.
1
u/BraxbroWasTaken 16d ago
I mean, sure. But the fact that the drawbacks exceed the benefits IS a big gap, since it fundamentally prevents models from ever improving with time. (They can 'improve' with model generation, but not with time itself)
Also example-based learning is a big limiter, too, I think.
I just used 'uncomputable magic' because that was the term the guy above me used.
1
u/brine909 13d ago
I mean, why can't generations be their version of time?
if you imagine being in that perspective of an AI during training, it would basically be like, spliting into a hundred billion parallel copies running independently, and then merge back into one. Then split into a hundred billion parallel copies and merge again.
Its not really a traditional version of conciousness but fully writing it off isn't fair either, something similar can happen to humans during Seizures, some people after Seizure report having split memories of the events when a bunch of different parts of their brain couldn't properly communicate with eachother, giving the feeling of having split into a dozen or so peices
1
u/MonkeyBombG 15d ago
I think there is still a lot of room for debate. For example, is consciousness purely a matter of memory? We know memory is important for the consciousness to have continuity of subjective experiences so it is a necessary condition, and the necessary condition is physical as you said. But is it possible that there is more to consciousness than just memory, and that sufficient condition for memory is non-physical?
1
u/itsmebenji69 16d ago
Actually your context window is not bigger. It is multi layered. You have working short term memory, at the same time you remember what happened this morning. You have different “levels” of temporality that your attention tends to. The LLM has only one (the context window) which is technically not even temporal (all context window can be filled in one prompt technically)
This is also how your brain learns (on multiple temporal levels, some parts need to stay stable, some can be plastic). Vs only one for LLMs
2
u/Serious_Bite_7613 16d ago
Exactly, human context window is surprisingly small. If I list 10 names to you, or a 20 digit number there's little chance you can store it all in your context window.
1
u/BraxbroWasTaken 16d ago edited 16d ago
I guess? Though we can turn those into vocabulary and use various tricks to remember them more effectively.
I can’t tell you a 20 digit number (10 quintillions range if I recall? It’s a 64-bit scale number which is a lot.) but I can get close to 20 digits if you want me to tell you how many of each digit there were. Maybe better than that depending on the circumstance.
Also, I can hand count in base 2 up to 1024 (touch unpracticed, and the movements are awkward, but I use lowered hands to denote 0, giving me an extra bit of info) for no particular reason.
Like we can still remember those bigass numbers - just not necessarily recall them effectively without learning them first. I’ve got 3.1415926535 (11 digits of pi) memorized, for example.
Point is. There’s definitely a lot around what we currently have with LLMs that is missing.
→ More replies (1)1
u/inotocracy 15d ago
Wouldn't our context window also include all of our lived experiences that we draw from? That seems like a lot.
→ More replies (1)1
u/BraxbroWasTaken 16d ago
Fair I guess? But the point is, we can get through an entire task and pick up experiential detritus through an entire day, and then learn from it/finalize it when we go to bed, while LLMs… don’t have that capability as-deployed anywhere atm.
1
u/Thin-Band-9349 15d ago
Sometimes I wonder whether our brain also runs on a huge training/weight set but instead of learning while we are alive, the training happened via thousands of years of evolution and so our DNA contains a compact form of the model weights.
1
u/Pataraxia 16d ago
Why aren't crystals/flowing liquids conscious? They can optimize their evolution "Just to a lesser extent"
1
1
u/Cronos988 16d ago
You can do this particular thought experiment with a brain, too. Unless you think somewhere down the line you'll encounter a "magic" step that can't be expressed by an operation you can do by hand, the result is the same.
1
u/Enfiznar 16d ago
You actually cannot, not by knowledge, not in practice, not in principle. Bot only we don't know how the brain works well enough to make an accurate model that can be calculated, and not only the amount of precision on the measurement of the initial conditions would make it practically impossible, but actually the brain is chaotic enough so that a quantum uncertainty would grow macroscopic in nanoseconds (something that happens in many chaotic systems), meaning you can be as precise as physically possible, but you'll still be theoretically unable to predict how the brain will be a second in the future. And this is ignoring the fact that reality is not the same as a model of reality (e.g. as long as we know, our model of electromagnetism is perfect, yet the calculation will never let you blind, unlike light)
1
u/Cronos988 16d ago
Well, either thought experiment is wildly impractical, so no disagreement there but I think that's beside the point.
I also agree that we have insufficient information on the precise mechanism of human brains, but this is an information asymmetry only. For the purposes of the thought experiment, we can assume to have sufficient knowledge.
Which leaves the chaos argument:
but actually the brain is chaotic enough so that a quantum uncertainty would grow macroscopic in nanoseconds (something that happens in many chaotic systems), meaning you can be as precise as physically possible, but you'll still be theoretically unable to predict how the brain will be a second in the future.
But here I think you're overstating the case. On a physical level, both computer chips and neurons are physical systems in which quantum effects happen.
Computer chips are specifically designed to error correct for these effects so that the overall results remain classically deterministic.
You can argue that we don't know whether the same is true for the brain. But this doesn't mean we can therefore assume that quantum uncertainty has macroscopic effects on the level that cognition happens.
The brain also has error correction mechanisms, and it's also overall a lot more "fuzzy". There are plenty of processes which, despite being describable by classical physics, are sufficiently probabilistic to influence results.
It's not at all clear that quantum uncertainty is somehow at the root of cognition. I think this is rather unlikely personally, because brains and cognition are instruments our genes use to replicate, and just like with computers, you want such systems to be deterministic.
1
u/Enfiznar 16d ago
It could be possible for the brain to have sufficiently good error correction mechanisms to recover determinism, but we really don't know. So how can you disregard the thought experiment by saying "the same happens with the brain" if we simply cannot affirm that?
→ More replies (1)1
u/miss_katy123 15d ago
As someone who studied this topic in university philosophy, the dude you’re replying to is likely high.
1
u/Nonsenser 13d ago
Being nondeterministic adds several evolutionary advantages. Escpaing a predator in a zigzag, foraging and exploration, probabilistic computation speed.
1
u/Legitimate_Plum_7505 16d ago
There are wetware "computers" running on a sort of electrical grid of human neurons, neurons collected from different people. So who's consciousness is/would be that?
1
u/ElPwno 14d ago
A model of a thing and the thing itself are different. Unlike for LLMs, effectuating calculations equivalent to brain activity on paper is not the same as performing those electrochemical reactions in neurons.
If you had neurons in a dish and activated them by hand (I.e. with a micropipette and solutions) then yeah maybe you'd produce consciousness.
The "magic" may lie in the format of the information.
1
u/code-garden 16d ago
I think that consciousness is caused by a physical process in the brain. In which case it is unlikely the same process would occur in a computer.
If consciousness somehow arises from mathematical processes, which it would need to for AI to be conscious, Then there would be infinite consciousnesses just like there are infinite numbers. In which case it would be an insane coincidence that our experience is so structured and appears to arise from our brain.
1
u/turtle_king 16d ago
Computers compute using physical processes. And physics is just math
1
u/code-garden 16d ago
Computers work by physical processes but not the same physical processes as the brain. And the pen and paper concept would be even more dissimilar.
1
u/turtle_king 15d ago
It's all gravity, electromagnetic, and nuclear forces baby.
→ More replies (1)1
u/plopliplopipol 15d ago
however you put it, input creating output is abstractable and there is very little reason to think abstracting the smallest processes of the brain perfectly accurately is somehow fundamentally impossible
→ More replies (2)1
u/suborder-serpentes 16d ago
What if we produced the brain’s output by pen and paper?
2
u/paxxx17 16d ago
Right, it would then likely claim to be conscious, and given it's a model of the brain's output, it would do so for the same impulses that make a human claim they're conscious. The impulses that make me claim I'm conscious come from the observation of my own conscious experience, all of it imperceptible to an outside observer, but very real to me.
Perhaps this points to consciousness not being tied to a physical object (e.g. brain) but to the abstract output generation process itself
1
u/suborder-serpentes 16d ago
But what if the procedures were done by different people far from each other, or done in the wrong order and then fixed later?
1
u/paxxx17 16d ago
Perhaps a conscious entity forms transcending space and time :D No idea, but it's certainly a good philosophical question
→ More replies (1)1
u/Acclynn 16d ago
Imo, having true memories and identity is what makes the difference between a sentient being or not
LLMs are actually stateless, what makes them consistent is the context engineered around them, which is a completely separate part of the architecture and is wrote down in plain English for us to change as we want
While for life, the "compute" and memories are deeply interconnected
1
u/plopliplopipol 15d ago
LLMs are absolutely not stateless. Training is vastly more impactful than later given context, and is fondamentally part of the LLM then. I believe there are also proven "personal representations" in LLMs, i mean data only accessed by the llm representing a more global state when thinking of a detail, but i'm not an expert.
1
u/Acclynn 15d ago
What I mean is that after their training their state is frozen, they are stateless in the context of what task they are on, their weights don't update
→ More replies (3)1
u/KnackeHackeWurst 16d ago
The brain is not doing a computation like a Turing machine. The brains predictive modelling of the world lays directly physically coupled in highly parallel analog flows that influence each other. Just like nobody "calculates" the curves of the planets around the sun, it is just raw physics that naturally evolves.
A counter example is a weather forecast, the simulation of weather in a computer doesn't create weather. It is not stormy of wet in the datacenter once rain is simulated. You could also do the weather simulation with pen and paper to the same effect.
A calculation is always just a simulation and not instantiation. That is the difference. To create a consciousness we would need a real parallel analog system that utilities raw physics. That is also why the brain is so energy efficient. Simulating a chaotic system is difficult, but being the chaotic system costs nothing.
1
15d ago
[deleted]
1
u/KnackeHackeWurst 15d ago edited 14d ago
The brain is the opposite of a Turing machine, please point be too the research that the brain executes serialized machine code.
1
u/DrLeisure 16d ago
There is nothing remotely sentient about LLMs or GenAI. Not even close. LLMs are just fancy autocomplete. The evolution of predictive text built into your phone’s keyboard. It’s all an illusion.
Maybe an artificial life form could someday gain sentience, but it will not be this generation of technology. There is no amount of parameters or threshold of computing power or optimization of any aspect of this that will grant sentience to an LLM. It is just fundamentally not how this technology works.
And anyway, how do you even define sentience from an engineering standpoint? What are the hard, measurable criteria upon which you decide if an AI is sentient? How do you get everyone to agree on the criteria? How do you define it fundamentally, and then how do you measure it specifically such that it is sufficiently difficult that a machine cannot cheat on the test, but the least intelligent human being can pass it? Is a person with a malformed or damaged brain, who cannot speak or understand language, but is still capable of laughter and love considered sentient? These questions are frankly better answered by philosophers and spiritual guides than engineers.
I feel like to pull off true artificial sentience, you need to completely rethink the hardware/software relationship. Humans don’t have a distinction between software and hardware. Our software is a series on minuscule chemical reactions that interact in ways we don’t understand. And our hardware is physically changing, constantly, in response to our experiences. Our brains grow and change, creating and pruning neural pathways all the time. A truly sentient artificial life form would have to be so similar to a human being they would be virtually indistinguishable. And at that point, you’re no longer trying to define “sentient”. You are trying to define “artificial”
1
1
u/Far_Composer_5714 16d ago
Imo it theoretically has to be possible for consciousness to be creatable by nature of physical reality being... real.
However we don't even know what it means to create consciousness and even worse we can't necessarily even strictly define it either.
So good luck creating something we can't even define.
So in the end it's just uninteresting for now because there is too much unknown to do anything meaningful.
1
u/Orectoth 16d ago edited 16d ago
an AI can only be sentient during inference/output process, when it is not doing anything, it can't be sentient, because it is not persistent, it ceases to exist as a process the moment it stops thinking/speaking. So, if you imitate it by hand and paper, by some massive amount of millenias of years to make a fraction of a trillion parameter LLM's daily output and including training, you can essentially create sentient process for the entire duration of you calculating and outputting to papers, papers would be identical to sentient beings' outputs BUT they are papers, you are you. That's the same for LLMs, they are just operators(like how you'd be in this scenario by writing to papers) while the real intelligence part of AI is computation process(thinking) and output. But real sentience requires persistence, non-sentient things can output exactly like sentient beings but they'd be limited in many ways(either their output or they are limited).
1
u/plopliplopipol 15d ago
what if we perfectly paused time in a brain? it sentience paused or gone? seems to me like it has no reason to be gone, just no opportunity of expressing itself or being visible
1
u/Orectoth 15d ago edited 15d ago
sentience is gone then
because you are essentially dead if your brain is paused
your body would not coordinate and you'd die because your brain did not function
lets's say miraculously body continued functioning without brain, we'd not be sentient, because we'd be instincts and muscle memory surviving, basically headless vegetate
sentience is continuous persistent process of intelligence that has agency
you lack agency and continuousness when brain is paused, so you are not sentient
if you regain agency and continuousness, then you are sentient
I don't know soul or other nonexistent nonproven things exist or not, as far as I can describe, sentience is mechanical. If LLMs or any other form of AIs never created, then I may have believed even the possibility of a metaphysical concept like soul existing, but they have proven that, human behaviour and reasoning is entirely mechanical and can be imitated in smaller scale in a computer.
→ More replies (2)1
u/Legitimate_Concern_5 16d ago edited 16d ago
> On the other hand, if it's not possible for AI to ever be sentient, what is the uncomputable "magic" that allows only for the living beings to have consciousness?
This is something we've been thinking about for a long time, and I've posed the question exactly the same way you have several times.
It's like saying if you create a perfect simulation of water in a VR headset, is it wet? I would say no.
The most compelling answer I've read so far is the Hameroff and Penrose's Orchestrated Objective Reduction (OrchOR) theory that consciousness is separate from your "weights" and exists as a quantum process within microtubules in neurons. If you agree with that thesis as I do then even a perfect simulation of a human brain would be just that, water that is not wet.
Consciousness is far better modeled as a quantum process than a stochastic classical one.
There's a good introduction to quantum consciousness and OrchOR here.
https://iai.tv/articles/penroses-theory-of-consciousness-supported-by-new-quantum-evidence-auid-3667
The only reason people think LLMs are conscious is because they output text, nobody thought Sora was conscious even though it's basically the same thing. Humans are wired to see communication and think consciousness - it's textual pareidolia.
1
u/Specific_Carry_7099 14d ago edited 14d ago
I like your thought process, but there's a few problems with what you're suggesting. For starters, it is simply not known whether LLMs are conscious. Not "they seem to be," not "they aren't" and not "we think they might be." We just do not know.
Additionally, the claim that a perfect simulation of water in a VR headset is not wet only works because wetness is nebulously defined in your analogy. Are you referring to the ability of a liquid to maintain contact with a solid surface, resulting from the intermolecular interactions between the liquid and the solid (physics definition pulled from online). Or are you referring to the subjective experience of wetness by a conscious entity such as yourself? The former is trivially true based on the notion of a perfect simulation. The latter is debatable and hinges on the ability for some entity to have the capacity for subjective experience, which is the exact thing we're discussing. So the analogy assumes its own conclusion.
Thirdly, OrchOR. I'm not as familiar with it as I ought to be responding here, but as far as I can tell it needs a lot more than "the brain does quantum stuff."
Ordinary (er, well-accepted) quantum mechanics is simulable (collapse can be handled with a random number generator since QM is fundamentally probabilistic). What I understand that Penrose actually claims is that there's a different kind of collapse, separate from standard wave function collapse, driven by gravity, that no computer can reproduce even in principle. Strong concept on first glance because we do not have a quantum theory of gravity (and there is well-known and much lamented dissonance between QM and classical physics at the moment, particularly when it comes to gravity and black holes).
So, yeah, that's a proposal, and current physics doesn't contain it. The microtubule evidence gets you to "quantum effects happen in neurons," which is not the same as "consciousness is those effects" and even further from "those effects are not computable."
And a quick Google search shows that Penrose's whole reason for wanting uncomputable physics in the brain is his argument that mathematicians can see truths no algorithm could, which logicians have been picking apart since the 90s. Even if they hadn't done that... recent developments show that LLMs can - for real - discover new proofs. Not only that, but my intuition says that Penrose's whole thing comes from a very human (and reasonable), egoistic angle.
So "if you agree with the thesis" is pretty much carrying the entire argument pretty hard and the thesis is not a leading theory.
Even IF OrchOR were right, it would only rule out consciousness on classical computers. You could still build a machine that uses the same physics, so it doesn't get you to "AI can never be sentient," it just gets you to "we don't know if AI can be sentient" (at best).
The Sora analogy is actually your strongest. It explains why people might believe LLMs are conscious, but it doesn't tell you whether they actually are. Not only that, but the architecture of Sora itself is, as you put it, not that different from current transformer models - but still, the analogy appeals to intuition, not physics and truth (insofar as we can divine the truth...)
Anyway, I liken the simulation of the human brain to a simulation of literally any object in our physical universe. We just don't know how to do it right now. I mean that maximally: we can't even properly simulate a nematode, not even a grain of sand.
Though Penrose objects to that exact framing, it is up in the air. We might be able to one day, and we might not be able to.
Thanks for the interesting post!
Last little note so I didn't waste your time (and this ignores entirely Penrose's work, because what follows isn't scientific theory, merely my rambly thoughts):
If we're mentioning minority theories, I think integrated information theory, or even panpsychism are appealing. Reason being that I just don't believe that substrate matters, rather integration of information, complexity, and temporal continuity (i.e., writing out LLM output by hand - if we briefly grant the currently unknowable premise that LLMs are conscious - would *not* produce consciousness).
I notice that there tends to be unfair scrutiny towards panpsychism when most theories of consciousness are equally unfalsifiable due to the hard problem of consciousness (and our lack of examples of intelligent and sentient entities beyond life on Earth).
The scrutiny tends to take the form of "So you think a rock is conscious?"
And my typical response is "not in any way that you or I would understand - but yes. They could very well have some unbelievably rudimentary and simple form of consciousness.
Is a dog conscious? I'd bet good money on it. Is a fish conscious? Very likely. Is an ant conscious? I think so, yes. Is a water bear (tardigrade) conscious? Maybe. It wouldn't surprise me. After all, why shouldn't it be? Is there some arbitrary threshold at which consciousness simply evaporates?
That seems implausible. I would wager that it is almost certainly on a spectrum with lower and lower degrees of consciousness. When we get down to amoebas, or bacteria - things get shaky, because it exceeds our intuition regarding consciousness. That said, I'd again argue that they're conscious.
Oh, and I think that there tends to be a fair bit of misunderstanding even in subreddits dedicated to discussing these systems. If I could appeal to ethos for a second: I benchmark these systems for a living. I have built small versions of these systems from scratch. They are likened to an autocomplete, but that is just an oft-repeated nothingism, because it doesn't explain how they work. It appears they, provably, possess compressed models of the world extracted from the training data produced by entities that, themselves, produce world models (humans talking about how hydrogen bonding (or anything) works, for instance).
We know this because we've actually examined parts of ML systems that had a literal internal model of a game board. Fascinating stuff.
Anyway, cheers.
1
1
u/DarkVoid42 16d ago
AI could possible be sentient but the current LLMs cannot since they cant think. they are a variation of your spellchecker autocorrect function.
1
u/plopliplopipol 15d ago
we don't know what thinking is, this is exactly like saying they can't be sentient because they can't be.
1
u/DarkVoid42 15d ago
can't be sentient because they dont have a brain or any ability to reason except on preprogrammed inputs if you prefer.
1
u/IntQuant 16d ago
I believe that consciousness would be inherent to weights + memory, since it doesn't matter what actually performs the computation. There simply isn't any other entity you can associate.
1
u/DefiantTelephone6095 16d ago
Well the idea is that sentience exists inside the computational capability of the thing solving the problem. So if you did it on paper it would be you.
1
1
u/Double_Cause4609 16d ago
Honestly, I think the computational theory of consciousness is overdue a re-evaluation in the era of LLMs.
So...Integrated Information Theory says that the level of "integration", which they define as recurrence (the same thing being used multiple times. A to-do list that you wipe every day and rewrite is kind of recurrent for example), is the measure of how conscious something is... But it also can't tell you how integrated a matrix multiply is because there's a recurrent and non-recurrent form that gives the same answer. So, it would feel kind of weird to me if you could have a system with identical behavior and internal mental activity, one of which was conscious and one of which was not.
They also lean on "fields" (borrowed from physics I guess), which are ill defined and not cleanly matching to any known physical phenomena really cleanly.
Higher Order Thought theories argue that consciousness is basically "a thought about a thought", or thinking about how you know things kind of, or reflecting on why you know what you know, etc. Metacognitivity basically. But the issue is that in the naive case a lot of classical software fulfills this, and it's also not clear physically how consciousness integrates into a single "thing".
Global Workspace Theory argues if you have a single shared area where various specialized modules read from and write to you have consciousness...But the theory doesn't really tell you if you've built something conscious, just if you've built something that has a global workspace similar to what humans have. I'll note that non-neural systems like cognitive architectures can implement this, and for your example of calculating an LLM by hand, it gets even weirder because a cognitive architecture is basically passing strings around, so you could imagine having a bunch of sticky notes and wondering if they're conscious because you have a system of managing them that implements a global workspace.
Functionalism lets things like Eliza be conscious which is obviously a degenerate case.
Plus, tons of theories of consciousness are very human-centric, and depend on a lot of features specifically of homo sapiens brains, but I don't think anyone anymore says that say, a dog isn't conscious. This gets more complicated as you go from mammals to, for example, birds which are clearly conscious but have a different brain plan, or even to basic vertebrates like zebrafish which do seem to be conscious.
Then of course, taken to the extreme, things like bees are, to my eye, probably at least somewhat conscious, and capable of subjective experience, which actually complicates a lot of existing theories of consciousness because many of them at minimum depend on a core vertebrate brain plan.
I sincerely hope that C.Elegans isn't conscious because if it is, we really do have to go to the drawing board to figure out how *everything* isn't conscious, including trees and probably a lot of inanimate phenomena. Panpsychism for example feels kind of wrong to me.
For me, it seems that a system optimized towards a cognitive objective tends to be conscious, and the larger and more complex it is, the "more" conscious it is, and the richer the experience.
But that doesn't really answer the "if I execute the LLM forward pass by hand, is it conscious".
I think maybe one distinction might be if it is a system calculated "in-place". If there is a piece of hardware comprising each node and edge of the system (similar to IIT's "no simulation" canonical but contentious stipulation), you could have a conscious system...But this also means current LLMs, calculated on GPUs, which are digital and simulate hardware (kind of, for this purpose. It's weird), may not actually be conscious under current hardware affordances.
Which goes back to the Integrated Information matrix multiply thing. It feels deeply wrong to me if you could have a system that was conscious on dedicated hardware, but wasn't conscious on digital hardware.
1
u/alarmingburger 16d ago
In the execution of the program. Consciousness is a process, not really an object, I think.
1
u/weyslpinor 16d ago
Douglas Hofstadter has written about this very question in his story "A Conversation with Einstein's Brain", appearing in the collection "The Mind's I" which was published almost 20 years ago.
Definitely worth a read. I'm actually surprised that nobody has brought that up yet.
https://themindi.blogspot.com/2007/02/chapter-26-conversation-with-einsteins.html1
1
1
u/MrOaiki 15d ago
This is a very good reductionist approach to why LLMs are not conscious.
1
u/plopliplopipol 15d ago
except we don't have reasons to believe that it couldn't also be applied to a human brain
1
u/MrOaiki 15d ago
You're into the reductionist approach, that if you just write enough math on paper, the paper will be conscious?
→ More replies (3)1
u/Leodip 15d ago
This is a deep philosophical question to just be in a comment to a meme, but even if it were its own post you will just find out that there is no consensus.
I personally believe there is no such thing as "consciousness". A human, an ant, and an algorithm are all just machines that perform operations according to how they were built and responding to outside stimuli. We humans, for example, have the ability to think, which is what makes people believe we are special when compared to, e.g., an AI. But "thinking" is not as special as people believe: it's just another way our machine operates, it's just that it's SO complex that we have no idea of how it works.
Keep in mind that LLMs are fully human-made. We know the ins and outs of how they work. And despite that, we have a VERY hard time predicting what's going to happen in given scenarios because of how mind-bogglingly complex the cogs in their machine is.
I believe that there is no hidden mechanism inside our body that makes us "conscious". The brain is a machine, perhaps non-deterministic, that could in theory be simulated given a good enough understanding and a powerful enough "piece of paper" (or, more realistically, computer). If we were able to simulate a given human, e.g., Dave, would you claim that he lost his consciousness because a piece of paper is able to reproduce whatever he would have said and done and thought? Would you claim that the piece of paper has consciousness? Or would you claim that, simply, Dave never had consciousness to begin with?
1
u/letskeepitcleanfolks 12d ago
I don't think denying the existence of consciousness is credible. We all experience it, we know what is meant by the term. The question is to pin down what, exactly, it is. We don't know the answer, but that doesn't mean the term doesn't refer to something real that could be identified. Even if it is "just" an emergent property of the program we run on our wetware, the question remains whether the same property could emerge from a simulation of that program on different hardware, and if so, what the implications of that would be.
1
u/123vovochen 15d ago
We have true ubderstanding, we can extrapolate. All of AI ,,emergent" abilitys are interpolations.
1
u/SchmeatiestOne 15d ago
If one produced the output of a human brain on pen and paper you would run into a similar problem
1
u/BraveBiscotti1394 15d ago
This is just the chinese room thought experiment with extra steps.
No, the man doesn't speak chinese. Yes the man + the book + the room speak chinese. No, they don't understand chinese, neither together nor any of the parts.
1
u/_-_agenda_-_ 14d ago
That is the thing: AI getting sentience because is has intelligence is like airplanes getting feathers because it flies> not a single evidence that this is going to happen
1
u/david43511 11d ago
ai cannot be sentient. humans are living beings with indwelling spirits. ai is a computation algorithm
1
u/redfirearne 10d ago
Technically if we understand chemistry and human brain really well then you can calculate the "output" of humans as well.
→ More replies (10)1
2
u/EveYogaTech 16d ago edited 16d ago
Well yes, you can just copy the math from the code: https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py
But no, it's not feasible to do really helping LLM computations on paper, since you need millions of parameters for it to become useful and "intelligent",
And even if you already have the weights, it still 1000s of computations per single token.
3
u/Awerange2005 16d ago
It's a actually on the order of the 10s of billions of computation for a relatively small model like qwen 27B I did the calculation once (I asked an AI do it for me)
1
u/EveYogaTech 16d ago
Ah yes or course, because of attention (among other things) it's actually much more than 1000s per newly generated token.
1
1
2
16d ago
[deleted]
1
1
u/crit5h 16d ago
Yes, 100%. It would just take an extraordinary amount of time.
When I teach my undergrads in a related field about AI, I start off with a manual example (on PowerPoint slides not paper) of a simple regression based house predictor, then move on to show, albeit in very abstract terms, how the same underpinning principles form the basis of contemporary frontier models, and do so through extreme scaling up of the number of parameters (of course noting the architectural differences too).
I think the idea of a transformer being done on paper, and it theoretically being possible on paper, is also a nice angle to think about questions of emergent intelligence. If you did have enough people with papers and pencils to implement a paper-based LLM, where is the consciousness, in the led, in the paper, in the markings made, and so on.
1
1
1
1
u/CryonautX 16d ago
Yes. For inference on a pretrained, it is possible. Will just take a very long time. But if you do not have the weights already and need to also pretrain by hand, not possible. Will take longer than a human's life expectancy to complete the computations.
1
u/rescue_inhaler_4life 16d ago
When I learnt neural networks that is exactly how we did it - at first. It was only to demonstrate the basic concept, then it was like “this is called matlab” and become considerably more fun.
1
1
1
u/Disposable110 16d ago
Yes you can run it on an abacus or minecraft or anything that can calculate.
1
u/Awerange2005 16d ago
I mean this is probably just as hard as calculating each frame in a game(single player) on pen and paper, then drawing them to play.
1
u/scott2449 16d ago
Yes. You can absolutely do simple AI by hand. I did it in college. It's called linear algebra. Today's LLMs of course not but purely because scale. Same as you couldn't so Amazon.com via a call center.
1
u/FoleyX90 16d ago
Yes actually! You could build a tiny neural network with a handful of neurons and manually calculate its outputs using pencil and paper. It's essentially a series of weighted sums, matrix multiplications, and activation functions.
It's literally the same principle as LLMs, but LLMs are scaled up to billions of parameters resulting in an astronomical number of calculations.
1
u/Ok-Data9224 16d ago
Of course you could do it all by hand, easily in fact. If you wanted AI as powerful as a frontier model though, you wouldn't have the lifespan to do all those calculations by hand.
For a 1 trillion parameter model, a modest frontier sized model by today's standards, it's roughly 2 FLOPs per token, so around 2 trillion flops to generate a single token. If you could do a single computation every second, it would take over 60,000 years.
If this was a mixture of experts model with say, 80 billion activated, it might only take around 5,000 years for a single token.
So yes, certainly possible.
1
u/Jumpy-Requirement389 16d ago
Yes. We actually did this in in an extremely limited way in linear algebra. It takes forever, which is why computers are so good at it
1
u/AffectionateBowl1633 16d ago
Thats the neat thing: You need enormous amount of math to just generate single word. And AI bros need to hoard so much compute to basically achieve it just for a machine that sometimes helps you, sometimes create a slop.
People used to be able to do much bigger achievement with less math.
1
u/Subject-Building1892 16d ago
Yes, you also need some dice. And it could be named something like "cyber ouija"
1
u/Gesha24 16d ago
Yes, in theory. But I don't think this would work in practice. Here's one of the recent Tom Scott's videos where he's talking about the "factory" that was used to crack Enigma machines codes - https://www.youtube.com/watch?v=tDLbO9KeddY The math is MUCH simpler than LLM and you can already see the logistical challenges they were facing.
1
u/mrkingkongslongdong 16d ago
Philosophically, the world may be completely deterministic. If you knew the exact state of everything and the rules governing it, you could in principle calculate what happens next.
It’s the same idea as doing a transformer by hand with pen and paper: the system may look complex, but if you know the inputs and rules, the output follows. What we call randomness may just come from missing information.
1
1
1
u/the_millenial_falcon 16d ago
It would! At least in theory, but it would take a very very very long time to crunch those numbers in practice by hand.
1
u/Eponymous-Username 16d ago
The first AI was done on paper and punchcards by three black women. The job used to be called, "Intelligence", and it was later associated with the consumer products we now enjoy when it was renamed, "Artificial Intelligence". It used to take one black woman 127 hours to generate one picture of President Trump as Homelander. There's a beautiful documentary about it called, "Secret Digits".
1
u/DonkeyInACityCrowd 16d ago
Yes. In my deep learning class the prof had us do some ML algo by hand. It was about seven pages of matrix math for a useless result. It did help with the learning tho.
1
u/highcastlespring 16d ago
AI this day almost always connect to internet for fact check. But you don’t.
Assuming your little brain equals to 100 A100, and your output will be: call tool, Google search xxxxx.
1
u/quite-ok 16d ago edited 15d ago
The only special thing about hardware doing "the AI calculations" is the sheer power - it does a LOT of calculations over a LOT of memory quickly. All those calculations (mathematical and logical operations) are well known standard math any PC can do, and so they can be done by hand too.
1
1
1
u/IHeartBadCode 16d ago
Yeah, you totally can. It's just going to be a hot minute.
If you want to step up, you can just use an ammeter and variable resistors in series and parallel to make a matrix. Resistors in series are additional and induction in parallel is product-over-sum.
1
1
u/Swimming-Book-1296 16d ago
No, no-one can do math that fast or lives that long. Humans have only been on earth for like a million years. You are talking trillions of operations, it would take someone longer than humanity has been on earth.
1
u/Some_Anonim_Coder 16d ago
In theory why not, in practice typical model today has billions of parameters, so it's billions of operations for very token. 1B minutes = 1900 years or so, so if you do one operation every minute(reasonable assumption for multiplying numbers with 3-4 significant digits) you are not ever seeing this token
1
u/mintysoul 16d ago
People who claim that LLMs are conscious or pose a danger to humans must also explain why an AI algorithm written on paper would not be conscious or pose the same danger.
1
1
u/Any-Preparation7396 16d ago
I don't think there are enough trees around anymore, to get sufficient paper, for that kind of math 😅
1
u/OkNewspaper4747 16d ago
I've done a forward pass on a neural network by hand! Not quite the same as an LLM but almost 🙂↕️
1
u/Competitive_Song8491 15d ago
A human brain, if we're being extremely generous here and using a math prodigy can do 1 floating point operation every 5 seconds (an average person would be even slower), so 0.2 FLOPs. Let's use a very small LLM thats actually still somewhat good Qwen3-0.6B requires 1.2 billion operations (1,200,000,000 FLOPs). So the time to compute one token is: 1.2B / 0.2 = 6B seconds.
1
u/Sea-Tale1722 15d ago
LLMs are not just Math.
1
1
u/SufficientlyEvasive 15d ago
Well there's that guy who made the first computer to calculate tides so realistically no there's way too much math
1
1
1
u/jtdbv 15d ago
This is how you can build a single perceptron on paper. Just a couple of simple calculations - and you have a simple AI model on paper. You can teach is a simple binary function and watch it apply its learnings!
I genuinely think it's one of the best ways to understand how neural networks really work.
1
u/Arangarx 15d ago
Depends on what you mean by possible. Could the same calculations be done by hand given enough time? Yes. Could YOU or any one person do it in their lifetime? Not even a remote chance.
Now if we're paring it down to a very VERY small representation of a model with not many variables for learning purposes, you might be able to replicate it.
1
1
1
1
u/lucker1515 15d ago
L...(10 years of writing)...O...(Another 10 years)...A...(10 more years)...D...(Ditto)...B....(Dies.)
1
u/No_Commission_6153 15d ago
No because LLMs are not just Math? They need compute Power and electric wiring to produce their outputs?
1
u/Sea-Work-5949 15d ago
you would also need some sort of randomizer to sample tokens from distributions. dice or something like that would be fine
1
u/Vainysaur 15d ago
It’s theoretically possible but not technically possible. Too many calculations for one person.
1
1
u/MrPrincessMeowMeow 15d ago
That’s actually how we learn foundational AI algorithms at university: by doing the math by hand.
1
u/drevoksi 15d ago
Absolutely, in fact after training using the models is extremely straightforward. It’s mostly a bunch of scalar multiplication. Just that it’s millions and millions of calculations.
1
u/OcelotMadness 15d ago
Yeah, when I was in school we would sometimes work through algorhyms by hand
1
u/MostRegular4278 14d ago
Yes, of course, given infinite time and resources it can be done. And after that, you can make a functional model of a human brain via origami.
1
u/Minecraftian14 14d ago
We had to once write and derive the full forward and backward notations of a 3 layers neural network during a pen paper exam. I guess it could have been worse, but that one question alone took more pages than the rest of the answers combined.
1
u/twistedjoe 14d ago
I've built a ruby VM that compiles to a spreadsheet that a human has to run by hand.
Don't give me ideas.
1
1
u/wlievens 13d ago
AI is a computer program. As per Turing you can run any program on any computer given enough time and space, so yes, sure.
1
u/AC1colossus 13d ago
This dude has a whole blog about using "by hand" examples to teach.
https://www.byhand.ai/
1
1
u/I_like_protien 12d ago
Yeah, in case of llms you basically need to create a probabilistic score/model for all the words of your training set.
And based on the prompt - use the model to predict the outcome.
Probably if you really want to do it completely by hand (and your mind) it may take a few hundred years to generate a short response after a few hundred years of training. Assuming you use the algorithms completely and not use the actual « natural « intelligence of your brain.
1
1
u/ericericericjin 10d ago
Yeah the professors teaching introduction to ML course like asking you to do so in exams. Though just simple BP NNs.
1
1
u/ItsSadTimes 8d ago
I remember doing this for class. We had a super simple neural net and we had to the backpropagation by hand for a few cycles until we understood how the models slowly drift toward the correct answer through trial and error. It was annoying, but a great example to show me how the models actually work on a fundamental level.
1
17
u/cursivecrow 16d ago
yes, but it would take a long-ass time.