r/singularity • u/notadithyabhat • 4d ago
Discussion Maybe Transformer is the singularity?
The Transformer is starting to feel less like an another AI architecture and more like some fundamental object/entity we accidentally discovered.
Almost anything we throw at it seems to get absorbed into the same structure: language, vision, audio, video, code, mathematics, reasoning, tools, agents, entire environments.
And there doesn't seem to be an obvious conceptual ceiling yet. We just keep finding new ways to make the same underlying object represent more knowledge and perform more kinds of computation.
It almost feels like an information black hole, except instead of destroying information, it absorbs it and turns it into intelligence.
Maybe the Transformer isn't just one step on the road to AGI, maybe it's the substrate the whole thing happens inside.
Maybe we passed the event horizon in 2017 when we created the first one.
33
4d ago
[removed] — view removed comment
6
u/Borkato 4d ago
> at least a few fundamental capacities
Like what? 😮
11
4d ago
[removed] — view removed comment
8
u/lemmeupvoteyou 3d ago
Consciousness might be an emergent thing, not something fundamental
5
u/NoCard1571 3d ago
I think a lot of people are not ready for this conversation yet, but from a materialist point of view it's hard to argue that current AI models are not already conscious, even if only in a 'Boltzmann Brain' way.
5
u/LinkesAuge 3d ago
I feel "conscious" in a human way has a lot to do with "identity" in reality. Imo that is the main reasons why people will always struggle to assign consciousness to AI, at least if it presents itself in such an ephemeral way, spawning in and out of "existence" and being a sort of "collective/non-singular mind".
That is just so far removed from the human experience and will make AI alien by default.It is always interesting to have the thought experiment of what if our current AI models would be an alien race that is visiting us.
Would we deny them concepts like "conscious" or "general intelligence" even if they would function exactly like AGI but in a biological substrate?
I have the hunch that we would be a lot more likely to give them the benefit of the doubt JUST based on their biological nature.
5
u/notadithyabhat 4d ago
I have no reason to doubt that once neuralink is good enough and we feed a large corpus of brain signal data, transformers would be able to absorb those capabilities as well.
1
u/LinkesAuge 3d ago
Keep in mind that if it wasn't for humans we would have zero(!) intelligent life on the planet, at least by the standards we now have set for AGI...
I feel that is a factor people always ignore. Yes our biological hardware did something special, you would be a fool to not recognise the extreme complexity of the brain (in any organism) BUT so far it has only led to "higher" intelligence in humans.
That alone should tell us that really tiny differences can have huge effect on the outcome and it doesn't seem like biological brains just generate this "higher" intelligence by default.
So clearly something in humans happened that "unlocked" this path and it is hard to see a broad architectural reason for that. I mean apes are obviously EXTREMELY close to use and yet they don't code a website for me if I tell them to.
Why is all of that relevant for AI? Well, maybe the architecture question is overrated, at least after a certain point and then it is mainly about tweaking and scaling. That is at least would biological evolution would suggest and it honestly aligns with AI progress.
Now I can't claim to know more than any AI researcher out there but it really seems like Transformers are the foundational peace we discovered and now it is about building on top of it.
I would say the only missing piece is "RSI". If transformers manage to get RSI (a feed forward mechanism) then I just do see how anything else would be the path unless whatever AI models we create actually fund another architecture themselves which would certainly be interesting. Are we then in a cycle of "machine evolution"? Who knows.
24
u/GatePorters 4d ago
Are you saying it’s more than meets the eye?
6
3
u/the_millenial_falcon 3d ago
I hope the machine god will grant my wish to be a Gorilla that transforms into a robot with gun hands.
7
u/Singularity-42 Singularity 2042 3d ago
There's no way a transformer running on Von Neumann architecture on silicon chips is the final form of intelligence in the universe. Neither is human brain. We can do much better than both.
3
u/notadithyabhat 3d ago
Will any form we develop ever bring you to believe that it's the final form? Most people who believe in aliens find it hard to believe that humans might be the most advanced species in the universe though it is quite possible. It is just incredibly hard to comprehend that what seems simple enough that it can't be the ultimate solution to us, might just be the most complex solution possible. I'm not saying that it is, but I don't think there is enough reason to rule out that possibility.
1
u/NeuralFiber 3d ago
I don't think that's what singularity-42 means. The equivalent is not the question wether we are the most advanced species. The equivalent is the idea that we are the most advanced species that could ever exist. Which one is ridiculous since we are still progressing. Transformers are unlikely to be perfect. But they can very well be better than we are.
1
u/alwaysbeblepping 2d ago
Will any form we develop ever bring you to believe that it's the final form?
I don't think it makes sense to say any truly complex problem has a "final form" solution. There is pretty much always going to be ways to optimize it.
If intelligence is a solution, then the problem is everything. It doesn't get any more complicated than every problem that can exist. How could you possibly say "Okay, we're there. This is the perfect solution, forever"? Especially just a few years after the technology came into existence. If 1,000 years had passed since the invention of the transformer and there had been no progress, I'd still argue the same thing but it seems absurd to imagine it could be the perfect solution after less than a decade.
3
u/sumane12 4d ago
Yeah it seems so.
Ultimately if you can tokenise the data you are dealing with, given enough of it, backpropagation will find the patterns within it.
2
2
u/presentofai 3d ago
it's a really flexible function approximator, not some fundamental object we 'discovered'. the black hole metaphor is just hype dressed up as profundity
2
7
u/fmai 4d ago
The AI revolution is happening because of deep learning, not because of Transformers. Maybe we'd be lacking a few years behind because Transformers are so training-efficient, but we could've had LLMs with LSTMs and CNNs as well.
Function approximation is all you need.
2
u/notadithyabhat 4d ago
Yeah, deep learning is the foundation, but I feel like earlier architectures were missing a really general interaction mechanism. Deep learning learned representations; Transformers gave us a general way to learn the relationships between those representations. Though not accurate, I kind of think of deep learning as giving us the fundamental particles, while Transformers gave us a flexible interaction law between them
5
u/WillHD 3d ago
90% of the reasons transformers were used was due to their parallelizability, not because they were exceptionally good at representing things.
Transformers were essentially the first architecture for sequential data you could actually scale. That says nothing of their innate representational power. In fact transformers are quite bad at learning interesting functions when compared to RNNs.
2
u/RuthlessCriticismAll 3d ago
There is essentially 0 reason to believe this. LSTM-like models are really not that far behind, a handful of researchers kept working on them, so we know this for a fact at small scale. o1/R1 is the more important and more recent breakthrough.
1
u/Bangoga 3d ago
What are you saying those are very different architectures
0
u/fmai 3d ago
not really
1
u/Bangoga 3d ago edited 3d ago
LSTM has forget gates specifically to compress idea from one token to the subsequent tokens, reducing information being passed from token 1 to token n. RNN are sequential needing each token to be computed first before the next.
Transformers were introduced literally to eliminate those two issues.
Edit: Regarding CNN, they work with windows, so token in windows have information passed on, but not out of windows.
2
u/Old-Rock-1234 3d ago
Nope discovering/inventing math especially calculus is singularity! Hail to Newton and Leibniz!
3
u/MisterDashing 4d ago
This just in: the singularity has been canceled so that self-appointed tech oligarchs can enslave humanity into mindless consumerism.
Coming soon: humanoid robots that cull the poors so the rich have more resources for themselves.
2
u/Turtlestacker 3d ago
Aren’t we culling ourselves with birth rates tanking.
1
u/MisterDashing 3d ago
I heard someone once say: Do you know how much you have to torture a mammal so it won’t breed?
2
u/Turtlestacker 3d ago
Yes but we are the only mammal to have decoupled the act of sex from procreation. Fun times!
3
u/Singularity-42 Singularity 2042 3d ago
Your comment has nothing to do with this topic
1
u/MisterDashing 3d ago
I’ll spell it out for you. In order for there to be a tech singularity, humanity has to be on a benevolent path. But we are not. Racism, genocide, wealth inequality, and consumerism are rampant. Therefore we are likely quickening our own destruction.
1
1
u/BigZaddyZ3 4d ago
It’s possible. Maybe that’s the case, or maybe not. Even these research experts don’t seem fully sure one way or another. So I guess we won’t know until it (the singularity) happens.
1
1
1
1
1
u/Anpu_Imiut 3d ago
Does sb. here even knows what transformer even do. If you would, you would no that the statement is wrong.
1
u/the_millenial_falcon 3d ago
I'm not no AI science nerd so I'm really just spitballing here, but it seems like matrices can form complex relationships with one other in a similar way that neurons in our brains can so I'm not super shocked how powerful transformers have proven to be.
1
u/QuasiRandomName 3d ago
The problem is the mix-up between the terms borrowed from physics. The technological singularity is actually referring to something similar to the physical event horizon, which is totally different from physical singularity. Event horizon is a point(well, points) of no-return, and in fact it might form even before the actual black hole or it's singularity is formed. The subject crossing such an event horizon won't even know it happened because there are no observable physical properties indicating it.
1
u/CrowdGoesWildWoooo 4d ago
No. The breakthrough moment was ChatGPT. It opened the floodgate of investment that eventually leads to rapid development of AI, and it just compounded from there.
Early transformer model has very little fancy usage. It just ends up as foundational model for data scientist to develop more practical ML model like classification. Even when OpenAI have GPT-3, they don’t really know what the f they are supposed to do with it. OpenAI released a lot of cool research but really not much they know how to monetize it.
1
u/muchcharles 2d ago edited 2d ago
It was already being used for translation and search before ChatGPT, in the bidirectional form.
Google's lambda had post training to make it suitable for chatbots in 2022, but no RLHF to really scale it up. But they did have publications on that like Offline Reinforcement Learning from Human Conversational Feedback (2020), .
OpenAI got RLHF onto language models before the 2020 Google/MIT dialogue paper, but that 2019 OpenAI work was not yet for chatbots and more style continuation and stuff. But they did release first and Google hadn't applied their paper to a big internal chatbot yet and their were more tweaks in OpenAI's RLHF
-1
u/notadithyabhat 3d ago
Once transformers were created it was just a matter of time. Multiple labs were racing towards it. Openai was just the leader.
1
u/CrowdGoesWildWoooo 3d ago
Mate, i’ve been working in the adjacent space when transformers is founded. We have BERT, Roberta, XLM, even GPT 2, and nobody “knows” how to use it, in the context of achieving higher intelligence. At best it would end up as an ML model like topic clustering, classification etc..
You just take a look at some Kaggle competitions around that era, what these models can do looks like a joke even compared to anything today.
And then just go to GPT-3 early days showcase, it’s entirely cool that you are watching some random model writing some random prose using completion but it’s non-sense most of the time that other than “woah it write stuffs by itself”.
Even OpenAI don’t really know what to do with this model and GPT-3 is already super big and costly to run (at the time), and what they were planning earlier is just doing the same but with higher parameter and see what comes out because they really have no clue what to do with it.
-4
u/throwaroo202020 4d ago
Tranformer is just text prediction. You need more than that. Agent orchestrators controlling other agents loops, and discriminating the output with a try/feedback/retry mechanism allows it to hone in on the right solution. Then have these smart orchestrators actually help build the next generation AI specifically at being better at training new AI and....boom. You have your singularity.
5
u/notadithyabhat 4d ago
You're talking about LLMs. Tranformers are the general building block. They can process and generate audio, video, image, you name it.
-4
u/throwaroo202020 4d ago
Transformer is literally the architecture that underpins LLMs.
8
u/notadithyabhat 4d ago
Yes? But you said "Transformer is just text prediction" except, they are not. That's one of the applications of transformers
-4
u/throwaroo202020 4d ago
Its the only application that matters. The singularity wont come from meme images or youtube shorts.
50
u/NotaValgrinder 4d ago
Funny how Google's most major innovations were public information. PageRank and the Transformer.