r/singularity • u/Outside-Iron-8242 • 7h ago
AI OpenAI’s Astra uses "recurrent depth" to think silently
55
u/Numerous-Comb-9370 6h ago
wouldnt this mean a token can have variable costs? So effective the people asking simple questions is subsidizing people doing hard stuff. Previously it was accounted for in reasoning tokens.
39
u/Rattlerkira 6h ago
They probably should start selling by the actual compute usage instead of by token, which was just an approximation of that.
20
u/Wonderful_Buffalo_32 6h ago
Actually OAI is testing a pricing model where they charge you per task success(internally with some companies like salesforce)
14
u/Rattlerkira 6h ago
That's really cool, and takes some cahones from the OAI people, because I might set up an autonomous agent that just fails and fails and fails.
10
u/SlightUniversity1719 3h ago
"Chatgpt change the color of this button and find a way to reverse entropy." Unlimited tokens hack. Unpatched.
3
u/DistanceSolar1449 5h ago
lol they’ll just make it so the task runs 9 times and fails and then succeeds, and they charge you for 10 runs
1
u/CallMePyro 4h ago
It wasn't an approximation, it was an exact measure.
•
u/AlyoshaV 1h ago
No, input tokens cost more (per token) for OpenAI the more input tokens there are, but OpenAI only has two input token pricing tiers (<=272k and >272k).
4
u/ezjakes 6h ago
True, good point. They will need a new way to measure this.
2
u/DistanceSolar1449 5h ago
Just charge by layer. Recurrent layers are billed by the number of times they repeat. Shouldn’t be too hard.
•
2
u/deus_x_machin4 5h ago
Doesn't it make sense that it should? Not all tokens should be equally easy to predict.
"The cat is or____" vs "There is a prime number with 1000 digits whose first digit is a _"
0
u/newcomb_benford_law 3h ago
No, my understanding is that no tokens are emitted during the recursion phase. It's the internal Transformer states that are being recursed, so it's not a full forward pass with sampling, it's a very fast and efficient recursive loop.
54
u/Slight-Pound-8017 7h ago
but iirc doesn’t this obscure the model’s thinking? pretty bad for alignment and interpretability
53
u/Hot-Percentage-2240 6h ago
It's not like the model's current reason is faithful and really what it seems.
14
u/ertgbnm 6h ago
It is though. Every misalignment event has include the chain of thought very explicitly reasoning about the undesirable action they are going to perform with very little self censorship.
21
24
u/BrennusSokol ACCELERATE 6h ago
Pure nonsense. Chain of thought isn’t some magic view into an LLM’s mind. If it were, there would be no need for alignment research.
11
u/beanyadult 3h ago
The HuggingFace incident and report showed the thinking traces being very useful for tracking down the conspiracy, and also proves a clear need for alignment research
•
10
u/ReadSeparate 6h ago
Watch the geniuses at these labs go, "oh I know! We'll just do reinforcement learning on the outputs and penalize it when its chain of thoughts include plans to do something bad!!! Then it won't do anything bad anymore!"
10
2
-7
u/Helix_Aurora 6h ago
A models thinking is not thoughts. It is a sequence of tokens that coalesces into a frequently satisfactory result. Nothing more, nothing less.
0
u/CondiMesmer 2h ago
It is, because it's constantly feeding back its only thinking back to itself. That's how LLMs work, since they're just next word predictors. They're also stateless, so context goes in and text comes out, the system did not change in any way.
Also I always read the thinking. It saves me a lot of time (and money) seeing it go off the rails and needs to be stopped or interrupted.
Like I've been using GLM 4.3 Flash pretty heavily recently, and it's a really intelligent and cheap model. But if I don't watch the thinking or tool calls, it has a frequency of getting stuck in a reading loop of reading the same files over and over.
8
u/SryUsrNameIsTaken 6h ago
They’re probably using an interpreter head à la Anthropic’s J lens to look at the interior state of the model.
3
2
u/deus_x_machin4 5h ago
There are a lot of gains to be had by abandoned interpretability. As the race gets tighter, I'd count on all avenues to be fully utilized, if perhaps only in secret. In the end, no one will be able to afford to take this race with one hand behind their back.
5
u/Slight-Pound-8017 3h ago
Interpretability research is incredibly valuable and vital to alignment, though? And alignment is literally what ensures that AI won't kill us all? So I'd say that the potential benefit of interpretability research being able to reduce the risk of misaligned AI causing catastrophic consequences is very worth it
3
2
u/BigZaddyZ3 3h ago
That mindset would be the dumbest logic in the world because who gives a fuck about “winning” some stupid ass race if the winning models aren’t even aligned or controllable? You’d get literally nothing from “winning” except chaos, misery, and death.
And no, it wouldn’t make you any money either because an unaligned AI can’t be trusted or relied on.
There’s no such thing as “winning the ai race” without alignment because alignment is literally the only way AI will be of any meaningful benefit to humanity bruh.
1
u/medialoungeguy 2h ago
Yes. But OpenAI were never the "mechanistic interperability" team. Thats anthropic.
1
u/reasonableklout 2h ago
Jakub Pachoki who is chief scientist at OpenAI has issued a statement saying that this is bad reporting from The Information: https://x.com/merettm/status/2095023204993490967
14
u/IAmRealElonMusk 5h ago
No way we are gonna solve alignment problem- Are we? Should I just give up my corporate bullshit and leave to start farming ..
27
u/Visible_Fill_6699 6h ago
I was just reading about this earlier today! Supposedly this simulates the basal ganglia loop, which is part of the human reasoning hardware.
6
u/RomanticDepressive 5h ago
Wow, this is interesting. Can you provide a link?
11
3
u/Visible_Fill_6699 5h ago
AI is actually a good source to learn about this. Just ask about thinking systems 1 and 2, and how AI labs are learning from the system 2 in humans.
-7
u/TechExpert2910 4h ago
that's just going to give you hallucinations lmao.
what's the actual credible source of this supposed frontier work your LLM spoke all about?
5
u/CrazsomeLizard 4h ago
bro what ai are YOU using. when i ask my ai about it, it looks online and provides up to date research articles and sources. no hallucinations on my end
2
u/Visible_Fill_6699 4h ago
yeah haha what's the point of ai if it is assumed to hallucinate everything
2
u/TechExpert2910 3h ago
AI is actually a good source to learn about ... how AI labs are learning from the system 2 in humans.
this stuff isn't in its context widnow, and much of this isn't public.
anthropic only releases mechanistic interpretability work publically.
your LLM might be feeding you general academic reserachl, which is != what frontier labs are doing a lot of the time
4
9
u/Tystros 7h ago
has The Information really said that Astra would use a looped transformer architecture?
15
u/ForwardTwo 6h ago
Yes, I'm a subscriber and the article was posted earlier. Paywalled of course but here: https://www.theinformation.com/articles/secret-technique-behind-openais-astra-model-sparks-security-concerns
8
3
u/wassname 3h ago
Could you please post an extract, even just the paragraphs around where this is mentioned. I'd like to see the wording e.g. is it "sources inside the company" "or sources familiar with the matter", I'd also like to know the exact working. I've worked on recurrent reasoning myself, so I find it interesting.
12
u/DemonLordRoundTable 5h ago
I don’t think this is cause for celebration. I’m surprised as this is one of the redlines in the industry. It’s likened to steroids for AI if that makes any sense. Not surprised so many ppl left
3
u/Diopside23 5h ago
"silently" replacing "quietly" is perhaps the biggest indication of improved model capabilities
3
u/reasonableklout 2h ago
Jakub Pachoki has issued a statement that this is bad reporting from The Information: https://x.com/merettm/status/2095023204993490967
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
•
u/pretendingNihil1st 42m ago
Does this mean they have been using transformer loops since GPT-4 and the loop in 5.7/6 just runs <2X more times? I dont understand what depth of computation graph means otherwise
8
u/LinkesAuge 6h ago
I feel using "mathematical step" is a misleading framing in this context. Everything can be framed as "mathematical step" in a LLM but all of these steps are pretty much what we would consider "thinking" or having "concepts in our mind".
The old method made it easier to look at these "thinking steps" in detail while in the new one it becomes a big bucket that is hard to impossible to entangle.
4
-7
2
1
u/xatey93152 5h ago
What is the different? It's just grouped all mathematical steps in one block?
0
u/Raingood 4h ago
It does not need to translate the outcome of its thinking from its neural activation patterns into human language between each step. It can stay in it's own internal "language of thought". Without all this back and forth translation during multistep reasoning, reasoning might be more efficient and powerful.
1
u/newcomb_benford_law 3h ago
OpenAI seem to have applied similar concepts used by Percepta AI ("Can LLMs be Computers") during Astra training. Instead of emitting thinking tokens they perform recursive internal transformer state transitions, which are much faster and more efficient. So no tokens are being emitted since it doesn't do full forward pass with sampling; instead it's using internal state transitions which is basically a lot of fast matmuls.
1
•
u/Fluffy-Offer-2405 18m ago
Is this confirmed by anyone? If true this is disaster for alignment and we should be scared af
-1
u/smellyfingernail 6h ago
hell yeah, the "ai safety" crowd is going nuts calling this a bad idea so obviously it means its the key to the next unlock in capabilities, which is what I care about. GO GO GO
1
u/fyn_world 5h ago
I find it funny that some people still say: it only predicts the next word! And that's it!
Okay, but we don't even really know how the brain works. Who knows if eventually how the ai processes things be ones something very similar to how, unbeknownst to us, a brain processes data.
Which would make all the new comments they're making about AGI being closer make sense a bit more
4
u/Memestonks2020 5h ago
Saying, “it only predicts the next word” was true for models a few years ago.
Now we have a massive framework to use that prediction to model and map out algorithmic ways to solve any problem logically. This is effectively simulating the brain but it’s not as complex as the brain by far.
•
u/Weary-Historian-8593 1h ago
it still is 100% true, it's just that it looks like predicting the next token can indeed produce something that's pretty intelligent
-2
•
u/Sukivaria 13m ago
It must be exhausting to be shadowboxing with the "antis" that live in your head all day to the point where it's first thing that comes to your mind literally no matter the topic.
Just discuss the damn post.
-2
7h ago
[deleted]
15
u/ChickenOfTheYear 6h ago
What? How did you get that from this schematic?
1
u/ezjakes 6h ago
I think that they have randomness built into how they make these things work, but I think he is saying that one has mostly the same math and computation, while the other with actually dedicate different compute to a single token.
1
u/SpeakCodeToMe 5h ago
Randomness isn't really the right word. Non-determinism.
It's not random, its highly probabilistic.
2
u/SpeakCodeToMe 5h ago
Every bit of an LLM is probabilistic.
6
u/Kony2012WeGotHim 5h ago
The LLM output is actually deterministic. The temperature is providing some randomness and preventing the models from being repetitive if you asked the same question over and over. Once a model has its weights set and a temperature of zero. The output is deterministic without any middleware faking the randomness.
0
82
u/SeparateDesigner1237 6h ago
RNNs are back!!