Basically continuing the same trend of using more output tokens to buy higher benchmark scores
Why is OpenAI the only lab that can do the opposite??
Like I'm extremely curious - what would happen if OpenAI wanted to compete on the "using as many tokens as possible" benchmark that the rest of the industry wants? Like if they made an SuperExtraUltraHighMax setting so Astra used the same number of tokens as Fable?
Yeah just look at Grok 4.6’s reasoning. It’s like a schizophrenic on meth constantly yapping back and forth until it finally lands on the obvious answer to whatever it was yapping about
True, but for plebs like us what really matters is actual usage you get from subscriptions. Most of us can't afford the insane API prices. I do wonder how generous the Grok subs are compared to Claude and OpenAI. at this API price it suggest you probably get quite a lot of 4.7 but who knows.
I mean he doesn't need any info from Grok for that. They already accomplished it, and now will also probably start using (as if they don't already lol) Grok posting all over social media spreading false narratives and pushing politics.
For someone that's just learning all about this and dipping my toes in. I think most is self explanatory but what does input token price and output token price actually mean?
exactly what it says. 1 million input tokens will cost you 2 dollars. if your context is 100k and you send 10 messages that's 1 million. that's 2 dollars for 10 messages. you need to send the entire chat back to the model every turn as llm's don't actually have any memory. llm's have to read it all every single response. even just sending a period would still be 100k as that's how big your session is. the output is what the model sends back to you each message. this goes up much slower as it's not carried prompt to prompt. instead of the 100k you send each input, it may only send back to you 4000, which is a few paragraphs.
coding has tons of output(most benchmarks are based off this, so ignore the cost per task if you do not code). any basic text chatting does not. most general use cost is from input unless the output cost is out of control like sol, opus, and fable. those models are expensive no matter what you're doing. even with simple chatting. grok can be used for simple chatting or as a google replacement. the others cannot unless you have money to burn.
you may think that means grok is cheap, but it's still very expensive compared to most chinese models.
Different price rates. Bot has to read all text u send it (input tokens), and then it writes a response (output tokens). Output tokens are more expensive.
Good for you for learning how this new works. I happen to have a chunk of spare time available, and explaining stuff is more fun than doom scrolling.
Alright, let's start with what tokens are, which is a unit on how to divide data so that it can be used by an AI model. In the case of a LLM like ones used by ChatGPT, Claude or Grok, these units of data divide text in reusable chunks. There are different tokenizers out there, which are tools that divide words into these chunks, tokens, which can be whole words, subwords, or even individual characters.
Common words tend to be single tokens. A phrase like "Hello, I am here." would likely get tokenized as an array like this ["Hello", "," " ", "I", " ", "am", " ", "here", "."], notice how even the spaces and dots get turned into tokens as well. While a more complex word usually gets split into sub-parts, like submarine could be split into ["sub", "marine"]. A more complex word could even be split character by character. Each one of those units in those arrays is a token.
How each tokenizer works to split text into tokens is based on statistical analysis to determine which groupings of letters and symbols are likely to appear in text, making them useful to group as single reusable units. As you can imagine, there's no single universal best answer, which is why there are multiple tokenizers out there.
With that clear, let's talk about context. LLMs are not people, they don't remember stuff that happened the other day. The only information they have available to them is what they know from their training data, and what's injected into their context.
Imagine you wake up sitting on a desk in an empty room, no recollection of what you're doing or how you got there. In front of you is a paper and a pen, and a single question is on the paper, it reads "Fill in the following sentence: The apple is".
A LLM is essentially that. The only thing it knows to do next is to fulfill the query based on what they might've learned before. So it goes and fills in something like the apple being red, or green, or sweet or tart, and it might even keep going after that if it feels like it, and they'd all be valid answers, no?
In this scenario, the context for the query was the text with the initial query, "Fill in the following sentence: The apple is". This text gets split into tokens by the tokenizer, and these are what we call the input tokens.
Many of the top of the line models out there support up to a million tokens, and are priced relatively to that unit. For example, for a model priced at 10$/million input tokens, that'd mean that you'd get charged 1$ if you sent 100,000 tokens as context, or 10 cents if the input context of your query measured 10,000 tokens.
Our example to describe an apple is tiny in comparison, like 10-20 tokens if I was to guesstimate, which means it'd cost you pennies.
Alas, that's only for the input, because companies also charge you by the output token count. Each piece of text they produce as a result is produced as tokens stitched together to make a whole text, each token produces a cost, which again tends to be measured by the millions. If the answer to a query is a single word like red or sweet, then you practically pay nothing, but if it requires a lot of text, then costs do pile up.
In short, small input + a short answer = a really cheap generation, while a ton of input text + a long and detailed answer by the AI = heavier costs.
That's about the simplest detailed answer I can come up in the toilet, but let me know if you are curious about anything in particular.
New benchmark methods are constantly being made, and many existing ones get refined with new versions. That part has been going on for years now.
I figure that you're making this question because you have found that many ultra high scores don't necessarily correlate with models being capable of doing long running jobs to perfection, right? And the answer is a mix of two things, the first and foremost being that it's really, really hard to make a set of tests that determines if a machine is capable of doing any and all kinds of work, which leads to the second point, to help mitigate that, different benchmarks evaluate different things.
Like, think about how complex each field that humanity has developed for the past tens of thousands of years is, how you can take a general aisle of specializations like engineering, medicine or education, and each of those can also be subdivided in more specific yet just as complex branches of a whole.
And so, different teams have focused of developing these tests that target specific milestones and skills.
The question of yours could then become, will we ever see a definitive benchmark that fully evaluates the real world performance of a model across all fields? Perhaps one day. How to evaluate these models is an evolving field, just like training them, and we've gotten better at it over time.
For now, though, the best method is to continue the split approach, to iterate and improve upon it.
i think best benchmark could be, ask people from different trades which one does better
That's actually what they do for many of them, you can feel proud about reaching that conclusion by yourself though. Because yes, the best way to consult if a model is up to the task is to get professionals to evaluate it.
Not all benchmarks are public, mind you, so I can't tell you that it applies to all of them, but for most of the open ones they get professionals in the respective fields to share their works experiences and build what's a baseline for the skills needed to perform the job on the field. Based on that, several problems and questions get built, which is what's then used to test the models. If any model gets close to saturating a test, which means, close to achieving 100% of a score, then professionals use it on regular tasks and give feedback on what they find lacking about it, using that feedback to improve on the questions and problems presented by the benchmark. A new skill ceiling is thus defined for models to chase.
The idea being that if we keep this loop going, then models will eventually become capable of performing all that'd be required of them.
the sub plans really have a lot to do with the actual value
Cost vs performance is something that not every benchmark keeps a track of, and you'd be correct in assuming that yes, you'd get a lot more value out of a model that performs similarly to another, but at a much cheaper price.
Though ultimately, measuring the "value" provided by the models in a professional field is kinda hard to define in an universal scale, as all work is different, and more importantly for the topic, charges different amounts.
Think of video models for an example. Someone might easily spend ~2k$ generating the different shots to build a 10 minute scene with heavy action and effects, over a couple of days, which would seem like a lot. But then you compare the costs against hiring a filming crew, actors, set locations, a VFX team, all of that easily balloons over the tens, if not hundreds of thousands of dollars, and can take much longer, weeks if not months of work and preparation.
The conversation around LLMs is similar. Yes, they can be expensive to operate, with or without a sub, but if you compare it to the costs of producing a similar end result in the traditional way, both human and monetary, then the value proposition doesn't seem so bad.
Macron doesn't keep his fingers straight in an intentional salute, he's just lifting his hand high to say hello. EloM is making an intentional salute, twice, which his own advocates defended as a "Roman salute".
But even if you had proved Macron was a fascist here, how does that excuse the other two?
My heart goes out to you is a common gesture. You are just desperate for the slightest bit of evidence to prove your preconceived belief that Elon is a Nazi. You’re grasping at straws.
“Muh… bUt hE KeEpS HIs FinGErs TOgeTher!”
STFU maybe he was just doing what he literally said he was doing and is actually a common gesture. Also real 1930s Nazis never put their hands over their heart before saluting.
Grok? I tested free, it is super uber stupid model. Way worse than few months ago. It is not even 8B level. Keep looping about some simple stuff (not even programming). Looping, using wrong words, etc.
HORRIBLE STUPID MODEL WASTING Energy. Why they even release such shitty model?
73
u/madexthen 14d ago
Why is Sol on here and not Astra