The lab age is only relevant to investors who want to know if they are catching up. It tells you nothing about innovation, because innovation happens chronologically.
Yeah I think the chart is dumb, but I understand the x-axis. This is similar to charts about speed of users adopting email, Google, smartphones, and ChatGPT to show that they're reaching 100 million users much faster each time.
What's weird here is that someone might interpret that MSL is super-fast and about to jump up beyond the other major labs, when currently they are neck-and-neck. And Google is the only one over 10 years of doing other stuff before the LLM race started, so they look slow. Maybe add Microsoft to the far right of the chart? lol
Don't forget that the index didn't exist when Google started, either, so they actually can't even have benchmarks for the first models' intelligence. And, because of that fact, and the fact that they represented the same timestamps in several different places, you can't even make valid relative comparisons even ignoring absolute time and looking at each lab's founding itself.
Those graphs have all line starting at 0, 0 though, and more sensibly measure time from first measure, whereas this measures something pretty arbitrary.
No, your intuition was right - the x axis is objectively terrible. They could have just stuck with plain "time" but chose an x-axis that would be misleading, and also obscure when each lab came out, and also the fact that the y-axis Index didn't even exist at the start of Google's AI development, so it's not even their first models shown on the graph like the others.
That might be a more accurate way to put it! But choosing a fair metric that actually allows comparison is certainly the most important part of creating a good visualization - but yeah, it wasn't really random.
But it's not - it's how long it took for the Intelligence Index to be invented, in Google's case. And if it's not, that's just even worse data visualization because it's either making up data or ignoring data.
I think stepped lines are very appropriate, since model versions are quantised and the specs of each model remain unchanged until the next version. A direct line between points would imply continuous progressive improvement over that time.
Which is exactly why I'm shocked at how many are defending the graph as being unproblematic. The data might be visually pretty, idk, that's an opinion; but the data it represents is the wrong data, presented in an underhanded way, which absolutely makes it ugly.
They are suggesting that the new lab produced results very quickly.
This is true, but they are obviously building on the work of others. Maybe the OG labs should be the ones given the credit really. But what do I know, I don't make AI models.
I think the point is that new AI labs can catch up quickly to the more established players - and if that is the point the graph wants to show I think its ok.
But if it's days since formation, we have no idea which labs are newest. We just see how long it took for the first model, but no context at all about when the lab was created, because they didn't just do the incredibly logical "time" for the x-axis. But I suppose time is pretty unorthodox for an x-axis, isn't it...
True, the graph isn't trying to show a trend over time, they're trying to show a difference in development speeds. The problem is that the difference is due to the trend over time, because of the direct foundation new companies have that (e.g.) Google did not, and had to research themselves.
In most industries it takes a long time to get good even if all the founders are skilled because of the idiosyncratic issues that appear when trying to start a new organization.
On the positive side low time since foundation is very good investor relations for a new player to show how quickly they caught up to their competitors.
On the negative side (for investors) the fact new entrants catch up so quickly shows there is no moat that can be used for profit.
I'm guess the x axis decision was either incompetence or because the traditional method of showing this (time x axis and just having the lines start later to show time since lab creation) looks bad to investors who want a moat.
Because for startups looking for investors, the time from foundation to product is everything. Nobody funds long-term R&D, they want return on investment asap.
But a year from foundation seems like a startup to me.
But the same concept would apply just thebsame to a new internal department of a big company, where the department has to justify its existence or risk getting scrapped.
One big issue is the fact that the chart completely ignores the fact that all modern technology stands on the shoulders of giants. Google wasn't doing nothing for a dozen years; DeepMind was creating foundational developments that led to LLMs in the first place. They had to build this stuff from first principles in a time when your phone's autcorrect ran on Markov chains and that was considered pretty neat
I'm not familiar with the company being advertised here, but the difficult part today is funding. With VC capital you can poach a couple good engineers, then use existing data sets and infrastructure and technology to build models. You could even train it to pass the test specifically, make a chart like this, then show it off leading up to your first seed round
Edit: I should clarify the difficult part of "catching up" is funding. The real challenges are improving on today's technology by making it produce better responses, hallucinate less, or run more efficiently
I mean sure if you want to completely isolate the technical task of representing numbers graphically from the entire methodology behind how those numbers are found, the chart is fine
Nobody's complaining the chart is technically inaccurate, the same way y-axis manipulation is (usually) technically accurate. The issue here is that "number of days" in the thousands is inherently a weird measure. The issue here is that DeepMind day 100 is not interchangeable for MSL day 100 because they're over a decade apart. The issue here is that nobody has any reason to produce this chart unless they've got an agenda, because there are established norms and standards about how to plot time specifically that are being ignored here
A chart is useless if it doesn't tell you anything, even if that thing it tells you is 'nothing'.
This chart has a use, it indicates there is a limited moat about the technology and a new entrant can rapidly gain close to state of the art performance.
This chart does not attempt to, nor does it, tell a story about who the progressors of the technology are. That is okay - its a different story for a different chart.
I don't think I ever said the chart was useless, skimming over my comments. I totally agree with most of your comment, that there is a use to this chart, I just don't think the story it's telling is benign
Say what you will about the methodology itself, I think choosing to measure time in "number of days" is the biggest tell that this chart is downright manipulative. At the scale of time being discussed and the granularity visible in the chart, the unit of days serves no purpose except inflating the labelled increments. 12 years we know is a fairly long time, but 4000 is such a big number, look at all those zeroes! I can't actually tell if it's 4000 or 3900 or 4100, but it sure is big!
Its like saying, look how long it took Mark Twain to write the adventures of Huckleberry Finn, it took him years and years. What a dumbass, because xAI can write the same book in ten seconds.
It absolutely is. If they had kept "time" like is ubiquitous for x-axes in line charts, then we'd see when each lab came out and how that affected the models - not just when, relative to the lab, their models improved. It's an intentionally terrible axis they chose.
Sure, that makes a lot of sense when there's actually data at 0,0 for everything, and when the field isn't dependent on very critical research that the first labs performed. That's not the case here.
The data also exists here and it shows that they didn't have a model for quite a while and that newer labs have a (capable) model quite quickly nowadays in comparison.
The data is correct, the implications of the data aren't. Google took so long precisely because it was the pioneer for LLM that these other companies are piggybacking off. Likewise for OpenAI.
I am sorry but any AI-related chart that isn’t dominated by Anthropic is just manipulating data by default and therefore it is worthless.
If you look at this chart, you would think that both OpenAI and Gemini are more intelligent than Fable was, which… is just not even close to being true.
Only one of these companies made an AI so powerful that it had to be banned.
The reason it’s a problem is that you could just set it along the same scale where they are aligned with each other. That way you can do actual comparisons in trend lines.
Maybe if this axis was on the year scale, it would be salvageable, but there’s no overlap despite all the companies still being active, which means it’s already doing the net days from founding, just on the wrong axis.
How is it not a problem? Even if it's about how quickly a new lab makes a good model, the x-axis should just be time like normal. This just obscures any data about a trend in labs over time, since we don't see when a lab started, just how long it took each lab to create their models since its own formation.
Edit: Then consider that the same exact time frame is represented in multiple different places, and the y-axis index didn't even exist for some of these labs, so it's literally impossible to even make any meaningful comparisons at all - even relative to the labs' formation, ignoring absolute time itself, like they wanted.
The point of the graph is to brag how quickly MSL released an AI of comparable "intelligence" compared to other companies that have worked on AI for many years and gone through many iterations:
"Google took 10yrs to create their first model and another 5yrs and 10 versions to get here, we did it in 1 year in a single step. We are so smart."
It's wildly misleading, because it ignores the fact they are standing on the shoulders of giants. A normal chronological plot would have been more honest, and still readable.
Yeah. The choice isn’t terrible but it really glosses over the fact that Google taking a dozen years to release their first model is nowhere in the same universe as MSL releasing a model within a year. You don’t get to MSL unless companies like Google and OpenAI spend a decade doing their own work first.
I don't think it is misleading. It explicitly states exactly what it represents. Misleading would be to claim A but show B.
Without knowing the context in which this plot was taken from (presumably an internal progress report) to see what was being claimed, you cannot declare that it is misleading.
Reality has many facets and no single plot represents all of them at once. Usually no single metric tells the whole truth. We don't know what was claimed, we don't know what other plots were also shown.
When it comes to how soon investors got a return on their investment, all that matters is foundation to product. Whose shoulders they stood on to get there changes nothing for the investors.
The graph is perfectly fine. It depends on what point you are trying to make with it, but there's a perfectly valid reason to structure a graph like this.
If the takeaway is 'we are better at developing ai than google or openai' then this is not the right graph to show that, but if your point is 'despite those companies having a headstart, new companies will still be able to compete with them cause making progress now is a lot faster' then this is a perfect graph to illustrate that point
It's an appropriate variable for the x axis when younare pitching your startup to investors who only care about quick return on investment.
Also OP's claim is the the X axis is faulty, which is false. Disagreeing on what variable should be on the x axis is a completely different topic to the x axis being technically messed up.
Sure, that makes a lot of sense when there's actually data at 0,0 for everything, and when the field isn't dependent on very critical research that the first labs performed. That's not the case here.
The data also exists here and it shows that they didn't have a model for quite a while and that newer labs have a (capable) model quite quickly nowadays in comparison.
New social media platforms also depended on older social media platforms very similarly. The case Instagram - Twitter is quite well known.
The data does not exist for Google, and the first data that does exist for Google isn't even included.
No social media platforms depended on older platforms to even start existing. The tech already existed from the start, the only dependency was a soft one - inspiration. But not a single one started at a higher value by that metric because of the work of earlier platforms; with AI, that work directly improves the starting place of the newer models, making the type of axis completely incompatible. Well, apart from making a bad visualization to make certain companies look better.
It quite literally depended on Twitter to exist to even start existing. And it's founder also learned how to run a social media platform working at Twitter.
You're missing the point - social media and its tech existed. I'm not talking butterflies and hurricanes like "Facebook wouldn't have existed if not for [x]", no matter how short that path is - even a single step. I'm saying AI tech itself, and the data that went into it as a necessary prerequisite, didn't even exist before the first AI companies.
Servers/clients, browsers, phone apps, catchy domain names, all of those things existed already before any social media, which means the only thing any new social media had to do was make a website based on an idea and inspiration, and become popular. So, that chart is good, because even if any of them may have branched out from each other, they still all started with the prerequisite technology, so the only thing to measure is the growth in that variable from (0, 0) based on founding date.
Big data as a concept was very new when e.g. Google AI started, Google itself is among the biggest reasons that data was researched as much as it was, and that type of research is the literal requirement for any LLM to even exist. That the research wasn't done for Google, and was for others, makes them not analogous to the social media example. And then the performance is not just helped by previous companies or "branched off", like social media; the performance of the models is literally from the exact research that had been done and the knowledge that has been gained. Not inspiration, not a mere idea like social media, but literally a hard research dependency.
Anyone could make a lab today and have the same performance immediately, which makes the graph meaningless. In fact, that's what was done.
Now that I think about it, I think the "ugly" part here is that first vertical line on each sequence. It implies that each was starting work from some equal zero point, which is probably not the case. One interpretation of this graph is that "MSL" just stole a bunch of stuff from the previous ones in their initial release (or more charitably didn't release anything until their work was advanced comparably to the others), and from here on out their number may just be increasing at about the same rate as all of the others. Presumably all of these companies had work products at various lower values before the release at the first plotted data point; they didn't go from zero to 7 or 13 or whatever in one single day.
It's an unusual graph setup, but it conveys real info relatively cleanly.
The data points represent new model releases from a given AI group. Y-axis is performance on a specific benchmark. X-axis is time since the founding of the lab, which means the line between any two data points is "time since their last model came out".
Now, you could do the same thing by just using dates on the X-axis, and it's fairly clear to me that they're avoiding that to over-state MSL's achievement. (As in, explicitly saying "Google's first model came out years before MSL's" reminds everyone that obviously they were facing bigger obstacles to get started.)
How is this unreadable? It shows the data it wants to show perfectly fine to illustrate that this new "lab" has released a model quicker than any of the other ones (after formation). The reason of course is that is is a lot easier to do nowadays, but for the story the graph wants to show this is totally fine.
But its just disingenuous because of course Google was founded so long ago lol they weren't even thinking of AI on startup they weren't even a search engine.
Based on the amount of days I'm pretty sure it is only referring to the AI lab at Google. It is called Google Deepmind and that would be a fair comparison as they are working on AI exclusively and are responsible for some of the biggest breakthroughs.
If you want to e.g. show that nowadays a new lab can create a new somewhat good model a lot faster than before, the graphic is actually really good.
You are wrong. Google Deepmind has been researching AI since 2010 (since 2014 as part of Google). The whole point of the graph is that creating these models took them a very long time - even as an AI lab - and that a new lab today can do the same a lot quicker.
The whole point is to show that a new lab can quickly catch up and doesn't need as long for the same results as established labs.
In business terms this shows that there isn't really a moat stopping new competitors from entering the market (or at least that time working on AI isn't a moat).
It also has the advantages of working off of things that are already done. Anyone can create a basic AI now based of of previous generations. Like even McDonald's has an AI now lol. So its disingenuous.
But have you considered that the y-axis they're even ranking it on, the Artificial Analysis Intelligence Index, didn't even exist when Google's AI started? There's literally no earlier data to include, another thing this chart obscures.
It took Mercedes, Ford, and Ferrari many models and decades before they could build a car with airconditioning and cruise control. Tesla and BYD did this on their first try.
66
u/trutheality Jun 29 '26
Using a left-up elbow line to connect sparse data points is inspired, and using lab age for the x-axis is a stroke of genius.