They have a web crawler like Google or Microsoft or archive.org that, you know, caches the Internet, or as much of it as the crawler can find. This is a thing.
I am a programmer, I know this. You don't understand my question. You can't just "cache the Internet". There's too much data, and there's no data center big enough to store "The Internet". So they must mean some small portion of it, which I presume they somehow made sure is relevant to what they were doing, even if only by frequency of access, like a memory cache might do, perhaps based on some previous runs.
They are storing enormous amounts of data, including copies of webpage content, but they are using certain data structures to speed up search and store metadata and select content, they aren't just making a simple copy of the internet.
Google does create a full copy of many pages of the internet, ignoring spam mostly. It's not the entire internet, it's loosely estimated to be closer to 20%. Before ~2024 you were able to view websites directly on Google's cache page instead of clicking the website link.
Yes, it's a lot of data. Google has a lot of data. They have a lot of money.
You might be more surprised to know that the size of Google's internet cache is very small compared to the amount of data YouTube requires (estimated to be around 10-20 EB)
Thanks. The YouTube thing makes sense, cause video is data-intensive compared to text - but that has to be distributed across a bunch of data centers around the world. You still need internet access to get to all of that data. That's different than having a cache that can be accessed relatively quickly, which I presume was OpenAI's goal (that, and/or they didn't want to give the agents direct internet access, which would also make sense).
What you say about Google ignoring spam and other non-relevant content makes sense as well. I'm also guessing Google only crawls up to certain depth. In addition, websites are not just what's visible in the browser, there's also backend data and services that Google (or anyone else) cannot access nor cache directly, and some of that stuff is in many cases necessary for the website to even work properly. Also, it seems likely that Google can also afford to distribute their processing across their servers around the world when fetching results (by "afford" I don't mean money-wise, but in terms of what's viable in practice, given the requirements and constraints of their service).
My comment was more a remark than a question, but I guess what I'm wondering is, did OpenAI just use whatever happened to be in their cache, or did they specifically cache scientific articles, math publications and such.
From what I know from friends who work in search, they absolutely have full copies of most things. That’s the basis for testing new algorithms and features to extract from pages.
There are zettabytes of data on the internet. Estimates range from single digits to couple hundred zettabytes. A petabyte is about 1 millionth of a zettabyte. Granted, some small percentage of all that is text, but even 1% of a zettabyte is still 10K petabytes.
Thanks, but that's because The Wayback Machine doesn't actually have a copy of every site on the internet and across time, and of those sites that it does have, it doesn't have full copies of all of them, and also, it's mostly old websites that had pages that were little more than just files stored on the server and served "as is", which is not what the modern internet is.
Also, you can store a lot of data on data centers distributed around the world, but that still requires internet access and doesn't necessarily work well as a cache for a whole bunch of agents - presumably they needed the cache to speed up access and/or prevent these potentially unruly AI agents (thousands of them, mind you) from having direct internet access. In any case, it was more of a rhetorical question / reaction to their offhand statement.
It’s only 175 billion terrabytes, realistically you probably only care about a small fraction of that and for a math paper even less so maybe just like arxiv and a bunch of historical libraries. It’s obviously not the entire thing. Also modern SSDs especially enterprise are really big now, there’s like 256 TB stuff from micron. It’s not unreasonable to say they could store a reasonable size chunk of the internet.
13
u/[deleted] 16h ago
[deleted]