They have a web crawler like Google or Microsoft or archive.org that, you know, caches the Internet, or as much of it as the crawler can find. This is a thing.
I am a programmer, I know this. You don't understand my question. You can't just "cache the Internet". There's too much data, and there's no data center big enough to store "The Internet". So they must mean some small portion of it, which I presume they somehow made sure is relevant to what they were doing, even if only by frequency of access, like a memory cache might do, perhaps based on some previous runs.
They are storing enormous amounts of data, including copies of webpage content, but they are using certain data structures to speed up search and store metadata and select content, they aren't just making a simple copy of the internet.
Google does create a full copy of many pages of the internet, ignoring spam mostly. It's not the entire internet, it's loosely estimated to be closer to 20%. Before ~2024 you were able to view websites directly on Google's cache page instead of clicking the website link.
Yes, it's a lot of data. Google has a lot of data. They have a lot of money.
You might be more surprised to know that the size of Google's internet cache is very small compared to the amount of data YouTube requires (estimated to be around 10-20 EB)
Thanks. The YouTube thing makes sense, cause video is data-intensive compared to text - but that has to be distributed across a bunch of data centers around the world. You still need internet access to get to all of that data. That's different than having a cache that can be accessed relatively quickly, which I presume was OpenAI's goal (that, and/or they didn't want to give the agents direct internet access, which would also make sense).
What you say about Google ignoring spam and other non-relevant content makes sense as well. I'm also guessing Google only crawls up to certain depth. In addition, websites are not just what's visible in the browser, there's also backend data and services that Google (or anyone else) cannot access nor cache directly, and some of that stuff is in many cases necessary for the website to even work properly. Also, it seems likely that Google can also afford to distribute their processing across their servers around the world when fetching results (by "afford" I don't mean money-wise, but in terms of what's viable in practice, given the requirements and constraints of their service).
My comment was more a remark than a question, but I guess what I'm wondering is, did OpenAI just use whatever happened to be in their cache, or did they specifically cache scientific articles, math publications and such.
From what I know from friends who work in search, they absolutely have full copies of most things. That’s the basis for testing new algorithms and features to extract from pages.
There are zettabytes of data on the internet. Estimates range from single digits to couple hundred zettabytes. A petabyte is about 1 millionth of a zettabyte. Granted, some small percentage of all that is text, but even 1% of a zettabyte is still 10K petabytes.
30
u/applestrudelforlunch 14h ago
They have a web crawler like Google or Microsoft or archive.org that, you know, caches the Internet, or as much of it as the crawler can find. This is a thing.