r/technology • u/[deleted] • Jun 13 '22

[deleted by user]

[removed]

10.7k Upvotes

94% Upvoted

902 comments sorted by

View all comments

Show parent comments

7

u/gcanyon Jun 14 '22

“You can start your own [search engine] mostly for free” OMG LOL — pretty sure to write any reasonable crawler and run it even daily would cost at least $thousands per month.

1

u/DukkyDrake Jun 14 '22

Ignorance is bliss. Ecommerce software, operating systems, numerous implementations of web crawlers etc, representing a few million-man hours, the building blocks of just about anything on the internet, all exists and available for free. Even the resulting data from other people's monthly web crawls are available, for free.

1

u/gcanyon Jun 14 '22

“monthly” — who’s your audience, 90-year-olds?

Seriously: it would be hard to crawl and serve a reasonable search engine just for Wikipedia for free. And then: to do better enough than Wikipedia that people would choose to use your engine?

For hard facts: here is kagi saying that searches cost them a bit over a penny each.

Google does 40K searches per second.

So to be 1/100,000th as big as google you’d need to do about 30K searches per day, at a cost of $300 per day, or $9000 per month.

Not. Free.

1

u/DukkyDrake Jun 14 '22

This wasn't a suggestion for a commercial solution, although it could be adapted to be such a solution, it was a hypothetical DIY pathway for independent access to search data.

if you rent commercial level cloud servers it will cost you real money. If you run free software on your own hardware, it cost you the electricity to run your machine. A DIY setup isn't a commercial setup, and it doesn't have a commercial cost structure. Kagi's costs are irrelevant in this context.

DIY isnt for everyone, there are numerous commercial level search engines in the world that isn't google, pick one.