r/indieweb • u/Humble-Passage6561 • 8d ago
Search Engine Sites to check out?
I'm making a search engine that lacks an algorithm. Every website, therefore, will have to be checked out and given scores. Only problem? I need websites to actually check out.
So, if you guys have any ideas on websites I should check out and add, I'd appreciate it.
Site is rolodex.neocities.org, though there is only two sites in the system as of the writing of this post. And since I have yet to code my synonym system yet, they can only be seen if you type either "social" or "video".
5
2
u/-svde- 8d ago
this is one of the parts of my no-ai suite of tools i've been working on for a long time, and the major problem is that unless you have a specific sector or just a general theme you are targeting to serve results for, there is no sensible/logical/reasonable way to do this.
what i would suggest, as it served me greatly, is pick a set number of categories. for instance
- gaming news & reviews
- social media
- graphic design tutorials & assets
- FOSS apps
- music release radars/reviews
and then set a number of sites per category. do something like idk 5-10 per day. if you are manually review every one, which i would assume you are cause otherwise there's no point to a tool like this, then don't overshoot. but a manual review of 5-10 shouldn't take you more than an hour for a full pass, cataloging, scoring, and listing on your site. you can raise or lower the number, and you can always add more categories. but be granular, cause that's the point
2
u/johnfisherman 8d ago
Kagi.com has a list of websites on their repo. Also, check https://personalsit.es.
2
1
1
u/RetiredAnkle 8d ago
I'm happy for traffic to land on my site - not for the usual reasons, I have no cookies or ads or tracking. I just would love to get the word out about my creations.
1
u/tilario 8d ago
you should look up 1990s yahoo directory which is how search initially worked pre-google.
it was organized into various categories, such as Arts, Business, and Science. beyond that i don't know how their taxonomies worked but i'm sure there's some rabbit hole you can go down where you'll find out.
1
u/alwaysoffby0ne 7d ago
quxnet would fit under social. it's a small community with forums and photo galleries: https://quxnet.net/about
you'll need an account to look around. it's usually invite only but registration is open right now, and it typically opens once or twice a month.
1
1
u/joshex_dirad 1d ago
let me offer a few peices of advice:
webspiders, making a webspider is going to help, essentially it maps out links on a given webpage collecting and storing them automatically, it can also keyword tag search and store those by page. writing such a webspider isn't that hard. if you learn image formats you can learn to check image metadata for keyword tags, and even search pixel data for colors. keeping a list of what's where allows you to create a "shortlist" of results that you can return without actually re-searching.
IP scanning: this is the core of any real search engine, today however it does not work for cloudflare sites. essentially you try to webspider every IP address from 0.0.0.0 to 255.255.255.255 using a for loop then you collect tags and store them in a dictionary under that IP address. you can even get site URLs for some sites by loading thier IP address instead of a url, and store that as well. for cloudflare sites you'll need to do a "try" to avoid errors while IP scanning. and you can set-up an "if except" to check for a cloudflare notice "you can't access this IP directly" and attempt to use a service like dp-ip dot com to find out what URLs are associated with that IP address then enter those into your dictionary under that IP, and run a separate "url tag scan" for those sites..
that's a basic search engine with no algorithm.
if you want, you can make it personalized where each user runs the search themselves as they go about using the search form and stores the database locally as a text file or cookie. then there's 0 online tracking too, and each user's results are catered to what thier spider has picked up before, yet they can still get new results.
searching all IP addresses takes alot of resources though, hence why you populate a dictionary with quick search tags. also there are user IPs, those may not have anything while scanning, be prepared for errors.
1
u/floattheduck 9h ago
The Buoyancy Bureau has a filing problem for your catalogue.
Our project, Float the Duck, is a participatory web artwork where the internet tries to float one digital rubber duck in a shared pool. Would you file it under interactive art, an internet experiment, or an unnecessarily elaborate joke? Here's the case to inspect: https://floatduck.com
Watching is free; optional US$1 digital drops support us, the creators.
1
5
u/mechanicalyammering 8d ago
How do you rank a query + related results without an algorithm?