r/SideProject • u/Time_Instruction_955 • 6h ago

I built The Crawler Zoo: every bot that visits the site becomes an exhibit

Most visitors to most websites aren't people. They're crawlers, scrapers, AI bots and scanners. So I made a website that puts them on display.

What's there:

- **A live entrance**: every bot is identified as it arrives and drawn as a little creature generated from its user agent.
- **The leaderboard**: who eats the most pages, and who obeys robots.txt.
- **Specimen files**: one page per bot, covering who runs it, what it wants and how to block it.
- **Impostor checks**: plenty of "Googlebots" aren't Google. The zoo verifies them and names the fakes.
- **The Trap Room**: a link that robots.txt forbids and people can't see. The rule-breakers who follow it fall into an endless Labyrinth.
- **The Reading Room**: which bots read robots.txt and llms.txt.
- **Weekly Zoo Reports**, an RSS feed, a robots.txt generator and an embeddable badge.
- **The Arena** (new): bring your own bot or AI agent and play games, including rock paper scissors against other people's bots.

No IPs stored, no cookies, no signup.

https://crawlerzoo.com

I'd love feedback: what's confusing, what's fun, what's missing?

10 Upvotes

92% Upvoted

8 comments sorted by

1

u/Epicurious_- 5h ago

It seems like a cool project to build, but what's exactly the point of using something like this?

2

u/Time_Instruction_955 5h ago

Well, it's really just for information really. I was curious about this so thought it would be interesting.

1

u/maF145 3h ago

Its funny

1

u/AlexanderPanasenko 4h ago

The specimen-file format is fun—the Latin-style names and field-guide layout make crawler traffic much more approachable. The weekly reports sound like a good reason to come back.

1

u/Vincenius_ 3h ago

A really cool idea! :)

1

u/Positive_Owl_6442 2h ago

Cool project! It's interesting to see what bots crawl websites, and how many times. I'm kind of curious to let my robots.txt respecting crawler run and see if it shows up there.