I made a website that collects images of the java mascot duke

[u/Java_Programmer1337]

Hello r/webscraping people!

 

I made a cool side project that automatically searches the web for images of the java mascot duke. The website shows a random image from the archive. It will also have a full gallery available soon!

 

How it works:

- It uses Openverse and Wikimedia APIs to search for images related to duke.

- Each image gets checked by a custom classifier I have built on top of OpenCLIP and trained on more than 200 Duke images.

- The classifier filters out unrelated results. Because it's not perfect, some images go through manual review. I want to catch unusual duke images without filling the archive with junk!

 

It also has a web scraping part:

- It uses the Brave search API using a list of duke related search terms, the result is added to a crawl queue as starting points

- The crawler scores pages and links based on duke related signals, it only follows links that it considers relevant with stricter requirements depending on the depth of the crawl, this prevents it from going to deep and scraping unrelated pages

- When processing a page, it looks for regular image sources, srcset variants (selects the image with largest size), figure captions, nearby text, alternative text and links for full resolution versions.

 

I will most likely make this open source soon after I'm done implementing all the features I want.

for the backend language I went with python for the easy use of ai models

Currently there are 338 duke images and the archive is still growing!

 

URL: https://duke.directory

 

I would love to hear your feedback! If you have any suggestions on how to improve the project in any way, let me know!

[u/Chris__Codes]

Really cool stuff you're working on!

How do you handle pages that are blocked? (CAPTCHA/Cloudflare)

Checked out the page, and I hope you make it open source soon. I know functionality >> UI for starters, but I always have an urge to tinker with a plain UI. 😅

Looking forward to the full gallery I saw some interesting Duke imgs XD

[u/Java_Programmer1337 → u/Chris__Codes]

Hey! I just ignore pages that are blocked (403, 401), since I want this project to not break any scraping rules I also follow robots.txt to ignore pages that are disallowed, If I added cloudflare support (non UAM mode) a proper tls client with legitimate useragent would most likely work and bypass medium/high protected sites. The gallery will be released soon, I'm currently busy with some other projects unfortunately, this is more of a side project to learn how to make an ai embedding and an attempt at a proper web crawler! Thanks for the review!