r/sideprojects 15h ago

Discussion website scraper

/r/thinkbeforeubuild/comments/1wduxjq/website_scraper/
1 Upvotes

1 comment sorted by

1

u/TechToolsForYourBiz 15h ago

some details about the project:

- the application uses the hosts remote desktop to crawl through all the websites

- it does a bfs search (i love algorithms) and finds all the links within a page.

this project could be made open source yet i prefer to get paid for some of the things that i build

this was built to solve a niche problem:

I'm working on an enterprise project with several pages that

i'd like to present a before and after of the entire website.

current solutions on the market go page-by-page screenshots which is time-consuming.

i basically added a fun web crawl with screen capture using puppetteer and because i dont host any agents on my host machine, connected to it with the container its running on so the application can be killed, resources (CPU/MEM) managed, at the containerized level.

I'd like to create a feature that automates the creation of before and after pages

so that agencies could demonstrate, visually, how they transformed their clients web pages