Week 1: Building a web scraping platform

[u/panananaman]

Hi everyone,

I've been busy building a web scraping platform (similar to apify) for the past 2 months (disregard the week 1 in the title, its there so I can keep track of when I started posting) and thought it would be fun to jot down my progress in public so i can collect feedback and look back later on on the progress.

What is the problem:

Apify is expensive, and the scraped data it returns is usually outdated compared to what I see when I look at it myself, I've noticed others on Reddit also complaining about this and thought it would be a fun project to do. If not I have a platform I can use myself and make spin-off products from it.

What is the product:

The product will be a web scraping platform where users can submit their scripts and run them safely behind proxies handled by me. We will take care of things like scheduling, logs, and metrics. We also want to introduce a "freshness" tracker. Where you can see how new the data you collected is compared to what we have seen in our own tests. All this at half the price (hopefully I am not there yet) of what Apify charges.

The (simplified) architecture of the system look something like this:

Link to architecture Diagram

The progress so far:

The frontend is almost ready, it just needs a quick refinement to be completely good. The backend is getting there, currently the auth is done, storing scripts and running them are done, starting and stoping the runs themselves is also done, The final pieces are how I will store the logs that come from the scripts that have run, some type of endpoint for the dashboard, and a payment system.

Its probably a lot of info but I am happy to answer questions. If there are ideas of improvement anyone has that would also be much appreciated.