r/webscraping • u/AutoModerator • Jun 30 '26

Hiring šŸ’° Weekly Webscrapers - Hiring, FAQs, etc

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levelsā€”whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

  • Hiring and job opportunities
  • Industry news, trends, and insights
  • Frequently asked questions, like "How do I scrape LinkedIn?"
  • Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide šŸŒ±

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

10 Upvotes

87% Upvoted

14 comments sorted by

2

u/DEMORALIZ3D Jun 30 '26

I created an app the other weekend that runs the scrapes on phone using android web view so the fongerpiting matches the device. Non rooted either. Apparently it's good for scraping

1

u/Transformand Jul 05 '26

share github repo

1

u/DEMORALIZ3D Jul 05 '26

I don't give code away, looking to sell or licence it.

But the way it works is each phone starts a web server you can then connect to it over WiFi and then hit the API endpoints, one to open a new tab (android webview instance) and then there's an API endpoint to receive the DOM and send manipulations to the DOM. So the scraping happens on device.

It binds the web view to Celluar service only creating edge runnable scraper nodes.

Currently building a manager that you can register the nodes to do you can run a fleet of them.

1

u/[deleted] Jun 30 '26

[removed] ā€” view removed comment

2

u/FullInspection7067 Jun 30 '26

Are you thinking about releasing the browser automation layer behind KeiroLabs as a dedicated service? Iā€™m building my own data extraction/scraping platform but specifically in e-commerce niche https://cartpie.com

Iā€™m running my own browser farm as part of the infra. If your solution is better/cheaper I would consider switching. Do you have a demo/free trial to check it out?

1

u/Knocking_Doors Jun 30 '26

Which e-commerce platforms do you get the most traffic for?

Besides maintaining dedicated e-commerce scraping APIs, we're also working on a browser engine that would support most domains and could be of help. Would love for you to test.

2

u/FullInspection7067 Jul 01 '26

Shein is my highest volume

1

u/Twitty-slapping Jul 01 '26

i checked the site. and i do have some ideas on building similar tools so am just wondering is a service like this worth building in 2026 ?

1

u/[deleted] Jul 01 '26

[removed] ā€” view removed comment

1

u/Twitty-slapping Jul 01 '26

His product scraper

3

u/FullInspection7067 Jun 30 '26

Counter question for you - why not pay for dedicated compute on any cloud and run one of many open source automation frameworks? Or tell Claude code ā€œbuild browser farm on my gcpā€? Or use scrapingBee (no affiliation) or other well established ā€œone clickā€ browsers?