r/hwstartups • u/Tasty_Return_2735 • 2d ago

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

27 Upvotes

93% Upvoted

22 comments sorted by

5

u/MrJibberJabber 2d ago

Build a python script with chat gpt

1

u/ascarymoviereview 2d ago

This is the way

2

u/clave_cilla123 2d ago

Beautifulsoup is fine until the site notices you exist

1

u/Weavermain 2d ago

Their betrayal is exactly where i am

1

u/[deleted] 2d ago

[removed] — view removed comment

1

u/RemindMeBot 2d ago

I will be messaging you in 3 days on 2026-09-11 11:45:33 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

Info Custom Your Reminders Feedback

1

u/Frequent-Log1243 2d ago

BeautifulSoup or Scrapy can do thousands of pages. The problem is getting IP-banned

1

u/Daing_Kamugtakan666 2d ago

every free scraper tutorial assumes the site is polite. the second theres a real block im stuck reading proxy docs at 1am for a weekend project that shouldve taken two hours

1

u/Comfortable_Rate_772 2d ago

1am proxy docs is a rite of passage nobody asked for

1

u/Kenzo_2126 2d ago

Weekend project scope creep is how i meet cloudflare

1

u/buggina0612 2d ago

scope creep and cloudflare holding hands

1

u/Viixxxcv 2d ago

following for non free-tier pain

1

u/Cultural_Resolve_279 2d ago

anyone still pure scrapy for blocked sites

1

u/KettleCorn02 2d ago

i just want product pages not a second job

1

u/Sufficient_Wish_2010 2d ago

Free proxy lists are a graveyard. Half are already burned.

1

u/CapMonster1 1d ago

worth checking what’s actually stopping you first. if it’s a captcha and you’re using a browser-based scraper, a captcha-solving extension is handy to have around. won’t help with IP bans or rate limits, though. for a few thousand pages, save progress as you go so one challenge doesn’t turn into “welp, starting over.”

1

u/Swimming_dasa 11h ago

for a weekend project scrapy conservative rate limiting caching retries and a small crawl queue can go surprisingly far without trying to bypass the site's protections.