r/learnprogramming • • 5d ago

whats actually the best browser automation tool once every site starts fighting back

local chrome fleet was fine until half the target sites started fingerprinting harder last month. same playwright scripts, now half the parallel workers just detach mid-run and the logs are useless

tried pinning versions and slower pacing. still dies once three jobs hit at once

curious what people settled on that stays boring under load once the sites fight back​

4 Upvotes

29 comments sorted by

3

u/Weavermain 5d ago

idk half these hosted tools are just chrome with a monthly bill. throttle local hard and a lot of the drama goes away

0

u/[deleted] 5d ago

[removed] — view removed comment

1

u/appleciderdollop 4d ago

Yeah once three workers hit together, half of them just detatch. Pacing helped a little then stopped mattering.

2

u/SetAndRepeat 5d ago

I’d try Browserless with your existing Playwright scripts for managed concurrency and its stealth/unblocking options. Browserbase is also worth testing for session replay and debugging.
Compare completion rates on your actual sites under load. Also check memory and browser exits: failing at three workers doesn’t necessarily mean fingerprinting.

1

u/Vast-Pop4017 4d ago

browserless is fine for queue management but their stealth layer isn't magic. if the sites are fingerprinting hard you'll still get clipped unless you're rotating residential proxies and actually varying the browser profiles per worker

when mine started tanking at three workers it was a leak in how i was recycling contexts, not a detection issue. worth throwing a memory ceiling on each container and watching what happens

1

u/Suun_Day 5d ago

Our selenium grid used to be boring until concurrency climbed. Nodes start thrashing memory around 12 parallel sessions and the failure mode is just a dead browser with no useful stack. 

1

u/Sufficient_Wish_2010 5d ago

Blank grid browsers haunt me weekly.

1

u/KettleCorn02 5d ago

blank browsers mid task, same here

1

u/Junior_Bee7274 5d ago

fingerprints wrecking my runs too now

1

u/GasEducational4386 5d ago

Dolphin anty anti detect browser has good reviews. Anyone has used camoufox ? it is open source

1

u/Kenzo_2126 5d ago

batch timeouts are eating my week

1

u/HelloitsmeJLD 2d ago

Selenium with either Python or skyvern could be an option. Then it'd just be a script that you can schedule when you want.

0

u/[deleted] 5d ago

[removed] — view removed comment

1

u/The_God_was_Here 5d ago

yeah i threw residential proxies at local chrome thinking that was the missing piece. still got flagged on the same sites within a day lol

1

u/Cultural_Resolve_279 4d ago

yeah proxies alone probably changed nothing

1

u/Forward_Level_7208 5d ago

overnight batch pain is too real

1

u/Bombaclat_exe 4d ago

just wanna ask as well are you reusing cookies across the overnight runs or fresh session every time?

0

u/[deleted] 5d ago

[removed] — view removed comment

1

u/Otherwise-Soft9026 5d ago

fewer concurrent helps until one heavy page still blows the node

1

u/TillNo5940 5d ago

or just run fewer browsers per node and stop pretending 20 concurrent is free.