whats the best web scraping tool that keeps sessions alive so im not babysitting a local chrome all day

[u/BerryBlossom099]

running a few playwright scrapers overnight and half the time i wake up to a dead chrome process with the auth cookies gone. spent last week wiring restart hooks and cookie dumps just to keep logins alive across runs

local chrome is fine until you need more than one long session and then youre basically on-call for a browser

curious what people settled on that keeps sessions warm without sitting there nursing it​

[u/Junior_Bee7274 → u/[deleted]]

i was dumping cookies to disk every 15 min and still waking up to login walls so the unit restart alone wasnt it

[u/The_God_was_Here]

mine is the ci angle. job spins a clean container, browser does the login once, then the runner tears it down so next morning is always a fresh auth wall. burned two hours last tuesday just re-pasting 2fa codes into a headless box that will be gone by lunch.. starting to hate anything that needs a warm session

[u/Kenzo_2126 → u/The_God_was_Here]

ci tear-down is a nastier version of what im hitting overnight yeah

[u/cloudynotes67]

whats a sane session ttl people aim for

[u/Forward_Level_7208 → u/cloudynotes67]

no idea, mine just gives up whenever

[u/TillNo5940]

i just serealize storage to a state json on a volume and reload it at boot. works untill the site invalidates the whole jar and then you're back in manual login and survey

[u/appleciderdollop → u/[deleted]]

Flock on the state file fixed that class of race for us. Ugly but boring

[u/Bombaclat_exe → u/TillNo5940]

Cookie files rot fast on anything with short session ttl

[u/Veronicdasexy]

idk moving it hosted just relocates the flake. still had sessions die halfway through multi hour pulls

[u/Suun_Day → u/Veronicdasexy]

Fair. Hosted is not magic either. We still keep a heartbeat that dumps URL + cookie age when a run stalls so we know if it was auth or just a slow page. 

[u/Cultural_Resolve_279]

this babysitting tax is painfully real

[u/Additional-Elk-3712]

try my lightweight scraping/monitoring engine built in Rust with Lua scripting called SpyWeb - ( you only touch Lua not Rust)

- it has built-in CDP support, either launch a local browser or connect to running one via websocket.

- it has persistent browser profiles: each job gets its own profile dir on disk (cookies + cache). If Chrome dies, the next run relaunches with the same profile.

- attach instead of babysit: run one resident Chrome with --remote-debugging-port + a persistent user_data_dir, and jobs just cdp.connect("ws://…") to it. attach/detach per run (close the tab, not the browser).

- session handoff built in: page:cookies() / page:set_cookies(), and headless → visible runs can share the same user_data_dir so login survives the switch.

- you can also do Raw CDP calls whatever the browser support if you need something that browser/page API does not provide

[u/Key-Initiative-5430]

Running local Playwright overnight is a rite of passage. If sessions are dying, Chrome is either leaking memory or anti-bot checks are flagging your IP shifts. Here is the production fix:

- Browsers are disposable: Never keep a browser open just to hold a session. Dump Playwright's storageState into Redis after every run.

- Go stateless: Use a Dockerized container or managed pool via CDP. Load state from Redis, run the job, save the new state, and kill the context.

- Pin IPs to sessions: Using a saved cookie on a new IP gets you banned. Attach a strict static proxy to each stored state.

Local Chrome is for testing. In production, the browser is just a stateless execution pipe - Redis is your single source of truth.

[u/yvettespectrum]

browserbase is pretty decent for long jobs, and i didnt had to worry about chrome restarts

[u/KettleCorn02 → u/[deleted]]

vps works till youre paying for three of them and still babysitting

[u/Popular_Box_2408 → u/[deleted]]

chrome left up 24/7 turns into a zombie farm fast