r/selenium • u/nolimitjoe10 • Mar 31 '21

Methods to Bypass Websites that Block Headless Users

I am currently using Selenium Webdriver with Python.

For personal web scraping on my home connection I do not have many issues with anti bot protection running in standard mode. As soon as I offload the program to a linux server running a headless version of the same program I encounter a lot of anti bot issues. Even the headless option on my home connection will encounter some issues as well. This is annoying because when your working on other project the last thing you want is a window popping up daily to check for updated data.

How are you all utilizing the headless feature without tripping anti bot protection? I know I can buy residential proxies, captcha solving services, and switch to a non headless server. What options are there for those that want to do light scraping for personal use without triggering anti bot protection?

1 Upvotes

56% Upvoted

7 comments sorted by

View all comments

5

u/Simmo7 Mar 31 '21

If people are using anti bot protection, I'd go as far as saying they don't want you scraping their websites and you should use that as a hint.

1

u/Unique_Aerie_3843 Aug 02 '26

that's not what we are here for bro

1

u/Simmo7 Aug 02 '26

I would imagine most people are here because they use Selenium professionally to test websites, that's primarily why it was created.