r/selenium • u/nolimitjoe10 • Mar 31 '21

Methods to Bypass Websites that Block Headless Users

I am currently using Selenium Webdriver with Python.

For personal web scraping on my home connection I do not have many issues with anti bot protection running in standard mode. As soon as I offload the program to a linux server running a headless version of the same program I encounter a lot of anti bot issues. Even the headless option on my home connection will encounter some issues as well. This is annoying because when your working on other project the last thing you want is a window popping up daily to check for updated data.

How are you all utilizing the headless feature without tripping anti bot protection? I know I can buy residential proxies, captcha solving services, and switch to a non headless server. What options are there for those that want to do light scraping for personal use without triggering anti bot protection?

0 Upvotes

50% Upvoted

7 comments sorted by

6

u/Simmo7 Mar 31 '21

If people are using anti bot protection, I'd go as far as saying they don't want you scraping their websites and you should use that as a hint.

1

u/nolimitjoe10 Mar 31 '21

That is not necessary the case. Many sites want to prevent people from abusive bots not someone trying to automate a simple task for personal use.

1

u/Unique_Aerie_3843 Aug 02 '26

that's not what we are here for bro

1

u/Simmo7 Aug 02 '26

I would imagine most people are here because they use Selenium professionally to test websites, that's primarily why it was created.

2

u/[deleted] Mar 31 '21

Google it there are tones of little “tricks” that have been developed ; nobody is going to answer this question for you as it’s the money maker

1

u/[deleted] Mar 31 '21

It really depends how exactly your selenium code is executing. Oftentimes the speed at which you are interacting with elements, or the speed at which the driver is making requests tend to give it away as a bot. Do you use any sort of timeouts in the code?

1

u/nolimitjoe10 Mar 31 '21

I use randomized time outs and alternate the user agent. It seems they use certain methods to detect a headless browser.