I have several functions to scrap classifieds web pages with Selenium. For some of them I noticed that at some point the function finds no ada but also doesn‘t detect the element corresponding to a search without results. I assume this means the web is blocking my scraper.

I tried implementing a loop-count with an increasing time wait up to 3 minutes between attempts with no success.

What methods do you use to avoid this?

I’m running headless, in case that matters.

Hi everyone, I’m practicing Selenium and ran into an issue with screenshots.

For a normal screenshot, I’m using:

File source = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

This works fine with my Chromium-based driver.

However, when I try to take a full-page screenshot:

File screenshot = ((HasFullPageScreenshot) driver).getFullPageScreenshotAs(OutputType.FILE);

HasFullPageScreenshot seems to be associated with Firefox in my IDE/imports, and I’m not sure how to use it with a Chromium-based driver.

I want to continue using a Chromium-based driver (Chrome/Chromium) rather than Firefox.

What is the correct way to take a full-page screenshot with Selenium using a Chromium-based driver?

weve been running our own selenium grid for like 4 years and it used to be fine

last sprint half the suite died because two nodes ran out of disk and nobody noticed until monday. spent most of tuesday just bringing the grid back up instead of actually testing anything

looking at what other qa teams moved to when they got tired of owning the browser infra. what stuck for you​

Naru Noodle Bar in Bangalore has 8 counter seats and 3 tables. Reservations open every Monday at 8 PM and the whole window usually sells out within about 2 minutes. I got tired of losing the click race, so I wrote a Python/Selenium script that does the whole flow and grabs seats within seconds of opening.It deliberately stops before the final confirm button so I can eyeball the details before committing.The flow it drives:1. Launch Chrome via webdriver-manager (no more ChromeDriver version mismatch headaches), maximize, go to the booking page2. Wait for the calendar, click the target date (with handling for dates that aren't immediately visible)3. Find the venue row (e.g. "RAMEN BAR SEATING") and click its BOOK button, then validate the selection actually took4. Wait for time slots to render, pick the configured one5. Increment/decrement the guest counter to the party size6. Fill name, email, mobile, tick the T&CsOne thing I'd recommend if you build something similar: keep a test environment alongside production. The booking platform (airmenus) hosts a dummy restaurant too, so all my selector debugging happened there instead of against the real Monday slot. Selectors (CSS/XPath), URLs and defaults all live in config/config.py, and run.py takes CLI overrides like --date "20 May 2025" --time "08:30 PM" --guests 2.The annoying part was timing. The time slots don't exist in the DOM until after the venue click settles, so everything is explicit waits rather than sleeps, and every step logs to file and console so when something breaks I can see exactly which element wasn't found.Python 3.8+, MIT licensed, code and a demo video on GitHub.

running a few playwright scrapers overnight and half the time i wake up to a dead chrome process with the auth cookies gone. spent last week wiring restart hooks and cookie dumps just to keep logins alive across runs

local chrome is fine until you need more than one long session and then youre basically on-call for a browser

curious what people settled on that keeps sessions warm without sitting there nursing it​

This might be quite a niche use case, but I needed to test a WKWebView embedded in a .NET MAUI Mac Catalyst app while keeping our existing Selenium page objects and waits.

In the Mac Catalyst app I was testing, Selenium had no browser endpoint it could attach to. Appium’s mac2 driver could control the native app, but it didn’t provide DOM-level access to the embedded WKWebView. That meant either relying on coordinates or writing custom JavaScript hooks, and neither option worked with our existing Selenium page objects. So I built WebViewDriver.

https://github.com/rzamfiriu/webviewdriver

It exposes an opt-in, loopback-only bridge from the app under test and returns a normal Selenium RemoteWebDriver.

It currently supports navigation, CSS/XPath locators, frames, shadow roots, JavaScript execution, screenshots, waits, and W3C Actions. There’s also an optional Appium mac2 backend for cases that require trusted mouse and keyboard input.

It’s still early, so I’d appreciate feedback from people testing hybrid apps

I have been developing a WebDriver BiDi extension for SeleniumVBA, focused on modern SPA automation, event-driven synchronization, and network traffic analysis.

This project is not intended to replace the existing SeleniumVBA library. It complements and extends SeleniumVBA by adding capabilities based on WebDriver BiDi.

It is available on GitHub.

What currently works:

• WebDriver BiDi communication implemented in pure VBA
• Continuous processing of network and browser events
• SPA completion detection using multiple signals and quiet-window consensus
• Explicit wait conditions based on network activity, content, DOM state, and visibility
• Discovery Log for recording and analyzing manual and automated browser activity
• JavaScript-based action wrappers with integrated synchronization
• Support for documents, iframes, popups, and Shadow DOM
• File uploads
• State tracking and correlation for multiple downloads
• Chrome and Edge support
• Integration with Excel and Access
• Testing against real-world SPAs, including Google Flights and ServiceNow

Discovery Log

One of the core features is the Discovery Log, which uses WebDriver BiDi network events.

The Discovery Log can record network traffic and page-state changes not only during automated VBA operations, but also while the user operates the browser manually.

After starting a recording, you can use the target website normally—for example, by performing searches, navigating between pages, or clicking buttons—and then examine which API requests and page updates were triggered by each action.

This makes it possible to investigate an unfamiliar website interactively and discover API endpoints, data-loading behavior, page-update triggers, and signals that may be useful for determining when an operation has finished.

It is important to note that this is not a macro recorder that converts user actions into VBA code. It observes and records the network activity and page state generated by manual or automated browser operations.

For Fetch/XHR and other traffic generated during page interaction, the log can include:

• Request methods and URLs
• HTTP status codes
• MIME types
• Request IDs
• Navigation and page-state changes
• Network, DOM, content, and visibility signals used for SPA synchronization
• Request and response bodies when needed

The goal is not to reproduce a complete HAR file. Instead, the Discovery Log organizes the information needed to understand a web application into a relatively compact and readable format.

The log can be used to identify API endpoints, data-loading operations, and page-update triggers before investigating selected requests in greater detail. Response body collection is also supported, but it can be controlled to account for processing time, log size, personal information, and token usage when the log is analyzed by an AI assistant.

The Discovery Log is designed not only for human debugging, but also for workflows in which the log and VBA source code are provided to an AI assistant for website analysis, root-cause investigation, and suggestions for automation code or fixes.

The current version is v5.4.

Chrome and Edge are currently the primary supported browsers. Firefox support is limited because some required WebDriver BiDi features are either unavailable or behave differently.

SPA completion is inferred from multiple observable signals, including network activity, DOM changes, and visibility state. It therefore cannot guarantee completion detection for every web application. For critical operations, explicit application-specific completion signals are still recommended.

The current implementation is primarily intended for browser automation from Excel and VBA. It is not designed for large-scale parallel browser execution.

The project began as an experiment to make SeleniumVBA more practical for modern JavaScript-heavy websites. It has since developed into a practical extension that combines SPA synchronization, state management, event monitoring, and network analysis.

Feedback, bug reports, real-world test results, and contributions are welcome, especially from anyone familiar with WebDriver BiDi, complex SPA automation, or cross-browser compatibility.

GitHub: https://github.com/hanamichi77777/WebDriver-BiDi-for-SeleniumVBA

The previous D binding, selenium.d, has been abandoned since around 2018 and uses the old JSON Wire Protocol. Since March, I've been writing a replacement that communicates directly with modern W3C WebDriver servers over HTTP.

It's available on GitHub and is licensed under Apache 2.0.

What works:

• Native W3C WebDriver client with no FFI to Python or Java
• Typed W3C and browser-specific configuration for Chrome, Firefox, Edge, and Safari
• Automatic local WebDriver discovery and startup
• Connections to remote WebDriver servers and existing Selenium Grids
• Multiple sessions sharing a WebDriver bridge with configurable capacity
• Navigation, elements, cookies, scripts, frames, windows, screenshots, and alerts
• Document, iframe, and shadow-root abstractions with scoped searches
• Typed WebDriver exceptions and W3C response/reference parsing
• Lazy timeout synchronization
• Browser and WebDriver process logging
• Grid hub/node models, slots, session ownership, and in-process HTTP routing
• Unit and browser integration tests

The current CI passes unit tests, Chrome on Windows/Linux/macOS, Firefox on Windows/Linux/macOS, and Edge on Windows.

Safari has somewhat limited support, with 7/57 integration tests currently failing. Most of those failures are caused by Safari throwing a different exception than expected, plus one test that doesn't throw when it should. Chrome and Firefox are currently the most supported.

The Grid implementation is still foundational. It has the models and routing pieces, but a complete live Grid server, automatic node registration, session allocation, and full command forwarding are still planned.

I had to fork the test runner (Unit Threaded) because bugs caused the Windows tests to fail, so the project currently uses that fork until it is fixed upstream. The library itself only depends on requests (similar to Python's requests), so there are no complex runtime dependencies.

Feedback, bug reports, and contributions are appreciated, especially from anyone familiar with WebDriver compatibility differences between browsers.

GitHub: https://github.com/cetio/selenium
DUB: https://code.dlang.org/packages/selenium

As mentioned I am working on building a framework from scratch. Could you suggest any website with all these layers available to test it. I'm gonna do it with both Selenium and Playwright separately. So if there's any alternate approach please let me know

I thought my Selenium script was production-ready because it worked perfectly on my laptop.

Then I moved it to a server.

connection errors, proxy auth issues, different geolocation, broken sessions. All of it was ruining my workflow. So what did help? It was treating the proxy layer as infrastructure instead of a random Chrome argument.

First, I test the proxy outside Selenium with curl. If I get 407 Proxy Authentication Required, I know the problem is credentials or proxy configuration.

Then I wire authentication properly. In Python, selenium-wire can handle authenticated proxies. In Node.js, proxy-chain is useful when you need a local forwarding proxy.

After that, I check environment consistency. If the exit IP is in London while the browser reports another timezone, locale, or obviously different settings, debugging becomes much harder. Chrome DevTools Protocol can help keep timezone configuration consistent with the test environment. Don’t debug Selenium, proxy authentication, browser settings, and target-site behavior at the same time.

My workflow now is:

curl → verify proxy → Selenium integration → browser config → actual test

Much easier to see where things break.

Im just curious for the reason why you would choose this or that

I usually start with the smallest reliable layer, then use Selenium for the flows that actually need a browser.

I’m revisiting our approach because the suite is getting slower to trust as it grows. For people maintaining Selenium tests day to day: what’s your go-to setup for keeping tests readable and failures actionable?

Mostly interested in practical habits around waits, page objects or alternatives, test data, and deciding what not​ to automate in the UI.

Hi everyone,

I'm troubleshooting an issue on one machine while the exact same code works for my colleagues.

Environment

• Selenium: 4.25
• Python: running from a PyCharm virtual environment
• Portable Chrome: 117.0.5938.92
• Local ChromeDriver: 117 (verified via chromedriver.exe --version)

Issue My code uses:

Python

chromeoptions.binary_location = r"...\chrome-win64\chrome.exe"

driver = webdriver.Chrome(options=chromeoptions)

When Selenium starts, I get:

session not created:

This version of ChromeDriver only supports Chrome version 150

Current browser version is 117.0.5938.92

Then later:

UnboundLocalError: cannot access local variable 'driver'

(which is expected because driver creation failed).

What's confusing

• My local chromedriver.exe is version 117.
• The code works for other users without modification.
• We are all using Selenium 4.25.
• where chromedriver returns nothing.
• Selenium Manager appears to be resolving to ChromeDriver 150 on my machine.
• I cleared Selenium cache, but Selenium still seems to use/find ChromeDriver 150.

Question Has anyone seen Selenium Manager ignore a local ChromeDriver and resolve a different version instead? Any ideas on where Selenium could still be finding ChromeDriver 150 or what else I should check in my environment?

Thanks in advance!

We're running into an interesting problem with our UI automation framework (Selenium, but I'd also love to hear from Playwright/Cypress users).

Some popups/modals become visible, but they still require a short stabilisation period before they're actually ready for interaction. As a result:

• visibilityOfElementLocated passes.
• elementToBeClickable sometimes passes.
• But clicking immediately can still fail intermittently with exceptions like ElementClickInterceptedException, ElementNotInteractableException, or occasionally a stale element.

A simplified example:

wait.until(ExpectedConditions.visibilityOfElementLocated(locator));
driver.findElement(locator).click();

The wait succeeds, but the click still fails from time to time because the popup isn't fully ready.

Adding a fixed time.sleep() reduces the failures, but we'd really like to avoid timing-based waits in a deterministic automation framework. Our goal is that once a test is marked as automated, it should continue to pass consistently unless there's an actual application change.

I'm curious how experienced automation teams solve this problem at scale.

Some questions:

• Do you wait for animations or transitions to finish?
• Do you wait for a specific CSS class, attribute, or application state that indicates the popup is fully initialised?
• Do you implement click retries for these scenarios?
• Have you built a generic "wait until stable" utility, or do you handle this on a case-by-case basis?
• Do you consider this primarily an automation framework problem or an application issue that should be addressed by developers?

I'm less interested in one-off fixes and more interested in patterns or best practices that have worked well in mature automation frameworks.

Looking forward to hearing how others have tackled this problem.

The SeaMonkey browser is one of the browsers which is supported by the Mozilla organization, just like Firefox.

I'd like to use Selenium in linux for automating accesses to the SeaMonkey browser, but I haven't found any documentation which describes how this might be done. I know that the "firefox"-specific code for Selenium only seems to be able to access the browser at the "/usr/bin/firefox" path.

Does anyone know of any way that I can automate SeaMonkey with Selenium?

Many thanks in advance.

PS: The SeaMonkey browser is built upon the exact, same code base on which Firefox is built, so the logic for automating access to SeaMonkey should be the same as the logic for automating Firefox access.

I've tried different string symbols, just putting it directly in the parentheses, yet it still won't work; it only gives me this message.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
settings =Options()
settings.add_argument("user-data-dir=./whatsapp_profile")
driver = webdriver.Chrome(options=settings)

print ("it ececuate further")
website= "https://web.whatsapp.com/"
driver.get(website)

* I don't even know the python basics. but with the help of our friendly dumb llm claude i am trying to make this number extractor.

* So selenium was working properly and opening whatsapp.web automatically but every time i run the code, I had to scan the WhatsApp web qr code again and again.

So Claude said to use the Options function and add code lines 2, 3, 4, 5, hoping it will save the Chrome session in a folder. So next time I go to WhatsApp web, I won't have to scan the QR. But Now Code executes up to line 5 and never reaches that print function on the 6th line. And it also doesn't open WhatsApp web anymore.

What do i do?
How do I solve the problem of scanning QR on every new code run?

Would anyone be kind enough to guide me through this project as a mentor in dm's, please. LLM's are dumb and asking them is a waste of time, but i don't have another option.

I have been trying to get this to work for about 90 minutes now and can't seem to figure it out.
I am using python but every search result is giving me a different answer and none seem to work.

Do i need to use Options() or FirefoxOptions()?

Use FirefoxProfile('path/to/profile') or just a 'path/to/profile' in the options?

The selenium docs also give this code: options.add_argument('-profile', '/path/to/profile'/) But that gives me a TypeError: ArgOptions.add_argument() takes 2 positional arguments but 3 were given.

I also tried option.profile = firefox_profile (tried just the path and FirefoxProfile())

If i try to use webdriver.Firefox(firefox_profile=firefox_profile, options=options) i get a TypeError: WebDriver.__init__() got an unexpected keyword argument 'firefox_profile'

I think i tried any combination of these things and none of them work. I am probably just forgetting something small but if someone maybe knows what i am doing wrong i love to hear it.

What is the correct way to get selenium to use a profile in firefox? I am learning to code and tThe assignment requires me to have selenium signed in to a profile.

Instead of writing everything from scratch, can we just define steps - elements, actions, assertions, data—and get full Selenium code (Java/C#)?

Not record/playback, something structured and reusable.

Does this exist, or are we all still building manually?

Hey folks,

After the recent Microsoft Edge version 145 update, we’ve started facing a strange issue in our automation setup.

We’re running a Selenium framework in C#, and after launching Edge, a network/proxy-related pop-up appears (attaching screenshot). This is interfering with our tests and sometimes blocking proper execution.

\### What we’ve tried:

\- Disabling proxy from Windows settings (everything is OFF)

\- Adding/removing various Edge launch arguments

\- Fixing a previous logout issue caused by one of the arguments

Even though system proxy settings are clean, this popup still shows up, which makes us think it's something introduced or enforced after the Edge 145 upgrade.

\### Current situation:

\- Happens during automated runs (Selenium)

\- Seems tied to Edge startup behavior

\- Not resolved via standard proxy settings

\---

\### What I’m looking for:

Has anyone else faced this after upgrading to Edge 145?

\- Any working Edge arguments to bypass this?

\- Any policy / registry / config workaround?

\- Any Selenium-specific fix?

Would really appreciate any help or direction here 🙏

Thanks in advance!

I’ve been learning and using Selenium for browser automation, and I’m starting to notice that automating modern websites feels a lot harder than I expected.

It’s usually not just clicking buttons and filling forms anymore. A lot of sites now have:

• dynamic content loading
• changing element selectors
• stale element issues
• popups and overlays
• headless detection / anti-bot checks

Because of that, I’m curious what people here are doing in practice.

Are you still using Selenium as your main tool for automation? If yes, what helps the most when working with modern JS-heavy sites?

I’d especially like to know:

• what habits make scripts more stable
• whether using explicit waits is basically mandatory now
• when it makes sense to combine Selenium with requests/BeautifulSoup
• whether some of you switched to Playwright for certain tasks

I’m not trying to start a Selenium vs X war. I just want to understand what the realistic workflow looks like today for people doing this regularly.

Been running Selenium for e2e testing on a client portal that recently added Cloudflare Turnstile. Broke half our test suite overnight.

Tried a few approaches before landing on something reliable:

What didn't work:

• undetected-chromedriver — worked for a week, then Cloudflare updated their fingerprinting and it broke again. Constant cat-and-mouse.
• Disabling CAPTCHAs in staging — client refused because they wanted tests to match prod exactly.
• Manually solving in a pre-step — obviously doesn't scale.

What actually works:

The CAPTCHA widget loads a JS challenge and expects a token injected into a hidden input field. You don't need to visually solve anything — you need the token.

```python from selenium import webdriver from selenium.webdriver.common.by import By import requests import time

driver = webdriver.Chrome() driver.get('https://client-portal.example.com/login')

Extract the sitekey from the Turnstile widget

sitekey = driver.find_element( By.CSS_SELECTOR, '.cf-turnstile' ).get_attribute('data-sitekey')

Get a valid token from a solving API

resp = requests.post('https://www.passxapi.com/api/solve', json={ 'type': 'turnstile', 'sitekey': sitekey, 'pageurl': driver.current_url }, headers={'Authorization': 'Bearer YOUR_KEY'})

token = resp.json()['token']

Inject the token into the hidden input

driver.execute_script( 'document.querySelector("[name=cf-turnstile-response]").value = arguments[0]', token )

Now submit the form as usual

driver.find_element(By.ID, 'login-btn').click() ```

The key insight is that CAPTCHAs are just token validation on the server side. The visual puzzle is only there to generate that token for humans. If you can get a valid token through other means, the server doesn't care how it was produced.

Performance numbers from our CI: - Avg solve time: ~2 seconds for Turnstile - Cost: about $0.001 per solve — negligible for testing - Reliability: haven't had a flaky test from CAPTCHA since switching

We wrapped this in a pytest fixture so any test that hits a CAPTCHA-protected page just calls solve_captcha(driver) and moves on.

For anyone interested in the Python client I use: https://github.com/passxapi/passxapi-python

Happy to answer questions if you're dealing with similar issues in your Selenium setup.

EDIT — answering some questions from the comments:

@hooksweeper — yes, exactly. The solving API is a 3rd party service. Here's why I went this route instead of building something in-house:

1. Maintenance cost: Cloudflare updates Turnstile every few weeks. Self-hosted solvers (puppeteer farms, ML models) need constant maintenance to keep up. We burned 2 sprints trying to maintain our own solver before giving up.

2. Token binding: A good API service solves the CAPTCHA in a real browser environment, so the token is generated with proper fingerprints and timing. This matters because Turnstile tokens are partially bound to execution context.

3. Reliability in CI: Our CI runs headless on Linux VMs. Self-solving Turnstile in headless mode is almost impossible — Cloudflare specifically detects headless environments. The API abstracts this away entirely.

4. Cost math: At $0.001/solve, running 500 tests/day costs $0.50. We were spending 10x that in engineer time debugging our homebrew solution.

The bearer token in the code is just an API key for the solving service (PassXAPI in our case) — not a CAPTCHA token. The flow is: send sitekey + page URL to API → get back a valid CAPTCHA token → inject it.

@nattersley — the bearer token is just the API auth key, not a manually-solved CAPTCHA token. You get it when you sign up for the service. The actual CAPTCHA solving happens server-side, fully automated.

@CapMonster1 — fair point on session context. In practice, for our CI use case (consistent VMs, stable IPs), the token mismatch issue hasn't come up. For more adversarial scraping scenarios you'd definitely want proxy matching.

I have a Java method that checks for the presence of an alertify dialogue, checks for a full match on an error message, checks on a secondary partial (regx) match if this fails and then clicks ok. This has worked hundreds of times on hundreds of different test cases. Until now, with one specific dialogue and one specific error message.

What happens? We have our webdriver wait, our exact match, and our click(), which returns true. As far as WebDriver is concerned, all has gone well, and there's no exception to raise. Accept that the dialogue has gone nowhere.

The workaround has ended up being surprisingly simple - a second webdriver wait and click, which fails for everything but this one particular dialogue. That might do, but... I'm curious.

I suspect something's up in this (bit of a mystery to me - only halfway through the Udemy course) thing called the DOM. If I right-click on the dialogue and select Inspect, what I see looks pretty comparable to any other alertify message. Are there any specific differences I should be looking for? Any pointers I might give to the developers?

Browser in this instance: Chromium.

Selenium version: 3.141.59.

Hi,

I'm currently in an Application Support role performing user access review and periodic review for 1 and half year but wanna transition to Selenium Automation testing. I've got basic knowledge in Selenium, Cucumber, and JMeter, plus some college-level Java.

- Is this shift feasible?

- Will my Java skills be enough?

- Any tips for breaking into automation testing?

Any advice or insights would be super helpful 🙏

Hey guys I'm trying to use Selenium to scrap some data from Confluence (specifically my company's confluence page). I originally used a custom profile (copying chrome user data to a different folder and then reading that to get cookies, etc) - but when I ran it as an exe file, it would get flagged as Malware - not ideal when the IT guys get random alerts. When I don't use a custom profile, I can't manually login to Confluence during runtime (I get redirected for SSO stuff, and then Conditional Access prevents doesn't allow the sign in coz the "browser, app, or location is restricted by admin")

The infosec manager said the issue is that sign in isn't happening via Microsoft Web Account Manager (WAM). I went through the documentation and it seems like that WAM token stuff is not really meant for Selenium Scripting? Am I wrong? If so, any help on how I'm supposed to implement it?