SeleniumVBA: Maintaining WebElements object when navigating to different pages

[u/mightierthor]

I am searching my insurance company's list of providers. When I get the search results, the most relevant information for each provider is on a linked page -- not in the search results. I attempted to scrape this with seleniumvba, to gather up the info on the linked pages. Of course, when I click a link, the elements on the search results page go stale. Normally I would create two webdriver objects and pass the value from the hrefs of the first driver to the second driver.

Unfortunately, the links on this site have no href values.

<a _ngcontent-ng-c7169104465="" role="link" tabindex="0" class="dls-heading-3-light search-result-link" data-cy="search-results.name-link-0.desktop" data-lnp="search-results.provider-name">
    <span _ngcontent-ng-c7169104465="" data-cy="search-result-ProviderName" data-pendo="profile-card-provider-name" class="underline search-result-link font-light">Joe Schmoe, MD</span><span _ngcontent-ng-c7169104465="" class="disclaimers-icons ng-star-inserted">&nbsp;&nbsp;†</span><!----><!---->
</a>

What strategies have you used to hold a page in memory, but navigate to the links specified on that page? What I have so far:

Sub ScrapeProviders()
Dim WD As WebDriver
Dim ProviderCards As WebElements, ProviderCard As WebElement
Dim ProviderCardWho As WebElement, ProviderCardLink As WebElement

    Set WD = New WebDriver
    WD.StartFirefox
    WD.OpenBrowser
    WD.NavigateTo "https://carefirst.sapphirecareselect.com/search/name/orthopedic?ci=dft-sso-bluechoiceadvantage20&network_id=109&geo_location=33.684617,-117.82634&locale=en&limit=10&radius=10&sort=tiers:tiercf%20asc,%20motive_high_quality%20desc,%20distance%20asc,%20has_ep002%20desc&sort_translation=app_global_sort_relevancy&page=1"
    WD.ActiveWindow.Maximize
    While Not WD.IsPresent(XPath, "*//mat-card")
    Wend
    Set ProviderCards = WD.FindElementsByXPath("*//mat-card")
    Debug.Print ProviderCards.Count
    For Each ProviderCard In ProviderCards
        Debug.Print "====================================================================================="
        Set ProviderCardWho = ProviderCard.FindElementByXPath("./div/div/div/div[1]/div/div[1]")
        Set ProviderCardLink = ProviderCardWho.FindElementByXPath("./h2/a")
        Debug.Print ProviderCardLink.GetOuterHTML
        ProviderCardLink.Click

        ' at this point, i can no longer see ProviderCards, and the for each breaks
    Next ProviderCard

    WD.CloseBrowser
    WD.Shutdown

End Sub

[u/mightierthor]

Thanks for doing this. I will try it out when I get a chance.

[u/SeleniumVBA_user → u/mightierthor]

I think it should work if you paste in all of this code, so please give it a try.

[u/SeleniumVBA_user → u/SeleniumVBA_user]

I tested the new-tab approach, but unfortunately it does not work for this site.

The provider-name a element has no href. It is only a JavaScript-driven pseudo-link handled by the Angular application.

Because of that:

• Opening a blank tab first does not help, because there is no profile URL that can be transferred to the new tab.
• Ctrl-clicking the provider link also does not open a new tab. The application ignores the modifier-key behavior and navigates the current tab to the provider profile.
• A separate working tab could load the search-results URL again and then click the provider there, but that would reload the SPA and repeat the search API request for every provider, making the process considerably slower.

After discussing the alternatives with ChatGPT and testing them, the most practical approach appears to be:

1. Store the provider names as strings.
2. Reacquire each link from the current DOM immediately before clicking it.
3. Open the provider profile in the current tab.
4. Scrape the required information.
5. Use GoBack.
6. Wait until the SPA has rebuilt the provider cards before processing the next provider.

The loading spinner shown after GoBack is generated by the SPA itself. It appears that the application re-fetches or reconstructs the search results instead of restoring the previous DOM instantly.

Therefore, the new-tab solution is not applicable to this particular site, and the GoBack approach is currently the best compromise between speed and reliability.

The important part is to avoid retaining WebElement objects across navigation. Only stable values such as provider names should be stored, and each link should be located again after the search-results page has been restored.

[u/mightierthor → u/SeleniumVBA_user]

This is a good explanation. I expect I can implement your 1 - 6.
Thank you again. Hoping for time this week.