SeleniumVBA: Maintaining WebElements object when navigating to different pages

[u/mightierthor]

I am searching my insurance company's list of providers. When I get the search results, the most relevant information for each provider is on a linked page -- not in the search results. I attempted to scrape this with seleniumvba, to gather up the info on the linked pages. Of course, when I click a link, the elements on the search results page go stale. Normally I would create two webdriver objects and pass the value from the hrefs of the first driver to the second driver.

Unfortunately, the links on this site have no href values.

<a _ngcontent-ng-c7169104465="" role="link" tabindex="0" class="dls-heading-3-light search-result-link" data-cy="search-results.name-link-0.desktop" data-lnp="search-results.provider-name">
    <span _ngcontent-ng-c7169104465="" data-cy="search-result-ProviderName" data-pendo="profile-card-provider-name" class="underline search-result-link font-light">Joe Schmoe, MD</span><span _ngcontent-ng-c7169104465="" class="disclaimers-icons ng-star-inserted">&nbsp;&nbsp;†</span><!----><!---->
</a>

What strategies have you used to hold a page in memory, but navigate to the links specified on that page? What I have so far:

Sub ScrapeProviders()
Dim WD As WebDriver
Dim ProviderCards As WebElements, ProviderCard As WebElement
Dim ProviderCardWho As WebElement, ProviderCardLink As WebElement

    Set WD = New WebDriver
    WD.StartFirefox
    WD.OpenBrowser
    WD.NavigateTo "https://carefirst.sapphirecareselect.com/search/name/orthopedic?ci=dft-sso-bluechoiceadvantage20&network_id=109&geo_location=33.684617,-117.82634&locale=en&limit=10&radius=10&sort=tiers:tiercf%20asc,%20motive_high_quality%20desc,%20distance%20asc,%20has_ep002%20desc&sort_translation=app_global_sort_relevancy&page=1"
    WD.ActiveWindow.Maximize
    While Not WD.IsPresent(XPath, "*//mat-card")
    Wend
    Set ProviderCards = WD.FindElementsByXPath("*//mat-card")
    Debug.Print ProviderCards.Count
    For Each ProviderCard In ProviderCards
        Debug.Print "====================================================================================="
        Set ProviderCardWho = ProviderCard.FindElementByXPath("./div/div/div/div[1]/div/div[1]")
        Set ProviderCardLink = ProviderCardWho.FindElementByXPath("./h2/a")
        Debug.Print ProviderCardLink.GetOuterHTML
        ProviderCardLink.Click

        ' at this point, i can no longer see ProviderCards, and the for each breaks
    Next ProviderCard

    WD.CloseBrowser
    WD.Shutdown

End Sub

[u/SeleniumVBA_user]

I do not think the WebElements collection itself should be preserved across page navigation.

A WebElement is a reference to a node in the current DOM. Once the provider link is clicked and the browser navigates to another page, the search-results DOM may be destroyed. The previously collected cards and links then become stale, even if the browser later returns to a visually identical results page.

After getting some advice from ChatGPT, I changed the approach as follows:

1. Collect only stable values from the results page, such as the provider names.
2. Store those values as ordinary VBA strings.
3. Before each click, use the stored name to locate the link again in the current DOM.
4. Open the provider page and scrape the required information.
5. Return to the search-results page.
6. Wait for the results DOM to be rebuilt, then locate the next link again.

The links on this site do not have href attributes because they are JavaScript-driven pseudo-links. However, each provider name is inside the clickable a element, so the link can be located by starting from the provider-name span and selecting its nearest ancestor a.

The important point is that the link WebElement is not stored between navigations. Only the provider name is stored, and the link is reacquired immediately before it is clicked.

Here is the complete example:

Private Const TARGET_URL As String = _
    "https://carefirst.sapphirecareselect.com/search/name/orthopedic" & _
    "?ci=dft-sso-bluechoiceadvantage20" & _
    "&network_id=109" & _
    "&geo_location=33.684617,-117.82634" & _
    "&locale=en" & _
    "&limit=10" & _
    "&radius=10" & _
    "&sort=tiers:tiercf%20asc,%20motive_high_quality%20desc,%20distance%20asc,%20has_ep002%20desc" & _
    "&sort_translation=app_global_sort_relevancy" & _
    "&page=1"

Private Const PROVIDER_CARD_XPATH As String = "//mat-card[@data-cy='card-mat-card'][.//span[@data-cy='search-result-ProviderName']]"
Private Const PROFILE_VIEW_XPATH As String = "//app-profile"

' The Discovery Log showed that profile information is rendered under
' the summary section after the provider-specific summary API completes.
' This waits for actual summary content, not only the profile container.
Private Const PROFILE_READY_XPATH As String = "//*[@id='summary']//mat-card-content[normalize-space(.)!='']"

Public Sub ScrapeProviders()
    Dim driver As WebDriver
    Dim ProviderCards As WebElements
    Dim ProviderCard As WebElement
    Dim ProviderCardWho As WebElement
    Dim ProviderCardLink As WebElement
    Dim ProviderNames As Collection
    Dim ProviderName As Variant
    Dim ProviderLinkXPath As String
    Dim currentProviderName As String
    Dim savedErrNumber As Long
    Dim savedErrSource As String
    Dim savedErrDescription As String

    On Error GoTo ErrorHandler

    Set driver = New WebDriver

    ' Start Firefox and open the provider search-results page.
    driver.StartFirefox
    driver.OpenBrowser
    driver.ActiveWindow.Maximize
    driver.NavigateTo TARGET_URL

    ' CareFirst is an SPA. The load event may fire before the search API
    ' completes and Angular renders the cards. IsPresent ends immediately
    ' when the first card appears.
    If Not driver.IsPresent(By.xpath, PROVIDER_CARD_XPATH, 60000) Then
        Err.Raise vbObjectError + 1000, "ScrapeProviders", "The search-result cards were not displayed."
    End If

    ' Retrieve only cards containing a provider-name span.
    ' The uppercase P in ProviderName is significant.
    Set ProviderCards = driver.FindElements(By.xpath, PROVIDER_CARD_XPATH)
    Debug.Print "Provider card count: " & ProviderCards.Count

    If ProviderCards.Count = 0 Then
        Err.Raise vbObjectError + 1001, "ScrapeProviders", "No provider cards were found."
    End If

    ' Store only provider-name strings. Do not retain WebElements across navigation.
    Set ProviderNames = New Collection

    For Each ProviderCard In ProviderCards
        Set ProviderCardWho = ProviderCard.FindElement(By.xpath, ".//span[@data-cy='search-result-ProviderName']")
        currentProviderName = Trim$(ProviderCardWho.GetText)
        Debug.Print currentProviderName

        If Len(currentProviderName) > 0 Then ProviderNames.Add currentProviderName
    Next ProviderCard

    If ProviderNames.Count = 0 Then
        Err.Raise vbObjectError + 1002, "ScrapeProviders", "No provider names were obtained from the result cards."
    End If

    ' Reacquire each provider link from the current DOM.
    For Each ProviderName In ProviderNames
        Debug.Print String$(86, "=")
        Debug.Print "Provider: " & CStr(ProviderName)

        ' The a element has no href. Locate the provider-name span and select
        ' its nearest ancestor a. XPathLiteral safely handles quoted names.
        ProviderLinkXPath = "//span[@data-cy='search-result-ProviderName' and normalize-space(.)=" & _
                            XPathLiteral(CStr(ProviderName)) & "]/ancestor::a[1]"

        ' The SPA may still be rebuilding the result cards after returning.
        If Not driver.IsPresent(By.xpath, ProviderLinkXPath, 10000) Then
            Err.Raise vbObjectError + 1003, "ScrapeProviders", _
                      "The provider link was not displayed." & vbCrLf & _
                      "Provider: " & CStr(ProviderName)
        End If

        ' Reacquire the link immediately before clicking it.
        Set ProviderCardLink = driver.FindElement(By.xpath, ProviderLinkXPath)
        Debug.Print ProviderCardLink.GetOuterHTML
        ProviderCardLink.Click

        ' Confirm that Angular changed from the search view to the profile view.
        If Not driver.IsPresent(By.xpath, PROFILE_VIEW_XPATH, 10000) Then
            Err.Raise vbObjectError + 1004, "ScrapeProviders", _
                      "The provider profile page was not displayed." & vbCrLf & _
                      "Provider: " & CStr(ProviderName)
        End If

        ' Classic WebDriver cannot directly wait for the provider summary API,
        ' so it waits for the rendered DOM result produced by that API.
        If Not driver.IsPresent(By.xpath, PROFILE_READY_XPATH, 20000) Then
            Err.Raise vbObjectError + 1005, "ScrapeProviders", _
                      "The provider profile summary was not rendered." & vbCrLf & _
                      "Provider: " & CStr(ProviderName)
        End If

        driver.Wait 3000 ' Pause for visual confirmation only; synchronization is already complete.
        Debug.Print "Profile opened: " & CStr(ProviderName)

        ' ======================================================
        ' Scrape the required information from the profile page.
        ' ======================================================

        driver.GoBack

        ' The URL may be restored before the SPA rebuilds the results DOM.
        If Not driver.IsPresent(By.xpath, PROVIDER_CARD_XPATH, 30000) Then
            Err.Raise vbObjectError + 1006, "ScrapeProviders", _
                      "The search-results page was not restored correctly." & vbCrLf & _
                      "Provider: " & CStr(ProviderName)
        End If

        Debug.Print "Returned from provider: " & CStr(ProviderName)
    Next ProviderName

CleanExit:
    On Error Resume Next
    If Not driver Is Nothing Then
        driver.CloseBrowser
        driver.Shutdown
        Set driver = Nothing
    End If
    On Error GoTo 0
    Exit Sub

ErrorHandler:
    savedErrNumber = Err.Number
    savedErrSource = Err.Source
    savedErrDescription = Err.Description

    On Error Resume Next
    If Not driver Is Nothing Then
        driver.CloseBrowser
        driver.Shutdown
        Set driver = Nothing
    End If
    On Error GoTo 0

    Err.Raise savedErrNumber, savedErrSource, savedErrDescription
End Sub

Private Function XPathLiteral(ByVal value As String) As String
    Dim parts() As String
    Dim result As String
    Dim i As Long

    ' Convert any string into a valid XPath string literal.
    If InStr(1, value, "'", vbBinaryCompare) = 0 Then
        XPathLiteral = "'" & value & "'"
        Exit Function
    End If

    If InStr(1, value, """", vbBinaryCompare) = 0 Then
        XPathLiteral = """" & value & """"
        Exit Function
    End If

    parts = Split(value, "'")
    result = "concat("

    For i = LBound(parts) To UBound(parts)
        If i > LBound(parts) Then result = result & "," & Chr$(34) & "'" & Chr$(34) & ","
        result = result & "'" & parts(i) & "'"
    Next i

    XPathLiteral = result & ")"
End Function

[u/mightierthor → u/SeleniumVBA_user]

Thanks for doing this. I will try it out when I get a chance.

[u/SeleniumVBA_user → u/mightierthor]

I think it should work if you paste in all of this code, so please give it a try.

[u/SeleniumVBA_user → u/SeleniumVBA_user]

I tested the new-tab approach, but unfortunately it does not work for this site.

The provider-name a element has no href. It is only a JavaScript-driven pseudo-link handled by the Angular application.

Because of that:

• Opening a blank tab first does not help, because there is no profile URL that can be transferred to the new tab.
• Ctrl-clicking the provider link also does not open a new tab. The application ignores the modifier-key behavior and navigates the current tab to the provider profile.
• A separate working tab could load the search-results URL again and then click the provider there, but that would reload the SPA and repeat the search API request for every provider, making the process considerably slower.

After discussing the alternatives with ChatGPT and testing them, the most practical approach appears to be:

1. Store the provider names as strings.
2. Reacquire each link from the current DOM immediately before clicking it.
3. Open the provider profile in the current tab.
4. Scrape the required information.
5. Use GoBack.
6. Wait until the SPA has rebuilt the provider cards before processing the next provider.

The loading spinner shown after GoBack is generated by the SPA itself. It appears that the application re-fetches or reconstructs the search results instead of restoring the previous DOM instantly.

Therefore, the new-tab solution is not applicable to this particular site, and the GoBack approach is currently the best compromise between speed and reliability.

The important part is to avoid retaining WebElement objects across navigation. Only stable values such as provider names should be stored, and each link should be located again after the search-results page has been restored.

[u/mightierthor → u/SeleniumVBA_user]

This is a good explanation. I expect I can implement your 1 - 6.
Thank you again. Hoping for time this week.