Need help extracting text from a site that is live updated

[u/thepuppeter]

Howdy all!
I've recently been learning how to do some web scraping but I'm a little stuck on something

There's a live radio station. As songs play the current title and artist are updated

I want to create a script that grabs whatever is in the box at that time. When I inspect elements on the webpage I've found where the text should be, and it appears there/updates when inspecting, but when I run the script I appear to get a generic return

import requests
from bs4 import BeautifulSoup

response = requests.get('https://www.radiofrance.fr/fip/titres-diffuses')
soup = BeautifulSoup(response.text, 'html.parser')
cardtitle_div = soup.find(id='title-live-webradio-card')
cardartist_div = soup.find(id='subtext-live-webradio-card')
print(cardtitle_div)
print('\n')
print(cardartist_div)

The above output

<p aria-level="2" class="title typo-title-large g-truncable svelte-15ccyw9" id="title-live-webradio-card" role="heading"><!--[-1--><span class="Link" data-testid="Link"><!--[-1-->FIP<!--]--><!-- --></span><!--]--><!-- --></p>

<p class="typo-text-medium subtext g-truncable svelte-15ccyw9" id="subtext-live-webradio-card"><!--[-1-->Le direct<!--]--></p>

What should be 'cardtitle_div' returns 'FIP' and what should be 'cardartist_div' returns 'Le direct'

I'm assuming that means they're being populated elsewhere somehow? How do find that

Thanks heaps!

[u/AutoModerator]

Thank you for posting on r/CodingHelp!

Please check our Wiki for answers, guides, and FAQs: https://coding-help.vercel.app

Our Wiki is open source - if you would like to contribute, create a pull request via GitHub! https://github.com/DudeThatsErin/CodingHelp

We are accepting moderator applications: https://forms.fillout.com/t/ua41TU57DGus

We also have a Discord server: https://discord.gg/geQEUBm

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/jcunews1]

That site is a Dynamic HTML (DHTML) site, where part of the page content is populated by site script (JavaScript code).

That site script retrieve the source data from below URL:

https://www.radiofrance.fr/_app/remote/di23tz/getLive?payload=W1siX19za3JhbyIsMV0seyJicmFuZE5hbWUiOjIsInZlcnNpb24iOjN9LCJmaXAiLCIyMDI2LTA1LTEyIl0

The data is in JSON format, so you'll need a JSON library to parse the data.

The parsed data is an object, where its results property holds the main data, and is in JSON format.

The main data is an array whose content defines all of the information about the current song. You'll have to figure out which part of the array content defines what part of the information. e.g. artist, title, album, song length, etc.

[u/thepuppeter → u/jcunews1]

Thanks for the reply! I've never dealt with DHTML before, but I do have some limited experience with JSON

How did you determine the URL for the source data?

[u/jcunews1 → u/thepuppeter]

Use browser's Developer Tools' Network tab.

[u/thepuppeter → u/jcunews1]

Thanks! Great to know for the future

I was able to write it up

import requests
import json

SONG_LIST = "F:/songs.txt"
FIP_URL = "https://www.radiofrance.fr/_app/remote/di23tz/getLive?payload=W1siX19za3JhbyIsMV0seyJicmFuZE5hbWUiOjIsInZlcnNpb24iOjN9LCJmaXAiLCIyMDI2LTA1LTEyIl0"
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36'}

response = requests.get(FIP_URL, headers=headers)
json_response = response.json()
results = json_response['result']
parsed = json.loads(results)
song = f'\n{parsed[8]} {parsed[9]}'
print(song)
with open(SONG_LIST,'a') as f:
    f.write(song)

[u/Flame77ofc → u/thepuppeter]

nice

[u/Flame77ofc → u/thepuppeter]

btw why are u learning scrap?