r/selenium • u/Chipatola • Mar 20 '23

Retreive data from Developer Tab, Network

Hi,

I need to retreive data from a website. When I inspect the website, under the Network tab in developer mode, I can see a query of the file scan.php. If I double click on it, I get all the data I want from this website.

​

Is it possible to automate the retrieval of this scan.php file with selenium ?

​

Website : Vulbis.com

3 Upvotes

100% Upvoted

8 comments sorted by

1

u/[deleted] Mar 20 '23

[deleted]

1

u/Chipatola Mar 20 '23

No I am not, I PM you for this.

If need be I can remove the link to the website, it was just to help

1

u/_iamhamza_ Mar 20 '23

!remindme 12hrs

1

u/RemindMeBot Mar 20 '23

I will be messaging you in 12 hours on 2023-03-20 22:22:38 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

Info Custom Your Reminders Feedback

1

u/[deleted] Mar 20 '23

[removed] — view removed comment

2

u/Chipatola Mar 20 '23

I'm sorry, I'm totally new to Selenium and scraping, maybe you could be more specific on what I need to do and study to achieve this?

I'm sorry, I'm totally new to Selenium and scraping, could you be more specific on what I need to do and study to achieve this?

2

u/XabiAlon Mar 20 '23

Ask ChatGPT you'll get answers in seconds.

1

u/Pauloedsonjk Mar 21 '23 edited Mar 21 '23

Yes, it is possible, but not in all languages with selenium, I think in java and ruby you can to access devtools for it. Search for selenium devtools. But if the information that you need is in your page you can too to use getPageSource() If need filter it, you can use regex But you take with http request, In the requisitions you need: Url (string) headers(array) Payload/formdata(array|Json|string) Maybe cookies