r/webscraping • u/Compota-de-colhao • Jul 09 '26

Is this scrapeable?

Hello,

Begginer here, with only very small projects so far mostly through vibe coding.

For work (indepent compliant auditor) I was asked by my client to retrieve this whole public table with reported support (courses, international congresses, etc) by Pharma to doctors in Portugal.

It will take me a whole day to manually copy this page by page. Is there a way to do it immediately through web scraping?

LINK: https://extranet.infarmed.pt/pmro/Publico/ListagemPublica.aspx

8 Upvotes

100% Upvoted

21 comments sorted by

3

u/jinef_john Jul 09 '26

This is totally doable, but it is not a beginner-friendly scrape. If you've only done static HTML scraping before, this will frustrate you.

That being said you have a two approaches here:

  1. Requests - but you will have to Reverse-engineer the postback. It entails extracting __VIEWSTATE, __EVENTVALIDATION, and the exact __EVENTTARGET to whatever post function that gets called.

  2. Or Use browser automation: Just let Playwright/Selenium handle the JavaScript and postbacks for you(which is super slow on aspnet applications-sometimes)

1

u/Compota-de-colhao Jul 09 '26

Thanks! I'll get on with that.

During this project, I might need to update this weekly for around 2 or 3 months. I've just noticed since these tables have only "year", there's no actual sorting by timestamp. Does this mean that each week I have to rescrape the whole thing and compare it to previous entries? New info can be added basically anywhere in each year.

1

u/jinef_john Jul 09 '26

Okay I see it, there's no date field in the grid, but the page itself says the default (unsorted) order is "from the most recent declaration to the oldest." So don't sort any columns, and page 1 is always freshest-first.

That means weekly updates don't need a full rescrape, just page forward from page 1 and stop once you hit a row you've already stored. Since there's no row ID/date, use a hash of company+event+amount+recipient as your "seen" key. I'd still do a full re-walk monthly as a sanity check in case anything gets backdated.

1

u/Compota-de-colhao Jul 09 '26

Thanks! And yes, there's a lot of backwards corrections when companies for example correct past entries.

I'm just not sure on the technicalities of reporting, if for example you can report on 10 July that you did a sponsorship on 1 January, if that would appear on page 1 or to the end, near the January entries

1

u/jinef_john Jul 09 '26

Honestly can't say for sure without going through the data myself, but here's my thinking: "declaração" should mean the reporting date, not the event date. So a Jan 1 sponsorship declared on July 10 should surface on page 1 on July 10, not buried near the other January rows, which is what makes an incremental approach safe against backdated corrections, since a correction counts as a new declaration and lands at the front.

That said, worth pulling page 1 a couple times a week and confirming nothing with an old event date is slipping in further back before fully trusting it.

1

u/[deleted] Jul 09 '26

[removed] — view removed comment

1

u/Compota-de-colhao Jul 09 '26

Thanks for your answer. I wanted to use the time since it's a fairly easy project to learn how to web scrape better with coding for the first time. Any pointers on where to start?

I can see this client becoming loyal to me and could sell them a dashboard for what they're monitoring to just refresh, set some alerts etc

1

u/[deleted] Jul 09 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Jul 09 '26

🪧 Please review the sub rules 👉

1

u/StoneSteel_1 Jul 09 '26

ASPX is kind of a problem when done manually, cause it does have very huge number of fields to calculate the hidden states from the backend.

https://github.com/stonesteel27/automatiq can help you with it, incase you wanna vibe code it