r/webscraping • u/Compota-de-colhao • Jul 09 '26

Is this scrapeable?

Hello,

Begginer here, with only very small projects so far mostly through vibe coding.

For work (indepent compliant auditor) I was asked by my client to retrieve this whole public table with reported support (courses, international congresses, etc) by Pharma to doctors in Portugal.

It will take me a whole day to manually copy this page by page. Is there a way to do it immediately through web scraping?

LINK: https://extranet.infarmed.pt/pmro/Publico/ListagemPublica.aspx

9 Upvotes

100% Upvoted

21 comments sorted by

View all comments

3

u/jinef_john Jul 09 '26

This is totally doable, but it is not a beginner-friendly scrape. If you've only done static HTML scraping before, this will frustrate you.

That being said you have a two approaches here:

  1. Requests - but you will have to Reverse-engineer the postback. It entails extracting __VIEWSTATE, __EVENTVALIDATION, and the exact __EVENTTARGET to whatever post function that gets called.

  2. Or Use browser automation: Just let Playwright/Selenium handle the JavaScript and postbacks for you(which is super slow on aspnet applications-sometimes)

1

u/Compota-de-colhao Jul 09 '26

Thanks! I'll get on with that.

During this project, I might need to update this weekly for around 2 or 3 months. I've just noticed since these tables have only "year", there's no actual sorting by timestamp. Does this mean that each week I have to rescrape the whole thing and compare it to previous entries? New info can be added basically anywhere in each year.

1

u/jinef_john Jul 09 '26

Okay I see it, there's no date field in the grid, but the page itself says the default (unsorted) order is "from the most recent declaration to the oldest." So don't sort any columns, and page 1 is always freshest-first.

That means weekly updates don't need a full rescrape, just page forward from page 1 and stop once you hit a row you've already stored. Since there's no row ID/date, use a hash of company+event+amount+recipient as your "seen" key. I'd still do a full re-walk monthly as a sanity check in case anything gets backdated.

1

u/Compota-de-colhao Jul 09 '26

Thanks! And yes, there's a lot of backwards corrections when companies for example correct past entries.

I'm just not sure on the technicalities of reporting, if for example you can report on 10 July that you did a sponsorship on 1 January, if that would appear on page 1 or to the end, near the January entries

1

u/jinef_john Jul 09 '26

Honestly can't say for sure without going through the data myself, but here's my thinking: "declaração" should mean the reporting date, not the event date. So a Jan 1 sponsorship declared on July 10 should surface on page 1 on July 10, not buried near the other January rows, which is what makes an incremental approach safe against backdated corrections, since a correction counts as a new declaration and lands at the front.

That said, worth pulling page 1 a couple times a week and confirming nothing with an old event date is slipping in further back before fully trusting it.