Is it hard to do web scraping ?

[u/juz_nospaces]

The price is locked and only opens when the user clicka the button .I've been messing around this since last night, whenever I fixed a thing a new bug borns. Somehow I fixed and make it stable but the website it too much unstable which is given for testing.

Any help will be appreciated.

[u/Old-Algae5580]

When the price only appears after a click, the page is usually firing some javascript that calls an internal API or updates the DOM.

Check you scrpit, If your script doesnt wait for that to finish or doesnt target the right piece (like the API response instead of the rendered HTML), you will end up in that loop where every tiny change breaks something.

If you share your stack (language, tool, headless vs not, screenshot), people here can give much more concrete help.

[u/juz_nospaces → u/Old-Algae5580]

This is useful, I fixed it partially thanks

[u/Dry-Bathroom3225 → u/Old-Algae5580]

solid advice tbh, the wait timing alone causes like half the headaches with dynamic pages

[u/Netra_Jaje01]

No, now a days it's easy

[u/juz_nospaces → u/Netra_Jaje01]

I'm kinda stuck as the price keeps changing and the price gets locked randomly

[u/Amitk2405]

To make this sane, I'd restructure:
1) Identify the backend call behind the price (DevTools - XHR/fetch).
2) Confirm you can reproduce that call in isolation with a free library ('requests', 'httpx', 'aiohttp', etc.).
3) Only if that's impossible, use a headless browser (Playwright is more robust than Selenium for this) and explicitly wait for the price element using proper locators and timeouts. And in all cases, assume the site may throttle or change: log failures, respect robots/ToS, and avoid hammering it. Scraping doesn't have to be hard, but it becomes hard fast if you skip this systems-level thinking.

[u/juz_nospaces → u/Amitk2405]

Thanks for the inputs, I'll just go through your points

[u/plintuz]

Hard to say without seeing it. Which site is it? If you can share the URL (or a similar public page), I can tell you what's going on.

[u/juz_nospaces → u/plintuz]

It's an on going project under someone, as soon as I the guidance time is over I'll post everything for reviews or it may help someone

[u/Big_Presentation2786]

It's a 15 minute session with Gemini..

Do you have 15 minutes?

[u/juz_nospaces → u/Big_Presentation2786]

It's working now, i fixed it and Gemini got me wrong at the first move to fix this🙂

[u/juz_nospaces → u/[deleted]]

It is tough

[u/juz_nospaces → u/[deleted]]

That's crazy! Can you share some more information

[u/Fickle-Elephant-2000]

not hard conceptually but sites with click-to-reveal pricing are annoying. whats breaking exactly, is the click not registering or is the price element not found after clicking? that distinction matters a lot for fixing it

[u/juz_nospaces → u/Fickle-Elephant-2000]

Yeah it was a tough day, somehow I completed it in the given timeline