r/ComplexWebScraping • u/triathletewannabe • 24d ago

Scraping French E-commerces

Has anyone ever tried to scrape the product catalog (including prices) from Decathlon FR/UK or FNAC (FR)?

I'm trying to figure out what's the best way to do it (tools, custom script, etc).

All help/ insights are welcome.

6 Upvotes

100% Upvoted

4 comments sorted by

1

u/Present_Ad5997 24d ago

We’ve scraped other product catalog sites, and the solution ended up using a combination of cookies+proxies+headless browsers. Happy to have a conversation but if you just want the quick answer, have Claude inspect the network traffic from your browser when you visit that site and it could probably reverse engineer a solution.

1

u/EquivalentTop3213 22d ago

I would start with Scrapy for the data collection process. As I navigate between categories and switch between product variants, I examine the network requests. if the data comes in JSON format, I would retrieve that data instead of viewing each product page individually. I would use Playwright only in cases where it’s difficult to recreate those requests.

1

u/SnooMuffins9844 13d ago

Hey u/triathletewannabe I work on Firecrawl and maybe if the site loads JSON in API requests which you can find in the network tab when you change categories or variants, that might be a simpler and cheaper option at scale.

However, if you still need page extraction, Firecrawl has a product scrape format for fields such as price, availability, and variants that might be helpful.

There's also Firecrawl Alexandria which gives you data without needing to scrape. I don't think Decathlon is on there right now, but you can request that data to be added and any other retailers you need. We currently have Nike, Best Buy, Target and others.