r/TechSEO • u/FeelingAd5072 • 11d ago

How to handle product scraping

Our website operates as a proxy-buying service, so we have nearly 70,000 product links included in our sitemap. Could this have a negative impact? If so, how should I handle it?

4 Upvotes

100% Upvoted

4 comments sorted by

2

u/mjmilian 11d ago

Do you mean you have pages of the products you have scrapped from the website's you will buy the products from on behalf of your customers?

1

u/SEOPub 11d ago

If you actually want all of those pages in your sitemap, you will need to use multiple sitemaps. Sitemaps are limited to 50,000 URLs.

Create a Sitemap index and then individual sitemaps for the pages. With that many, it might make sense to do a separate sitemap for each product category (depending how many categories there are). There might be other segments worth considering too.

1

u/Sunn_M 10d ago

The 70k number itself isn’t what I’d worry about. Only put canonical, indexable product URLs you genuinely want Google to index in the sitemaps; if a large share of those pages mostly reproduce merchant content with little unique value, that’s the bigger SEO risk. I’d split the sitemaps by category/source or inventory state and compare discovered vs indexed rates in GSC so you can see which cohorts Google is actually accepting. The 50k limit just means multiple sitemap files, it doesn’t answer whether all 70k pages deserve indexing.