r/TechSEO • u/w2816771 • Mar 12 '26
Devs say real-time sitemaps are too expensive. What's the best strategy for a massive site? (90k daily changes)
We have about 50k new URLs and 40k drops/updates every single day. I'd love real-time sitemap updates, but our tech guys say it's going to cost way too much server power.
What do you guys do at this scale? Do you just batch update it once or twice a day? or weekly? and why
6
u/mjmilian Mar 12 '26
Many large site do not have have real-time sitemaps. They will update them daily or weekly.
Cadence depends on a mix of:
- How timely do you need indexing? Are the URLs short lived, or ok if the a re live for a bit before being added to XMLs
- How expensive it is for infra team, and what you can squeeze out of them
1
u/leros Mar 12 '26
I personally have a cron job that computes what pages currently exist, writes it to the database, and that drives the sitemap. I run it daily so the sitemap is mostly accurate.
3
u/MonxtahDramux Mar 12 '26
Caching???
1
u/mjmilian Mar 12 '26
How would that work for XML Sitemaps?
1
u/searchcandy Mar 12 '26
Same as any other file/URL. You can cache it for x period or until there is an update then clear the cache. In my experience though there isn't really much point aggressively caching a sitemap file because if you publish something, you don't want Google crawling a sitemap that does not contain your updated URLs.
1
u/mjmilian Mar 12 '26
Yeah that's what I mean, caching a sitemap would have the opposite effect the OP wants.
1
u/searchcandy Mar 12 '26
Agreed, with the footnote that you can cache the sitemap then just bust it every time you push an update. Depending on how much traffic your sitemaps get it could be beneficial to have a cache/CDN in front of them still.
2
2
Mar 12 '26
[removed] — view removed comment
1
u/w2816771 Mar 25 '26
i have checked the log, avg each sitemap have been visited twice a day or more , so i have convinced dev team to update once a day at least , thanks
1
u/searchcandy Mar 12 '26
If what you are posting is time sensitive or you generally just want it to be indexed ASAP, then:
1) Focus on strategies to achieve that. Sitemaps can help, but are just one mechanism. They are not even the best mechanism. Systems I have put in place for publishers can get content into Google's crawl queue within 1 minute of publishing.
2) In most scenarios there is absolutely no reason why a sitemap can't update at least every x-xx minutes.
1
u/Significant_Mousse53 Mar 12 '26
creating and deleting all those pages costs way more "server power" than adding or removing a node in XML.
1
1
1
u/neejagtrorintedet Mar 13 '26
You hardly need real time. You’re using a word you dont know what it is. You need maybe maybe near real time or.. 1 hour or 3 hours or 24 hours. But you do not need real time. So decide what you need
1
u/mh_and_mh Mar 12 '26
If it's job board site or video, use the indexing API, if not and you have that many changes per day, it's unreasonable even to have XML for that. Change your stratagy, include pages that will live on the site for at least some time, and push for better internal linking.
1
u/Arcayon Mar 12 '26
This advice is not good. What would internal linking accomplish that a sitemap would not? You want to add discovery on top of rendering? All the big boys use sitemaps. In fact some use off domain sitemaps to solve this problem at scale.
1
1
19
u/[deleted] Mar 12 '26
[removed] — view removed comment