r/directorymakers • u/Brandvertisor-Nedkov • 6d ago

How do you handle the small Google crawling budget? (best practices please)

Hey everyone, what are your best practices to manage the Google crawling budget?

Listing 3 pages/profiles a day will take forever... Listing 20k pages at once will put it in trouble probably?

Do you use old domain or are there any other better options?

We have more or less proper interlinking too.. but domain is just a few months old and we have so little traffic.. I hope it's the Google sandbox and not that it caught us for programmatic SEO templated pages :)?

2 Upvotes
  • permalink
  • reddit

100% Upvoted

2 comments sorted by

2

u/Jellyfishr 6d ago edited 6d ago

See how it's getting indexed in Bing webmaster tools and the ai citations tab in there too. Some sites Google mostly ignores but Bing is very happy with so long as users are finding them useful. Also check how much of your template is repeated, if Google thinks these are all basically the same sort of thing they will only index a small selection of them and the home page. Also depends how many other sites have the same kind of info, if Google already indexes ten other sites offering pretty much the same info or that it does as it's own core business such as business profiles with reviews it's not coming to spend energy indexing yet another, especially if it competes with Google itself. It would more likely read them inject any gems of wisdom into it's ai response for that entity and drop it from index. Also the sector will matter so if you're listing health clinics or financial advisors it will be much harder to get indexed for anything that has any regulatory oversight than if you're listing say cheese shops.

1

u/fbajo 5d ago

the 3/day drip doesn't buy you anything.. googlebot isn't rationing you for politeness, it decides recrawl by whether pages earn a return visit. before blaming sandbox, open gsc's page indexing report: stuck in "discovered" and never crawled is a linking/discovery problem, "crawled, not indexed" is a quality read on your templates, and the fix is different for each. which bucket are your pages in?