Crawl budget for ecommerce: where Google spends its time on your store
Google only crawls so many pages of a site in a day. On a small store that never matters. On a store with thousands of products, filters and tags, it decides how quickly new and updated products are seen — and whether some are seen at all.
Start your free trial
When crawl budget is a real problem
Google’s own guidance is that crawl budget matters mainly for large sites, or medium-sized sites whose content changes often, and for sites with many pages stuck in Discovered – currently not indexed. Online stores fit that description more often than any other kind of site, because platforms generate addresses: every filter, sort order, tag and page of a category is a new URL.
Most crawl budget advice stops at “reduce duplicate URLs”. The useful question is which ones, on your site, and how many. Your sitemap says what you claim exists; 90 days of Search Console says what Google has actually shown anyone. The gap between the two is where crawling is being wasted.
Crawl budget is also about demand. Google crawls more of a site whose pages it values, so thin and duplicate product descriptions lower the crawl rate for the whole store, not just themselves.
Where store crawl budget usually goes
- Faceted navigation: Every filter combination — colour, size, price, brand — as its own crawlable address.
- Paginated archives: Page 7 of a category or blog archive listed in the sitemap.
- Tag pages: Thin copies of categories that list the same products.
- Tracking and query parameters: utm and session parameters creating duplicate addresses.
- Internal search results: Search pages that can be generated without end.
- Feeds and alternate formats: RSS and other formats listed as if they were landing pages.
- Errors and redirects in the sitemap: Addresses Google is sent to that are broken or moved.
- Pages never shown: Sitemap URLs with no Search Console impressions in 90 days.
How to win crawl budget back
- Keep the sitemap to live, canonical, indexable pages: no redirects, errors, parameters, feeds or deep pagination.
- Keep internal search and pointless filter or sort combinations out of crawling, and canonicalise filter pages that should not rank.
- Noindex or remove tag pages nobody browses.
- Make server responses fast and stable — Google crawls less when a site is slow or errors.
- Strengthen the pages you do want crawled: unique content and internal links from pages Google already visits.
Questions people ask
- Does my store have a crawl budget problem?
- Probably not if it has a few hundred pages and Google indexes them. Look closer if you have thousands of products, many pages in “Discovered – currently not indexed”, or new products that take weeks to appear.
- How does NicheSEO Pro find crawl waste?
- Its crawl budget report compares your sitemap with 90 days of your Search Console data and groups the waste by kind, with a count of real addresses and examples for each — no generic tips.
- Should I block filter pages in robots.txt?
- Only combinations no one searches for, and only if they do not carry a canonical or noindex you need Google to see. Filter pages that match real searches can be worth indexing — see collection page SEO.