.png){kind=logo}
Crawl Budget: Why Google Isn't Indexing All Your Products
You have 3,000 products and Search Console shows 900 indexed. This is one of the more common and more confusing e-commerce SEO situations, and the usual explanation is crawl budget — the practical limit on how much of your site search engines will crawl in a given period. Here's what that actually means and, more usefully, what wastes it.
Key Takeaways
- ✓ Crawl budget only becomes a real constraint on large sites, typically thousands of URLs.
- ✓ Parameter and filter URLs consume the majority of wasted crawl on most stores.
- ✓ Slow server response reduces how much gets crawled, independent of site size.
- ✓ Thin or low-value pages compete for the same budget as your important ones.
What Crawl Budget Actually Is
Search engines allocate a finite amount of crawling to each site, based roughly on how important the site appears and how well the server handles requests. For a 40-page brochure site this is irrelevant — everything gets crawled easily. For a store generating tens of thousands of URLs, it becomes a genuine constraint on what gets discovered and refreshed.
The Biggest Waste: Parameter URLs
On most stores, the majority of crawl budget goes to URLs nobody should be landing on. Every filter combination, every sort order, every tracking parameter creates a crawlable URL. A category with six filters can generate hundreds of URL variations, all containing broadly the same products. Crawlers work through them while your actual product pages wait.
- — Filter and sort parameter URLs
- — Session IDs or tracking parameters in URLs
- — Infinite pagination on large categories
- — Internal search result pages left crawlable
- — Out-of-stock or discontinued products still linked
Server Speed Is a Crawl Factor
This is under-appreciated. If your server responds slowly, crawlers reduce their request rate to avoid overwhelming it. A slow site therefore gets crawled less, independent of how large it is. Improving server response time can increase how much of your catalogue gets discovered and how often it gets refreshed — a genuine SEO benefit from a purely technical change.
Thin Pages Compete With Good Ones
Every low-value page — a tag archive with two items, an empty category, a product with three words of description — consumes crawl that could have gone to a page worth ranking. Consolidating or removing thin pages isn't only about quality signals; it's about not spending a finite resource on pages that will never earn anything.
How to Tell If It's Actually Your Problem
Search Console's crawl stats report shows how many requests Google makes and what it spends them on. If a large share of crawl requests go to URLs with parameters, that's your answer. If your indexed page count is far below your product count and your crawl requests are dominated by filter URLs, the two facts are connected.
Where This Sits Relative to Other Problems
Honest caveat: crawl budget is frequently blamed for problems that are actually about content quality or internal linking. A product page with no internal links and a thin description isn't unindexed because of crawl budget — it's unindexed because nothing suggests it's worth indexing. Rule out orphan pages and thin content before concluding it's a crawl constraint.
Making Large Catalogues Crawlable — With Devrex Digital
Devrex Digital builds stores with parameter handling, canonical structure and server performance configured so crawl budget goes to pages that matter. If a large share of your catalogue has never been indexed, the cause is usually identifiable from Search Console's crawl stats within an hour.
FAQs
It generally starts mattering in the thousands of URLs rather than hundreds. Below that, unindexed pages are almost always caused by thin content or missing internal links rather than crawl constraints.
Ready to start your project? Devrex Digital is a web development agency in Islamabad building custom coded websites for businesses across Pakistan.
Get a Free Quote →