Pagination And Crawl Waste

Search engines allocate a finite amount of attention to any site. On a small service site that never matters. On a site with filters, sorts, paginated archives and parameter URLs, it matters a lot.

The symptom is simple: the pages you care about get crawled slowly, and the index fills with things you never wanted in it.

Where the waste comes from

Filter and sort parameters. One category page with five filters and three sort orders can generate dozens of URLs containing the same products in a different arrangement.

Session and tracking parameters, which create a unique URL for every visitor if the configuration allows it.

Endless pagination, archives running to page 40 where pages 5 onward contain nothing anybody searches for.

Tag archives, which on most blogs are automatically generated, thin, and duplicative of the category pages.

Internal search result pages, which should never be indexable and frequently are.

Printer friendly versions, staging copies left live, and trailing slash variants of the same page.

Each of these is a URL the crawler will fetch, assess and mostly discard, using attention that could have gone to a service page.

Fixing the parameter problem

Canonical tags on every filtered and sorted variant, pointing at the clean version of the page. This is the main control and it has to be consistent.

Do not link to parameter URLs internally, because internal links are the strongest instruction you give about what matters.

Robots rules for the paths that should never be crawled: internal search, cart, account pages, anything behind a parameter that exists only for the interface.

Noindex rather than a robots block where a page needs to be crawlable but should not appear in results. Blocking a page in robots means the noindex is never read, which is the mistake that keeps pages in the index forever.

Pagination done properly

The question on any paginated archive is whether page 3 has a reason to exist in search.

Usually it does not, which means page 1 is canonical and indexable, and the rest are crawlable, linked, and not indexed. The articles themselves are reached from the sitemap and from internal links rather than from deep archive pages.

What matters is that the crawler can still reach everything. Pagination that is loaded by script with no real links in the markup is a wall, and the content behind it may as well not exist.

On very long archives, the answer is usually better structure rather than better pagination: real category pages with editorial content, each covering a topic, instead of a chronological list forty pages deep.

Faceted navigation on a product site

The hardest version of this problem, and the one worth real thought.

A small number of filter combinations have genuine search demand and deserve indexable, linked, canonical pages with their own copy. Material and application, say, or product type and size.

Everything else is interface. Crawlable if it has to be, never indexed, never in the sitemap, never linked from a template that multiplies it.

Getting that split right is the difference between a site with twenty useful category pages and a site with four thousand variants of the same one.

Keep the sitemap honest

The sitemap is a statement about what you want indexed. It should contain only canonical, indexable, 200 responding URLs.

Which means no redirects in it, no 404s, no noindexed pages and no parameter variants. A sitemap full of junk is treated as a less reliable signal, and on a large site it is worth splitting into separate files by section so coverage can be read per type rather than as one number.

How to see the problem

Search Console coverage, read by category rather than by total. Crawled and not indexed in the thousands on a site with two hundred real pages is the signal.

Then the server logs, which show what is actually being fetched and how often. On a large site that is the only honest view of where attention is going.

What this unlocks: crawl effort spent on the pages that earn money rather than on permutations of a filter.