The Deliberate Detour: On Building a Path for the Lost Crawler
We spend so much time thinking about how to guide crawlers to our most important pages. We optimize, we interlink, we submit sitemaps. But we rarely consider the opposite scenario: what about the pages we don’t want them to waste time on? The ones that are necessary for a functioning site but are meaningless to an index? I’m talking about the labyrinth of ‘Thank You’ pages, the infinite session IDs, the dynamically generated filters that lead nowhere. Left unchecked, a crawler can get lost in these cul-de-sacs, burning its precious budget on digital drywall.
This isn’t about blocking them outright with a robots.txt sledgehammer. That’s a crude tool, and often, these pages need to be accessible to users and some crawlers. The more elegant technique, the one we often overlook, is the deliberate use of the ‘nofollow’ link attribute. We typically think of ‘nofollow’ as a way to sculpt PageRank or flag untrusted content. But its most practical, daily use is as a simple signpost for the crawler: ‘Do not turn down this road.’
Here’s the concrete how-to. Audit your site’s pathways. Find every point where a user action generates a non-essential, one-off, or parameter-heavy URL. The ‘Log Out’ button. The ‘Add to Cart’ action. The ‘Filter by Color: Burnt Sienna’ link. For each of these, in your HTML, you will add `rel="nofollow"` to the hyperlink.
This isn't a directive not to crawl; it's a suggestion not to follow. It’s a whisper to the crawler that the link it’s about to pursue is not part of the primary narrative of your site. It’s a detour, a utility tunnel, a backstage pass. By marking these links, you are not hiding anything. You are simply clarifying the hierarchy of your content. You are telling the crawler, ‘The real substance is over here, in the main articles and product pages. Don’t bother with the receipt printer.’
The effect is subtle but profound. You are not cutting off access; you are optimizing the journey. The crawler, with its limited time and resources, will spend more of its visit tracing the connections between your truly valuable content. It will index more deeply instead of more broadly. It’s a act of curation, a way of building a better, more intelligible map for the machine that is trying to read your work. By creating these deliberate detours, we aren't building walls. We're just putting up better street signs, ensuring the crawler's time is spent not wandering alleys, but exploring the city we actually built.
Notes & further reading
A few pages I came back to while writing this: