The False Economy of the 'Perfect' Sitemap
There’s a piece of advice that gets passed around webmaster circles with the solemnity of ancient wisdom: your XML sitemap must be pristine. It should be a perfectly curated list, they say, containing only your most valuable, canonical pages. To include anything less is to waste the crawler’s precious time and, by extension, your crawl budget. It sounds logical, efficient even. But in practice, this dogma of perfection can become a form of self-sabotage.
The logic behind the 'perfect sitemap' is seductive. By presenting a crawler with a flawless itinerary, you’re supposedly guiding it directly to the gold, preventing it from wasting cycles on the rough. It’s the digital equivalent of only showing a tourist the Eiffel Tower and the Louvre, sparing them the back alleys. But a web crawler isn't a tired tourist with limited vacation days; it's a relentless, curious machine. Its entire purpose is to explore alleys.
This philosophy makes a critical, flawed assumption: that we, as site owners, have a perfect and unchanging understanding of what a search engine will deem valuable. We don't. I’ve seen it time and again. A deeply nested support article, hastily added to a sitemap during a site migration just to 'be thorough,' unexpectedly becomes a top-traffic driver because it perfectly answers a nascent query. A dated blog post, deemed unworthy of the main list, gathers backlinks over years and suddenly becomes an authority page—if only a crawler could be reminded it exists.
By overly sanitizing our sitemaps, we aren't just hiding our 'rough' pages; we're hiding our potential. We are making a calculated bet that our present judgment is superior to the crawler’s future discovery. We are silencing pages before they have a chance to speak for themselves. The sitemap isn’t just a list of recommendations; it’s a shout into the void. Why are we whispering?
The real waste of crawl budget isn't a crawler briefly glancing at a thin page. It’s the crawler failing to find a page that later becomes crucial because we were too cautious to list it. It’s the internal link that never gets followed because the page was deemed unworthy of the master map. The goal should not be a perfect sitemap, but a complete and honest one. Let the crawler see the whole store, not just the showroom floor. Trust it to do its job, which is to discern value, not just to consume a pre-packaged meal. Sometimes, the most valuable discoveries are waiting in the alleys we were too afraid to mark on the map.
Notes & further reading
A few pages I came back to while writing this: