The Cartographer's Dilemma: On the Ghost Streets in Our Sitemaps

Every cartographer knows the pull of the phantom street. It’s the lane drawn from a hundred-year-old survey, the alleyway that has since been swallowed by a new development, the path that leads only to a brick wall. It exists on the map with perfect clarity, a promise of passage, but on the ground, it is pure absence. In the quiet work of guiding the web’s indexers, we face our own version of this dilemma with every sitemap we file away.

A sitemap, in its ideal form, is a gesture of pure honesty. It’s a website’s way of sitting down with a search engine and saying, “Here is my entirety. These are the places I have built for visitors. This is the path you should take.” It is a document of intent. But intentions, like cities, change over time. Pages are retired; products are discontinued; blog posts are pruned. The sitemap, however, can linger in its last submitted state, a proud chart of a territory that no longer matches the one we actually inhabit.

This creates a peculiar kind of ghost. The crawler, a diligent student of our provided maps, dutifully follows the coordinates to a page that promises a ‘Summer Workshop Schedule’ or a ‘Latest Product Launch.’ But when it arrives, it finds only a 404 silence or a redirect to a generic homepage. The crawler expected a bustling plaza and found an overgrown lot. It’s not that the page was hidden or forbidden; it was simply forgotten, its retirement party never communicated to the mapmaker.

The dilemma is subtle. On one hand, we want our sitemaps to be comprehensive, a complete atlas of our domain. We fear that by removing an entry, we are shrinking our digital footprint, erasing a potential point of discovery. But the greater loss is one of trust. Each time a crawler encounters one of these ghost streets, it learns a little less faith in our guidance. It begins to question the reliability of the map itself. Why should it invest its limited crawl budget following paths that often lead to dead ends? The sitemap shifts from being a trusted guide to a historical document of questionable accuracy.

And so, the modern cartographer’s most crucial task is not just addition, but diligent subtraction. It is an act of curation that requires a certain ruthlessness. It means regularly walking the routes yourself, comparing the lived reality of the site to the perfect grid of the XML file. It means accepting that a smaller, perfectly accurate map is infinitely more valuable than a vast one dotted with phantoms. The integrity of the path is what guarantees the journey will be worth taking, both for the mechanical crawler and, ultimately, for the human visitor it hopes to bring our way.

Notes & further reading

A few pages I came back to while writing this: