The City Planner's Mistake: On Building Roads Without Asking Why

In the rush to get indexed, it’s easy to think like a road engineer, not a city planner. The engineer’s question is simple: how do I efficiently connect point A to point B? They lay asphalt in the most logical, direct lines. They build interchanges. They optimize for throughput. In our world, this is the sitemap: a pristine, efficient blueprint of every destination we have. It’s the XML file that says, "Here are all my pages. Here are the roads. Drive here." It feels like the responsible, complete thing to do.

But a city planner asks a different, more humbling question first: why would anyone want to go there? They look at land use, at community, at desire paths worn in the grass by feet taking a shortcut the asphalt ignored. They understand that a road to nowhere, no matter how perfectly paved, is a monument to wasted effort. It decays from neglect. It becomes a burden to maintain.

This is the subtle trap of the perfect crawl. We hand the search engine's crawler a master-planned grid of every cul-de-sac and utility shed in our digital territory, and then we wonder why its budget is spent idling in empty lots instead of visiting our bustling town square. The crawler, dutiful but unimaginative, follows the roads we built. It wastes its finite visits—its crawl budget—on pages that have no reason to exist beyond our own bureaucratic completionism. We built the road, so someone must drive on it, right?

The lesson from urban planning is to stop building roads for the sake of having a complete map. Instead, we must first cultivate destinations worth discovering. A page should earn its pathway. Its reason to exist should be woven into the fabric of the site through genuine, thematic connections from other pages—those are the organic desire paths. A sitemap should then function less like a master blueprint and more like a helpful signpost for destinations that have already proven their worth to travelers.

The Maintenance of Ghost Towns

Every abandoned page in your sitemap is a digital ghost town. The crawler visits, finds no life, no fresh water, no reason to stay, and moves on, having consumed fuel for the journey. Over time, these phantom roads divert traffic and resources from the thriving parts of your site. The city planner knows that blight spreads; a neglected block affects the whole neighborhood's value.

Our work, then, is not in the frantic paving of new roads, but in the thoughtful curation of a living city. Before you add another URL to the sitemap, ask the planner’s question: why would anyone, or any crawler, want to go there? If the answer isn’t clear, if the page isn’t already connected by the foot traffic of your own internal links, then you’re not building a road to discovery. You’re drafting a plan for future decay. Let the crawler find the vibrant places first. The best maps are drawn by the people who actually live there.

Notes & further reading

A few pages I came back to while writing this: