The Urban Planner's Lesson: Designing Streets, Not Just Lists, for the Crawler

We often talk about web architecture as if we’re building a library. We arrange shelves (directories), write labels (title tags), and create a master catalog (the sitemap). It’s a tidy, contained metaphor. But what if our site is less like a quiet repository and more like a bustling city? And what if the key to being discovered lies not in the buildings we erect, but in the streets that connect them? This is the lesson borrowed from urban planning: to design for flow, not just for footprint.

An urban planner knows that a grid of empty, identical lots is a ghost town. What creates a thriving district is the infrastructure between them—the width of the sidewalks, the frequency of intersections, the clarity of signage. They design for pedestrian traffic, understanding that the ease of moving from one place to another determines which shops succeed and which neighborhoods feel alive. On the web, the crawler is that pedestrian. It doesn't just land on a plot; it arrives via a street—a link. Our obsession is often with the plot (the page), while the crawler's experience is dominated by the journey.

Consider a common, well-intentioned mistake: the 'resources' page. It's a city block with a hundred doorways, each labeled, but with no connecting sidewalks. You've listed every tool, every guide, every article in one massive directory. From a library perspective, it's a comprehensive index. From an urban planner's perspective, it's a traffic dead-end. The crawler arrives, sees a hundred exits with no hierarchy or reason to choose one over another, and may retreat, having expended its “walking energy” on a single, confusing intersection. You built a list, not a street network.

The alternative is to design connective tissue. This means creating thematic boulevards. Instead of one page for “100 Gardening Guides,” you might have a few cornerstone pages: “Starting Your Vegetable Garden,” “Pruning Perennials,” “Seasonal Soil Care.” From each, a clear network of internal links guides the visitor (and the crawler) down specific, related paths. These are your main streets. The individual guides become storefronts along them, easily reached from a logical thoroughfare. The crawler flows, discovering page after page because the path forward is always obvious and meaningful.

In the end, the urban planner teaches us that crawlability is a byproduct of thoughtful connection. A sitemap is a useful aerial map, but the crawler lives on the ground. It experiences your site one link at a time, one corner at a turn. By designing streets—clear, hierarchical, purposeful internal link pathways—we don't just show the crawler where our buildings are. We invite it in for a walk, ensuring it discovers the vibrant districts we've worked so hard to build, simply because the journey itself is so compelling.

Notes & further reading

A few pages I came back to while writing this: