The Urban Planner's Grid: On Structure, Serendipity, and the Crawlable City
Urban planners don't build cities from the top down. They lay down a grid—a foundational structure of streets and avenues—and then let life fill in the spaces between. A website is a city of information, and the way we plan its structure, or lack thereof, dictates how effectively a search engine's crawler can navigate its streets and discover its inhabitants: the pages.
Consider the rigid, predictable grid of Manhattan. Its numbered streets and avenues create a logical, highly navigable system. A crawler, like a tourist with a map, can efficiently traverse from 5th Avenue to 10th, knowing exactly what to expect. This is the ideal of a well-structured information architecture: clear hierarchies, logical internal linking, and a clean sitemap. The crawler’s path is optimized, its budget spent on thorough exploration, not on getting lost in dead-end alleys.
But the most interesting parts of a city are often the exceptions to the grid. The diagonal slash of Broadway cutting through the order, creating dynamic intersections like Times Square. These are the unexpected connections, the links from a deeply technical blog post to a whimsical ‘about us’ page. A crawler, in following these diagonal paths, experiences serendipity. It discovers content not because it was next in a logical sequence, but because a thoughtful connection was made. This is the value of contextual, deep linking—it introduces the crawler to the city’s vibrant, organic life beyond the master plan.
The Cost of Sprawl
Urban sprawl is a planner's nightmare: inefficient, resource-draining, and difficult to service. Website sprawl is a crawler's nightmare. Endless pagination, thin content on countless similar URLs, and orphaned pages in distant suburbs of your site—this is how crawl budget is squandered. The crawler becomes a exhausted commuter, wasting fuel on long, repetitive journeys that yield little new value. It may never reach the truly valuable, unique neighborhoods of your content because it’s stuck in traffic on a highway to nowhere.
The lesson from urban planning is not to eliminate spontaneity in favor of pure, sterile order. A city with only straight lines is a boring city. The goal is to establish a strong, sensible grid—your site’s core structure—that makes navigation fundamentally efficient. Upon that grid, we then build the delightful diagonals of internal links that encourage discovery and exploration. We zone our digital real estate to prevent content sprawl, ensuring our most valuable properties are easy to find and connected to the main thoroughfares.
We must build our sites not as mazes to be solved, but as vibrant, walkable cities. We provide the clear street signs of breadcrumbs and sitemaps, the efficient public transit of a robust internal link structure, and we actively demolish the dilapidated buildings of low-value content. In doing so, we invite the crawler to be a flâneur, an explorer who can both efficiently cover ground and still stumble upon our hidden gems. A well-planned city is a discoverable city. A well-structured site is a found site.
Notes & further reading
A few pages I came back to while writing this: