The Urban Planner and the Crawl Path: On Designing for Discovery
I was reading about a city planner the other day, a woman tasked with revitalizing a neglected historic district. Her challenge wasn't just about restoring old buildings; it was about re-weaving the district back into the fabric of the living city. She spoke not of demolition and new construction, but of creating desire paths—of making it intuitively obvious how to move from the bustling main avenues into the quieter, interesting side streets. It struck me that her philosophy is a perfect, if unexpected, blueprint for how we should think about structuring a website for discovery.
Search engine crawlers are not unlike new visitors to a city. They arrive with limited time and a mandate to map what’s important. They start on the main thoroughfares—your homepage, your primary category pages—and then look for clear, logical paths to explore further. The urban planner knows that if you only build wide, empty plazas with no clear exits, people will get confused and leave. Similarly, a site with a beautiful homepage but a convoluted, deep, or broken internal linking structure is a city with blocked alleyways and dead-end streets. The crawler, like the visitor, will simply turn back.
The Architecture of Flow
This planner’s key insight was ‘permeability’—the ease with which one can move through a space. A highly permeable neighborhood has multiple connection points, short blocks, and visible destinations. Translating this to our realm, a highly permeable website has a clean, logical information hierarchy with strong, contextual internal links. It doesn’t hide its valuable content behind five clicks and a JavaScript-rendered menu. It creates a visible trail of breadcrumbs from the central square all the way to the artisan’s workshop in the old quarter.
Her second lesson was in prioritizing maintenance. A city doesn’t just build new roads; it relentlessly repairs the old ones. It fixes potholes, clears debris, and updates signage. For us, this is the unglamorous work of SEO hygiene: ruthlessly finding and fixing broken links (the collapsed bridges of the web), removing or updating outdated content, and ensuring redirects are in place for moved pages. A crawler, upon hitting a 404 error, is like a driver hitting a roadblock—it wastes its limited journey time and fails to reach the destination. A well-maintained site assures the crawler that its time is valued, encouraging a deeper, more thorough exploration.
We often get lost in the technical specifications of crawling—the `robots.txt`, the XML sitemap, the crawl budget metrics. These are our zoning laws and official maps. They are necessary, but they are not the city itself. The true experience, for both a human and a bot, is defined by the lived-in architecture of the place. By borrowing the urban planner’s mindset, we stop just building pages and start designing an entire environment for discovery. We become less like isolated architects and more like civic planners, thoughtfully designing a space that invites thorough, intuitive, and valuable exploration.
Notes & further reading
A few pages I came back to while writing this: