The Myth of the Perfectly Structured Site

Conventional wisdom in our field often sounds like advice from a meticulous urban planner: create a clean, logical hierarchy, with broad avenues (categories) leading to neat side streets (subcategories) and finally to individual houses (pages). We are told that a well-structured site is a crawler’s paradise, a place where discovery is a straightforward, predictable process. The sitemap is the official map, the internal links are the signposts, and the entire operation is meant to be a model of efficiency. I’d like to propose a different, somewhat messier truth: that this obsession with perfect structure might be quietly stifling the very discovery we aim to facilitate.

Consider how humans explore a city. Yes, we use the main streets, but the memorable discoveries—the hidden café, the quirky bookshop, the vibrant courtyard—are almost always found by wandering down an alley, by following a crowd, or by a chance recommendation. The perfectly planned city can feel sterile, devoid of surprise. Similarly, a website with an immaculate, rigid hierarchy often lacks the natural pathways that lead to unexpected connections. When every link is purely functional, serving only to move a user or a bot predictably up or down a taxonomic tree, we remove the potential for accidental discovery, which is often the source of the most valuable engagement.

This over-reliance on ‘correct’ structure places an enormous burden on the sitemap and the main navigation. It assumes that the crawler’s journey should be a systematic traversal of our pre-defined architecture. But crawlers, in their relentless pursuit of links, are also agents of chaos. They don’t just follow the tour guide; they follow the graffiti, the footprints, the whispers. This is where the underappreciated power of tangential linking comes in. A link from a deeply nested blog comment, a footnote in a long-form article, or a related-but-not-directly-connected resource page creates what I call ‘discovery capillaries.’ These are the small, often illogical connections that carry the lifeblood of a crawler’s attention to parts of the site that the main infrastructure might neglect.

The Beauty of a Little Chaos

This isn’t an argument for anarchy. A sound informational structure is crucial for human usability. Instead, it’s a plea to embrace a controlled wildness within that structure. Think of it as cultivating a garden rather than building a parking lot. A garden has planned beds and paths, but it also allows for self-seeding plants, for the way vines can unexpectedly connect two distant trellises.

In practical terms, this means creating content and linking strategies that aren’t solely dictated by the site’s primary taxonomy. It means allowing for, and even encouraging, links that are relevant but not hierarchical. A technical spec sheet for a product might link to a foundational research paper buried in the ‘About Us’ section. A personal story in a newsletter might link to a forgotten blog post from years ago. These connections are not mistakes in the blueprint; they are the features that give a website depth, character, and a more resilient network for crawlers to explore. They acknowledge that the map is not the territory, and that sometimes the most efficient way to be found is to create a web so rich and interconnected that discovery becomes inevitable, not just orderly.

Notes & further reading

A few pages I came back to while writing this: