The Broken Compass: On Why Getting Lost Might Be the Point

There is a well-worn path in the discourse of web discovery, one beaten flat by years of earnest advice: that our websites require meticulous structure, comprehensive sitemaps, and a clear hierarchy to be properly understood. We are told to be cartographers of our own domains, leaving no page uncharted, no link unaccounted for. This advice is as sensible as it is stifling. We have become so focused on building the perfect internal compass for the crawler that we’ve forgotten the value of letting it, on occasion, get a little lost.

The obsession with total control stems from a fear of the crawl budget, that mythical portion of a search engine’s attention it’s willing to spend on our little corner of the web. We’re terrified of wasting it, so we sterilize our sites. We prune back internal links until only the most “valuable” pages remain connected. We funnel journeys down pre-approved corridors, afraid that a detour into a forgotten archive or a tangential thought might be seen as a squandering of resources. The goal, we are told, is to make discovery frictionless for both machine and human. But in the process, we strip away the very texture that makes a website feel like a place to be explored, not just a vending machine for predefined content.

The Serendipity of the Broken Link

Consider the physical library. The most memorable finds are rarely the books you went in for. They are the volumes shelved upside down, the one peeking out from behind another, the discovery made while wandering stacks ordered by a system you don’t quite understand. This serendipity is a feature, not a bug, of analog discovery. Our digital spaces, by contrast, have been engineered to eliminate this “inefficiency.” But what are we losing?

An overly pristine sitemap and a rigid internal link architecture create a predictable, sterile browsing experience. They tell the crawler—and by extension, the user—exactly what we think is important. This is a form of curation, yes, but it is also a declaration of limits. It says, “Your journey ends here. There is nothing more to see.” When we leave a few threads dangling, when we allow a section of the site to be reachable only by a few obscure, tangential links, we are not being negligent. We are creating pockets of mystery. We are building in the possibility for a crawler, stumbling upon a page through an unexpected route, to index a connection we hadn’t thought to explicitly authorize.

This isn’t an argument for chaos. Technical errors—like broken links or pages blocked by robots.txt that are still linked to—are genuine problems. The argument is rather for intentional imperfection. For allowing a small, curated amount of the “unnecessary” to persist. A labyrinth, after all, is not a maze; it is a single, winding path designed to provoke contemplation. A well-designed website can have a little of that labyrinthine quality. It can have a page that is three clicks off the main path, accessible only if you followed a particular line of thought in a blog comment or an old, archived newsletter.

By fixating on the crawl budget as a scarce commodity to be hoarded, we risk optimizing the soul out of our sites. Perhaps the real resource we should be mindful of is not the crawler’s time, but the user’s capacity for wonder. Sometimes, the most profound discovery happens not when you follow the compass, but when you put it away and let the path itself be the guide. A little deliberate disorientation, it turns out, might be the very thing that makes your corner of the web worth staying in.

Notes & further reading

A few pages I came back to while writing this: