The Lighthouse Keeper's Folly: On the Danger of Being Too Easy to Find

We are taught, from the moment we first learn to structure a website, to make discovery effortless. The prevailing wisdom is a gospel of clarity: a pristine sitemap, a logical hierarchy, a clean internal link structure. We aim to be the well-lit path in a dark forest, guiding the crawler’s every step with the gentle glow of a perfect information architecture. We want to be found. But what if, in our zeal to be perfectly navigable, we have built not a path but a prison?

The common advice assumes the crawler is a simple, grateful creature, one that will reward our meticulousness with its full attention. We imagine it as a guest in a meticulously curated home, admiring each room we present. But a crawler is not a guest; it is a force of nature. It is a tide that follows the path of least resistance, and in building that resistance-free channel, we may inadvertently instruct it to ignore the very shores we wish it to explore.

The Tyranny of the Perfect Path

Consider a website with a flawless, comprehensive sitemap and a primary navigation that leaves no page behind. To a crawler, this is a directive. It will dutifully follow this pre-approved tour, ticking off each URL on the list. Its budget—that finite amount of attention—is spent entirely on this official itinerary. The crawler sees the map, not the territory. It never has a reason to wander, to get delightfully lost down an alley of tangential links, to stumble upon a deeply nested but invaluable piece of content that exists outside the main flow.

This is the folly of the lighthouse keeper who shines his beam only on the main channel, leaving the unique and treacherous coves—the very places that might hold the most interesting wreckage—in perpetual darkness. By making the primary route too obvious, we signal that everything else is secondary, perhaps even irrelevant. We centralize discovery to the point of suffocating serendipity.

There is a counterintuitive value in a degree of obscurity. A page that is only reachable through a meandering journey of three or four clicks from an unlikely hub page isn’t lost; it is curated for a more intentional discoverer, be it human or algorithmic. This journey imbues the page with context and relational value that a flat sitemap entry can never provide. The links that lead to it are votes of confidence, not just from us, but from the content itself. It forces the crawler to engage with the fabric of the site, not just its outline.

The goal is not to hide our content, but to architect a landscape, not just a roadway. It is to build a site with a subtle topography that encourages exploration rather than mere transit. Sometimes, the best way to be found is not to stand directly under the brightest light, but to wait in the intriguing shadows just beyond it, confident that the most curious explorers always bring their own.

Notes & further reading

A few pages I came back to while writing this: