The Librarian's Unwritten Rule: On the Quiet Power of the Unsubmitted Page
Conventional wisdom in our field is a gospel of submission. We are told to meticulously craft our sitemaps, to submit them proudly to search consoles, to announce our updates and demand the attention of the crawler. We treat discovery as a formal request, a document filed in triplicate. The underlying belief is that a page unsubmitted is a page unseen, a whisper lost in the wind. But what if this act of eager submission is, in some cases, the very thing that muffles the whisper?
Consider the library. A patron does not formally submit a request for a book to the head librarian for every volume they wish to read. They find it through a card catalog, a recommendation from a friend, or by browsing the shelves and noticing a familiar author or an intriguing title. The most organic discoveries happen not by decree, but by connection and context. The web, at its heart, is the same. Its greatest strength is not a top-down directory but a sprawling, interconnected web of context.
By focusing exclusively on the sitemap as the primary key for discovery, we risk creating a sterile, clinical version of our site for the crawler—a site stripped of the very context that gives its pages meaning. A page that is found naturally, through a link from a deeply relevant external site or through a thoughtful internal contextual link, carries a different kind of signal. It doesn’t just say "I exist." It says "I exist here, among these ideas, as a part of this conversation." This is the difference between a name on a census form and an introduction from a trusted colleague.
The Uncharted Path Holds the Strongest Scent
Crawlers are designed to follow scent. The logic of a link, especially one with thoughtful anchor text, is a powerful trail of breadcrumbs. A page that is only listed in a sitemap has a location, but no journey to get there. It is a coordinate on a map with no roads leading to it. A page found via a link, however, has a history and a reason for being found. It suggests value, relationship, and relevance that a mere XML entry cannot convey.
This isn’t to say sitemaps are without merit. For orphaned content or sites with poor internal linking, they are a vital safety net. But treating them as the first and best tool for discovery encourages a lazy architecture. It allows us to neglect the richer, more meaningful work of weaving our content into a cohesive tapestry where pages naturally lead to one another. The goal should not be to simply inform a crawler of a page’s existence, but to demonstrate its necessity within the ecosystem of your site. Sometimes, the most powerful statement a page can make is to be found without ever having asked to be seen.
Notes & further reading
A few pages I came back to while writing this: