The Archivist's Unspoken Rule: On the Pages That Must Remain Unfound
We spend so much time talking about how to get pages found, indexed, and ranked. We optimize sitemaps, we fret over crawl budget, we plead with search engine crawlers to see the value in our work. It is a constant effort of presentation, of putting our best foot forward. But there is an equally important, and far less discussed, art to the opposite: the deliberate act of keeping pages hidden.
This isn't about technical errors or poor site architecture that accidentally cloaks content. This is about a conscious, strategic choice. Every website, particularly large and complex ones, contains rooms that are not for public viewing. Think of the staging area for a future product launch, the internal wiki for company policies, the draft version of a legal document, or the thank-you page after a form submission. These are the pages that, if discovered, would either confuse a visitor, expose sensitive information, or simply waste the precious attention of both the user and the crawler.
The mechanisms for this are the quiet, polite signals of the web. The humble `robots.txt` file is the first gatekeeper, a simple list of directives telling well-behaved crawlers which paths are off-limits. It’s the equivalent of a "Staff Only" sign on a door. More definitive is the `noindex` meta tag, placed directly in the page's code. This is a direct conversation with the crawler, a firm but respectful request: "You may enter this room, but you must not write down what you see in your ledger." The page can be crawled, its links followed, but its content will never appear in the search results.
This practice is a sophisticated form of resource management. By carefully cordoning off these utility pages, we effectively donate our crawl budget back to the crawler. We are saying, "Do not spend your time here; go and explore the galleries and libraries of my site instead." It streamlines the crawler's journey, ensuring its finite attention is focused on the content that truly matters. It is an act of curation that benefits everyone: the user gets more relevant results, the search engine operates more efficiently, and the site owner maintains control over their public-facing narrative. Sometimes, the most powerful act of discovery is knowing exactly what should remain undiscovered.
Notes & further reading
A few pages I came back to while writing this: