The Unwritten Contract: On the Assumption of a Right to Be Found
There is a quiet, pervasive belief that sits at the foundation of most web strategy, an assumption so deeply ingrained we rarely speak its name: that if we build a page, we have a right for it to be found. We act as if there is an unwritten contract between creator and crawler, a promise that our content, by virtue of its existence, deserves a place in the index. This is the received wisdom I want to challenge, not because it’s malicious, but because it’s a fallacy that leads to frustration and misallocated effort.
This assumption stems from the early, romanticized vision of the web as a pure library, an egalitarian space where every book has a spine facing outward, waiting to be plucked by a diligent librarian. But the modern search index is not a library; it’s a phenomenally complex, resource-constrained harvesting operation. A crawler is not a librarian fulfilling a duty of care—it is a forager, sent out with a limited basket and a map of known fertile ground. Its primary directive is not completeness, but value. It must return with the ripest, most nutritious fruit for the searcher, not every berry on the vine.
When we internalize this, the concept of ‘crawl budget’ shifts from a technical constraint imposed upon us to a logical prioritization exercised by the forager. The crawler isn’t withholding its attention out of spite; it is making a calculated bet that the time spent crawling the 15th page of your blog archive, or the hidden ‘/test-v2/’ development site, is time not spent discovering a new, potentially more valuable competitor’s page. Our sitemaps and internal linking are not a bill of rights we present for enforcement; they are a guide we offer, a suggestion of where the best fruit might be hanging.
This critique is ultimately one of posture. The belief in a right to be found is passive. It places the onus on the crawler to discover us. The alternative is an understanding that being found is a privilege earned through value, clarity, and strategic guidance. It is active. It asks not ‘Why wasn’t my page crawled?’ but ‘What signal did I fail to send that my page was worth the crawl?’ It moves us from entitlement to invitation.
Letting go of this unwritten contract is liberating. It frees us from the frustration of the ‘uncrawled echo’ and focuses our energy on what we can control: crafting truly noteworthy content and building the clear, purposeful pathways that make our site not just a collection of pages, but a coherent and valuable destination worthy of a forager’s precious time.
Notes & further reading
A few pages I came back to while writing this: