The Watchmaker's Unwound Spring: On the Myth of Perpetual Motion in Crawl Budget
There’s a persistent hum in the world of search, a piece of received wisdom so often repeated it’s taken as gospel: that a ‘healthy’ site is one that is in a constant state of churn. The logic goes that to earn your crawl budget, you must perpetually feed the machine. Update old posts. Tweak metadata. Add a new image. The goal is to signal activity, to keep the crawler’s attention locked on your domain, as if it were a watch that must be wound daily lest it stop ticking entirely.
This is the myth of perpetual motion, applied to the digital realm. And like its physical counterpart, it’s an impossibility built on a fundamental misunderstanding of energy. A crawler’s attention is not a reward for busywork; it is a finite resource allocated based on perceived value. When we engage in trivial, superficial updates purely to ‘show activity,’ we are not winding the watch. We are merely jiggling the hands, creating the illusion of progress while the mainspring—the actual, substantive content—remains untouched and potentially neglected.
The crawler, over time, is not fooled. It learns. It begins to recognize the pattern of these insignificant tremors. The page that sees a comma added and a publication date changed every Tuesday starts to look less like a vibrant source of new information and more like a static entity pretending to be dynamic. The crawler’s precious attention, its budget, may initially be spent re-inspecting this mirage, but it will soon learn to allocate its resources more wisely elsewhere. You haven’t earned more crawl budget; you’ve taught the engine to distrust your signals.
The Virtue of a Still Pool
Consider instead the profound value of a page that is perfectly, completely finished. A definitive guide, a timeless essay, a technical specification that will not change. Its power lies not in its change log, but in its unwavering constancy. It is a deep, still pool of information. A crawler, upon its first visit, drinks deeply and understands the page’s comprehensive nature. It logs its quality and its relevance. On subsequent visits, it need only dip a finger to test the waters—to ensure the page is still present, accessible, and unaltered. This is an incredibly efficient use of crawl resources.
The page is not neglected; it is respected. Its lack of change is not a signal of decay but a badge of authority and completeness. The crawler trusts it. It knows that when it returns, it will find the same reliable information, and that trust frees up the engine’s resources to go and discover what is truly new and unknown.
Our obsession with perpetual motion stems from a fear of being forgotten. But in the vast library of the web, the most revered volumes are not the ones with the most dog-eared pages from constant, minor revisions. They are the foundational texts, the authoritative sources that are referenced time and again. They are trusted because they are stable. Our goal should not be to keep the crawler constantly busy, but to give it a reason to have unwavering faith in what we have already built. Sometimes, the most powerful signal you can send is the confident silence of a thing that is complete.
Notes & further reading
A few pages I came back to while writing this: