The Siren's Call of the Freshness Fetish: On Crawling's Obsession with the New

We talk a lot about crawl budget, about sitemaps and robots.txt, about making our sites discoverable. But beneath these technical discussions lies a deeper, almost philosophical assumption: that a crawler’s primary purpose is to find what’s new. We structure our sites, our publishing schedules, and our very content strategies around this idea. The ‘latest’ post, the ‘just updated’ page, the ‘breaking’ news—these are the prizes we believe crawlers are designed to seek. I’ve come to think of this as the Freshness Fetish, and it’s a mindset that subtly warps our understanding of the web.

This obsession with novelty isn’t without reason. Search engines themselves have trained us to value it, prioritizing recent results for time-sensitive queries. But in our eagerness to be seen as fresh, we often forget what a crawler truly is: a collector. Its mandate isn’t merely to chronicle the birth of pages, but to understand the entire ecosystem of a site, including its deep and enduring wisdom. By focusing all our energy on the front door, we risk letting the library’s most valuable, timeless volumes gather dust in corners the crawler is never urged to revisit.

The consequence is a web that increasingly feels ephemeral, a river of content where only the most recent droplets are deemed worthy of attention. We pour resources into generating the new thing, hoping to catch the crawler’s eye on its next pass, while older, cornerstone content—the definitive guide, the foundational essay, the patient tutorial—is left to fend for itself. We assume that because it was crawled once, its work is done. But a page, once published, is not a static artifact; it exists in a network of links that change, with rankings that fluctuate, and alongside new content that may reference or invalidate it.

A more holistic view of discovery would see a crawler not just as a scout for new territory, but as a steward of an entire domain. It would value the re-crawl as much as the first crawl, understanding that the significance of a page evolves over time. Our strategies, then, shouldn’t just be about announcing novelty with sitemap pings, but about actively maintaining relevance. This means internally linking to evergreen content with the same fervor we promote new posts, and understanding that a crawler’s ‘budget’ should be spent not only on discovery but on remembrance.

Let’s break the spell of the new. Let’s start speaking to crawlers in a language that values depth and endurance as much as it does freshness. The goal isn’t just to be found today, but to remain found, to be a persistent and evolving source of value in the vast, forgetful expanse of the web. A crawler can be a curator of legacy, not just a herald of the latest thing. We just have to ask it to listen.

Notes & further reading

A few pages I came back to while writing this: