The Geologist's Map: On Deep Time and the Crawler's Patience
Geologists operate on a timescale that can shame the most patient among us. They speak of epochs and eras, of shifts so incremental they are imperceptible over a human lifetime, yet which sculpt continents. The central tool in their work isn't just a hammer or a microscope; it’s the geological map. It’s a representation of the surface that tells a profound story about the deep, invisible structures below. It occurred to me that we, as stewards of the web’s landscape, are often tasked with a similar form of cartography. Our sitemaps are not just simple directories; they are documents that attempt to chart a domain's structure for an entity that perceives time and space very differently than we do.
A geologist knows that the map is a hypothesis, not an absolute truth. The surface can be deceptive. A layer of sedimentary rock might be exposed in one cliff face, only to plunge deeply underground a mile away, obscured by foliage or a shift in the terrain. They map what they can see and infer the rest, knowing the map will be revised as new boreholes are drilled, new surveys conducted. Our websites are no different. We build a sitemap with our best understanding of the site’s architecture. But what about the pages tucked away behind complex forms? The legacy content in an unlinked directory? The seasonal pages that lie dormant for eleven months of the year, like a fossil bed waiting for the right conditions to be exposed? These are our site’s subterranean layers.
The crawl budget, then, is not merely a resource to be optimized like a quarterly report. It is more akin to a field season. A geologist doesn’t arrive at a mountain range and bulldoze the entire thing in a week. They take core samples. They follow promising outcrops. They spend days on a single rock face, piecing together a story. A crawler, similarly, must be trusted to explore. It uses the sitemap as a primary guide, but its most valuable work often happens when it follows the links it discovers—the subtle strata of our internal linking—revealing connections we may not have formally documented. This exploratory drilling is where the true architecture is understood, not just by the machine, but often by us.
Embracing the Rhythm of Tectonic Shifts
Most critically, geology teaches us to respect processes that operate on a non-human rhythm. A crawler’s perception of ‘frequent’ and ‘important’ is dictated by the immense scale of the web. A page that hasn’t been touched in six months might seem ancient to us, but to the crawler, it might simply be a stable layer of bedrock, checked on a longer, more leisurely schedule because its content type signals low volatility. Our impulse is often to force the issue: to ping, to resubmit, to demand re-crawls. This is like trying to hurry an ice age.
Instead, we should focus on creating a landscape that is legible and stable. A clear, logical site structure is the stable tectonic plate. Consistent, semantic markup is the distinct layering of rock that makes a formation easy to identify. A clean, fast server is the stable ground on which to conduct the survey. When we build with this geological patience, we create an environment where the crawler can work efficiently and, more importantly, accurately. It can distinguish between the ephemeral topsoil of a daily blog post and the bedrock of a cornerstone article, allocating its ‘field season’ accordingly. We stop trying to command the crawler and instead learn to communicate with it in a language of structure and signal that transcends our own frantic sense of time. The goal is not to control the exploration, but to be a reliable cartographer of the territory we’ve built.
Notes & further reading
A few pages I came back to while writing this: