The Unwatched Pot: On the Boil That Happens When You Stop Staring

I have been told for years, and I have told others, to manage the crawl budget. To shepherd the crawlers like a border collie, directing their energy away from the thin, weedy patches of the site and toward the lush, valuable content. To prune the XML sitemap like a bonsai tree, presenting only the most perfect branches. The logic is seductive: finite attention from a giant, so spend it wisely. But I have begun to wonder if this gospel of meticulous control is, in many cases, an elaborate performance for an audience of one—ourselves.

The counterintuitive truth is that for a vast majority of sites, the most effective way to get your pages found is not to obsessively optimize for discovery, but to build something worth discovering and then, crucially, to get out of the way. We treat crawlers as dim, easily confused creatures that need the nutritional paste of a sitemap and the guardrails of perfect internal linking. But modern discovery engines are profoundly curious, deeply associative, and astonishingly patient. They are less like finicky tourists with a tight itinerary and more like foraging animals with an excellent sense of smell.

The Efficiency of Inefficiency

Consider the crawl budget. The term itself implies scarcity, a zero-sum game. But this scarcity is largely a specter for sites that aren’t the size of Amazon or the BBC. For your local bakery’s blog, your portfolio site, or even a robust niche publication, Google’s crawler has more than enough time and interest. By hyper-focusing on blocking ‘wasteful’ crawls to your tag pages or filtered views, you might be severing the very exploratory pathways a bot uses to understand context, hierarchy, and the hidden relationships within your content. That ‘inefficient’ crawl of a monthly archive might be the thread that leads it to a two-year-old article that’s suddenly becoming relevant again.

And what of the sitemap? We treat it as a mandate, a required submission for consideration. But it is, in its purest form, a confession of poor information architecture. It says, ‘My site’s own connective tissue is so weak that I must provide this external diagram.’ A crawler that finds your content organically, through the natural link-laden pathways you’ve built for human visitors, understands that content in a richer, more meaningful context than it ever could from a sterile list in a sitemap.xml file. The sitemap should be a backup, a safety net for orphaned pages, not the primary blueprint.

This isn’t an argument for chaos. Good structure matters immensely—for people. Clear navigation, logical categorization, and contextual linking create an environment where both humans and bots can forage successfully. The shift is in intent. Build the site for the human who lingers, reads, and clicks. The bot is a ghost in that machine, recording the paths of real interest, not the ones you preordained in a search console.

So, stop staring at the pot. Stop adjusting the flame under your sitemap, fretting over the exact second the crawler last passed by. Tend to your garden—write the compelling thing, link to the relevant older thing, build a space that makes sense to a curious person. Then, have faith that the foragers will come. They have a nose for what’s truly boiling, even if you’re not there to point it out.

Notes & further reading

A few pages I came back to while writing this: