The Gardener Who Drowned His Roses: On the Folly of Forced Crawl Submission
There is a persistent and seductive belief in our corner of the web, a piece of received wisdom passed around like a worn-out tool: if you want a search engine to find your pages, you must show it the way. You must build a sitemap. You must submit it. You must, in effect, grab the gardener by the hand, lead him to every single plant, and personally introduce each leaf and petal. This logic feels impeccable. It’s proactive, it’s thorough, it’s the mark of a diligent steward of digital space. But I’ve come to see it as a kind of anxious overwatering, a well-intentioned act that can drown the very roots you’re trying to nourish.
We imagine the crawler as a tentative visitor, a guest who needs constant guidance lest they get lost in our hallways. This is a fundamental misreading of its nature. The modern crawler is not a guest; it’s a surveyor with an innate, relentless drive to map terrain. Its primary language isn’t the sitemap—that’s a helpful footnote, a translation aid—but the link. The link is the path worn by footsteps, the desire line across a field. It is the organic signal of value and connection. When we sideline this in favor of the official directory, we’re not just being helpful; we’re subtly telling the web that our pages lack the most natural currency of discovery: the reason for another page to point to them.
The Echo in the Empty Room
Think of what a sitemap submission really is: you are announcing the existence of pages directly to an index, bypassing the ecosystem. You are, in a sense, asking for pages to be known before they have earned their place in the conversational fabric of the web. This creates a strange, hollow kind of discovery. The page might be indexed, but does it have a context? Has it been vetted by the simple, brutal democracy of a human choosing to link to it? Or is it just a coordinate on a grid, waiting for a visitor who may never arrive because no other path leads there?
This practice fosters a brittle architecture. It encourages the creation of ‘content’ designed for the index, not for the network. Pages become isolated nodes, known only because their creator filled out a form. They lack the resilience of a page found through a trail of links, a page that is, by its very position in a web of references, validated as part of a larger conversation. When you force-feed a crawler, you risk building a library where every book is catalogued but none are ever cited by another.
This is not to say sitemaps are useless. For sprawling, new, or architecturally complex sites, they are a courtesy, a way to ensure no dark corner is permanently overlooked. But they are a backup system, a safety net. They should not be the primary engine of discovery. The real work—the work that builds lasting findability—is in weaving your pages into the web’s existing tapestry. It is in creating something so genuinely useful, insightful, or connective that others cannot help but build a path to it. That path, that single, earned link from a relevant place, speaks louder to a crawler than a thousand submitted URLs. It says: here is a place worth visiting. Not because its gardener said so, but because the rest of the garden points the way.
The best thing we can do for our pages is not to submit them, but to give them a reason to be found. Build them for people, connect them thoughtfully, and let the crawlers do what they do best: follow the trails we leave for each other. Stop drowning the roses. Just tend the soil, and let the roots find their own, deep way.
Notes & further reading
A few pages I came back to while writing this: