The Sourdough Starter: On the Persistent Culture of Link Discovery
I was cleaning out a kitchen cupboard last weekend, a task I’d put off for years, when I found it tucked behind a stack of mismatched lids: my jar of sourdough starter. A relic from a pandemic-era obsession, it was a sad, separated mass of greyish liquid and beige paste. The ‘hooch’ had risen to the top, a sure sign of neglect. My immediate instinct was to toss it. It looked dead. But a small, stubborn part of me remembered the care it once required, the daily feeding, the way it would bubble with life just a few hours after being nurtured. I wondered if anything was left alive in that jar.
As I weighed the decision to discard it, my mind, as it often does, drifted to my work. I thought about the web crawler, that indefatigable digital agent tasked with discovering every last page. We talk about sitemaps and robots.txt, about crawl budgets and server latency, as if the process is purely mechanical. But there’s a more organic, almost biological layer to it, especially when it comes to the most crucial element of discovery: the hyperlink. A link isn't just a pathway; it's a potential culture. It’s a piece of living data that, if healthy, can lead to the proliferation of discovered content.
This jar of starter was a perfect, if melancholic, analogy for a neglected link architecture. A website I’d worked on years ago came to mind. It had been vibrant once, with a blog that linked out thoughtfully to industry resources and, more importantly, linked deeply within its own structure. New articles would reference old ones, creating a rich, interconnected web. Then the blog was abandoned. No new posts were written, no new links were forged. The ‘starter’ of discovery began to stagnate. The external sites that once linked back to it updated their own content, and those reciprocal links decayed. The internal links remained, but without fresh ‘food’—without new pages to point to—the crawler’s visits became less frequent, less enthusiastic. The site’s presence in the index slowly faded, not because it was deleted, but because its culture of discovery had died.
Feeding the Culture
Just as I decided to take a small sample of my starter and mix it with fresh flour and water, a website can be revived by reintroducing that culture of linking. It’s not about a frenzied, artificial link-building campaign. It’s the quiet, consistent work of creating valuable content that naturally connects to other valuable content, both your own and others'. It’s the act of maintaining relationships, of being a part of the conversation. A single, well-placed link from a dusty corner of your site to a new, relevant page is like feeding the starter. It signals to the crawler that there’s still life here, that something is happening worth investigating.
This morning, I checked the jar. A few small bubbles had appeared on the surface. It’s not ready to bake a loaf yet, but it’s alive. The culture persisted. It reminded me that discovery on the web isn’t just about the initial programming of a spider. It’s about cultivating an environment where links can thrive, where pathways remain open and inviting. It’s a practice of persistence, of ensuring that the signals we send out into the vastness of the internet aren’t just one-time shouts, but part of a sustained, living conversation.
Notes & further reading
A few pages I came back to while writing this: