The Unseen Cobweb: On the Ghosts of Links That Never Let Go
There’s a corner in my hallway, high up near the ceiling, where the cobwebs gather. I see them when the afternoon light hits just right, revealing a delicate, dusty architecture I had no idea was there. I’ll get the duster, clear them away, and for a week or two, the corner is pristine. But without fail, a few weeks later, a new, faint silken thread will reappear in the exact same spot. The spider, or its successor, is compelled to build there again. It’s a prime location, a junction between walls where the air currents bring a steady supply of gnats and dust. The spot has a history of success written in pheromones and geometry, an invitation that is never truly rescinded.
This is not unlike the way a web crawler views the internet. We think of our websites as finished, polished edifices. We build a page, we link to it, and later, perhaps, we remove it. We drag the page to the digital trash can and hear the satisfying swoosh of deletion. But for the crawler, that page’s location—its URL—is a junction in the architecture. And like my hallway corner, its history matters more than its present state.
When you remove a page but leave its address active, even to return a ‘404 Not Found’, you haven’t erased its existence. You’ve merely changed its state. To the persistent crawler, that URL is a known coordinate, a point on its map that must be periodically re-checked. It’s a ghost, a page that isn’t there, but whose address maintains a stubborn presence in the crawler’s memory and schedule. It is allocated a portion of the crawl budget, a sliver of attention that is spent confirming an absence, over and over again.
The real trouble begins when other websites have linked to that ghost. Those external links are the strongest pheromones, the most compelling invitations. They signal to the crawler that this junction was once important, that it led somewhere valuable. The crawler, acting on this old intelligence, will return with the dutiful regularity of a spider rebuilding its web. It checks the spot, finds nothing, and departs—only to return again on its next circuit, compelled by the enduring signal from the wider web.
We are the keepers of these digital corners. The solution isn’t just to delete, but to properly decommission. A ‘410 Gone’ response is a stronger signal, a way of telling the crawler that the resource is not just missing, but intended to be removed permanently. It’s the equivalent of painting over the spot or moving the furniture. Better yet, a custom 404 page that gently guides both humans and crawlers toward active, living content can repurpose that attention. It’s about understanding that on the web, as in my hallway, locations have memory. And unless we consciously break the cycle, the crawler will always return to the empty space, checking on a ghost we forgot was still waiting to be found.
Notes & further reading
A few pages I came back to while writing this: