The Unvisited Chapel: On the Forgotten Corner of a Vast Digital Cathedral
There is a small chapel in the north transept of Lincoln Cathedral, England, known as the ‘Crazy Chapel’. It’s a quiet, out-of-the-way place, easily missed by tourists following the main guidebook routes. For centuries, it has sat in relative obscurity, its beauty known only to a handful of clergy, architects, and the occasional lost visitor who stumbles upon it. It is, by any measure, a page that has never been indexed by the world’s gaze.
I was reminded of this place recently while watching a crawler’s log for a client’s sprawling e-commerce site. Amidst the frantic polling of product pages and category lists, one line appeared, over and over, like a quiet but persistent whisper: a request for a long-dead URL from a marketing campaign from 2014. The crawler had found a single, ancient, and forgotten link in a footer, buried under layers of new content. And with the dogged determination of a pilgrim, it kept trying to visit this place that no longer existed.
This wasn’t a drain on the crawl budget or an error to be swiftly fixed. It was something more poignant. It was a map to a room that had been bricked up long ago. The crawler, in its own blind way, was the only thing that remembered the path.
We spend so much time optimising for discovery—polishing sitemaps, clearing crawl paths, building logical information architectures—that we forget there is a whole archaeology of the forgotten web living within our own sites. These are the legacy links, the experimental microsites, the campaign landing pages that outlived their campaigns. They are our own Crazy Chapels. They hold no SEO value, they serve no business purpose, and yet, the crawler finds them. It is drawn to them by the faintest of signals, a single hypertext breadcrumb left behind from a different era.
There’s a strange comfort in this. It suggests that the web crawler is not just a cold, efficient harvester of what we want it to see. It is also an accidental archivist, a digital spelunker charting every corridor, even the ones we sealed off. It remembers the paths we paved and then abandoned. In its relentless journey, it acknowledges that a website is not a perfect, finished product but a living document, layered with history, mistakes, and ghosts. The next time you see one of those 404 errors for a page you haven’t thought of in a decade, don’t just see a broken link. See a crawler paying its respects to a forgotten chapel, ensuring that someone, or something, still remembers the way.
Notes & further reading
A few pages I came back to while writing this: