The Night Soil Porter: On the Pages That Must Be Removed
In the quiet hours before the city of a website wakes, a different kind of worker makes their rounds. This isn’t the enthusiastic explorer mapping new territory, nor the librarian curating the grand archives. This is the night soil porter, a figure from a bygone urban reality, tasked with a necessary, unglamorous duty: the removal of waste. In the ecosystem of a site, some pages do not age into quaint archives; they fester. They become the digital night soil, and someone must carry them away.
Think of the porter, moving through the pre-dawn streets of an older metropolis. Their work is vital to public health, yet conducted out of sight, a service acknowledged only in its absence. On a website, the equivalent is the page that has soured. It’s the event listing for a concert that happened three years ago, still popping up in search results to frustrate a hopeful fan. It’s the product page for an item long out of stock, its 'Add to Cart' button a cruel joke. It’s the blog post boasting outdated information that now misleads more than it educates. These pages accumulate, cluttering the crawl pathways and drawing the crawler’s finite attention to dead ends.
The porter’s work is not destruction for its own sake. It is an act of curation as deliberate as any act of creation. Leaving these pages to decay in place pollutes the user experience and misdirects the site’s own internal cartography. When a crawler repeatedly trips over a 404 error from a broken internal link or spends its budget indexing content that serves no purpose, the entire site suffers. The fresh, relevant pages—the new market stalls, the current announcements—receive less attention. The crawler, like a visitor following misleading street signs, wastes its energy.
The most skilled porters, however, understand that removal is not the only tool. Sometimes, the material can be repurposed. A 301 redirect is a form of alchemy, transmuting the value of an old, stale URL into the currency of a new, living one. It’s not about hiding the past, but about updating the address. The porter carefully redirects the flow of visitors and crawlers alike from the abandoned cul-de-sac to the thriving new district, preserving the accumulated authority of the original path without the decay.
This work requires a quiet vigilance. It’s a continuous process of auditing the site’s back alleys and forgotten cellars, deciding what to remove, what to redirect, and what to simply leave to its peaceful obsolescence. It’s a thankless job, in a way. When done well, no one notices. The site just feels cleaner, faster, more coherent. The paths feel intentional. The crawler moves with purpose, and the visitors find what they need. The night soil porter ensures that what remains, what is discoverable, is worth the finding.
Notes & further reading
A few pages I came back to while writing this: