The Honeypot in the Closet: On Crafting a Decoy for the Lost Crawler

We spend so much time trying to get every important page crawled that we rarely consider the opposite problem: what to do with the pages we wish would simply vanish. The sprawling annex of a website—the legacy press releases from 2008, the deprecated API documentation, the holiday promo pages that never got taken down. They linger, consuming crawl budget, and more importantly, they create a signal-to-noise problem for the engines trying to understand what your site is about now. You can 404 them, but that feels abrupt, almost rude. You can noindex them, but the crawler still wastes time finding them. What if, instead of hiding the door, you offered a better one?

This is the technique of the decoy directory. It’s a deliberate, controlled space built not for users, but for the crawler’s own curious nature. The goal is not to block, but to redirect its inherent curiosity toward a more useful conclusion. You identify a cluster of those low-value, legacy pages—say, a whole section of `/old-products/`—and you don’t just delete it. First, you create a single, new page: `/old-products-archive/`. This page is your honeypot.

The Architecture of a Polite Dead End

This decoy page must be crafted with care. Its title and a clear, plain-language heading should state its purpose: “Historical Product Archive.” The body is a single, concise paragraph explaining that this is a preserved record of past offerings, and that current information lives elsewhere. You then provide one, maybe two, highly relevant links. Not a sprawling list, but a purposeful funnel: one link to your main products page, and perhaps one to a “What’s New” section. You then, crucially, interlink all those scattered, legacy `/old-products/example` pages to point only to this new archive page. You sever their ties to the living site and tether them to this quiet chamber.

Finally, you take your new `/old-products-archive/` page and list it plainly in your sitemap. You are essentially filing a formal change of address for an entire neighborhood of content. When the crawler next wanders down the dusty path of an old product URL, it is met not with a broken path or a confusing, outdated page, but with a gentle redirect (via link) to a page that explicitly contextualizes the dead end. The crawler learns the relationship, acknowledges the archive as the canonical hub for this retired content, and, most importantly, finds a clean, authoritative link from that hub back to your core content.

The outcome is a tidying of the mental model. Instead of dozens of orphaned pages weakly whispering unrelated keywords into the void, you have a single, well-defined node that tells a coherent story: “This is the past. That is the present.” It satisfies the crawler’s mandate to map what it finds, but it does so on your terms, guiding it out of the closet and back into the main hall. It turns a problem of crawl waste into an exercise in information architecture, a quiet pact with the machine to help it forget what no longer matters, by first showing it where to put those memories away.

Notes & further reading

A few pages I came back to while writing this: