The Fisherman in the Frozen Lake: On the Trap of Good Intentions

I was twelve years old when I learned that you can have all the right intentions and still break the ice. It was a bitter January morning, the kind that fogs your breath and makes the world feel hushed and brittle. My grandfather and I were visiting our summer cabin, a place abandoned to the deep cold. The lake, which in July was a chaos of laughter and leaping fish, was now a vast, grey-white sheet of silence.

We spotted it from the shore: a beautiful, hand-carved wooden duck decoy, stranded maybe thirty feet out on the ice. It was an artifact of last season, forgotten and forlorn. My grandfather loved that decoy. Without a second thought, fueled by a boy’s bravado and a genuine desire to help, I declared I would retrieve it. The ice looked thick enough. I took a tentative step, then another. The surface groaned a low, resonant complaint, but held. I was a hero in the making, inching toward the prize.

I’m often reminded of that morning when I look at a website’s crawl statistics. We build pages with the best of intentions – a beautifully crafted thank-you note after a form submission, a labyrinthine archive of past events, a meticulously tagged but ultimately duplicate version of a core page. We put them out there on the virtual ice, hoping they’ll be discovered, appreciated, maybe even loved. They are our decoys.

But the crawler, like the frozen lake, has its own physics. It doesn’t care about sentiment. It operates on a simple, unforgiving logic of structural integrity and resource allocation. Each step it takes towards one of our well-intentioned but low-value pages is a step not taken towards the pages that truly matter—the new content, the vital products, the core services. The crawl budget isn't infinite; it's a finite path of safe, thick ice leading to the heart of what we offer.

I was halfway to the decoy when the sound changed from a groan to a sharp crack, a sound like a thousand twigs snapping at once. A spiderweb of fractures erupted around my boots. I froze, not from the cold, but from a primal understanding of my mistake. My grandfather’s voice, calm but firm, cut through the panic: "Don't move. Now, slowly, lie flat." I spread my weight across the surface, my cheek pressed against the freezing pane, and inched my way back to solid ground, leaving the decoy behind.

The lesson wasn’t to stop venturing onto the ice. It was to understand the ice. We don’t serve the crawler by littering our site with thin, fragile pages that threaten to break its progress. We serve it by ensuring the path is strong, clear, and leads to substance. Every ‘thank you’ page that squanders a crawl, every archival page that offers no new scent, is a potential weak spot. It’s the trap of good intentions, a risk not just to the page itself, but to the entire journey. Sometimes, the wisest thing you can do for discovery is to not place the decoy on the ice in the first place, or to gently guide the crawler away from it. A site map isn't just a map of what's there; it's a recommendation of where to tread carefully, and where it's safe to run.

I never did get that decoy. It stayed out there until the spring thaw, when we recovered it, waterlogged but intact. It sits on my desk now, a quiet reminder that not everything we create needs to be found. Some things are just meant to be, waiting patiently for the right season.

Notes & further reading

A few pages I came back to while writing this: