The Broken Handrail: On the Crawler's Reluctance to Climb the Stairs

We think of discovery as a flash of light, a sudden connection, a beacon switched on. In the architecture of a website, we imagine search engines as meticulous inspectors, tapping on every wall, testing every door. But in my own old apartment building, I’ve learned a different truth about how things are found—or, more accurately, how they are not. The lesson is in the staircase, specifically in the broken handrail on the flight between the third and fourth floors.

The stairs themselves are fine. The light works. The destination—the fourth-floor landing with its three doors—is clearly reachable. Yet for over a year, most residents have avoided that flight. The missing four feet of polished oak isn’t a blockade, but it introduces a subtle, cognitive friction. You must shift your weight, grip the wall instead, and proceed with a fraction more caution. It’s a trivial inconvenience, yet it functionally demotes that entire upper floor. We go there only when we must.

A web crawler, in its way, is not so different. We build sites with clear ‘stairs’—logical, sequential pagination for article series, product listings, or archives. ‘Page 1, Page 2, Page 3…’ it seems as sturdy as steel. But we often forget to install the handrail: the firm, semantic, and authoritative signal that assures the crawler this path is both safe and worthwhile to ascend. A ‘rel=next’ tag that’s broken. A pagination link that’s a barely-styled ‘>’ symbol with no anchor text. A URL structure that suddenly shifts for pages 4 and beyond. These are our broken handrails.

The crawler will likely find the stairs. It may even climb a few steps. But at the point of friction—where the effort to parse the next link spikes relative to the perceived value—it will pause. Its budgeted time for your site is finite. Like a resident calculating the trip to the fourth floor, it must ask: Is the certainty of discovery up there worth the awkwardness of the climb? Often, the answer is a quiet ‘no.’ The crawler will turn back, its map of your site ending abruptly at page 3, leaving the rest in a kind of architectural shadow.

This is how pages that are perfectly linked, technically accessible, and full of good content simply wait, unseen. They aren’t blocked by a wall. They aren’t hidden in a vault. They are waiting at the top of a staircase we’ve made feel untrustworthy. The fix is rarely a grand redesign. It’s the work of a carpenter, not an architect: securing the connection, sanding the rough edge, ensuring the path of ascent feels continuous, expected, and supported. It’s about making the crawler, and the person who will follow it, feel welcome to climb all the way to the top.

Notes & further reading

A few pages I came back to while writing this: