The Humble Doorstop: On the Simple Physics That Keeps a Crawl Moving
I keep a small, rubber doorstop in my office. It’s an unremarkable thing, a wedge of black material that lives in the shadow of the door it’s meant to hold. Its entire purpose is to perform a single, simple task: to create enough friction against a floor to counteract a greater force, ensuring a passageway remains open. It doesn’t build the door, it doesn’t design the room, it doesn’t even open the door itself. It just makes sure that once opened, the door stays that way.
In the architecture of a website, we have our own version of this humble tool. We spend immense energy designing grand entrances (homepages), crafting intricate hallways (navigation), and building beautiful rooms (content). But we often forget the small, critical act of propping the door open. We assume that because we’ve built a path, it will naturally remain accessible to every visitor, including the most important one: the crawler.
This act of ‘propping the door open’ is the unglamorous work of internal linking. A single page, no matter how well-crafted, is a room with a closed door if no other page points to it. It exists in a state of potential, waiting for a force to overcome the friction of its isolation. A crawler is a creature of momentum, following the paths we explicitly lay out for it. It doesn’t speculate on what might lie behind a sealed door; it moves where the current of links directs it.
The Friction of Isolation
Every unlinked page on your site has a kind of existential weight, a resistance to being discovered. The crawler’s journey is a finite resource—a crawl budget—and it will not waste energy trying to open every possible door. It trusts the blueprint you provide through your link structure. A page without an inbound internal link has an infinite coefficient of friction against discovery. It is a door that cannot be opened from the outside.
The simple, corrective action of the doorstop—the internal link—isn’t about force. It’s about intelligent placement. It’s a contextual nudge from a product page to a related article, a navigational breadcrumb that leads back through the hallways, a ‘see also’ section that points to a deeper archive. It’s the acknowledgement that a page is not an island, but part of a contiguous structure. This lowly link does the quiet, essential work of reducing the friction of isolation to zero, allowing the crawler’s momentum to carry it through effortlessly.
We design grand architectures and create masterful content, but without these small, strategic acts of propping doors open, we are simply building beautiful, empty rooms no one can find. The crawl is a physics of movement, and sometimes the most powerful force is the simplest one: a wedge of intent, placed just so, to keep the way clear.
Notes & further reading
A few pages I came back to while writing this: