The Clockmaker and the Cartoonist: On Planning for the Crawler's Gaze

There are, I’ve come to believe, two fundamental dispositions when building something intended for an unseen audience. One is the Clockmaker, the other, the Cartoonist. Both are creators of intricate systems, but their philosophies on structure, and thus on how their work is discovered, could not be more different. The way their creations are crawled—the paths a search engine's bot might take—reveals a great deal about the nature of the web itself.

The Clockmaker’s website is a masterpiece of intention. Every gear, every spring, every link is placed with precision. The architecture is a logical tree, a perfect hierarchy where depth is calculated, and the journey from the trunk to the smallest leaf is deliberate. For the Clockmaker, the sitemap is not a mere list; it is the blueprint, the true representation of a perfect, discoverable order. Crawlers, in this world, are welcomed as keen-eyed inspectors, appreciated for their ability to follow the neat pathways laid out for them. The crawl budget is a known quantity, to be spent efficiently along these groomed trails. This approach has a certain elegance, a predictability that feels like control.

Contrast this with the Cartoonist. Their site is less a timepiece and more a sprawling, vibrant comic strip. It is a network, not a tree. The primary mode of discovery isn't a top-down map but the surprising, delightful link hidden within a panel. A footnote might lead to an entire new storyline; a background character in one post is the star of another, linked in a throwaway line. The Cartoonist builds for the human who clicks, who explores, who delights in the connection the creator has woven between seemingly disparate ideas. The crawler, here, becomes an explorer in a world of Easter eggs, following a trail of breadcrumbs that was designed not for efficiency, but for revelation.

So, which approach yields better discovery? The answer is unsatisfyingly contextual. The Clockmaker’s method is peerless for e-commerce, for documentation, for any domain where clarity and predictability are paramount. A user (and the crawler) should be able to find the product manual or the support article with minimal friction. The Cartoonist’s way, however, breathes life into blogs, artistic portfolios, and knowledge bases where the joy is in the journey itself. It encourages deep, engaged crawling, where the value isn't just on the destination page but in the curated path that led you there. It builds a sense of world.

The friction arises when we misunderstand our own role. A Clockmaker who tries to insert ‘surprise’ links willy-nilly risks creating a confusing mess, wasting a crawler’s budget on dead ends. A Cartoonist who imposes a rigid, hierarchical menu on their free-flowing ideas might suffocate the very connections that make their content unique. The key is not to choose one absolutely, but to understand which you are, and to build your house accordingly. Plan your crawl paths with the same intention you plan your content. Are you building a precision instrument, or are you sketching a universe? The crawler will faithfully report on the world you’ve built, whether it’s one of ordered gears or of wondrous, interlocking panels.

Notes & further reading

A few pages I came back to while writing this: