The Cartographer's Dilemma: On Drawing Maps Without Knowing the Territory

For centuries, cartographers faced a profound professional hazard: drawing coastlines for lands they had never seen. They worked from the sketches of sailors, from second-hand tales, and often, from the vast, blank spaces of their own ignorance. To fill these voids, they sometimes sketched in speculative mountain ranges or entire kingdoms, labeled with the hopeful phrase "Here be dragons." This act—creating a guide for an unknown space—isn't so different from how we often approach making our websites discoverable. We are all, in a sense, drawing maps for an explorer we can never fully see: the web crawler.

The core lesson from those old mapmakers is about the tension between completeness and accuracy. A map cluttered with guessed detail is worse than useless; it’s actively misleading. A cartographer knew that a blank space, while unsatisfying, was at least an honest admission. In our world, this translates directly to the structure we present for discovery. A sitemap stuffed with every possible URL, including those thin, duplicate, or transient pages, isn't a helpful guide. It’s a speculative coastline drawn from hope, not evidence. It asks the crawler to invest its limited time in territories that offer no real value, potentially causing it to miss the fertile, well-drawn continents of our core content.

The Honest Frontier

Where the old cartographers had dragons, we have our own mythical creatures: the "might be useful someday" page, the "just in case" duplicate, the orphaned section from a site redesign. These are our digital terra incognita. The disciplined approach, borrowed from the best mapmakers, is to sharply define the known world. A sitemap should be a curated atlas of your settled territories—the clear, canonical, and content-rich pages. Let the internal link structure, the clear pathways of your site's architecture, be the explorer's tool for finding the edges of that known world. A crawler following a strong, logical link is like a sailor charting a real coastline, providing genuine data for the map.

This also changes how we think about the "crawl budget"—not as a spending account to be maximized, but as a cartographer's precious ink. Do you waste it sketching fantastical beasts in the margins, or do you use it to reinforce the definitive lines of your continent's true borders? Every request a crawler wastes on a "dragon" is ink not spent perfecting the map of your city centers and trade routes.

The final, humbling parallel is acceptance. No map is ever the territory. The cartographer’s drawing is a static representation, while the land itself changes with seasons and tides. Similarly, our sitemaps and structures are static guides for a dynamic, living site. The goal is not a perfect, controlling diagram, but an honest, well-drawn one that correctly orients the explorer to the most important landmarks, trusting them to discover the authentic contours for themselves. It is an act of guided humility, leaving the blank spaces blank until we have something true to put there.

Notes & further reading

A few pages I came back to while writing this: