The Cartographer's Ghost: On the Forgotten Art of the Sitemap.txt

Before the sitemap index was a sprawling, automated XML file, it was a quiet, hand-drawn map. It lived in the root directory of a site, a simple text file named ‘sitemap.txt’. It had no schema, no validation, no priority tags. It was just a list. A cartographer’s ledger of places worth visiting.

I think about the people who made these. They weren’t engineers optimizing for crawl budget; they were guides. A librarian walking the stacks, noting down every volume on a slip of paper. A groundskeeper walking the paths, writing down each signpost and bench. The act was one of curation, of conscious selection. To add a URL to that list was to declare: This matters. This is part of the whole. It was a deliberate act of invitation.

That text file was a whisper to the early crawlers, which were more like curious travelers than industrial harvesting machines. It said, “Here is what I have to show you. Start here.” There was a humility to it, an understanding that discovery is a collaboration between the maker and the seeker. The site builder provided the map; the crawler consented to read it.

We’ve traded that simplicity for power and scale, and for good reason. The modern XML sitemap is a feat of engineering, capable of speaking a crawler’s language with immense precision. We can signal lastmod dates, change frequencies, and alternate hreflangs. We can index our indexes. It is a magnificent, chattering engine of discovery.

But in that transition, something quietly vanished: the human hand. The act of curation became an act of automation. The list is now generated by a plugin, a module, a script that parses a database. It is comprehensive, but it is no longer a guided tour. It is a census, not a recommendation.

The ghost of that old cartographer still lingers, though. You can find it in the care a developer takes to exclude thin or duplicate content, not just for efficiency, but for clarity. You can see it in the thoughtful structure of a site’s information architecture, which is a map in itself. The spirit of the sitemap.txt was never about the format; it was about the intent. It was the act of thoughtfully preparing a place for others to find, of drawing a map with the hope that someone, or something, would follow it.

That intention is the quiet, enduring art beneath all the noise of discovery. It’s the belief that what we make is worth finding, and that we have a responsibility to show the way.

Notes & further reading

A few pages I came back to while writing this: