The Archivist's Single, Stubborn Box: On the Page That Refuses to Be Inventoried

Years ago, I volunteered at a small historical society, a place that smelled of dust and decaying paper. My task was simple: to help a gentle, meticulous archivist named Arthur inventory a new donation—a collection of a local painter’s letters and sketches. We worked for days, opening bankers' boxes, carefully cataloging each item with a neat, handwritten number on a acid-free tag. The system was elegant. Every page had a place. Every box had a manifest. It was a perfect, closed universe of information.

Then we found the box. It was unremarkable, cardboard like any other, but it contained the painter’s final, frantic project: hundreds of tiny, nearly identical studies of a single oak tree at dawn, each on a scrap of vellum or the back of a napkin. They were fragile, smudged, and impossible to separate without damage. Arthur’s system, built for distinct entities, broke down. We couldn’t assign a single number to the chaotic whole, nor could we reasonably number each faint sketch. The box defied inventory. It sat to the side, a silent, stubborn anomaly in our otherwise orderly rows.

I think of that box every time I look at a site’s crawl diagnostics and see the outliers. Not the broken links or the redirect chains, but the pages that are built to be un-catalogable. The live inventory feed that generates a new, unique URL for every visitor and every refresh. The massive, infinitely-scrolling page of user-generated content that has no end and no pagination. The single-page app that presents a universe of content under one, unchanging address.

These are our digital stubborn boxes. To a crawler, they are maddening. They represent either a bottomless pit of duplicate or low-value content, or a vast repository of information hidden behind a single, flat door. They refuse to play by the rules of the map we’ve so carefully drawn in our sitemaps. The crawler arrives, expecting a finite list of rooms to enter, and instead finds a single, ever-changing chamber or a hallway that stretches into infinity.

The lesson from Arthur wasn’t to force the box into the system. It was to acknowledge its otherness. He eventually created a single catalog entry for the entire box, describing its nature as a cohesive, fragile whole, and stored it appropriately. He didn’t try to make it something it wasn’t. In our work, sometimes the answer isn’t to force a recalcitrant page into a crawl budget-friendly format, but to understand its inherent nature and ask if it even needs to be discovered in the traditional sense. Some pages are meant for human experience, not for a spider’s inventory. And sometimes, the most truthful thing we can do is to simply note their existence and let them be.

Notes & further reading

A few pages I came back to while writing this: