How Crawlers Decide What to Fetch Next

A crawler is mostly a scheduler. It keeps a frontier of known links and constantly re-ranks them by freshness, importance and politeness rules. Pages that change often and are linked from busy places rise to the top; quiet corners are visited rarely. Understanding that queue is the difference between a page being seen tomorrow or next quarter.