Crawl Traps
URL patterns that create infinite or near-infinite crawl paths that waste resources.
Definition
Crawl traps are site patterns—such as calendars, endless faceted combinations, or session parameters—that produce infinite URLs for crawlers to follow. These traps can consume significant crawl budget and hinder discovery of valuable pages. Detecting and neutralizing crawl traps preserves resources for important content.