Technical SEO
robots.txt
A file that instructs search engine crawlers which URLs they can or cannot access.
Definition
The robots.txt file, placed at the root of a domain, provides crawl directives to search engine bots. While it does not enforce indexation rules, it helps conserve crawl budget by disallowing low-value or sensitive paths and can specify sitemaps. Misconfigurations can block critical content from being crawled, so updates should be carefully tested. It complements other controls such as meta robots, canonical tags, and server-side redirects.