What is robots.txt?

A file telling crawlers which paths a site permits or disallows.

robots.txt is a file at a site's root that declares which URLs automated crawlers are asked to access or avoid. It is an advisory standard rather than a technical block, but respecting it is a core part of ethical and legally cautious scraping. Responsible data collection reviews robots.txt, honors crawl-delay directives, and avoids disallowed paths alongside respecting rate limits and terms of service.