robots.txt

robots.txt is the <a href="/facts/Filename/EGc4tA7n">filename</a> used for implementing the Robots Exclusion Protocol, a standard used by <a href="/facts/Website/SAsjJnca">websites</a> to indicate to visiting <a href="/facts/Web_crawler/ra7D1xyb">web crawlers</a> and other <a href="/facts/Internet_bot/yPTtRLyq">web robots</a> which portions of the website they are allowed to visit.
The standard, developed in 1994, relies on <a href="/facts/Voluntary_compliance/Yo9QxAKp">voluntary compliance</a>. Malicious bots can use the file as a directory of which pages to visit, though standards bodies discourage countering this with <a href="/facts/Security_through_obscurity/GvF87wz1">security through obscurity</a>. Some archival sites ignore robots.txt. The standard was used in the 1990s to mitigate <a href="/facts/Server_(computing)/dEaYFD7r">server</a> overload. In the 2020s, websites began denying bots that collect information for <a href="/facts/Generative_artificial_intelligence/ykT3GGyT">generative artificial intelligence</a>.
The "robots.txt" file can be used in conjunction with <a href="/facts/Sitemaps/V0Eppohg">sitemaps</a>, another robot inclusion standard for websites.

robots.txt open-in-new

robots.txt