From: brian
robots.txt
provides a means to request that robots limit their
activities at the site, or more often than not, to leave the site alone.
When the first robots were developed, they had a bad reputation for sending hundreds/thousands of requests to each site, often resulting in the site being overloaded. Things have improved dramatically since then, thanks to Guidelines for Robot Writers, but even so, some robots may exhibit unfriendly behavior which the webmaster isn't willing to tolerate, and will want to stop.
+When the first robots were developed, they had a bad reputation for +sending hundreds/thousands of requests to each site, often resulting +in the site being overloaded. Things have improved dramatically since +then, thanks to +Guidelines for Robot Writers, but even so, some robots may exhibit +unfriendly behavior which the webmaster isn't willing to tolerate, and +will want to stop.
Another reason some webmasters want to block access to robots, is to stop them indexing dynamic information. Many search engines will use the