From: brian Date: Mon, 26 Jan 1998 07:12:58 +0000 (+0000) Subject: PR: X-Git-Tag: APACHE_1_3b5~20 X-Git-Url: https://granicus.if.org/sourcecode?a=commitdiff_plain;h=bbc3f6d277e3b53cc23ebd07663087dba79ef941;p=apache PR: tsk tsk, randy. Can't find this on covalent.net either. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@80015 13f79535-47bb-0310-9956-ffa450edef68 --- diff --git a/docs/manual/misc/howto.html b/docs/manual/misc/howto.html index 98a1843e83..a7d6b38c3c 100644 --- a/docs/manual/misc/howto.html +++ b/docs/manual/misc/howto.html @@ -130,7 +130,14 @@ is then used by a search engine to help locate information.

robots.txt provides a means to request that robots limit their activities at the site, or more often than not, to leave the site alone.

-

When the first robots were developed, they had a bad reputation for sending hundreds/thousands of requests to each site, often resulting in the site being overloaded. Things have improved dramatically since then, thanks to Guidelines for Robot Writers, but even so, some robots may exhibit unfriendly behavior which the webmaster isn't willing to tolerate, and will want to stop.

+

When the first robots were developed, they had a bad reputation for +sending hundreds/thousands of requests to each site, often resulting +in the site being overloaded. Things have improved dramatically since +then, thanks to +Guidelines for Robot Writers, but even so, some robots may exhibit +unfriendly behavior which the webmaster isn't willing to tolerate, and +will want to stop.

Another reason some webmasters want to block access to robots, is to stop them indexing dynamic information. Many search engines will use the