From bbc3f6d277e3b53cc23ebd07663087dba79ef941 Mon Sep 17 00:00:00 2001 From: brian Date: Mon, 26 Jan 1998 07:12:58 +0000 Subject: [PATCH] PR: tsk tsk, randy. Can't find this on covalent.net either. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@80015 13f79535-47bb-0310-9956-ffa450edef68 --- docs/manual/misc/howto.html | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/manual/misc/howto.html b/docs/manual/misc/howto.html index 98a1843e83..a7d6b38c3c 100644 --- a/docs/manual/misc/howto.html +++ b/docs/manual/misc/howto.html @@ -130,7 +130,14 @@ is then used by a search engine to help locate information.

robots.txt provides a means to request that robots limit their activities at the site, or more often than not, to leave the site alone.

-

When the first robots were developed, they had a bad reputation for sending hundreds/thousands of requests to each site, often resulting in the site being overloaded. Things have improved dramatically since then, thanks to Guidelines for Robot Writers, but even so, some robots may exhibit unfriendly behavior which the webmaster isn't willing to tolerate, and will want to stop.

+

When the first robots were developed, they had a bad reputation for +sending hundreds/thousands of requests to each site, often resulting +in the site being overloaded. Things have improved dramatically since +then, thanks to +Guidelines for Robot Writers, but even so, some robots may exhibit +unfriendly behavior which the webmaster isn't willing to tolerate, and +will want to stop.

Another reason some webmasters want to block access to robots, is to stop them indexing dynamic information. Many search engines will use the -- 2.40.0