merge from 3.4

issue20753 - robotparser tests should not rely upon external resource when not required.

Specifically, it was relying a URL which gave HTTP 403 and used it to assert
it's methods, this changes undoes that and provides a local http server with
similar properties.

Patch contributed by Vajrasky Kok.