LWP::RobotUA - a class for well-behaved Web robots |
LWP::RobotUA - a class for well-behaved Web robots
use LWP::RobotUA; my $ua = LWP::RobotUA->new( 'my-robot/0.1', 'me@foo.com' ); $ua->delay(10); # be very nice -- max one hit every ten minutes! ...
# Then just use it just like a normal LWP::UserAgent: my $response = $ua->get('http://whatever.int/...'); ...
This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently.
But before you consider writing a robot, take a look at <URL:http://www.robotstxt.org/>.
When you use a LWP::RobotUA object as your user agent, then you do not
really have to think about these things yourself; robots.txt
files
are automatically consulted and obeyed, the server isn't queried
too rapidly, and so on. Just send requests
as you do when you are using a normal LWP::UserAgent
object (using $ua->get(...)
, $ua->head(...)
,
$ua->request(...)
, etc.), and this
special agent will make sure you are nice.
The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the same methods. In addition the following methods are provided:
Optionally it allows you to specify the WWW::RobotRules object to use. If you don't provide one, then this user agent will make its own internal database of robots.txt rules as needed.
delay([$minutes])
$ua->delay(10/60);
use_sleep([$boolean])
sleep()
if requests
arrive too fast (before $ua->delay minutes has passed). The default is
TRUE. If this value is FALSE then an internal SERVICE_UNAVAILABLE
response will be generated. It will have an Retry-After header that
indicates when it is OK to send another request to this server.
rules([$rules])
no_visits($netloc)
num_visits()
or
something like that. :-(
host_wait($netloc)
the LWP::UserAgent manpage, the WWW::RobotRules manpage
Copyright 1996-2002 Gisle Aas.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
LWP::RobotUA - a class for well-behaved Web robots |