|
Unix VPS
Support
FAQ
Network
Miscellaneous
|
| Gungho::Component::RobotRules(3) | User Contributed Perl Documentation | Gungho::Component::RobotRules(3) |
NAME
Gungho::Component::RobotRules - Respect robots.txtSYNOPSIS
---
components:
- RobotRules
METHODS
setup
setup_robot_rules_parser
setup_robot_rules_storage
handle_response
send_request
allowed($request)
Returns 1 if request is allowed to be fetched, 0 if not. -1 and -2 are returned when there is a pending request to fetch /robots.txtdispatch_pending_robots_txt
Dispatches requests that were pending because of a missing robots.txt entrypush_pending_robots_txt
Pushes a request in the wait queue for a robots.txtparse_robot_rules
Parses the robot rule and stores it| 2007-11-29 | perl v5.32.1 |
Visit the GSP FreeBSD Man Page Interface.
Output converted with ManDoc.