NAME

GunghoX::FollowLinks - Automatically Follow Links Within Responses

SYNOPSIS

  follow_links:
    parsers:
      - module: HTML
        config:
          rules:
            - module: HTML::SelectedTags
              config:
                tags:
                  - a
            - module: MIME
              config:
                types:
                  - text/html
      - module: Text
        config:
          rules:
            - module: URI
              config:
                match:
                  - host: ^example\.com
                    action: FOLLOW_ALLOW
  package MyHandler;
  sub handle_response
  {
    my ($self, $c, $req, $res) = @_;
    $c->follow_links($res);
  }

DESCRIPTION

The most common action that a crawler takes is to follow links on a page. This module helps you with that task.

METHODS

setup

follow_links

Parses the given HTTP::Response/Gungho::Response object and dispatches the appropriate parser from its content-type.

For each URL found, Automatically dispatches the rules given to the parser, and if the rules match, the URL is sent to Gungho->send_request.

Returns the number of matches found.

AUTHOR

LICENSE

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

See http://www.perl.com/perl/misc/Artistic.html