NAME

VERSION

SYNOPSIS

 use HOP::Lexer 'string_lexer';
  
 my @input_tokens = (
     [ 'VAR',   qr/[[:alpha:]]+/    ],
     [ 'NUM',   qr/\d+/             ],
     [ 'OP',    qr/[+=]/            ],
     [ 'SPACE', qr/\s*/, sub { () } ],
 );
  
 my $text  = 'x = 3 + 4';
 my $lexer = string_lexer( $text, @input_tokens );
  
 my @tokens;
 while ( my $token = $lexer->() ) {
     push @tokens, $token;
 }

EXPORT

FUNCTIONS

 my $lexer = make_lexer( $input_iterator, @tokens );

 [ $label, $matched_text ]

 [ $label, qr/$match/, &transform ]

  [ 'VAR', 'x' ]
  [ 'OP',  '=' ]
  [ 'NUM', 3   ]
  [ 'OP',  '+' ]
  [ 'NUM', 4   ]

  my $text = 'x = 3 + 4';
  my @text = ($text);
  my $iter = sub { shift @text };
  
  my @input_tokens = (
      [ 'VAR',   qr/[[:alpha:]]+/    ],
      [ 'NUM',   qr/\d+/             ],
      [ 'OP',    qr/[+=]/            ],
      [ 'SPACE', qr/\s*/, sub { () } ],
  );
  
  my $lexer = make_lexer( $iter, @input_tokens );
  
  my @tokens;
  while ( my $token = $lexer->() ) {
      push @tokens, $token;
  }

 my $lexer = string_lexer( $string, @tokens );

 my $lexer = string_lexer( $text, @input_tokens );

 my @text  = ($text);
 my $iter  = sub { shift @text };
 my $lexer = make_lexer( $iter, @input_tokens );

DEBUGGING

• Unlexed data

  The tokens returned by the lexer are array references. If any data cannot be lexed, it will be returned as a string, unchanged.

• Capturing parens

  Internally, Hop::Lexer uses capturing parentheses to extract the data from the provided regular expressions. If you need to group data in regular expressions, use the non-capturing parentheses "(?:...)". Otherwise, your code will break.

• Precedence

  It's important to note that the order of the described tokens is important. If you have keywords such as "while", "if", "unless", and so on, and any text which matches "qr/[[:word:]]+/" is considered a variable, the following fails:

    my @input_tokens = (
        [ 'VAR',     qr/[[:word:]]+/         ],
        [ 'KEYWORD', qr/(?:while|if|unless)/ ],
    );
      

  This is because the potential keywords will be matched as "VAR". To deal with this, place the higher precedence tokens first:

    my @input_tokens = (
        [ 'KEYWORD', qr/(?:while|if|unless)/ ],
        [ 'VAR',     qr/[[:word:]]+/         ],
    );
      

AUTHOR

BUGS

FURTHER READING

This has now been included in the distribution as HOP::Lexer::Article.

ACKNOWLEDGEMENTS

COPYRIGHT & LICENSE

ABOUT THE SOFTWARE

ELSEVIER SOFTWARE LICENSE AGREEMENT

LIMITED WARRANTY AND LIMITATION OF LIABILITY

YOU UNDERSTAND THAT ELSEVIER, ITS AFFILIATES, LICENSORS, SUPPLIERS AND AGENTS, MAKE NO WARRANTIES, EXPRESSED OR IMPLIED, WITH RESPECT TO THE SOFTWARE PRODUCT, INCLUDING, WITHOUT LIMITATION THE PROPRIETARY MATERIAL, AND SPECIFICALLY DISCLAIM ANY WARRANTY OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

IN NO EVENT WILL ELSEVIER, ITS AFFILIATES, LICENSORS, SUPPLIERS OR AGENTS, BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, WITHOUT LIMITATION, ANY LOST PROFITS, LOST SAVINGS OR OTHER INCIDENTAL OR CONSEQUENTIAL DAMAGES, ARISING OUT OF YOUR USE OR INABILITY TO USE THE SOFTWARE PRODUCT REGARDLESS OF WHETHER SUCH DAMAGES ARE FORESEEABLE OR WHETHER SUCH DAMAGES ARE DEEMED TO RESULT FROM THE FAILURE OR INADEQUACY OF ANY EXCLUSIVE OR OTHER REMEDY.

SOFTWARE LICENSE AGREEMENT

By installing, copying, or otherwise using the Software, you agree to be bound by the terms of this Agreement.

The parties agree as follows:

1 Grant of License

We grant you a nonexclusive license to use the Software for any purpose, commercial or non-commercial, as long as the following credit is included identifying the original source of the Software: "from Higher-Order Perl by Mark Dominus, published by Morgan Kaufmann Publishers, Copyright 2005 by Elsevier Inc".

2 Disclaimer of Warranty.

We make no warranties at all. The Software is transferred to you on an "as is" basis. You use the Software at your own peril. You assume all risk of loss for all claims or controversies, now existing or hereafter, arising out of use of the Software. We shall have no liability based on a claim that your use or combination of the Software with products or data not supplied by us infringes any patent, copyright, or proprietary right. All other warranties, expressed or implied, including, without limitation, any warranty of merchantability or fitness for a particular purpose are hereby excluded.

3 Limitation of Liability.

We will have no liability for special, incidental, or consequential damages even if advised of the possibility of such damages. We will not be liable for any other damages or loss in any way connected with the Software.