Token Streams

This document will describe only the token-stream model of input. Marpa allows other models of the input, but their use requires special method calls, which are described in the document on alternative input models.

new

    my $recce = Marpa::PP::Recognizer->new( { grammar => $grammar } );

The "new" method creates a recognizer object. The "new" method either returns a new recognizer object or throws an exception.

The arguments to the "new" method are references to hashes of named arguments. In each key/value pair of these hashes, the key is the argument name, and the hash value is the value of the argument. The named arguments are described below.

terminals_expected

    my $terminals_expected = $recce->terminals_expected();

Returns a reference to a list of strings, where the strings are the names of the terminals acceptable at the current location. In the default input model, the presence of a terminal in this list means that terminal will be acceptable in the next "read" method call. This is highly useful for Ruby Slippers parsing.

check_terminal

    my $is_symbol_a_terminal = $recce->check_terminal('Document');

Returns a Perl true when its argument is the name of a terminal symbol. Otherwise, returns a Perl false. Not often needed.

set

    $recce->set( { max_parses => 10, } );

The "set" method's arguments are references to hashes of named arguments. The "set" method can be used to set or change named arguments after the recognizer has been created. Details of the named arguments are below.

read

    $recce->read( 'Number', 42 );
    $recce->read( 'Multiply', );
    $recce->read( 'Number', 1 );
    $recce->read( 'Add', );
    $recce->read( 'Number', 7 );

The "read" method reads one token at the current parse location. It then advances the current location by 1.

"read" takes two arguments: a token name and a token value. The token name is required. It must be the name of a valid terminal symbol. The token value is optional. It defaults to a Perl "undef" when not specified. For details about terminal symbols, see "Terminals" in Marpa::PP::Grammar.

The parser may accept or reject the token. If the parser accepted the token, the "read" method returns the number of tokens which are acceptable at the new current location. This number may be helpful in guiding Ruby Slippers parsing.

"read" may return zero, which means that no tokens will be acceptable at the next earleme. This is turn means that the next "read" call will fail. In the default input model, where the "read" method is the only means of inputing tokens, a zero return from a "read" method means that the parse is exhausted -- that no more input is possible. More details on "exhaustion" are in a section below.

Marpa may reject a token because it is not one of those acceptable at the current location. When this happens, "read" returns a Perl "undef". A rejected token need not end parsing -- it is perfectly possible to retry the "read" call with another token. This is, in fact, an important technique in Ruby Slippers parsing. For details, see the section on Ruby Slippers parsing.

For other failures, including an attempt to "read" a token into an exhausted parser, Marpa throws an exception.

value

show_earley_sets

    print $recce->show_earley_sets()
        or die "print failed: $ERRNO";

An advanced, internals-oriented tracing method, which will not be of interest to most users. Most users will want to use the "show_progress" method instead. "show_earley_sets" returns a multi-line string listing every Earley item in every Earley set.

show_progress

    print $recce->show_progress()
        or die "print failed: $ERRNO";

Returns a string describing the progress of the parse. With no arguments, the string contains reports for the current location. With a non-negative argument N, the string contains reports for location N.

With two numeric arguments, N and M, the arguments are interpreted as a range of locations and the returned string contains reports for all locations in the range. The first argument, N, must be a non-negative integer, and is always the number of the location which begins the range. If both arguments are non-negative integers, the range is from location N to location M. If the second argument is a negative integer, -M, the end of the range is the Mth location from the furthest location. For example, if 42 was the furthest location, -1 would be location 42 and -2 would be location 41. The method call "$recce->show_progress(0, -1)" will print progress reports for the entire parse.

"show_progress" is Marpa's most powerful tool for debugging application grammars. It can also be used to track the progress of a parse or to investigate how a parse works. A much fuller description, with an example, is in the document on debugging Marpa grammars.

grammar

too_many_earley_items

Marpa parses from any BNF, and can handle grammars and inputs which produce large Earley sets. But parsing that involves large Earley sets can be slow. Large Earley sets are something most applications can, and will wish to, avoid.

By default, Marpa calculates an Earley item warning threshold based on the size of the grammar. The default threshold will never be less than 100. If the Earley item warning threshold is set to 0, warnings about large Earley sets are turned off.

trace_file_handle

trace_terminals

At a trace level of 1 or higher, Marpa produces a trace message for each terminal as it is accepted or rejected by the recognizer. At a trace level of 2 or higher, the trace messages include, for every location, a list of the terminals expected. In practical grammars, output from trace level 2 can be voluminous.

warnings

An Example

• Modifying existing tokens, and
• Creating new tokens.

The second technique, the creation of new, "virtual", tokens is used by Marpa::HTML to deal with omitted start and end tags. The actual HTML grammar that Marpa::HTML uses takes an oversimplified view of the HTML -- it assumes, even when the HTML standards do not require it, that start and end tags are always present. For most HTML files of interest, this assumption will be contrary to fact.

Ruby Slippers parsing is used to make the grammar's over-simplistic view of the world come true for it. Whenever a token is rejected, Marpa::HTML looks at the expected tokens list. If it sees that a start or end tag is expected, Marpa::HTML creates a token for it -- a completely new "virtual" token that gives the parser exactly what it expects. Marpa::HTML then resumes input at the point in the original input stream where it left off.

end

closures

When an action name is a key in the "closures" named argument, the usual action resolution mechanism of the semantics is bypassed. One common use of the "closures" named argument is to allow anonymous subroutines to be semantic actions. For more details, see the document on semantics.

max_parses

Marpa allows extremely ambiguous grammars. "max_parses" can be used if the user wants to see only the first few parse results of an ambiguous parse. "max_parses" is also useful to limit CPU usage and output length when testing and debugging.

trace_actions

trace_values