langof

  $language = langof($text);

To get the most probable language and also the percentage of its probability, do:

  ($language, $probability) = langof($text);

If you want a hash where each active language is mapped into its percentage, use this:

  %languages = langof($text);

OPTIONS

langof can also be given some configuration parameters, in this way:

  $language = langof(\%config, $text);

These parameters are detailed here:

• extract-from

  When the size of the input exceeds the C'max-size', "langof" analyzes only the beginning of the file. You can specify which part of the file is analyzed with the 'extract-from' option:

    langof( { 'extract_from' => 'tail' } , $text );
      

  Possible values are 'head' and 'tail' (for now).

  You can also specify more than one part of the file, so that text is extracted from those parts:

    langof( { 'extract_from' => [ 'head', 'tail' ] } , $text );
      

  (this will be useful when more than two possibilities exist)

  You can also specify different values for each part of the file (not necessarily for all of them:

   langof( { 'extract_from' => { head => 40, tail => 60 } } , $text);
      

  The line above, for instance, retrives 40% of the text from the beginning and 60% from the end. Note, however, that those values are not percentages. You'd get the same behavior with:

   langof( { 'extract_from' => { head => 80, tail => 120 } } , $text);
      

  The percentages would be the same.

• max-size

  By default, "langof" analyzes only 1,000,000 bytes. You can specify how many bytes (at the most) can be analyzed (if not enough exist, the whole input is still analyzed).

    langof( { 'max-size' => 2000 }, $text);
      

  If you want all the text to be analyzed, set max-size to 0:

    langof( { 'max-size' => 0 }, $text);
      

  See also "set_max_size".

• method

  You can choose which method or methods to use, and also the relevance of each of them.

  To choose a single method to use:

    langof( {method => 'smallwords' }, $text);
      

  To choose several methods:

    langof( {method => [qw/prefixes2 suffixes2/]}, $text);
      

  To choose several methods and give them different weight:

    langof( {method => {smallwords => 0.5, ngrams3 => 1.5} }, $text);
      

  To see the list of available methods, see section METHODS OF LANGUAGE IDENTIFICATION.

  If no method is specified, the configuration for this parameter is the following (this might change in the future):

    method => {
      smallwords => 0.5,
      prefixes2  => 1,
      suffixes3  => 1,
      ngrams3    => 1.3
    };
      

• mode

  By default, "Lingua::Identify" assumes "normal" mode, but others are available.

  In "dummy" mode, instead of actually calculating anything, "Lingua::Identify" only does the preparation it has to and then returns a bunch of information, including the list of the active languages, the selected methods, etc. It also returns the text meant to be analised.

  Do be warned that, with langof_file, the dummy mode still reads the files, it simply doesn't calculate language.

    langof( { 'mode' => 'dummy' }, $text);
      

  This returns something like this:

    { 'methods'          => {   'smallwords' => '0.5',
                                'prefixes2'  => '1',
                            },
      'config'           => {   'mode' => 'dummy' },
      'max-size'         => 1000000,
      'active-languages' => [ 'es', 'pt' ],
      'text'             => $text,
      'mode'             => 'dummy',
    }
      

langof_file

Currently, langof_file assumes the files are regular text. This may change in the future and the files might be scanned to check their filetype and then parsed to extract only their textual content (which should be pretty useful so that you can perform language identification, say, in HTML files, or PDFs).

To identify the language a file is written in:

  $language = langof_file($path);

To get the most probable language and also the percentage of its probability, do:

  ($language, $probability) = langof_file($path);

If you want a hash where each active language is mapped into its percentage, use this:

  %languages = langof_file($path);

If you pass more than one file to langof_file, they will all be read and their content merged and then parsed for language identification.

OPTIONS

langof_file accepts all the options langof does, so refer to those first (up in this document).

  $language = langof_file(\%config, $path);

langof_file currently only reads the first 10,000 bytes of each file.

You can force an input encoding with "{ encoding => 'ISO-8859-1' }" in the configuration hash.

confidence

You can find more about the likeliness of the results to be accurate by computing its confidence level.

  use Lingua::Identify qw/:language_identification/;
  my @results = langof($text);
  my $confidence_level = confidence(@results);
  # $confidence_level now holds a value between 0.5 and 1; the higher that
  # value, the more accurate the results seem to be

The formula used is pretty simple: p1 / (p1 + p2) , where p1 is the probability of the most likely language and p2 is the probability of the language which came in second. A couple of examples to illustrate this:

English 50% Portuguese 10% ...

confidence level: 50 / (50 + 10) = 0.83

Another example:

Spanish 30% Portuguese 10% ...

confidence level: 30 / (25 + 30) = 0.55

French 10% German 5% ...

confidence level: 10 / (10 + 5) = 0.67

As you can see, the first example is probably the most accurate one. Are there any doubts? The English language has five times the probability of the second language.

The second example is a bit more tricky. 55% confidence. The confidence level is always above 50%, for obvious reasons. 55% doesn't make anyone confident in the results, and one shouldn't be, with results such as these.

Notice the third example. The confidence level goes up to 67%, but the probability of French is of mere 10%. So what? It's twice as much as the second language. The low probability may well be caused by a great number of languages in play.

get_all_methods

Small Word Technique - smallwords

This is usually a good method for big texts, especially if you happen to have few languages active.

Prefix Analysis - prefixes1, prefixes2, prefixes3, prefixes4

The methods are, respectively, for prefixes of size 1, 2, 3 and 4.

Suffix Analysis - suffixes1, suffixes2, suffixes3, suffixes4

The methods are, respectively, for suffixes of size 1, 2, 3 and 4.

Ngram Categorization - ngrams1, ngrams2, ngrams3, ngrams4

Ngrams are a very good way for identifying languages, given that the most common ones of each language are not generally very common in others.

This is usually the best method for small amounts of text or too many active languages.

The methods are, respectively, for ngrams of size 1, 2, 3 and 4.

THE BASIC EXAMPLE

  use Lingua::Identify qw/langof/;

  my $text = join "\n", <>;

  # identify the language by letting the module decide on the best way
  # to do so
  my $language = langof($text);

IDENTIFYING BETWEEN TWO LANGUAGES

  use Lingua::Identify qw/langof set_active_languages/;
  set_active_languages(qw/pt en/);

  my $text = join "\n", <>;

  # identify the language by letting the module decide on the best way
  # to do so
  my $language = langof($text);