high-level functions

    # read a file with encoding from the BOM:
    open_bom(FH, $file)
    open_bom(FH, $file, ':utf8') # the same but with a default encoding

    # get encoding too
    $encoding = open_bom(FH, $file, ':utf8');

    # open a potentially unseekable file:
    ($encoding, $spillage) = open_bom(FH, $file, ':utf8');

    # change encoding of an open handle according to BOM
    $encoding = defuse(*HANDLE);
    ($encoding, $spillage) = defuse(*HANDLE);

    # Decode a string according to leading BOM:
    $unicode = decode_from_bom($string_with_bom);
    
    # Decode a string and get the encoding:
    ($unicode, $encoding) = decode_from_bom($string_with_bom)

PerlIO::via interface

    # Read the Right Thing from a unicode file with BOM:
    open(HANDLE, '<:via(File::BOM)', $filename)

    # Writing little-endian UTF-16 file with BOM:
    open(HANDLE, '>:encoding(UTF-16LE):via(File::BOM)', $filename)

lower-level functions

    # read BOM encoding from a filehandle:
    $encoding = get_encoding_from_filehandle(FH)

    # Get encoding even if FH is unseekable:
    ($encoding, $spillage) = get_encoding_from_filehandle(FH);

    # Get encoding from a known unseekable handle:
    ($encdoing, $spillage) = get_encoding_from_stream(FH);

    # get encoding and BOM length from BOM at start of string:
    ($encoding, $offset) = get_encoding_from_bom($string);

variables

    # print a BOM for a known encoding
    print FH $enc2bom{$encoding};

    # get an encoding from a known BOM
    $enc = $bom2enc{$bom}

symbols

• open_bom()
• defuse()
• decode_from_bom()
• get_encoding_from_filehandle()
• get_encoding_from_stream()
• get_encoding_from_bom()
• %bom2enc
• %enc2bom

tags

• :all

  All of the above

• :subs

  subroutines only

• :vars

  just %bom2enc and %enc2bom

%bom2enc

The keys of this hash are strings which represent the BOMs, the values are their encodings, in a format which is understood by Encode

The encodings represented in this hash are: UTF-8, UTF-16BE, UTF-16LE, UTF-32BE and UTF-32LE

%enc2bom

Note that UTF-16, UTF-32 and UCS-4 are not included in this hash. Mainly because Encode::encode automatically puts BOMs on output. See Encode::Unicode

open_bom

    $encoding = open_bom(HANDLE, $filename, $default_mode)

    ($encoding, $spill) = open_bom(HANDLE, $filename, $default_mode)

opens HANDLE for reading on $filename, setting the mode to the appropriate encoding for the BOM stored in the file.

On failure, a fatal error is raised, see the DIAGNOSTICS section for details on how to catch these. This is in order to allow the return value(s) to be used for other purposes.

If the file doesn't contain a BOM, $default_mode is used instead. Hence:

    open_bom(FH, 'my_file.txt', ':utf8')

Opens my_file.txt for reading in an appropriate encoding found from the BOM in that file, or as a UTF-8 file if none is found.

In the absence of a $default_mode argument, the following 2 calls should be equivalent:

    open_bom(FH, 'no_bom.txt');

    open(FH, '<', 'no_bom.txt');

If an undefined value is passed as the handle, a symbol will be generated for it like open() does:

    # create filehandle on the fly
    $enc = open_bom(my $fh, $filename, ':utf8');
    $line = <$fh>;

The filehandle will be cued up to read after the BOM. Unseekable files (e.g. fifos) will cause croaking, unless called in list context to catch spillage from the handle. Any spillage will be automatically decoded from the encoding, if found.

    e.g.

    # croak if my_socket is unseekable
    open_bom(FH, 'my_socket');

    # keep spillage if my_socket is unseekable
    ($encoding, $spillage) = open_bom(FH, 'my_socket');

    # discard any spillage from open_bom
    ($encoding) = open_bom(FH, 'my_socket');

defuse

    $enc = defuse(FH);

    ($enc, $spill) = defuse(FH);

FH should be a filehandle opened for reading, it will have the relevant encoding layer pushed onto it be binmode if a BOM is found. Spillage should be Unicode, not bytes.

Any uncaptured spillage will be silently lost. If the handle is unseekable, use list context to avoid data loss.

If no BOM is found, the mode will be unaffected.

decode_from_bom

    $unicode_string = decode_from_bom($string, $default, $check)

    ($unicode_string, $encoding) = decode_from_bom($string, $default, $check)

Reads a BOM from the beginning of $string, decodes $string (minus the BOM) and returns it to you as a perl unicode string.

if $string doesn't have a BOM, $default is used instead.

$check, if supplied, is passed to Encode::decode as the third argument.

If there's no BOM and no default, the original string is returned and encoding is ''.

See Encode

get_encoding_from_filehandle

    $encoding = get_encoding_from_filehandle(HANDLE)

    ($encoding, $spillage) = get_encoding_from_filehandle(HANDLE)

Returns the encoding found in the given filehandle.

The handle should be opened in a non-unicode way (e.g. mode '<:bytes') so that the BOM can be read in its natural state.

After calling, the handle will be set to read at a point after the BOM (or at the beginning of the file if no BOM was found)

If called in scalar context, unseekable handles cause a croak().

If called in list context, unseekable handles will be read byte-by-byte and any spillage will be returned. See get_encoding_from_stream()

get_encoding_from_stream

    ($encoding, $spillage) = get_encoding_from_stream(*FH);

Read a BOM from an unrewindable source. This means reading the stream one byte at a time until either a BOM is found or every possible BOM is ruled out. Any non-BOM bytes read from the handle will be returned in $spillage.

If a BOM is found and the spillage contains a partial character (judging by the expected character width for the encoding) more bytes will be read from the handle to ensure that a complete character is returned.

Spillage is always in bytes, not characters.

This function is less efficient than get_encoding_from_filehandle, but should work just as well on a seekable handle as on an unseekable one.

get_encoding_from_bom

    ($encoding, $offset) = get_encoding_from_bom($string)

Returns the encoding and length in bytes of the BOM in $string.

If there is no BOM, an empty string is returned and $offset is zero.

To get the data from the string, the following should work:

    use Encode;

    my($encoding, $offset) = get_encoding_from_bom($string);

    if ($encoding) {
        $string = decode($encoding, substr($string, $offset))
    }

Reading

Because of a limitation in PerlIO::via, read() always works on bytes, not characters. BOM decoding will still be done but output will be bytes of UTF-8.

    open(BOM, '<:via(File::BOM)', $file);
    $bytes_read = read(BOM, $buffer, $length);
    $unicode = decode('UTF-8', $buffer, Encode::FB_QUIET);

    # Now $unicode is valid unicode and $buffer contains any left-over bytes

Writing

There is a "Wide character in print" warning generated when the via(File::BOM) layer doesn't receive utf8 on writing. This glitch was resolved in perl version 5.8.7, but if your perl version is older than that, you'll need to make sure that the via(File::BOM) layer receives utf8 like this:

    # This works OK
    open(FH, '>:encoding(UTF-16LE):via(File::BOM):utf8', $filename)

    # This generates warnings with older perls
    open(FH, '>:encoding(UTF-16LE):via(File::BOM)', $filename)

Seeking

    # Seek to end of BOM (not start of file!)
    seek(FILE_BOM_HANDLE, 0, SEEK_SET)

Telling