NAME

SYNOPSIS

  use Test::Cmd;
  use Test::More tests => 3;

  my $test = Test::Cmd->new( prog => 'outerr', workdir => '' );
  $test->run();

  is( $test->stdout, "out\n", 'standard out' );
  is( $test->stderr, "err\n", 'standard error' );
  is( $? >> 8,       1,       'exit status' );

Where "outerr" is the shell script:

  $ cat outerr 
  #!/bin/sh
  echo out
  echo >&2 err
  exit 1
  $ chmod +x outerr

See below for other examples. Otherwise, the full list of available methods is:

  use Test::Cmd;

  $test = Test::Cmd->new(prog => 'program_or_script_to_test',
                        interpreter => 'script_interpreter',
                        string => 'identifier_string',
                        workdir => '',
                        subdir => 'dir',
                        match_sub => $code_ref,
                        verbose => 1);

  $test->verbose(1);

  $test->prog('program_or_script_to_test');

  $test->basename(@suffixlist);

  $test->interpreter('script_interpreter');

  $test->string('identifier string');

  $test->workdir('prefix');

  $test->workpath('subdir', 'file');

  $test->subdir('subdir', ...);
  $test->subdir(['sub', 'dir'], ...);

  $test->write('file', <<'EOF');
  contents of file
  EOF
  $test->write(['subdir', 'file'], <<'EOF');
  contents of file
  EOF

  $test->read(\$contents, 'file');
  $test->read(\@lines, 'file');
  $test->read(\$contents, ['subdir', 'file']);
  $test->read(\@lines, ['subdir', 'file']);

  $test->writable('dir');
  $test->writable('dir', $rwflag);
  $test->writable('dir', $rwflag, \%errors);

  $test->preserve(condition, ...);

  $test->cleanup(condition);

  $test->run(prog => 'program_or_script_to_test',
                interpreter => 'script_interpreter',
                chdir => 'dir', args => 'arguments', stdin => <<'EOF');
  input to program
  EOF

  $test->pass(condition);
  $test->pass(condition, \&func);

  $test->fail(condition);
  $test->fail(condition, \&func);
  $test->fail(condition, \&func, $caller);

  $test->no_result(condition);
  $test->no_result(condition, \&func);
  $test->no_result(condition, \&func, $caller);

  $test->stdout;
  $test->stdout($run_number);

  $test->stderr;
  $test->stderr($run_number);

  $test->match(\@lines, \@matches);
  $test->match($lines, $matches);

  $test->match_exact(\@lines, \@matches);
  $test->match_exact($lines, $matches);

  $test->match_regex(\@lines, \@regexes);
  $test->match_regex($lines, $regexes);

  $test->diff_exact(\@lines, \@matches, \@output);
  $test->diff_exact($lines, $matches, \@output);

  $test->diff_regex(\@lines, \@regexes, \@output);
  $test->diff_regex($lines, $regexes, \@output);

  sub func {
        my ($self, $lines, $matches) = @_;
        # code to match $lines and $matches
  }
  $test->match_sub(\&func);
  $test->match_sub(sub { code to match $_[1] and $_[2] });

  $test->here;

DESCRIPTION

The "Test::Cmd" module makes no assumptions about what constitutes a successful or failed test. Attempting to read a file that doesn't exist, for example, may or may not be an error, depending on the software being tested.

Consequently, no "Test::Cmd" methods (including the "new()" method) exit, die or throw any other sorts of exceptions (but they all do return useful error indications). Exceptions or other error status should be handled by a higher layer: a subclass of Test::Cmd, or another testing framework such as the Test or Test::Simple Perl modules, or by the test itself.

(That said, see the Test::Cmd::Common module if you want a similar module that provides exception handling, either to use directly in your own tests, or as an example of how to use "Test::Cmd".)

In addition to running tests and evaluating conditions, the "Test::Cmd" module manages and cleans up one or more temporary workspace directories, and provides methods for creating files and directories in those workspace directories from in-line data (that is, here-documents), allowing tests to be completely self-contained. When used in conjunction with another testing framework, the "Test::Cmd" module can function as a fixture (common startup code for multiple tests) for simple management of command execution and temporary workspaces.

The "Test::Cmd" module inherits File::Spec methods ("file_name_is_absolute()", "catfile()", etc.) to support writing tests portably across a variety of operating and file systems.

A "Test::Cmd" environment object is created via the usual invocation:

    $test = Test::Cmd->new();

Arguments to the "Test::Cmd::new" method are keyword-value pairs that may be used to initialize the object, typically by invoking the same-named method as the keyword.

TESTING FRAMEWORKS

Note that you should not intermix multiple testing frameworks in a single testing script.

    use Test::Cmd;

    print "1..5\n";

    $test = Test::Cmd->new(prog => 'test_program', workdir => '');
    if ($test) { print "ok 1\n"; } else { print "not ok 1\n"; }

    $input = <<_EOF;
    test_program should process this input
    and exit successfully (status 0).
    _EOF_

    $wrote_file = $test->write('input_file', $input);
    if ($wrote_file) { print "ok 2\n"; } else { print "not ok 2\n"; }

    $test->run(args => '-x input_file');
    if ($? == 0) { print "ok 3\n"; } else { print "not ok 3\n"; }

    $wrote_file = $test->write('input_file', $input);
    if ($wrote_file) { print "ok 4\n"; } else { print "not ok 4\n"; }

    $test->run(args => '-y input_file');
    if ($? == 0) { print "ok 5\n"; } else { print "not ok 5\n"; }

    use Test::Simple tests => 5;
    use Test::Cmd;

    $test = Test::Cmd->new(prog => 'test_program', workdir => '');
    ok($test, "creating Test::Cmd object");

    $input = <<_EOF;
    test_program should process this input
    and exit successfully (status 0).
    _EOF_

    $wrote_file = $test->write('input_file', $input);
    ok($wrote_file, "writing input_file");

    $test->run(args => '-x input_file');
    ok($? == 0, "executing test_program -x input_file");

    $wrote_file = $test->write('input_file', $input);
    ok($wrote_file, "writing input_file");

    $test->run(args => '-y input_file');
    ok($? == 0, "executing test_program -y input_file");

    use Test::Unit;
    use Test::Cmd;
    
    my $test;
    
    $input = <<'EOF';
    test_program should process this input
    and exit successfully (status 0).
    EOF
    
    sub set_up {
        $test = Test::Cmd->new(prog => 'test_program', workdir => '');
        $test->write('input_file', $input);
    }
    
    sub test_x {
        my $result = $test->run(args => '-x input_file');
        assert($result == 0, "failed test_x\n");
    }
    
    sub test_y {
        my $result = $test->run(args => '-y input_file');
        assert($result == 0, "failed test_y\n");
    }
    
    create_suite();
    run_suite;

    use Test::Cmd;

    $test = Test::Cmd->new(prog => 'test_program', workdir => '');
    Test::Cmd->no_result(! $test);

    $input = <<EOF;
    test_program should process this input
    and exit successfully (status 0).
    EOF

    $wrote_file = $test->write('input_file', $input);
    $test->no_result(! $wrote_file);

    $test->run(args => '-x input_file');
    $test->fail($? != 0);

    $wrote_file = $test->write('input_file', $input);
    $test->no_result(! $wrote_file);

    $test->run(args => '-y input_file');
    $test->fail($? != 0);

    $test->pass;

    use Test::Cmd::Common;

    $test = Test::Cmd::Common->new(prog => 'test_program', workdir => '');

    $input = <<EOF;
    test_program should process this input
    and exit successfully (status 0).
    EOF

    $wrote_file = $test->write('input_file', $input);

    $test->run(args => '-x input_file');

    $wrote_file = $test->write('input_file', $input);

    $test->run(args => '-y input_file');

    $test->pass;

METHODS

"new"

Create a new "Test::Cmd" environment. Arguments with which to initialize the environment are passed in as keyword-value pairs. Fails if a specified temporary working directory or subdirectory cannot be created. Does NOT die or exit on failure, but returns "undef" if the test environment object cannot be created.

"verbose"

Sets the verbose level for the environment object to the specified value.

"prog"

Specifies the executable program or script to be tested. Returns the absolute path name of the current program or script.

"basename"

Returns the basename of the current program or script. Any specified arguments are a list of file suffixes that may be stripped from the basename.

"interpreter"

Specifies the program to be used to interpret "prog" as a script. Returns the current value of "interpreter".

"string"

Specifies an identifier string for the functionality being tested to be printed on failure or no result.

"workdir"

When an argument is specified, creates a temporary working directory with the specified name. If the argument is a NULL string (''), the directory is named "testcmd" by default, followed by the unique ID of the executing process.

Returns the absolute pathname to the temporary working directory, or FALSE if the directory could not be created.

"workpath"

Returns the absolute path name to a subdirectory or file under the current temporary working directory by concatenating the temporary working directory name with the specified arguments.

"subdir"

Creates new subdirectories under the temporary working dir, one for each argument. An argument may be an array reference, in which case the array elements are concatenated together using the "File::Spec-&"catfile> method. Subdirectories multiple levels deep must be created via a separate argument for each level:

    $test->subdir('sub', ['sub', 'dir'], [qw(sub dir ectory)]);
    

Returns the number of subdirectories actually created.

"write"

Writes the specified text (second argument) to the specified file name (first argument). The file name may be an array reference, in which case all the array elements except the last are subdirectory names to be concatenated together. The file is created under the temporary working directory. Any subdirectories in the path must already exist.

"read"

Reads the contents of the specified file name (second argument) into the scalar or array referred to by the first argument. The file name may be an array reference, in which case all the array elements except the last are subdirectory names to be concatenated together. The file is assumed to be under the temporary working directory unless it is an absolute path name.

Returns TRUE on successfully opening and reading the file, FALSE otherwise.

"writable"

Makes every file and directory within the specified directory tree writable ("rwflag" == TRUE) or not writable ("rwflag" == FALSE). The default is to make the directory tree writable. Optionally fills in the supplied hash reference with a hash of path names that could not have their permissions set appropriately, with the reason why each could not be set.

"preserve"

Arranges for the temporary working directories for the specified "Test::Cmd" environment to be preserved for one or more conditions. If no conditions are specified, arranges for the temporary working directories to be preserved for all conditions.

"cleanup"

Removes any temporary working directories for the specified "Test::Cmd" environment. If the environment variable "PRESERVE" was set when the "Test::Cmd" module was loaded, temporary working directories are not removed. If any of the environment variables "PRESERVE_PASS", "PRESERVE_FAIL", or "PRESERVE_NO_RESULT" were set when the "Test::Cmd" module was loaded, then temporary working directories are not removed if the test passed, failed, or had no result, respectively. Temporary working directories are also preserved for conditions specified via the "preserve" method.

Typically, this method is not called directly, but is used when the script exits to clean up temporary working directories as appropriate for the exit status.

"run"

Runs a test of the program or script for the test environment. Standard output and error output are saved for future retrieval via the "stdout" and "stderr" methods.

Arguments are supplied as keyword-value pairs:

"pass"

Exits the test successfully. Reports "PASSED" on the error output and exits with a status of 0. If a condition is supplied, only exits the test if the condition evaluates TRUE. If a function reference is supplied, executes the function before reporting and exiting.

"fail"

Exits the test unsuccessfully. Reports "FAILED test of {string} at line {line} of {file}." on the error output and exits with a status of 1. If a condition is supplied, only exits the test if the condition evaluates TRUE. If a function reference is supplied, executes the function before reporting and exiting. If a caller level is supplied, prints a simple calling trace N levels deep as part of reporting the failure.

"no_result"

Exits the test with an indeterminate result (the test could not be performed due to external conditions such as, for example, a full file system). Reports "NO RESULT for test of {string} at line {line} of {file}." on the error output and exits with a status of 2. If a condition is supplied, only exits the test if the condition evaluates TRUE. If a function reference is supplied, executes the function before reporting and exiting. If a caller level is supplied, prints a simple calling trace N levels deep as part of reporting the failure.

"stdout"

Returns the standard output from the specified run number. If there is no specified run number, then returns the standard output of the last run. Returns the standard output as either a scalar or an array of output lines, as appropriate for the calling context. Returns "undef" if there has been no test run.

"stderr"

Returns the error output from the specified run number. If there is no specified run number, then returns the error output of the last run. Returns the error output as either a scalar or an array of output lines, as apporpriate for the calling context. Returns "undef" if there has been no test run.

"match"

Matches one or more input lines against an equal number of expected lines using the currently-registered line-matching function. The default line-matching function is the "match_regex" method, which means that the default is to match lines against regular expressions.

"match_exact"

Compares two arrays of lines for exact matches. The arguments are passed in as either scalars, in which case each is split on newline boundaries, or as array references. An unequal number of lines in the two arrays fails immediately and returns FALSE before any comparisons are performed.

Returns TRUE if each line matched its corresponding line in the other array, FALSE otherwise.

"match_regex"

Matches one or more input lines against an equal number of regular expressions. The arguments are passed in as either scalars, in which case each is split on newline boundaries, or as array references. Trailing newlines are stripped from each line and regular expression. An unequal number of lines and regular expressions fails immediately and returns FALSE before any comparisons are performed. Comparison is performed for each entire line, that is, with each regular expression anchored at both the start of line (^) and end of line ($).

Returns TRUE if each line matched each regular expression, FALSE otherwise.

"diff_exact"

Diffs two arrays of lines in a manner similar to the UNIX diff(1) utility.

If the Algorithm::DiffOld package is installed on the local system, output describing the differences between the input lines and the matching lines, in diff(1) format, is saved to the $output array reference. In the diff output, the expected output lines are considered the "old" (left-hand) file, and the actual output is considered the "new" (right-hand) file.

If the Algorithm::DiffOld package is not installed on the local system, the Expected and Actual contents are saved as-is to the $output array reference.

The "lines" and "matches" arguments are passed in as either scalars, in which case each is split on newline boundaries, or as array references. Trailing newlines are stripped from each line and regular expression.

Returns TRUE if each line matched its corresponding line in the expected matches, FALSE otherwise, in order to conform to the conventions of the "match" method.

Typical invocation:

        if (! $test->diff_exact($test->stdout,
                                \@expected_lines,
                                \@diff)) {
                print @diff;
        }
    

"diff_regex"

Diffs one or more input lines against one or more regular expressions in a manner similar to the UNIX diff(1) utility.

If the Algorithm::DiffOld package is installed on the local system, output describing the differences between the input lines and the matching lines, in diff(1) format, is saved to the $output array reference. In the diff output, the expected output lines are considered the "old" (left-hand) file, and the actual output is considered the "new" (right-hand) file.

If the Algorithm::DiffOld package is not installed on the local system, the Expected and Actual contents are saved as-is to the $output array reference.

The "lines" and "regexes" arguments are passed in as either scalars, in which case each is split on newline boundaries, or as array references. Trailing newlines are stripped from each line and regular expression. Comparison is performed for each entire line, that is, with each regular expression anchored at both the start of line (^) and end of line ($).

Returns TRUE if each line matched each regular expression, FALSE otherwise, in order to conform to the conventions of the "match" method.

Typical invocation:

        if (! $test->diff_regex($test->stdout,
                                \@expected_lines,
                                \@diff)) {
                print @diff;
        }
    

"match_sub"

Registers the specified code reference as the line-matching function to be called by the "match" method. This can be a user-supplied subroutine, or the "match_exact", "match_regex", "diff_exact", or "diff_regex" methods supplied by the "Test::Cmd" module:

        $test->match_sub(\&Test::Cmd::match_exact);

        $test->match_sub(\&Test::Cmd::match_regex);

        $test->match_sub(\&Test::Cmd::diff_exact);

        $test->match_sub(\&Test::Cmd::diff_regex);
    

The "match_exact", "match_regex", "diff_exact" and "diff_regex" subroutine names are exportable from the "Test::Cmd" module, and may be specified at object initialization:

        use Test::Cmd qw(match_exact match_regex diff_exact diff_regex);
        $test_exact = Test::Cmd->new(match_sub => \&match_exact);
        $test_regex = Test::Cmd->new(match_sub => \&match_regex);
        $test_exact = Test::Cmd->new(match_sub => \&diff_exact);
        $test_regex = Test::Cmd->new(match_sub => \&diff_regex);
    

"here"

Returns the absolute path name of the current working directory. (This is essentially the same as the "Cwd::cwd" method, except that the "Test::Cmd::here" method preserves the directory separators exactly as returned by the underlying operating-system-dependent method. The "Cwd::cwd" method canonicalizes all directory separators to '/', which makes for consistent path name representations within Perl, but may mess up another program or script to which you try to pass the path name.)

ENVIRONMENT

"PRESERVE"

If set to a true value, all temporary working directories will be preserved on exit, regardless of success or failure of the test. The full path names of all temporary working directories will be reported on error output.

"PRESERVE_FAIL"

If set to a true value, all temporary working directories will be preserved on exit from a failed test. The full path names of all temporary working directories will be reported on error output.

"PRESERVE_NO_RESULT"

If set to a true value, all temporary working directories will be preserved on exit from a test for which there is no result. The full path names of all temporary working directories will be reported on error output.

"PRESERVE_PASS"

If set to a true value, all temporary working directories will be preserved on exit from a successful test. The full path names of all temporary working directories will be reported on error output.

"VERBOSE"

When set to a true value, enables verbose reporting of various internal things (path names, exact command line being executed, etc.).

PORTABLE TESTS

The best and most comprehensive set of portability guidelines is the standard "Writing portable Perl" document at:

        http://www.perl.com/pub/doc/manual/html/pod/perlport.html

To reiterate one important point from the "WpP" document: Not all Perl programs have to be portable. If the program or script you're testing is UNIX-specific, you can (and should) use the "Test::Cmd" module to write UNIX-specific tests.

That having been said, here are some hints that may help keep your tests portable, if that's a requirement.

Use the "Test::Cmd-&"here> method for current directory path.

The normal Perl way to fetch the current working directory is to use the "Cwd::cwd" method. Unfortunately, the "Cwd::cwd" method canonicalizes the path name it returns, changing the native directory separators into the forward slashes favored by Perl and UNIX. For most Perl scripts, this makes a great deal of sense and keeps code uncluttered.

Passing in a file name that has had its directory separators altered, however, may confuse the command or script under test, or make it difficult to compare output from the command or script with an expected result. The "Test::Cmd::here" method returns the absolute path name of the current working directory, like "Cwd::cwd", but does not manipulate the returned path in any way.

Use "File::Spec" methods for manipulating path names.

The File::Spec module provides a system-independent interface for manipulating path names. Because the "Test::Cmd" class is a sub-class of the File::Spec class, you can use these methods directly as follows:

        if (! Test::Cmd->file_name_is_absolute($prog)) {
                my $prog = Test::Cmd->catfile(Test::Cmd->here, $prog);
        }
    

For details about the available methods and their use, see the documentation for the File::Spec module and its sub-modules, especially the File::Spec::Unix modules.

Use "Config" for file-name suffixes, where possible.

The standard Config module provides values that reflect the file-name suffixes on the system for which the Perl executable was built. This provides convenient portability for situations where a file name may have different extensions on different systems:

        $foo_exe = "foo$Config{_exe}";
        ok(-f $foo_exe);
    

(Unfortunately, there is no existing $Config value that specifies the suffix for a directly-executable Perl script.)

Avoid generating executable programs or scripts.

How to make a file or script executable varies widely from system to system, some systems using file name extensions to indicate executability, others using a file permission bit. The differences are complicated to accommodate in a portable test script. The easiest way to deal with this complexity is to avoid it if you can.

If your test somehow requires executing a script that you generate from the test itself, the best way is to generate the script in Perl and then explicitly feed it to the Perl executable on the local system. To be maximally portable, use the $^X variable instead of hard-coding "perl" into the string you execute:

        $line = "This is output from the generated perl script.";
        $test->write('script', <<EOF);
        print STDOUT "$line\\n";
        EOF
        $output = `$^X script`;
        ok($output eq "$line\n");
    

This completely avoids having to make the "script" file itself executable. (Since you're writing your test in Perl, it's safe to assume that Perl itself is executable.)

If you must generate a directly-executable script, then use the $Config{'startperl'} variable at the start of the script to generate the appropriate magic that will execute it as a Perl script:

        use Config;
        $line = "This is output from the generated perl script.";
        $test->write('script', <<EOF);
        $Config{'startperl'};
        print STDOUT "$line\\n";
        EOF
        chdir($test->workdir);
        chmod(0755, 'script');  # POSIX-SPECIFIC
        $output = `script`;
        ok($output eq "$line\n");
    

Addtional hints on writing portable tests are welcome.

SEE ALSO

Alternative command-testing modules include:

Test::Exit, Test::Output, or using Capture::Tiny with one of the above test modules, for example Test::More.

A rudimentary page for the "Test::Cmd" module is available at:

        http://www.baldmt.com/Test-Cmd/

The most involved example of using the "Test::Cmd" package to test a real-world application is the "cons-test" testing suite for the Cons software construction utility. The suite uses a sub-class of Test::Cmd::Common (which in turn is a sub-class of "Test::Cmd") to provide common, application-specific infrastructure across a large number of end-to-end application tests. The suite, and other information about Cons, is available at:

        http://www.dsmit.com/cons

REPOSITORY

AUTHORS

This module is now being maintained by Neil Bowers <neilb@cpan.org>.

COPYRIGHT

ACKNOWLEDGEMENTS

        http://www.dsmit.com/cons/

The general idea of managing temporary working directories in this way, as well as the test reporting of the "pass", "fail" and "no_result" methods, come from the testing framework invented by Peter Miller for his Aegis project change supervisor. Aegis is an excellent bit of work which integrates creation and execution of regression tests into the software development process. Information about Aegis is available at:

        http://www.tip.net.au/~millerp/aegis.html

Thanks to Michael Schwern for all of the thoughtful work he's put into Perl's standard testing methodology, including the Test::Simple and Test::More modules, and enhancement and maintenance of the Test and Test::Harness modules. Thanks also to Christian Lemburg for the impressively complete Test::Unit framework of modules. Ideas from both have helped keep "Test::Cmd" flexible enough to be useful in multiple testing frameworks.