WARNING -- smoke testing is risky

"start()"

"start()" supports several optional arguments:

• "clean_cache_after" -- number of distributions that will be tested before checking to see if the CPAN build cache needs to be cleaned up (not including any prerequisites tested). Must be a positive integer. Defaults to 100
• "list" -- if provided, this list of distributions will be tested instead of all of CPAN. May be a reference to an array of distribution names or may be a filename containing one distribution name per line. Distribution names must be of the form 'AUTHOR/Dist-Name-0.00.tar.gz'
• "restart_delay" -- number of seconds that must elapse before restarting smoke testing. This will reload indices to search for new distributions and restart testing from the most recent distribution. Must be a positive integer; Defaults to 43200 seconds (12 hours)
• "skip_dev_versions" -- if true, unindexed distributions will not be tested. Valid values are 0 or 1. Defaults to 0.
• "set_term_title" -- toggle for whether the terminal titlebar will be updated with the distribution being smoke tested and the starting time of the test. Helps determine if a test is hung and which distribution might be responsible. Valid values are 0 or 1. Defaults to 1
• "status_file" -- during testing, the name of the distribution under test and a timestamp are written to this file. The file is removed after the test is complete. This helps identify a problem distribution if testing hangs or crashes the computer. If the argument includes a path, all directories to the file must exist. Defaults to "smoker-status-$$.txt" in File::Spec->tmpdir.
• "install" -- toggle for whether the distribution should be installed after successful testing. Can be useful to avoid prerequisite re-building and growing PERL5LIB for the cost of disk space used for installed modules. Valid values are 0 or 1. Defaults to 0
• "reverse" -- toggle the order in which releases are tested. When set to 1, testing starts from the older release not the most recent one (or the last distribution if --list is provided). Valid values are 0 or 1. Defaults to 0
• "random" -- toggle whether to randomize distribution test order. When set to 1, the list of releases is shuffled. Valid values are 0 or 1. Defaults to 0
• "force_trust" -- toggle whether to override CPAN's "trust_test_report_history" option. When set to 1, "trust_test_report_history" is set to 1. When set to 0, "trust_test_report_history" is left alone and whatever the user has configured for their CPAN client is used. Valid values are 0 or 1. Defaults to 0
• "reload_history_period" -- after this period in seconds, history of modules smoked will be reloaded when possible. Default value 1800 seconds (30 minutes).

Selection of distributions to test

For example, if Foo-Bar-0.01, Foo-Bar-0.02, Foo-Bar-0.03_01 and Foo-Bar-0.03_02 are on CPAN, only Foo-Bar-0.02 and Foo-Bar-0.03_02 will be tested, and in reverse order of when they were uploaded. Once Foo-Bar-0.04 is released and indexed, Foo-Bar-0.03_02 will not longer be tested.

To avoid testing script or other tarballs, developer distributions included must have a base distribution name that resembles a distribution tarball already indexed by PAUSE. If the first upload of distribution to PAUSE is a developer release -- Baz-Bam-0.00_01.tar.gz -- it will not be tested as there is no indexed Baz-Bam appearing in CPAN's 02packages.details.txt file.

Unauthorized tarballs are treated like developer releases and will be tested if they resemble an indexed distribution and are newer than the indexed tarball.

Perl, parrot, kurila, Pugs and similar distributions will not be tested. The skip list is based on CPAN::Mini and matches as follows:

     qr{(?:
           /(?:emb|syb|bio)?perl-\d
         | /(?:parrot|ponie|kurila|Perl6-Pugs)-\d
         | /perl-?5\.004
         | /perl_mlb\.zip
     )}xi,

Bundles and mod_perl distributions will also not be tested, though mod_perl is likely to be requested as a dependency by many modules. See the next section for how to tell CPAN.pm not to test certain dependencies.

Skipping additional distributions

The first step is configuring a directory for distroprefs files:

     $ cpan
     cpan> o conf init prefs_dir
     cpan> o conf commit

Next, ensure that either the YAML or YAML::Syck module is installed. (YAML::Syck is faster). Then create a file in the "prefs_dir" directory to hold the list of distributions to disable, e.g. call it "disabled.yml"

In that file, you can add blocks of YAML code to disable distributions. The match criteria "distribution" is a regex that matches against the canonical name of a distribution, e.g. "AUTHOR/Foo-Bar-3.14.tar.gz".

Here is a sample file to show you some syntax (don't actually use these, though):

     ---
     comment: "Tests take too long"
     match:
         distribution: "^DAGOLDEN/CPAN-Reporter-\d"
     disabled: 1
     ---
     comment: "Skip Win32 distributions"
     match:
         distribution: "/Win32"
     disabled: 1
     ---
     comment: "Skip distributions by Andy Lester"
     match:
         distribution: "^PETDANCE"
     disabled: 1

Please note that disabling distributions like this will also disable them for normal, non-smoke usage of CPAN.pm.

One distribution that I would recommend either installing up front or else disabling with distroprefs is mod_perl, as it is a common requirement for many Apache:: modules but does not (easily) build and test under automation.

     ---
     comment: "Don't build mod_perl if required by some other module"
     match:
         distribution: "/mod_perl-\d"
     disabled: 1

Distroprefs are more powerful than this -- they can be used to automate responses to prompts in distributions, set environment variables, specify additional dependencies and so on. Read the docs for CPAN.pm for more and look in the "distroprefs" directory in the CPAN distribution tarball for examples.

Using a local CPAN::Mini mirror

An alternative is to use CPAN::Mini to create a local CPAN mirror and to point CPAN's "urllist" to the local mirror.

     $ cpan
     cpan> o conf urllist unshift file:///path/to/minicpan
     cpan> o conf commit

However, CPAN::Reporter::Smoker needs the "find-ls.gz" file, which CPAN::Mini does not mirror by default. Add it to a .minicpanrc file in your home directory to include it in your local CPAN mirror.

     also_mirror: indices/find-ls.gz

Note that CPAN::Mini does not mirror developer versions. Therefore, a live, network CPAN Mirror will be needed in the urllist to retrieve these.

Note that CPAN requires the LWP module to be installed to use a local CPAN mirror.

Alternatively, you might experiment with the alpha-quality release of CPAN::Mini::Devel, which subclasses CPAN::Mini to retrieve developer distributions (and find-ls.gz) using the same logic as CPAN::Reporter::Smoker.

Timing out hanging tests

Warning -- on Win32, terminating processes via the command_timeout is equivalent to SIGKILL and could cause system instability or later deadlocks

This option is still considered experimental.

Avoiding repetitive prerequisite testing

As of version 1.92_56, CPAN supports the "trust_test_report_history" config option. When set, CPAN will check the last test report for a distribution. If one is found, the results of that test are used instead of running tests again.

     $ cpan
     cpan> o conf init trust_test_report_history
     cpan> o conf commit

Avoiding repetitive prerequisite builds (EXPERIMENTAL)

NOTE: Support for "build_dir_reuse" is highly experimental. Wait for at least CPAN version 1.92_62 before trying this option.

     $ cpan
     cpan> o conf init build_dir_reuse
     cpan> o conf commit

Stopping early if a prerequisite fails

As of version 1.92_57, CPAN supports the "halt_on_failure" config option. When set, a prerequisite failure stops further processing.

     $ cpan
     cpan> o conf init halt_on_failure
     cpan> o conf commit

However, a disadvantage of halting early is that no DISCARD grade is recorded in the history. The next time CPAN::Reporter::Smoker runs, the distribution will be tested again from scratch. It may be better to let all prerequisites finish so the distribution can fail its test and be flagged with DISCARD so it will be skipped in the future.

CPAN cache bloat

     $ cpan
     cpan> o conf init build_cache scan_cache
     cpan> o conf commit

CPAN verbosity

     $ cpan
     cpan> o conf init /verbosity/
     cpan> o conf commit

Saving reports to files instead of sending directly

     transport=File /path/to/directory

After review, send saved reports using Test::Reporter:

     Test::Reporter->new()->read($filename)->send()

Bugs / Feature Requests

Source Code

<https://github.com/cpan-testers/CPAN-Reporter-Smoker>

  git clone https://github.com/cpan-testers/CPAN-Reporter-Smoker.git