yarn - scenario testing of Unix command line tools
is a scenario testing tool: you write a scenario describing how a
user uses your software and what should happen, and express, using very
lightweight syntax, the scenario in such a way that it can be tested
automatically. The scenario has a simple, but strict structure:
GIVEN some setup for the test
WHEN thing that is to be tested happens
THEN the post-conditions must be true
As an example, consider a very short test scenario for verifying that a backup
program works, at least for one simple case.
SCENARIO backups can be restored
GIVEN some live data in a directory
AND an empty backup repository
WHEN a backup is made
THEN the data case be restored
Note the addition of AND: you can have multiple GIVEN, WHEN, and THEN
statements. The AND keyword makes the text be more readable. SCENARIO is also
necessary, and gives the title.
FINALLY is for cleanups. The FINALLY steps will be run regardless of whether the
scenario succeeds or not.
Scenarios are meant to be written in somewhat human readable language. However,
they are not free form text. In addition to the GIVEN/WHEN/THEN structure, the
text for each of the steps needs a computer-executable implementation. This is
done by using IMPLEMENTS. The backup scenario from above might be implemented
IMPLEMENTS GIVEN some live data in a directory
rm -rf "$DATADIR/data"
echo foo > "$DATADIR/data/foo"
IMPLEMENTS GIVEN an empty backup repository
rm -rf "$DATADIR/repo"
IMPLEMENTS WHEN a backup is made
backup-program -r "$DATADIR/repo" "$DATADIR/data"
IMPLEMENTS THEN the data can be restored
restore-program -r "$DATADIR/repo" "$DATADIR/restored"
diff -rq "$DATADIR/data" "$DATADIR/restored"
IMPLEMENTS FINALLY cleanup
echo nothing to do, actually
Each "IMPLEMENTS GIVEN" (or WHEN, THEN, FINALLY) is followed by a
regular expression on the same line, and then a shell script that gets
executed to implement any step that matches the regular expression. The
implementation can extract data from the match as well: for example, the
regular expression might allow a file size to be specified.
The above example is a bit silly, of course: why go to the effort to obfuscate
the various steps? The answer is that the various steps, implemented using
IMPLEMENTS, can be combined in many ways, to test different aspects of the
program being tested.
Moreover, by making the step descriptions be human language text, matched by
regular expressions, most of the test can hopefully be written, and
understood, by non-programmers. Someone who understands what a program should
do, could write tests to verify its behaviour. The implementations of the
various steps need to be implemented by a programmer, but given a
well-designed set of steps, with enough flexibility in their implementation,
that quite a good test suite can be written.
The shell commands in an IMPLEMENTS section are run in the directory in which
the user ran yarn
. The environment variable SRCDIR
is set to the
fully qualified path to that directory.
- allow scenarios to reference steps that do not exist, by warning about
them, but otherwise ignoring the scenarios
- opposite of --allow-missing-steps
- change to DATADIR when running commands
- opposite of --cd-datadir
- add NAME=VALUE to the environment when tests are run
- fill in manual page TEMPLATE
- -h, --help
- show this help message and exit
- write output to FILE, instead of standard output
- -q, --quiet
- be quiet, avoid progress reporting, only show errors
- opposite of --quiet
- require ASSUMING to always pass
- opposite of --require-assumptions
- -r, --run=SCENARIO
- run only SCENARIO (this option can be repeated)
- run IMPLEMENTS using SHELL
- use ARG when running shell
- -s, --shell-library=SHELL-LIBRARY
- include a shell library for the IMPLEMENTS sections to use
- make snapshots of test working directory after each scenario step; you
probably want to use this with --tempdir
- opposite of --snapshot
- stop if any scenario step fails, don't run more scenarios
- opposite of --stop-on-first-fail
- use DIR as the temporary directory for tests; it should be empty or not
- report wall clock time for each scenario and step
- opposite of --timings
- -v, --verbose
- make progress reporting be more verbose ("wall of text"),
instead of a one-line status info; this is turned automatically if there
is not terminal
- opposite of --verbose
- show program's version number and exit
- -n, --no-act, --dry-run, --pretend
- do not actually run any tests, merely print what would be run
- --no-no-act, --no-dry-run, --no-pretend
- opposite of --no-act
- add FILE to config files
- write out the entire current configuration
- write out all names of settings and quit
- show all options
- list all possible config files
- clear list of configuration files to read
- write log entries to FILE (default is to not write log files at all); use
"syslog" to log to system log, "stderr" to log to the
standard error output, or "none" to disable logging
- keep last N logs (10)
- log at LEVEL, one of debug, info, warning, error, critical, fatal
- rotate logs larger than SIZE, zero for never (default: 0)
- set permissions of new log files to MODE (octal; default 0600)
- make memory profiling dumps using METHOD, which is one of: none, simple,
or meliae (default: simple)
- make memory profiling dumps at least SECONDS apart
- Fully qualified pathname to a temporary directory, in which the tests can
use files. The temporary directory is removed at the end of the test
execution, unless the user specifies otherwise with --snapshot.
- Fully qualitifed pathname to the directory in which the user ran
yarn. This is useful when the tests want to change the
To run yarn
on all the scenarios in your current directory:
All the files will be treated together as if they had been one file.
To add a shell library to be included when running any IMPLEMENTS section:
yarn --shell-library mylib.sh *.scenario
You can repeat --shell-library
as many times as necessary.
The README.yarn file has more details on the scenario testing language.