NAME

shmux - Shell Multiplexor

SYNOPSIS

shmux [ -h | -V ]

shmux [ -bBdFmpqQstv ] [ -C timeout ] [ -M max ] [ -r rcmd ] [ -S mode ] [ -e list ] [ -E list ] [ -a analyzer ] [ -A condition ] [ -o dir ] [ -P timeout ] [ -T timeout ] -c command [ - | targets... ]

DESCRIPTION

shmux is program for executing the same command on many hosts in parallel. For each target, a child process is spawned by shmux, and a shell on the target obtained one of the supported methods: rsh, ssh, or sh. The output produced by the children is received by shmux and either (optionally) output in turn to the user, and/or written to files for later processing. The list of targets may either be specified on the command line or be read from the standard input.

The default method may be set by using the -r option, and may be overridden for any target by prefixing the target name by the method and a colon. Two special methods, ssh1 and ssh2 may be used to select the SSH protocol version 1 or 2, respectively. For the sh method, it is up to the specified command to contact the target. In order for the executed command to know which target it is being invoked for, the SHMUX_TARGET environment variable is set by shmux.

Before executing the specified command, shmux will optionally ping each target to ensure that it can be reached, and/or run a dummy test command to make sure that the target not only is alive, but that it is possible to cleanly execute a command on it. Both these tests are typically run with a fairly short timeout to quickly dismiss unavailable targets rather than waiting for the standard (longer) network timeout.

After executing the specified command, shmux will optionally process the output or run a user specified command to determine whether the command was successful or resulted in an error. Optionally, the command output is suppressed for successful targets.

Finally shmux will print out a summary of the results and list targets with errors.

OPTIONS

-h

Display a brief help message.

-V

Display the version information.

-c command

Specify the command to execute on targets.

-C timeout

Specify a timeout for the command being executed on targets. This should be a number followed by a time unit. The following are valid time units: s(econds), m(inutes), h(our), d(ays), w(eeks). See the PROCESS MANAGEMENT section for details on how this is handled.

-M max

Defines the maximum number of spawned processes. While there is no real (or hard coded) limitation for this, the system resources are typically bounded and this affects shmux in turn. The most critical resource is the maximum number of open files a process may have. But one should also consider the load imposed on the system by the creation of a (potentially) large number of processes.

-r rcmd

Defines the default method used to run a shell on targets.

-S mode

This options defines the strategy used by shmux when spawning children to run the command on targets. If the mode is set to "all", then processes will be spawned as fast as possible. When set to "check", processes will be spawned as long as commands succeed. Finally, the default mode, "one", is to spawn only one process at first, and wait for it to succeed, at which point shmux will switch to the "check" mode. In the "one" or "check" modes, when the command fails on a target, shmux will stop spawning new processes.

-F

When using either the "one" or "check" spawn modes (see -S), shmux will stop spawning new processes if the command fails on a target. By default, shmux then waits for user input if available, but reverts to spawning processes if such input cannot be obtained (because the interactive mode is disabled, either via -B or lack of a terminal). Using the -F option changes this behaviour to force shmux to quit once all existing processes have terminated after such a failure.

-e list

Defines which command exit codes should be considered errors and reported as such. The list should be a comma separated list of ranges of the form "x", "x-y", "x-" or "-y" where "x" and "y" are numbers between 0 and 255.

-E list

Defines which command exit codes should always be output to the screen. Codes that are considered errors (see the -e option) are always shown, but by default, other exit codes are only shown if the -v option is specified. This allows more flexibility.

-a analyzer

Defines how output should be analyzed by shmux after the command completes on a target. By default, nothing is done. Valid options are: lnregex, lnpcre, regex, pcre and run. This option requires -o to be used as well.

-A condition

When the -a option is used, it is also necessary to configure the chosen analyzer.

For line based regular expressions (lnregex and lnpcre), the -A option should be specified twice: once for the standard output and once for the standard error output. In both cases, the argument should be the name of a file containing a list of regular expressions, one per line. Each regular expression must be prefixed by the character '=' or '!'. The output of the command is analyzed line per line by checking whether it matches any of the specified regular expressions. If the first matching regular expression is prefixed by '=', then the command is considered to be successful (so far). Any other result (e.g. no match, or a match prefixed by '!') indicates an error condition. The default is to consider standard output normal, and standard error output indicative of an error.

For other regular expressions (regex and pcre), the -A option should be specified twice: once for the standard output and once for the standard error output. This option allows defining the expression to be applied to the output. If the condition starts with the character '=' then a match indicates that the command completed without error. If the condition starts with the character '!' then a failure to match indicates that the command completed without error. The second character of the condition should either be '=' if the regular expression follows or '<' if the regular expression should be read from the file which name follows. (The '=' characters may be omitted.) A condition must be specified for the standard output, but is optional for standard error output. The default for the latter is to consider any output as indicative of an error.

For the run analyzer, the -A must be specified at least once with the name of a program to run, and optionally a second time to specify a timeout which defaults to 15 seconds if unset. This timeout is implemented using alarm(3) and will not otherwise be enforced by shmux. The program will be called after a command completes on a target unless the command exit code is considered an error (see -e) with two arguments: the target name and the directory specified with the -o option. The output of the program will always be shown to the user. Exit codes other than 0 indicate that the command failed for the particular target.

-o dir

If specified, shmux will place the output and (for normal terminations) the exit code of the executed commands in files under this directory. For each target, the files will be named after the target name so as to make it easy to identify. The files must not already exist, so it is recommended that the directory be empty. This also means that each target must be unique. The directory will be created if it does not already exist.

-p

Ping targets to verify they are alive before doing anything. The target names must be unique or bad things will happen.

-P timeout

Defines the initial target ping timeout in milliseconds, see fping(8). (Implies -p.)

-t

Before executing the specified command for a target, send a simple "echo" test and verify that the output is correct. This step goes beyond the simple ping test as it verifies that it is possible to get a shell on the target. If this option is specified twice (or once when -T is used), output for failed tests will be displayed to help the user understand what went wrong.

-T timeout

Defines the test timeout in seconds. (Implies -t.)

-m

By default the output is displayed as soon as it is received. For multi-line outputs, this will typically result in output from several targets being mixed. This option may be used to keep each target output separate.

-b

Show the bare output from the executed commands instead of prefixing each line by the corresponding target name.

-B

Enables batch mode which forcefully disables the interactive mode, ignoring input from the terminal. See also -s and -F.

-s

Suppress the progress status line. See also -B.

-q

Used once, this will suppress the output from successful targets (as defined by the use of options options -e, -a and -A. If using the run analyzer, the output of all targets is suppressed. Note that using this option effectively implies -m in most cases as failure is ultimately determined upon completion of the command. (Only the lnregex and lnpcre analyzers are able to determine errors while the command is running.) When specified twice, this option allows to completely suppress output from the command run on targets. This also changes the default spawn stategy to "all". Using this option (once or twice) requires using -o.

-Q

Suppress the final summary of results.

-v

Display internal status messages.

-D

Display internal debug messages.

EXIT CODES

shmux will optionally report the exit code of the command it runs for each target. For the sh command, this will be the same as the command itself, but this is the exception. For other methods, the exit code reported by shmux really is the code returned by the program used to spawn command for that method.

In particular, rsh typically only returns either 0 or 1 while different implementations of ssh have different behaviours which may include returning the actual exit code of the remotely executed command or not.

PROCESS MANAGEMENT

Children processes spawned by shmux are made process group leaders (using setpgid(2)) in order to group all of their descendants in a single process group which allows to easily send signals to all of these processes. This is especially useful for the sh method which can generate many processes locally, but has no effect on remote processes.

shmux uses pipe(2) to capture the output generated by its children (even when this output is redirected to a file with -o) and will read from these until they are closed (by the child as well as its descendants, if any). Once there is nothing left to read, shmux will wait for the child and its descendants to terminate before considering that particular target done with.

The timeout is implemented by using alarm(3) which (unless canceled or intercepted) will terminate the child process upon receipt but is not inherited by descendants of that process. For that reason, shmux will send a SIGALRM signal to the child's process group when the child itself terminates upon such signal.

After the scheduled SIGALRM time, shmux will wait an additional 5 seconds for all of the processes in the group to terminate to give the child a chance to handle the timeout more gracefully if desired. It will then send a SIGTERM signal to the child's process group. After another 5 seconds, if any process is still alive in the group, a final SIGKILL signal is sent to the process group.

Note that only commands which terminate because of an ALRM signal or because of a TERM or KILL signal after such signal has been issued by shmux will be reported as having timed out in the final summary report produced by shmux upon exiting. This is not as accurate as it should be.

INTERACTIVE MODE

By default, shmux offers a minimal "interactive mode" while running: it reads commands from the terminal and acts upon them accordingly. This can be disabled using the -B option. The following commands are currently recognized:

q: Prevent shmux from spawning any more children, wen for active children to complete, and then quit.
Q: Quit immediately (e.g. abort), not waiting for any active children to terminate. This isn't recommended as children may be left running unattended. A better alternative is to press control-C which will be received by shmux and all running children. These will typically abort, and shmux terminate in a cleaner fashion.
space: Initiate a pause, this prevents shmux from spawning any more children to do its work. The following three commands may be used to resume work.
1: This is equivalent to specifying "-S more" on the command line, but may be used at any time.
return: This is equivalent to specifying "-S check" on the command line, but may be used at any time.
+: This is equivalent to specifying "-S all" on the command line, but may be used at any time.
F: This toggles the failure mode for the "one" and "check" spawn modes (see -S) between "pause" and "quit" (see -F).
p: Display the list of pending targets.
r: Display the list of targets for which a children process is currently running.
f: Display the list of targets for which shmux failed to run the command.
e: Display the list of targets for which the command completed with error(s).
s: Display the list of targets for which the command completed successfully.
a: Display the list of all targets with their status.
k: Allows sending a signal to a target.
v: Toggles whether internal messages are displayed or not.
D: Toggles whether debug messages are displayed or not.

ENVIRONMENT VARIABLES

shmux will use the following environment variables if set:

SHMUX_MAX: Specifies the maximum number of processes to spawn simultaneously. If the -M option is specified, it overrides this variable.
SHMUX_RCMD: Specifies the default command used to run a shell on targets. If the -r option is specified, it overrides this variable.
SHMUX_SH: Specify the shell to use for targets using the sh method. (Default is "/bin/sh".)
SHMUX_RSH: Specify an alternative command to "rsh" to use for targets using the rsh method.
SHMUX_SSH, SHMUX_SSH1, SHMUX_SSH2: Specify an alternative command to "ssh" to use for targets using the ssh, ssh1 and ssh2 method (respectively). The variable SHMUX_SSH is used for all methods, but superseded by the other specific variables.
SHMUX_SSH_OPTS, SHMUX_SSH1_OPTS, SHMUX_SSH2_OPTS: The variables may be used to specify replace the default "-xa -oLogLevel=ERROR" options passed to "ssh", "ssh1" and "ssh2" (respectively). The variable SHMUX_SSH_OPTS is used for all methods, but superseded by the other specific variables.
Multiple arguments may be supplied, using white-space characters to separate them from each other. You can use single- and/or double-quotes to preserve white-spaces in an argument (but not backslashes). Note that doubling a single-quote inside a single-quoted string gives you a single-quote; likewise for double-quotes (though you need to pay attention to which quotes your shell is parsing and which quotes shmux is parsing).
SHMUX_SPAWNMODE: Specifies the default spawn mode. If the -S option is specified, it overrides this variable.
SHMUX_ERRORCODES: Specifies the default list of exit codes to be considered errors. If the -e option is specified, it overrides this variable.
SHMUX_SHOWCODES: Specifies the default list of exit codes to be always display. If the -E option is specified, it overrides this variable.

EXIT STATUS

The following exit values are returned:

0: indicates that shmux was able to complete its work. However, this does not mean that all targets successfully ran the command, nor that shmux ran without any error.
1: indicates shmux was cleanly interrupted before it could finish its work.
2: indicates shmux aborted upon user request, possibly leaving some children running.
3: indicates shmux encountered a fatal error which caused it to abort, possibly leaving some children running.
4: indicates a syntax error or a fatal error which caused shmux to abort before it could spawn any children.

AVAILABILITY

The latest official release of shmux is available on GitHub at http://github.com/shmux.

AUTHOR

Christophe Kalt <ck@taranis.org>

BUGS

All rsh and some ssh implementations effectively achieve user authentication by using privileged ports. Since there are only 1024 such ports, there is an obvious limitation on how many connections may be open using these programs. To make matters worse such ports may be held for four minutes (by default) as per the TCP specification. When using shmux, one can very quickly run out of privileged ports, at which point failures will start happening.

All rsh and ssh implementations fail to accurately return the remote command exit code to the caller. Typically, rsh will simply not bother to return this exit code, and ssh will try its best but is limited by the fact it also tries to return an error code for SSH related errors. In either case, this may pose problems when using -e.

Under Cygwin, child processes do not inherit alarm(3) settings from the parent disabling this simple timeout enforcement system used by shmux.

Some shells will ignore SIGALRM, others die upon its receipt (regardless of any trap). Again, this will affect the simple timeout enforcement system used by shmux.

For bug reports, create an issue at https://github.com/shmux/shmux/issues, or send an email to shmux-bugs@taranis.org.