Input Modes

"stream"

In "stream" input-mode, rwflowpack processes the probe statements in the sensor.conf(5) file which specify how rwflowpack is to capture data from one or more sources. The data is assigned to a sensor based on the probe-sensor mapping in the sensor.conf file. rwflowpack then categorizes the records, converts them to the SiLK Flow format, and writes them to files.

The sources of data that rwflowpack supports are:

"fcfiles"

Instead of having rwflowpack capture data itself, you may configure rwflowpack to work in conjunction with one or more flowcap(8) daemons by specifying "fcfiles" as the input-mode.

In this configuration, each flowcap uses the probe statements in the sensor.conf(5) file to determine how to collect the data. flowcap supports the network-based capture methods specified for the "stream" input-mode---i.e., flowcap does not support directory polling. flowcap writes the data it captures into small files and includes the probe name in each file's header.

Typically, the flowcap processes run on separate machines near the router or flow meter that is generating the records. The rwsender(8) and rwreceiver(8) daemons are often used to move the files from the flowcap machines to the rwflowpack machine.

rwflowpack polls a directory for the files created by flowcap. Once it finds a file, it assigns those records a sensor based on the probe-sensor mapping in the sensor.conf file, it categorizes the records, and it writes the records to one or more output files.

Since rwflowpack does not capture data in "fcfiles" input-mode, rwflowpack does not use the probe statements in the sensor.conf file, and the statements do not need to be present.

The switches that rwflowpack uses in "fcfiles" input-mode are given below ("Flowcap Files Collection Switches (--input-mode=fcfiles)").

"pdufile"

Setting the input-mode to "pdufile" tells rwflowpack to read a single file containing NetFlow v5 PDU records and then exit. rwflowpack does not become a daemon in this input-mode; instead it remains in the foreground, processes the NetFlow file, and exits.

The file must be in the format created by NetFlow Collector: The file's size must be an integer multiple of 1464, where each 1464 byte chunk contains a 24-byte NetFlow v5 header and space for thirty 48-byte NetFlow records. The number of valid records per chunk is specified in the chunk's header. (This is also the format that rwflowpack requires in "stream" input-mode when it is polling a directory for NetFlow v5 files.)

To convert single PDU file to a stream of SiLK Flow records, consider using rwpdu2silk(1).

In "pdufile" input-mode, the sensor.conf file must define a sensor that maps to a probe that uses the read-from-file statement. However, the argument to that statement is ignored, and the argument is typically set to /dev/null. The NetFlow v5 file's location must be specified with the --netflow-file switch. If sensor.conf contains multiple sensor blocks, the --sensor-name switch is required to tell rwflowpack which sensor to use.

See the "PDU File Collection Switches (--input-mode=pdufile)" section below for additional configuration information.

"respool"

Sometimes it is desirable to pull existing SiLK Flow records from one data repository and use them to create a "mini" data repository (for example, for testing). The "respool" input-mode is one way to accomplish this.

In this configuration, rwflowpack polls a directory for SiLK flow files and it uses the existing class/type pair and sensor values to determine where to store the flow record. That is, rwflowpack puts the data into appropriate hourly file, but it does not change any other settings on the flow records.

To contrast "respool" input-mode with rwflowpack processing SiLK Flow files in "stream" input-mode: In "respool" input-mode, the existing class, type, and sensor values are used to determine each record's destination. In the latter, rwflowpack treats the records as it would any other newly arrived data, assigning the data to a sensor and re-categorizing the records to assign a class/type pair to them.

Since no categorization occurs in "respool" input-mode, the --sensor-configuration and --packing-logic switches are not required and not allowed, and their presence causes rwflowpack to exit with an error code.

Output Modes

In order for the record in that file to be retrievable by rwfilter(1), the file must be stored in a SiLK data repository, which is a directory tree of binary SiLK Flow files. The root of this directory tree is called the SILK_DATA_ROOTDIR. The structure of the tree under the root is determined by the path-format setting in the silk.conf(5) file.

There are two ways to get the files into the SILK_DATA_ROOTDIR; which method is used is determined by rwflowpack's --output-mode switch. This switch supports the following values:

"local-storage"

In "local-storage" output-mode, rwflowpack creates the hourly SiLK Flow files directly in the data repository, and it writes the records into these files. rwflowpack uses "local-storage" output-mode when the --output-mode switch is not provided.

"incremental-files"

When the output-mode is "incremental-files", rwflowpack does not create hourly data files directly. Instead, rwflowpack creates smaller files (called incremental files), and rwflowpack relies on the rwflowappend(8) daemon to combine the incremental files into hourly files in the final data repository.

In "incremental-files" output-mode, rwflowpack uses a single destination directory whose location is specified by the --incremental-directory switch. In this directory, rwflowpack creates a zero-byte place holder file and a corresponding working file. The name of the place holder file has a unique, random suffix, and the name of the working file is a dot (".") followed by the name of the place holder file.

rwflowpack first writes records to the working files. Periodically (as determined by the value specified to --flush-timeout), rwflowpack closes all the working files and moves them over the place holder files. Once rwflowpack has closed and moved a working file, it no longer takes responsibility for it.

The rwflowappend process may poll the incremental-directory itself, or another process may poll that directory and pass the files to rwflowappend. If rwflowpack and rwflowappend are on different machines, an rwsender(8)/rwreceiver(8) pair may be used to move the files between the machines.

Additional reasons one may want to use "incremental-files" output-mode are to support having multiple data repositories or to allow additional processing of the SiLK Flow records (such as by the Analysis Pipeline (<http://tools.netsa.cert.org/analysis-pipeline/>)).

Note: This output-mode was introduced in SiLK 3.6.0. This mode is similar to the "sending" output-mode and is meant to replace it. In "incremental-files" output-mode, all writing occurs within the incremental-directory, while the "sending" output-mode uses two directories.

"sending"

This output-mode works similarly to "incremental-files". The difference is that when rwflowpack flushes the open files, it moves the files from the incremental-directory and to the directory specified by the --sender-directory switch. Once a file is moved to the sender-directory, rwflowpack no longer takes responsibility for it.

As of SiLK 3.6.0, the "sending" output-mode is deprecated.

Input and Output Mode

--input-mode=MODE

Determine how rwflowpack gathers data. The default input MODE is "stream". The available input-modes are "stream", "fcfiles", "pdufile", and "respool".

--output-mode=MODE

Determine what rwflowpack does with the data as it is packed into SiLK binary files. The default output MODE is "local-storage". The available output-modes are "local-storage", "incremental-files" and "sending".

Stream Collection Switches (--input-mode=stream)

--sensor-configuration=FILE_PATH

Give the path to the configuration file that specifies how rwflowpack is to capture data and that tells rwflowpack whether a record represents an incoming or outgoing flow. The complete syntax of the configuration file is described in the sensor.conf (5) manual page; see also the SiLK Installation Handbook.

--packing-logic=PLUGIN

Specify the plug-in that rwflowpack should load, where the plug-in provides functions that determine into which class and type each flow record will be categorized and the format of the files that rwflowpack will write. When SiLK has been configured with hard-coded packing logic (i.e., when --enable-packing-logic was specified to the configure script), this switch will not be present on rwflowpack. A default value for this switch may be specified in the silk.conf(5) site configuration file (see the description of the --site-config-file switch). When PLUGIN does not contain a slash ("/"), rwflowpack attempts to find a file named PLUGIN in the directories listed in the "FILES" section. If rwflowpack finds the file, it uses that path. If PLUGIN contains a slash or if rwflowpack does not find the file, rwflowpack relies on your operating system's dlopen(3) call to find the file. When the SILK_PLUGIN_DEBUG environment variable is non-empty, rwflowpack prints status messages to the standard error as it attempts to find and open each of its plug-ins. rwflowpack exits if it does not have access to the packing logic functions.

--sensor-name=SENSOR

Cause rwflowpack to ignore all probes in the sensor configuration file except the probes for SENSOR. Only data for SENSOR is collected. This allows a common configuration file to be used by multiple rwflowpack invocations, yet also allow each rwflowpack instance only collect data for a single sensor. There must be a sensor definition for SENSOR in the configuration file. When this switch is not present, rwflowpack collects and packs data for all sensors.

--polling-interval=NUMBER

Specify the number of seconds rwflowpack waits between scans of the directories specified by the "poll-directory" settings of the probes in the sensor.conf file. The default is 15 seconds.

Flowcap Files Collection Switches (--input-mode=fcfiles)

--sensor-configuration=FILE_PATH

Give the path to the configuration file that rwflowpack consults to determine whether a record represents an incoming or outgoing flow. The complete syntax of the configuration file is described in the sensor.conf(5) manual page; see also the SiLK Installation Handbook.

--incoming-directory=DIR_PATH

Periodically scan the directory DIR_PATH for files that have been created by flowcap. As rwflowpack scans DIR_PATH, it ignores a file if its name begins with a dot (".") or if its size is 0 bytes. When a file is first detected, rwflowpack records its size, and the file must have the same size for two consecutive scans before rwflowpack processes it. After the file is successfully processed, rwflowpack either moves it to the archive-directory or deletes it. The interval between scans is set by --polling-interval. DIR_PATH must be a complete directory path.

--packing-logic=PLUGIN

Specify the plug-in that rwflowpack should load for the packing logic. For more detail, see the description above.

--polling-interval=NUMBER

Specify the number of seconds rwflowpack waits between polls of the incoming-directory for new files created by flowcap. If not given, the default value is 15 seconds.

PDU File Collection Switches (--input-mode=pdufile)

--sensor-configuration=FILE_PATH

Give the path to the configuration file that rwflowpack consults to determine whether a record represents an incoming or outgoing flow.

--netflow-file=FILE_PATH

Name the full path of the file from which rwflowpack reads NetFlow v5 PDUs. This switch is required in PDU File input-mode.

--sensor-name=SENSOR

Cause rwflowpack to ignore all probes in the sensor configuration file except the probes for SENSOR. There must be a sensor definition for SENSOR in the configuration file. This switch is required in this input-mode unless the sensor.conf file only defines a single sensor.

--packing-logic=PLUGIN

Specify the plug-in that rwflowpack should load for the packing logic. For more detail, see the description of this switch in the "stream" input-mode.

Respooling Switches (--input-mode=respool)

--incoming-directory=DIR_PATH

Periodically scan the directory DIR_PATH for SiLK Flow files to process. As rwflowpack scans DIR_PATH, it ignores a file if its name begins with a dot (".") or if its size is 0 bytes. When a file is first detected, rwflowpack records its size, and the file must have the same size for two consecutive scans before rwflowpack will process it. After the file is successfully processed, rwflowpack either moves it to the archive-directory or deletes it. The interval between scans is set by --polling-interval. DIR_PATH must be a complete directory path.

--polling-interval=NUMBER

Specify the number of seconds rwflowpack waits between polls of the incoming-directory. If not given, the default value is 15 seconds.

Local Storage Switches (--output-mode=local-storage)

--root-directory=DIR_PATH

Name the full path of the SILK_DATA_ROOTDIR; that is, the directory under which the files containing the packed SiLK Flow records are stored. rwflowpack creates subdirectories below DIR_PATH based on the data received and the path-format setting in the silk.conf(5) file.

Incremental-Files Switches (--output-mode=incremental-files)

--incremental-directory=DIR_PATH

Name the full path of the destination directory where incremental-files are both created and stored to await action by another process such as rwflowappend or rwsender. It is recommended that only a single rwflowpack process write to this directory.

Sending Mode Switches (--output-mode=sending)

--incremental-directory=DIR_PATH

Name the full path of the working directory under which incremental-files are initially created. Periodically (as determined by the --flush-timeout switch), rwflowpack closes the files in this directory and moves them to the sender-directory. An rwflowpack invocation assumes it has full control over the files in this directory. When rwflowpack starts, any files in this directory are moved to the sender-directory.

--sender-directory=DIR_PATH

Name the full path of the destination directory where incremental-files are moved to await action by another process such as rwflowappend or rwsender. Once rwflowpack moves files to this directory, it no longer takes responsibility for them. The other process (e.g., rwsender) is responsible for removing files from this directory. Multiple rwflowpack invocations may use a single sender-directory.

General Configuration

--no-file-locking

Do not use advisory write locks. Normally, rwflowpack obtains a write lock on an data file prior to writing records to it. The write lock prevents two instances of rwflowpack from writing to the same data file simultaneously. However, attempting to use a write lock on some file systems causes rwflowpack to exit with an error, and this switch may be used when writing data to these file systems.

--flush-timeout=VAL

Set the timeout for flushing any in-memory records to disk to VAL seconds. If not specified, the default is 2 minutes (120 seconds). When the output-mode is "local-storage", this value specifies how often the files are flushed to disk to ensure that any records in memory are written to disk. When using the "incremental-files" or "sending" output-mode, this value specifies how often to close and move the incremental files. See the "Output Modes" section for details.

--file-cache-size=VAL

Set the maximum number of data files to have open for writing at any one time to VAL. If not specified, the default is 128 files. The minimum file cache size is 4. This switch also determines how many files rwflowpack reads from simultaneously when using probes that poll directories for files (see sensor.conf(5)). The maximum number of input files open at any one time is limited to one eighth of VAL (with a minimum of 2), and the number of directory polling operations to perform simultaneously is limited to one sixteenth of VAL (minimum is 1).

--pack-interfaces

Allow one to override the default file output formats of the packed SiLK Flow files that rwflowpack writes. When this switch is present, rwflowpack writes additional information into the packed files: the router's SNMP input and output interfaces and the next-hop IP address. (When the sensor.conf file contains an "interface-values" attribute whose value is "vlan", the input and output fields contain the vlan IDs instead of SNMP interface values.) The extra data produced by this switch is useful for determining why traffic is being stored in certain files. Note that this switch only affects newly created files. New records are always appended to an existing file in the file's current output format to maintain file integrity.

--byte-order=ENDIAN

Set the byte order for newly created SiLK Flow files. When appending records to an existing file, the byte order of the file is maintained. The argument is one of the following:

--compression-method=COMP_METHOD

Specify the compression library to use when creating new files. When this switch is not given, newly created files are compressed using the default chosen when SiLK was compiled. When appending records to an existing file, the compression method of the file is maintained. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

--site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwflowpack searches for the site configuration file in the locations specified in the "FILES" section.

Disposal of Input Flow Files

--error-directory=DIR_PATH

Move input files that cannot be opened, have an unexpected format, contain an unrecognized probe name in "fcfiles" input-mode, or are not successfully processed into the directory DIR_PATH. DIR_PATH must be a complete directory path. If this switch is not provided, problem files remain in place and cause rwflowpack to exit.

--archive-directory=DIR_PATH

Move input files that rwflowpack processes successfully into the directory DIR_PATH. DIR_PATH must be a complete directory path. When this switch is not provided and the input-mode is "pdufile", the original NetFlow source file is not modified, moved, or deleted. In all other input-modes, no --archive-directory switch causes rwflowpack to delete each input file after successfully processing it. When the --flat-archive switch is also provided, incoming files are moved into the top of DIR_PATH; when --flat-archive is not given, each file is moved to a subdirectory based on the current UTC time: DIR_PATH/YEAR/MONTH/DAY/HOUR/. Removing files from the archive-directory is not the job of rwflowpack; the system administrator should implement a separate process to clean this directory. This switch is required when the --post-archive-command switch is present.

--flat-archive

When archiving input files via the --archive-directory switch, move the files into the top of the archive-directory, not into subdirectories of the archive-directory. This switch has no effect if --archive-directory is not also specified. This switch may be used to allow another process to watch for new files appearing in the archive-directory.

--post-archive-command=COMMAND

Run COMMAND on each input file after rwflowpack has successfully processed the file and moved the file into the archive-directory. Each occurrence of the string %s in COMMAND is replaced with the full path to the input file in the archive-directory, and each occurrence of "%%" is replaced with "%". If any other character follows "%", rwflowpack exits with an error. When using this feature, the --archive-directory switch must be specified. See also the rwpollexec(8) daemon.

Logging and Daemon Configuration

--log-destination=DESTINATION

Specify the destination where logging messages are written. When DESTINATION begins with a slash "/", it is treated as a file system path and all log messages are written to that file; there is no log rotation. When DESTINATION does not begin with "/", it must be one of the following strings:

--log-directory=DIR_PATH

Use DIR_PATH as the directory where the log files are written. DIR_PATH must be a complete directory path. The log files have the form

 DIR_PATH/LOG_BASENAME-YYYYMMDD.log
    

where YYYYMMDD is the current date and LOG_BASENAME is the application name or the value passed to the --log-basename switch when provided. The log files are rotated: At midnight local time, a new log is opened, the previous file is closed, and the command specified by --log-post-rotate is invoked on the previous day's log file. (Old log files are not removed by rwflowpack; the administrator should use another tool to remove them.) When this switch is provided, a process-ID file (PID) is also written in this directory unless the --pidfile switch is provided.

--log-pathname=FILE_PATH

Use FILE_PATH as the complete path to the log file. The log file is not rotated.

The following set of switches is optional:

--log-level=LEVEL

Set the severity of messages that are logged. The levels from most severe to least are: "emerg", "alert", "crit", "err", "warning", "notice", "info", "debug". The default is "info".

--log-sysfacility=NUMBER

Set the facility that syslog(3) uses for logging messages. This switch takes a number as an argument. The default is a value that corresponds to "LOG_USER" on the system where rwflowpack is running. This switch produces an error unless --log-destination=syslog is specified.

--log-basename=LOG_BASENAME

Use LOG_BASENAME in place of the application name in the name of log files in the log directory. See the description of the --log-directory switch. This switch does not affect the name of the process-ID file.

--log-post-rotate=COMMAND

Run COMMAND on the previous day's log file after log rotation. When this switch is not specified, the previous day's log file is compressed with gzip(1). When the switch is specified and COMMAND is the empty string, no action is taken on the log file. Each occurrence of the string %s in COMMAND is replaced with the full path to the log file, and each occurrence of "%%" is replaced with "%". If any other character follows "%", rwflowpack exits with an error. Specifying this switch without also using --log-directory is an error.

--pidfile=FILE_PATH

Set the complete path to the file in which rwflowpack writes its process ID (PID) when it is running as a daemon. No PID file is written when --no-daemon is given. When this switch is not present, no PID file is written unless the --log-directory switch is specified, in which case the PID is written to LOGPATH/rwflowpack.pid.

--no-chdir

Do not change directory to the root directory. When rwflowpack becomes a daemon process, it changes its current directory to the root directory so as to avoid potentially running on a mounted file system. Specifying --no-chdir prevents this behavior, which may be useful during debugging. The application does not change its directory when --no-daemon is given.

--no-daemon

Force rwflowpack to run in the foreground---it does not become a daemon process. This may be useful during debugging.

Help Options

--verify-sensor-config

--verify-sensor-config=VERBOSE

Verify that the syntax of the sensor configuration file is correct and then exit rwflowpack. If the file is incorrect or if it does not define any sensors, an error message is printed and rwflowpack exits abnormally. If the file is correct and no argument is provided to the --verify-sensor-config switch, rwflowpack simply exits with status 0. If an argument (other than the empty string and 0) is provided to the switch, the names of the probes and sensors found in the sensor configuration file are printed to the standard output, and then rwflowpack exits.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.