NAME

rwaggbagtool - Manipulate binary Aggregate Bag files

SYNOPSIS

  rwaggbagtool [{ --add | --subtract }]
        [--insert-field=FIELD=VALUE [--insert-field=FIELD2=VALUE2...]]
        [{ --remove-fields=REMOVE_LIST | --select-fields=SELECT_LIST
           | --to-ipset=FIELD [--ipset-record-version=VERSION]
           | --to-bag=BAG_KEY,BAG_COUNTER }]
        [--min-field=FIELD=VALUE [--min-field=FIELD=VALUE...]]
        [--max-field=FIELD=VALUE [--max-field=FIELD=VALUE...]]
        [--set-intersect=FIELD=FILE [--set-intersect=FIELD=FILE...]]
        [--set-complement=FIELD=FILE [--set-complement=FIELD=FILE...]]
        [--output-path=PATH]
        [--note-strip] [--note-add=TEXT] [--note-file-add=FILE]
        [--compression-method=COMP_METHOD]
        [--site-config-file=FILENAME]
        [AGGBAG_FILE [AGGBAG_FILE ...]]

  rwaggbagtool --help

  rwaggbagtool --version

DESCRIPTION

rwaggbagtool performs operations on one or more Aggregate Bag files and creates a new Aggregate Bag file. An Aggregate Bag is a binary file that maps a key to a counter, where the key and the counter are both composed of one or more fields. rwaggbag(1) and rwaggbagbuild(1) are the primary tools used to create an Aggregate Bag file. rwaggbagcat(1) prints a binary Aggregate Bag file as text.

rwaggbagtool processes the Aggregate Bag files listed on the command line. When no file names are specified, rwaggbagtool attempts to read an Aggregate Bag from the standard input. To read the standard input in addition to the named files, use "-" or "stdin" as a file name. If any input is not an Aggregate Bag file, rwaggbagtool prints an error to the standard error and exits with an error status.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--add: Sum each of the counters for each key for all the Aggregate Bag input files. All the Aggregate Bag files must have the same set of key fields and counter fields. (The values of the keys may differ, but the set of fields that comprise the key must match.) If no other operation is specified, the add operation is the default.
--subtract: Subtract from the first Aggregate Bag file all subsequent Aggregate Bag files. All the Aggregate Bag files must have the same set of key fields and counter fields. If a key does not appear in the first Aggregate Bag file, rwaggbagtool assumes it has a value of 0. If any counter subtraction results in a negative number, the key will not appear in the resulting Aggregate Bag file.

Field manipulation switches

The following switches allow modification of the fields in the Aggregate Bag file. The --remove-fields and --select-fields switches are mutually exclusive, and they reduce the number of fields in the Aggregate Bag input files. The --insert-field switch is applied after --remove-fields or --select-fields, and it adds a field unless that field is already present.

--insert-field=FIELD=VALUE: For each entry read from an Aggregate Bag input file, insert a field named FIELD and set its value to VALUE if one of the following is true: (1)the input file does not contain a field named FIELD or (2)the input file does have a field named FIELD but it was removed by either (2a)being listed in the --remove-fields list or (2b)not being listed in the --select-fields list. That is, this switch only inserts FIELD when FIELD is not present in the input Aggregate Bag, but specifying FIELD in --remove-fields removes it from the input. VALUE is a textual representation of the field's value as described in the description of the --fields switch in the rwaggbagbuild(1) tool. This switch may be repeated in order to insert multiple fields.
--remove-fields=REMOVE_LIST: Remove the fields specified in REMOVE_LIST from each of the Aggregate Bag input files, where REMOVE_LIST is a comma-separated list of field names. This switch may include field names that are not in an Aggregate Bag input, and those field names are ignored. If a field name is included in this list and in a --insert-field switch, the field is given the value specified by the --insert-field switch, and the field is included in the output Aggregate Bag file. If removing a key field produces multiple copies of a key, the counters of those keys are merged. rwaggbagbuild exits with an error when this switch is used with --select-fields, --to-ipset, or --to-bag.
--select-fields=SELECT_LIST: For each Aggregate Bag input file, only use the fields in SELECT_LIST, a comma-separated list of field names. Alternatively, consider this switch as removing all fields that are not included in SELECT_LIST. This switch may include field names that are not in an Aggregate Bag input, and those field names are ignored. When a field name is included in this list and in a --insert-field switch, the field uses its value from the input Aggregate Bag file if present, and it uses the value specified in the --insert-field switch otherwise. If selecting only some key fields produces multiple copies of a key, the counters of those keys are merged. rwaggbagbuild exits with an error when this switch is used with --remove-fields, --to-ipset, or --to-bag.

Filtering switches

The following switches remove entries from the Aggregate Bag file based on a field's value. These switches are applied immediately before the output is generated.

--min-field=FIELD=VALUE: Remove from the Aggregate Bag file all entries where the value of the field FIELD is less than VALUE, where VALUE is a textual representation of the field's value as described in the description of the --fields switch in the rwaggbagbuild (1) tool. This switch is ignored if FIELD is not present in the Aggregate Bag. This switch may be repeated. Since SiLK 3.17.0.
--max-field=FIELD=VALUE: Remove from the Aggregate Bag file all entries where the value of the field FIELD is greater than VALUE, where VALUE is a textual representation of the field's value as described in the description of the --fields switch in the rwaggbagbuild (1) tool. This switch is ignored if FIELD is not present in the Aggregate Bag. This switch may be repeated. Since SiLK 3.17.0.
--set-intersect=FIELD=SET_FILE: Read an IPset from the stream SET_FILE, and remove from the Aggregate Bag file all entries where the value of the field FIELD is not present in the IPset. SET_FILE may be the name a file or the string "-" or "stdin" to read the IPset from the standard input. This switch is ignored if FIELD is not present in the Aggregate Bag. This switch may be repeated. Since SiLK 3.17.0.
--set-complement=FIELD=SET_FILE: Read an IPset from the stream SET_FILE, and remove from the Aggregate Bag file all entries where the value of the field FIELD is present in the IPset. SET_FILE may be the name a file or the string "-" or "stdin" to read the IPset from the standard input. This switch is ignored if FIELD is not present in the Aggregate Bag. This switch may be repeated. Since SiLK 3.17.0.

Output switches

The following switches control the output.

--to-ipset=FIELD: After operating on the Aggregate Bag input files, create an IPset file from the resulting Aggregate Bag by treating the values in the field named FIELD as IP addresses, inserting the IP addresses into the IPset, and writing the IPset to the standard output or the destination specified by --output-path. When this switch is used, the only legal field name that may be used in the --insert-field switch is FIELD. rwaggbagbuild exits with an error when this switch is used with --remove-fields, --select-fields, or --to-bag.
--ipset-record-version=VERSION: Specify the format of the IPset records that are written to the output when the --to-ipset switch is used. VERSION may be 2, 3, 4, 5 or the special value 0. When the switch is not provided, the SILK_IPSET_RECORD_VERSION environment variable is checked for a version. The default version is 0.

0: Use the default version for an IPv4 IPset and an IPv6 IPset. Use the --help switch to see the versions used for your SiLK installation.
2: Create a file that may hold only IPv4 addresses and is readable by all versions of SiLK.
3: Create a file that may hold IPv4 or IPv6 addresses and is readable by SiLK 3.0 and later.
4: Create a file that may hold IPv4 or IPv6 addresses and is readable by SiLK 3.7 and later. These files are more compact that version 3 and often more compact than version 2.
5: Create a file that may hold only IPv6 addresses and is readable by SiLK 3.14 and later. When this version is specified, IPsets containing only IPv4 addresses are written in version 4. These files are usually more compact that version 4.

--to-bag=BAG_KEY,BAG_COUNTER: After operating on the Aggregate Bag input files, create a (normal) Bag file from the resulting Aggregate Bag. Use the BAG_KEY field as the key of the Bag, and the BAG_COUNTER field as the counter of the Bag. Write the Bag to the standard output or the destination specified by --output-path. When this switch is used, the only legal field names that may be used in the --insert-field switch are BAG_KEY and BAG_COUNTER. rwaggbagbuild exits with an error when this switch is used with --remove-fields, --select-fields, or --to-ipset.
--output-path=PATH: Write the resulting Aggregate Bag, IPset (see --to-ipset), or Bag (see --to-bag) to PATH, where PATH is a filename, a named pipe, the keyword "stderr" to write the output to the standard error, or the keyword "stdout" or "-" to write the output to the standard output. If PATH names an existing file, rwaggbagtool exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this switch is not given, the output is written to the standard output. Attempting to write the binary output to a terminal causes rwaggbagtool to exit with an error.
--note-strip: Do not copy the notes (annotations) from the input files to the output file. Normally notes from the input files are copied to the output.
--note-add=TEXT: Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.
--note-file-add=FILENAME: Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.
--compression-method=COMP_METHOD: Specify the compression library to use when writing output files. If this switch is not given, the value in the SILK_COMPRESSION_METHOD environment variable is used if the value names an available compression method. When no compression method is specified, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

none: Do not compress the output using an external library.
zlib: Use the zlib(3) library for compressing the output, and always compress the output regardless of the destination. Using zlib produces the smallest output files at the cost of speed.
lzo1x: Use the lzo1x algorithm from the LZO real time compression library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead.
snappy: Use the snappy library for compression, and always compress the output regardless of the destination. This compression provides good compression with less memory and CPU overhead.
best: Use lzo1x if available, otherwise use snappy if available, otherwise use zlib if available. Only compress the output when writing to a file.

Miscellaneous switches

--site-config-file=FILENAME: Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwaggbagtool searches for the site configuration file in the locations specified in the "FILES" section.
--help: Print the available options and exit.
--version: Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line.

To create two Aggregate Bag files, in.aggbag and inweb.aggbag, and then add the counters to create total.aggbag:

 $ rwfilter --type=in --pass=-                              \
   | rwaggbag --key=sport,dport,proto --counter=records     \
        --output-path=in.aggbag
 $ rwfilter --type=inweb --pass=-                           \
   | rwaggbag --key=sport,dport,proto --counter=records     \
        --output-path=inweb.aggbag
 $ rwaggbagtool --add in.aggbag inweb.aggbag --output-path=total.aggbag
 $ rwaggbagcat total.aggbag

To subtract inweb.aggbag from total.aggbag:

 $ rwaggbagtool --subtract total.aggbag inweb.aggbag    \
   | rwaggbagcat

Create an Aggregate Bag file:

 $ rwaggbag --key=sport,dport                       \
        --counter=sum-bytes,sum-packets data.rw     \
        --output-path=my-ab.aggbag

To get just the source port and byte count from the file my-ab.aggbag, you may either remove the destination port and packet count:

 $ rwaggbagtool --remove=dport,sum-packets my-ab.aggbag  \
        --output-path=source-bytes.aggbag

or you may select the source port and byte count:

 $ rwaggbagtool --select=sport,sum-bytes my-ag.aggbag    \
        --output-path=source-bytes.aggbag

To replace the packet count in my-ab.aggbag with zeros, remove the field and insert it with the value you want:

 $ rwaggbagtool --remove=sum-packets --insert=sum-packets=0  \
        my-ab.aggbag --output-path=zero-packets.aggbag

To create a regular Bag with the source port and byte count from my-ab.aggbag, use the --to-bag switch:

 $ rwaggbagtool --to-bag=sport,sum-bytes my-ab.aggbag  \
        --output-path=sport-byte.bag

The --to-ipset switch works similarly:

 $ rwaggbag --key=sipv6,dipv6 --counter=records data-v6.rw  \
        --output-path=ips.aggbag
 $ rwaggbagtool --to-ipset=dipv6 --output-path=dip.set

ENVIRONMENT

SILK_IPSET_RECORD_VERSION: This environment variable is used as the value for the --ipset-record-version when that switch is not provided.
SILK_CLOBBER: The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.
SILK_COMPRESSION_METHOD: This environment variable is used as the value for --compression-method when that switch is not provided.
SILK_CONFIG_FILE: This environment variable is used as the value for the --site-config-file when that switch is not provided.
SILK_DATA_ROOTDIR: This environment variable specifies the root directory of data repository. As described in the "FILES" section, rwaggbagtool may use this environment variable when searching for the SiLK site configuration file.
SILK_PATH: This environment variable gives the root of the install tree. When searching for configuration files, rwaggbagtool may use this environment variable. See the "FILES" section for details.

FILES

${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/data/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/local/share/silk/silk.conf
/usr/local/share/silk.conf: Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided.