Operation switches

--add

Sum the counters for each key for all Bag files given on the command line. At least one Bag file must be specified, and any number of additional Bag files may be given. If a key is not present in an input file, a counter of zero is used. When no operation switch is specified on the command line, the add operation is the default.

--subtract

Subtract from the first Bag file all subsequent Bag files. At least one Bag file must be specified, and any number of additional Bag files may be given. If a key does not appear in the first Bag file, rwbagtool assumes it has a value of 0. If subtracting a key's counters results in a non-positive number, the key does appear in the resulting Bag file.

--minimize

Cause the output to contain the minimum counter seen for each key. Keys that do not appear in all input Bags do not appear in the output. At least one Bag file must be specified, and any number of additional Bag files may be given.

--maximize

Cause the output to contain the maximum counter seen for each key. The output contains each key that appears in any input Bag. At least one Bag file must be specified, and any number of additional Bag files may be given.

--divide

Divide the first Bag file by the second Bag file. It is an error if only one Bag file or more than two Bag files are given. Every key in the first Bag file must appear in the second file; the second Bag may have keys that do not appear in the first, and those keys do not appear in the output. Since Bags do not support floating point numbers, the result of the division is rounded to the nearest integer (values ending in .5 are rounded up). If the result of the division is less than 0.5, the key does not appear in the output.

--scalar-multiply=VALUE

Multiply each counter in the Bag file by the scalar VALUE, where VALUE is an integer in the range 1 to 18446744073709551615. This switch requires a single Bag as input.

--compare=OPERATION

Compare the key/counter pairs in exactly two Bag files. It is an error if only one Bag file or more than two Bag files are specified. The keys in the output Bag are only those for which the comparison denoted by OPERATION is true when comparing the key's counter in the first Bag with the key's counter in the second Bag. The counters for all keys in the output have the value 1. Any key that does not appear in both input Bag files does not appear in the result. The possible OPERATION values are the strings:

Masking/Limiting switches

--intersect=SETFILE

Mask the keys in the intermediate Bag using the set in SETFILE. SETFILE is the name of a file or a named pipe containing an IPset, or the name "stdin" or "-" to have rwbagtool read the IPset from the standard input. If SETFILE does not contain an IPset, rwbagtool prints an error to stderr and exits abnormally. Only key/counter pairs where the key matches an entry in SETFILE are written to the output. (IPsets are typically created by rwset(1) or rwsetbuild(1).)

--complement-intersect=SETFILE

As --intersect, but only writes key/counter pairs for keys which do not match an entry in SETFILE.

--mincounter=VALUE

Cause the output to contain only those entries whose counter value is VALUE or higher. The allowable range is 1 to the maximum counter value; the default is 1.

--maxcounter=VALUE

Cause the output to contain only those entries whose counter value is VALUE or lower. The allowable range is 1 to the maximum counter value; the default is the maximum counter value.

--minkey=VALUE

Cause the output to contain only those entries whose key value is VALUE or higher. Default is 0 (or 0.0.0.0). Accepts input as an integer or as an IP address in dotted decimal notation.

--maxkey=VALUE

Cause the output to contain only those entries whose key value is VALUE or higher. Default is 4294967295 (or 255.255.255.255). Accepts input as an integer or as an IP address in dotted decimal notation.

Output switches

--invert

Generate a new Bag whose keys are the counters in the intermediate Bag and whose counter is the number of times the counter was seen. For example, this turns the Bag {sip:flow} into the Bag {flow:count(sip)}. Any counter in the intermediate Bag that is larger than the maximum possible key is attributed to the counter for the maximum key; to prevent this, specify "--maxcounter=4294967295" which removes all key-counter pairs whose counters do not fit into a key. (The --bin-ips switch on rwbagcat(1) allows one to invert a Bag file as it is being printed.)

--coverset

Instead of creating a Bag file as the output, write an IPset which contains the keys contained in the intermediate Bag.

--ipset-record-version=VERSION

Specify the format of the IPset records that are written to the output when the --coverset switch is used. VERSION may be 2, 3, 4, 5 or the special value 0. When the switch is not provided, the SILK_IPSET_RECORD_VERSION environment variable is checked for a version. The default version is 0. Since SiLK 3.11.0.

--output-path=PATH

Write the resulting Bag to PATH, where PATH is a filename, a named pipe, the keyword "stderr" to write the output to the standard error, or the keyword "stdout" or "-" to write the output to the standard output. If PATH names an existing file, rwbagtool exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this switch is not given, the output is written to the standard output. Attempting to write the binary output to a terminal causes rwbagtool to exit with an error.

--note-strip

Do not copy the notes (annotations) from the input files to the output file. Normally notes from the input files are copied to the output.

--note-add=TEXT

Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD

Specify the compression library to use when writing output files. If this switch is not given, the value in the SILK_COMPRESSION_METHOD environment variable is used if the value names an available compression method. When no compression method is specified, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

Adding Bag Files

 $ rwbagtool --add Bag1.bag Bag2.bag > Bag-sum.bag
 $ rwbagcat --key-format=decimal Bag-sum.bag
  1|   1|
  3|  10|
  4|   9|
  6|  14|
  7|  55|
  8|   4|

 $ rwbagtool --add Bag1.bag Bag2.bag Bag3.bag > Bag-sum2.bag
 $ rwbagcat --key-format=decimal Bag-sum2.bag
  1|   1|
  2|   8|
  3|  10|
  4|  19|
  6|  28|
  7|  67|
  8|   4|
  9|   8|

Subtracting Bag Files

 $ rwbagtool --subtract Bag1.bag Bag2.bag > Bag-diff.bag
 $ rwbagcat --key-format=decimal Bag-diff.bag
  3|  10|
  4|   5|
  6|  14|

 $ rwbagtool --subtract Bag2.bag Bag1.bag > Bag-diff2.bag
 $ rwbagcat --key-format=decimal Bag-diff2.bag
  1|   1|
  7|   9|

Getting the Minimum Value

 $ rwbagtool --minimize Bag1.bag Bag2.bag Bag3.bag > Bag-min.bag
 $ rwbagcat --key-format=decimal Bag-min.bag
  4|   2|
  7|  12|

Getting the Maximum Value

 $ rwbagtool --maximize Bag1.bag Bag2.bag Bag3.bag > Bag-max.bag
 $ rwbagcat --key-format=decimal Bag-max.bag
  1|   1|
  2|   8|
  3|  10|
  4|  10|
  6|  14|
  7|  32|
  8|   2|
  9|   8|

Dividing Bag Files

 $ rwbagtool --divide Bag2.bag Bag4.bag > Bag-div1.bag
 $ rwbagcat --key-format=decimal Bag-div1.bag
   1|   1|
   4|   1|
   7|   8|

When the order of the Bag file arguments is reversed an error is reported.

 $ rwbagtool --divide Bag4.bag Bag2.bag > Bag-div2.bag
 rwbagtool: Error dividing bags; key 6 not in divisor bag

To work around this issue, use the --coverset switch to create a copy of Bag4.bag that contains only the keys in Bag2.bag.

 $ rwbagtool --coverset Bag2.bag > Bag2-keys.set
 $ rwbagtool --intersect=Bag2-keys.set  Bag4.bag > Bag4-small.bag
 $ rwbagtool --divide Bag4-small.bag Bag2.bag > Bag-div2.bag
 $ rwbagcat --key-format=decimal Bag-div2.bag
   1|   1|
   4|   2|
   8|   3|

The following command is the same as the above except the IPset and Bag files are piped between the tools instead of being written to disk:

 $ rwbagtool --coverset Bag2.bag                \
   | rwbagtool --intersect=-  Bag4.bag          \
   | rwbagtool --divide -  Bag2.bag             \
   | rwbagcat --key-format=decimal
   1|   1|
   4|   2|
   8|   3|

Scalar Multiplication

 $ rwbagtool --scalar-multiply=7 Bag1.bag > Bag-multiply.bag
 $ rwbagcat --key-format=decimal Bag-multiply.bag
  3|  70|
  4|  49|
  6|  98|
  7| 161|
  8|  14|

Use two rwbagtool commands if multiple operations are desired.

 $ rwbagtool --add Bag1.bag Bag2.bag   \
   | rwbagtool --scalar-multiply=3 --output-path=Bag12-multi.bag
 $ rwbagcat --key-format=decimal Bag12-multi.bag
  1|   3|
  3|  30|
  4|  27|
  6|  42|
  7| 165|
  8|  12|

Comparing Bag Files

The following comparisons operate on Bag1.bag and Bag2.bag which have as common keys 4, 7, and 8.

Find counters in Bag1.bag that are less than those in Bag2.bag:

 $ rwbagtool --compare=lt Bag1.bag Bag2.bag > Bag-lt.bag
 $ rwbagcat --key-format=decimal Bag-lt.bag
  7|   1|

Find counters in Bag1.bag that are less than or equal to those in Bag2.bag:

 $ rwbagtool --compare=le Bag1.bag Bag2.bag > Bag-le.bag
 $ rwbagcat --key-format=decimal Bag-le.bag
  7|   1|
  8|   1|

Find counters in Bag1.bag that are equal to those in Bag2.bag:

 $ rwbagtool --compare=eq Bag1.bag Bag2.bag > Bag-eq.bag
 $ rwbagcat --key-format=decimal Bag-eq.bag
  8|   1|

Find counters in Bag1.bag that are greater than or equal to those in Bag2.bag:

 $ rwbagtool --compare=ge Bag1.bag Bag2.bag > Bag-ge.bag
 $ rwbagcat --key-format=decimal Bag-ge.bag
  4|   1|
  8|   1|

Find counters in Bag1.bag that are greater than those in Bag2.bag:

 $ rwbagtool --compare=gt Bag1.bag Bag2.bag > Bag-gt.bag
 $ rwbagcat --key-format=decimal Bag-gt.bag
  4|   1|

Making a Cover Set

 $ rwbagtool --coverset Bag1.bag Bag2.bag Bag3.bag > Cover.set
 $ rwsetcat --key-format=decimal Cover.set
  1
  2
  3
  4
  6
  7
  8
  9

One use of a cover set is to limit the contents of a Bag file to keys that are present in a second Bag file:

 $ rwbagtool --coverset --output-path=Cover.set Bag1.bag
 $ rwbagtool --intersect=Cover.set Bag2.bag > Bag1-mask-Bag2.bag
 $ rwbagcat --key-format=decimal Bag1-mask-Bag2.bag
  4|   2|
  7|  32|
  8|   2|

To mask the contents of Bag2.bag by the keys that are not present in Bag1.bag:

 $ rwbagtool --complement-intersect=Cover.set Bag2.bag \
        > Bag1-notmask-Bag2.bag
 $ rwbagcat --key-format=decimal Bag1-notmask-Bag2.bag
  1|   1|

Inverting a Bag

 $ rwbagtool --invert Bag1.bag > Bag-inv1.bag
 $ rwbagcat --key-format=decimal Bag-inv1.bag
  2|   1|
  7|   1|
 10|   1|
 14|   1|
 23|   1|

 $ rwbagtool --invert Bag2.bag > Bag-inv2.bag
 $ rwbagcat --key-format=decimal Bag-inv2.bag
  1|   1|
  2|   2|
 32|   1|

 $ rwbagtool --invert Bag3.bag > Bag-inv3.bag
 $ rwbagcat --key-format=decimal Bag-inv3.bag
  8|   2|
 10|   1|
 12|   1|
 14|   1|

When multiple Bag files are specified on the command line, the files are added prior to creating the inverted Bag. Even though the counter 2 appears three times in the files Bag1.bag and Bag2.bag, the key 2 is not present in the following since the add operation is performed first.

 $ rwbagtool --invert Bag1.bag Bag2.bag   \
   | rwbagcat --key-format=decimal
  1|   1|
  4|   1|
  9|   1|
 10|   1|
 14|   1|
 55|   1|

Masking Bag Files

 $ rwbagtool --intersect=Mask.set Bag1.bag > Bag-mask.bag
 $ rwbagcat --key-format=decimal Bag-mask.bag
  4|   7|
  6|  14|
  8|   2|

The --complement-intersect switch limits the output to only those keys that do not appear in the IPset file.

 $ rwbagtool --complement-intersect=Mask.set Bag1.bag > Bag-mask2.bag
 $ rwbagcat --key-format=decimal Bag-mask2.bag
  3|  10|
  7|  23|

See also the next section.

Restricting the Output

 $ rwbagtool --add --maxkey=5 Bag1.bag Bag2.bag > Bag-res1.bag
 $ rwbagcat --key-format=decimal Bag-res1.bag
  1|   1|
  3|  10|
  4|   9|

 $ rwbagtool --minkey=3 --maxkey=6 Bag1.bag > Bag-res2.bag
 $ rwbagcat --key-format=decimal Bag-res2.bag
  3|  10|
  4|   9|
  6|  14|

 $ rwbagtool --mincounter=20 Bag1.bag Bag2.bag > Bag-res3.bag
 $ rwbagcat --key-format=decimal Bag-res3.bag
  7|  55|

 $ rwbagtool --subtract --maxcounter=9 Bag1.bag Bag2.bag  \
        > Bag-res4.bag
 $ rwbagcat --key-format=decimal Bag-res4.bag
  4|   5|

Changing a File's Format

It is not possible to change the compression-method directly. A new file must be created first, and then you may then replace the old file with the new file.

To create a new file that uses a different compression-method of the Bag file A.bag, use rwbagtool with the --add switch and specify the desired argument:

 $ rwbagtool --add --compression=none --output-path=A1.bag A.bag

Changing the Key Type or Counter Type

 $ rwbagcat Bag1.bag    \
   | rwbagbuild --bag-input=- --output-path=Bag1-typed.bag  \
        --key-type=sport --counter-type=sum-bytes

Use rwfileinfo(1) to see the type of the key and counter.

 $ rwfileinfo --field=bag Bag1-typed.bag
 Bag1-typed.bag:
   bag          key: sPort @ 4 octets; counter: sum-bytes @ 8 octets

Alternatively, one may use PySiLK (see pysilk(3)) to modify the key type and counter type.

 $ cat bag-type.py
 import sys
 from silk import *

 key_type = sys.argv[1]
 counter_type = sys.argv[2]
 old_file = sys.argv[3]
 new_file = sys.argv[4]

 old = Bag.load(old_file, key_type=IPv4Addr)
 new = Bag(old, key_type=key_type, counter_type=counter_type)
 new.save(new_file)
 $
 $ python bag-type.py sipv4 sum-packets Bag1.bag Bag1-type2.bag
 $ rwfileinfo --field=bag Bag1-type2.bag
 Bag1-type2.bag:
   bag          key: sIPv4 @ 4 octets; counter: sum-packets @ 8 octets