SET INPUT

If the --key-type is "sip-country", "dip-country", or "any-country", each IP address is mapped to its country code using the country code mapping file (see "FILES") and that value is stored in the Bag file.

If the --key-type is "sip-pmap", "dip-pmap", or "any-ip-pmap", each IP address is mapped to a value found in the prefix map file specified in --pmap-file and that value is stored in the Bag file.

BAG (TEXTUAL) INPUT

A line may contain only a key or it may contain a key and counter separated by a delimiter. Use --delimiter to specify the delimiter; the accepted formats of the key are described below. If the delimiter character is not present on a line, the line must contain only a key, or a line may contain a key followed by a delimiter with no additional text on the line. In both cases, the count is set to 1. Otherwise, the line must contain a key before the delimiter and an integer counter after the delimiter. These lines may have a delimiter after the counter; this delimiter and any text following it are ignored.

The --default-count switch overrides any counter value present on the line, and any text appearing after the delimiter that follows the key is ignored.

For each key-count pair, the key is inserted into Bag with its count or, if the key is already present in the Bag, its total count is incremented by the count from this line. When using the --default-count switch, the count for a key that appears in the input N times is the product of N and DEFAULTCOUNT.

rwbagbuild prints an error and exits when a key or counter cannot be parsed.

Format of the Key

The key is a 32-bit integer, an IP address, a CIDR block, a SiLK IPWildcard, or a pair of numbers when the key-type is a protocol-port prefix map file.

For key-types that use fewer than 32-bits, rwbagbuild does not verify the validity of the key. For example, it is possible to have 257 as a key in Bag whose key-type is protocol.

rwbagbuild parses specific key-types as follows:

sIPv4, dIPv4, nhIPv4, any-IPv4

key is an IPv4 address or a 32-bit value; key-type set to corresponding IPv6 type when an IPv6 address is present. A CIDR block or SiLK IPWildcard representing multiple addresses adds multiple entries to the Bag

sIPv6, dIPv6, nhIPv6, any-IPv6

key is an IPv6 address. An IPv4 address is mapped into the ::ffff:0:0/96 netblock. All keys must be IP addresses.

flags, initialFlags, sessionFlags

key is the numeric value of the flags, 17 = FIN|ACK

sTime, eTime, any-time

key is seconds since the UNIX epoch

duration

key represents seconds

sensor

key is the numeric sensor ID

sip-country, dip-country, any-country

key is an IP address; the country_codes.pmap prefix map file is used to map the IP to a country code that is stored in the Bag

sip-pmap, dip-pmap, any-ip-pmap

key is an IP address; the specified --prefix-map file is used to map the IP to a value that is stored in the Bag

sport-pmap, dport-pmap, any-port-pmap

key is comprised of two numbers separated by a delimiter: a protocol (8-bit number) and a port (16-bit number). Those values are looked up in the specified --prefix-map file and the result is stored in the Bag. The delimiter separating the protocol and port may be set by --proto-port-delimiter. If not explicitly set, it is the same as the delimiter specified to --delimiter. The default delimiter is '"|"'.

attributes

these bits of the key are relevant, though any 32-bit value is accepted: 0x08=F, 0x10=S, 0x20=T, 0x40=C

class, type

key is treated as a number

An IP address or integer key must be expressed in one of the following formats. rwbagbuild complains if the key field contains a mixture of IPv6 addresses and integer values.

• Dotted decimal---all 4 octets are required:

   10.1.2.4
      

• An unsigned 32-bit integer:

   167838212
      

• An IPv6 address in canonical format (when SiLK has been compiled with IPv6 support):

   2001:db8:a:1::2:4
   ::ffff:10.1.2.4
      

• Any of the above with a CIDR designation---for dotted decimal all four octets are still required:

   10.1.2.4/31
   167838212/31
   2001:db8:a:1::2:4/127
   ::ffff:10.1.2.4/31
      

• SiLK IP wildcard notation. A SiLK IP Wildcard can represent multiple IPv4 or IPv6 addresses. An IP Wildcard contains an IP in its canonical format, except each part of the IP (where part is an octet for IPv4 or a hexadectet for IPv6) may be a single value, a range, a comma separated list of values and ranges, or the letter "x" to signify all values for that part of the IP (that is, "0-255" for IPv4). You may not specify a CIDR suffix when using the IP Wildcard notation.

   10.x.1-2.4,5
   2001:db8:a:x::1-2:4,5
      

Create a bag with IP addresses as keys from a text file

 192.168.0.1,5
 192.168.0.2,500
 192.168.0.3,3
 192.168.0.4,14
 192.168.0.5,5

To build a bag with it:

 $ rwbagbuild --bag-input=mybag.txt --delimiter=, > mybag.bag

Use rwbagcat(1) to view its contents:

 $ rwbagcat mybag.bag
     192.168.0.1|                   5|
     192.168.0.2|                 500|
     192.168.0.3|                   3|
     192.168.0.4|                  14|
     192.168.0.5|                   5|

Create a bag with protocols as keys from a text file

   1|      4|
   6|    138|
  17|    131|

use

 $ rwbagbuild --key-type=proto --bag-input=myproto.txt > myproto.bag
 $ rwbagcat myproto.bag
          1|                   4|
          6|                 138|
         17|                 131|

When the --key-type switch is specified, rwbagcat knows the keys should be printed as integers, and rwfileinfo(1) shows the type of the key:

 $ rwfileinfo --fields=bag myproto.bag
 myproto.bag:
   bag            key: protocol @ 4 octets; counter: custom @ 8 octets

Without the --key-type switch, rwbagbuild assumes the integers in myproto.txt represent IP addresses:

 $ rwbagbuild --bag-input=myproto.txt | rwbagcat
         0.0.0.1|                   4|
         0.0.0.6|                 138|
        0.0.0.17|                 131|

Although the --key-format switch on rwbagcat may be used to choose how the keys are displayed, it is generally better to use the --key-type switch when creating the bag.

$ rwbagbuild --bag-input=myproto.txt | rwbagcat --key-format=decimal 1| 4| 6| 138| 17| 131|

Create a bag and override the existing counter

 $ rwbagbuild --key-type=protocol --bag-input=myproto.txt  \
        --default-count=1 --output-path=myproto1.bag
 $ rwbagcat myproto1.bag
          1|                   1|
          6|                   1|
         17|                   1|

Create a bag from multiple text files

 $ cat X.txt Y.txt Z.txt                                \
   | rwbagbuild --bag-input=- --output-path=xyz.bag

For each key that appears in multiple input files, rwbagbuild sums the counters for the key.

Create a bag with IP addresses as keys from an IPset file

 $ rwbagbuild --set-input=myset.set --default-count=3  \
        --out=mybag2.bag

Create a bag from multiple IPset files

 $ rwsetcat A.set
 10.0.0.1
 10.0.0.2
 $ rwsetcat B.set
 10.0.0.2
 10.0.0.3
 $ rwsetcat C.set
 10.0.0.1
 10.0.0.2
 10.0.0.4

We want to create a bag file from these IPset files where the count for each IP address is the number of files that IP appears in. rwbagbuild accepts a single file as an argument, so we cannot do the following:

 $ rwbagbuild --set-input=A.set --set-input=B.set ...   # WRONG!

(Even if we could repeat the --set-input switch, specifying it multiple times would be annoying if we had 300 files instead of only 3.)

Since IPset files are (mathematical) sets, joining them together first with rwsettool(1) and then running rwbagbuild causes each IP address to get a count of 1:

 $ rwsettool --union A.set B.set C.set   \
   | rwbagbuild --set-input=-            \
   | rwbagcat
        10.0.0.1|                   1|
        10.0.0.2|                   1|
        10.0.0.3|                   1|
        10.0.0.4|                   1|

When rwbagbuild is processing textual input, it sums the counters for keys that appear in the input multiple times. We can use rwsetcat(1) to convert each IPset file to text and feed that as single textual stream to rwbagbuild. Use the --cidr-blocks switch on rwsetcat to reduce the amount of input that rwbagbuild must process. This is probably the best approach to the problem:

 $ rwsetcat --cidr-block *.set | rwbagbuild --bag-input=- > total1.bag
 $ rwbagcat total1.bag
        10.0.0.1|                   2|
        10.0.0.2|                   3|
        10.0.0.3|                   1|
        10.0.0.4|                   1|

A less efficient solution is to convert each IPset to a bag and then use rwbagtool(1) to add the bags together:

 $ for i in *.set ; do
        rwbagbuild --set-input=$i --output-path=/tmp/$i.bag ;
   done
 $ rwbagtool --add /tmp/*.set.bag > total2.bag
 $ rm /tmp/*.set.bag

There is no need to create a bag file for each IPset; we can get by with only two bag files, the final bag file, total3.bag, and a temporary file, tmp.bag. We initialize total3.bag to an empty bag. As we loop over each IPset, rwbagbuild converts the IPset to a bag on its standard output, rwbagtool creates tmp.bag by adding its standard input to total3.bag, and we rename tmp.bag to total3.bag:

 $ rwbagbuild --bag-input=/dev/null --output-path=total3.bag
 $ for i in *.set ; do
        rwbagbuild --set-input=$i  \
        | rwbagtool --output-path=tmp.bag --add total3.bag stdin ;
        /bin/mv tmp.bag total3.bag ;
   done
 $ rwbagcat total3.bag
        10.0.0.1|                   2|
        10.0.0.2|                   3|
        10.0.0.3|                   1|
        10.0.0.4|                   1|

Create a bag where the key is the country code

 $ rwbagbuild --bag-input=mybag.txt --delimiter=,       \
        --key-type=any-country --output-path=scc1.bag
 $ rwbagcat scc1.bag
 --|                 527|

 $ rwbagbuild --set-input=A.set --key-type=any-country  \
        --output-path=scc2.bag
 $ rwbagcat scc2.bag
 --|                   2|

Create a bag using a prefix map value as the key

For a prefix map that maps by IP addresses, use a key-type of "sip-pmap", "dip-pmap", or "any-ip-pmap". The input may be an IPset or text. The form of the textual input is the same as for a normal bag file.

 $ rwbagbuild --set-input=A.set --key-type=sip-pmap     \
        --pmap-file=ip-map.pmap --output=test1.bag

 $ rwbagbuild --bag-input=mybag.txt --delimiter=,       \
        --key-type=sip-pmap --pmap-file=ip-map.pmap     \
        --output-path=test2.bag

The prefix map file is not stored as part of the Bag, so you must provide the name of the prefix map when running rwbagcat(1).

 $ rwbagcat --pmap-file=ip-map.pmap test2.bag
          internal|                 527|

For a prefix map file that maps by protocol-port pairs, the textual input must contain either three column (protocol, port, counter) or two columns (protocol and port) which uses the --default-counter.

 $ cat proto-port-count.txt
 6| 25|  800|
 6| 80| 5642|
 6| 22
 $ rwbagbuild --key-type=sport-pmap                 \
        --bag-input=proto-port-count.txt            \
        --pmap-file=proto-port-map.pmap             \
        --output-path=service.bag
 $ rwbagcat --pmap-file=port-map.pmap service.bag
   TCP/SSH|                   1|
  TCP/SMTP|                 800|
  TCP/HTTP|                5642|

Delimiter examples

 $ cat sport.txt
 0
 1|
 2|3
 4|5|
 6|7|8|
 9|10|||||
 11|0
 $ rwbagbuild --bag-input=sport.txt --key-type=sport \
   | rwbagcat
          0|                   1|
          1|                   1|
          2|                   3|
          4|                   5|
          6|                   7|
          9|                  10|

The --default-counter switch overrides the count.

 $ rwbagbuild --bag-input=sport.txt --key-type=sport --default-count=1 \
   | rwbagcat
          0|                   1|
          1|                   1|
          2|                   1|
          4|                   1|
          6|                   1|
          9|                   1|
         11|                   1|

In fact, the --default-counter switch causes rwbagbuild to ignore all text after the delimiter that follows the key.

 $ echo '12|13 14' | rwbagbuild --bag-input=- --output=/dev/null
 rwbagbuild: Error parsing line 1: Extra text after count
 rwbagbuild: Error creating bag from text bag

 $ echo '12|13 14' | rwbagbuild --bag-input=- --default-count=1 \
   | rwbagcat --key-format=decimal
         12|                   1|